Review Reports
- Leonid Koliaskin 1,2,3,*,
- Danil Kazimirov 1,3,4,* and
- Vladimir V. Arlazarov 3,5
- et al.
Reviewer 1: Anonymous Reviewer 2: Anonymous Reviewer 3: Muhammed Ali Pala
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThis paper introduces STAMM3D, an open-access benchmark intended to evaluate the generalization of multimodal, non-deformable 3D image-registration methods. The benchmark contains 22 volumetric images covering medical, biological, and non-biomedical objects and provides an automated evaluation framework based on Target Registration Error. The inclusion of multiple domains and baseline results for four open-source registration tools may provide a useful resource for the research community. However, the limited dataset size, construction of the reference standard, evaluation design, and broad claims of universality require further clarification. My major comments are as follows:
-
The composition of the dataset should be described comprehensively. A table should report each object, imaging modality, voxel dimensions, spatial resolution, field of view, bit depth, preprocessing, and corresponding registration pairs. It is currently unclear how 22 images translate into the number of evaluated multimodal pairs.
-
The authors should justify whether 22 volumes are sufficient to evaluate modality-independent generalization. The dataset may provide a valuable proof of concept, but claims concerning universal multimodal registration should be moderated unless substantially broader object and modality coverage is demonstrated.
-
The ground-truth registration procedure is central to the benchmark and must be documented in detail. The authors should explain whether the reference transformations were obtained from synthetic perturbations, calibrated acquisition geometry, fiducial markers, expert alignment, or another procedure.
-
The Target Registration Error evaluation requires further specification.
-
The term “Similarity Transform Alignment” should be defined mathematically. The manuscript should clarify whether the transformation includes translation, rotation, isotropic scaling, reflection, or other operations. The reasons for selecting similarity transformations rather than rigid or affine registration should also be explained.
-
The evaluation protocol should separate parameter tuning from final benchmarking. If baseline methods were configured using the same image pairs on which they were evaluated, the results may overestimate generalization. The benchmark should provide fixed development and hidden or held-out test subsets wherever feasible.
-
Fairness of the baseline comparison needs greater attention.
-
TRE alone may not provide a complete account of registration quality.
-
The benchmark should include explicit reproducibility and maintenance provisions. These should cover data licenses, anonymization, metadata standards, download integrity, containerized baseline implementations, deterministic evaluation scripts, versioning, submission rules, and procedures for adding future modalities and datasets.
-
To broaden the discussion of volumetric image analysis, medical-data processing, material characterization, and trustworthy AI evaluation, the authors may consider citing the following relevant work “Leveraging Explainable Artificial Intelligence for Transparent and Trustworthy Cancer Detection Systems”
- Do not left any section (heading) blank.
It must be improved.
Author Response
We thank the reviewer for the careful reading of our manuscript and for the valuable comments and recommendations. In the revised version, we have expanded the dataset description, clarified the similarity-transform model and the construction of the reference transformations, and provided a more detailed explanation of the evaluation metrics and experimental protocol. We have also specified the arrangements for future benchmark maintenance.
The present work introduces STAMM3D as a proof-of-concept benchmark for evaluating non-deformable registration across selected imaging modalities and object domains. Its scope is a public dataset, a unified evaluation framework, and a baseline study with fixed configurations; exhaustive cross-domain generalization is not established.
Below, we provide point-by-point responses describing the revisions and explaining the methodological choices retained in the study. We believe these revisions improve the clarity of the manuscript and make its scope and limitations more explicit.
Comment 1: The composition of the dataset should be described comprehensively. A table should report each object, imaging modality, voxel dimensions, spatial resolution, field of view, bit depth, preprocessing, and corresponding registration pairs. It is currently unclear how 22 images translate into the number of evaluated multimodal pairs.
Response 1: We have extended the STAMM3D dataset description in Section 3.1 with information about 3D image volume sizes, voxel dimensions, field of view, and bit depth. We also provided a detailed extension for Table 1, which describes the construction of STAMM3D registration pairs.
Comment 2: The authors should justify whether 22 volumes are sufficient to evaluate modality-independent generalization. The dataset may provide a valuable proof of concept, but claims concerning universal multimodal registration should be moderated unless substantially broader object and modality coverage is demonstrated.
Response 2: As pointed out, STAMM3D consists of 22 high-resolution 3D images, each in the form of a large 2D image sequence. Most of the image acquisitions required time-consuming scanning, varying from hours to whole days, and involved sophisticated imaging setups with busy schedules, which caused the main difficulty for further expansion and is generally evident in the scarcity of real 3D image public datasets. As pointed out in Sec. 1, the majority of the datasets in the field of multimodal registration remain private, and existing public datasets mostly focus on a single modality pair and a specific domain of objects. Nevertheless, we were able to achieve a high level of data diversity in different modalities, covering many object domains: rat leg is related to the biomedical cancer research field, porous composites were studied for bio-resorbable implant usage, nutmeg and SD-card internal structure were examined for biology- and microelectronics-related reasons, respectively, fossil brachiopod belongs to the paleontology field, and validation with a CAD model is a common manufacturing task. In addition, as pointed out in Sec. 3, the STAMM3D repository includes data augmentation utilities, which provide infrastructure to create custom benchmark setups from the provided 22 3D images. Therefore, we declare the novel universality of STAMM3D in comparison to existing datasets and benchmarks, which provides meaningful insight into registration algorithms' performance in cases of modality and object diversity. We extended the discussion in Section 6 to clarify universality claims and address limitations arising from the number of data packages.
Comment 3: The ground-truth registration procedure is central to the benchmark and must be documented in detail. The authors should explain whether the reference transformations were obtained from synthetic perturbations, calibrated acquisition geometry, fiducial markers, expert alignment, or another procedure.
Response 3: The description of the Ground Truth construction in Section 3.1.7 has been extended.
Comment 4: The Target Registration Error evaluation requires further specification.
Response 4: We thank the reviewer for this remark. The description of the TRE evaluation in Section 3.2 has been extended. We now state the general definition of TRE for an arbitrary target point, specify that the error is computed in the fixed image domain, explain that the ground-truth transform available for every image pair allows TRE to be evaluated at any point of the volume without manual landmark annotation, and justify the choice of the fixed-volume corners as evaluation points.
Comment 5: The term “Similarity Transform Alignment” should be defined mathematically. The manuscript should clarify whether the transformation includes translation, rotation, isotropic scaling, reflection, or other operations. The reasons for selecting similarity transformations rather than rigid or affine registration should also be explained.
Response 5: We thank the reviewer. The similarity transform is defined explicitly in Equation (8), i.e., it consists of a translation, a rotation, and an isotropic scaling, while reflections, anisotropic scaling, and shear are excluded. We have also added the rationale for this choice. The scanned objects are rigid, so multimodal images differ by a rigid motion and a global scale change caused by the different voxel sizes of the devices. A rigid model cannot account for this scale difference, whereas an affine model introduces degrees of freedom that have no physical counterpart in this setting.
Comment 6: The evaluation protocol should separate parameter tuning from final benchmarking. If baseline methods were configured using the same image pairs on which they were evaluated, the results may overestimate generalization. The benchmark should provide fixed development and hidden or held-out test subsets wherever feasible.
Response 6: We thank the reviewer for this remark. We have extended Section 4 to emphasize that all the baseline methods are evaluated in the “out-of-the-box” scenario with fixed parameters for all STAMM3D registration pairs, and no baseline method configuration occurred. The pair-specific refinement used to construct the reference (ground truth) transformations is a separate procedure and was not used to configure the evaluated baselines. We agree that development and held-out test subsets are important when methods are trained, or parameters are selected using benchmark data. Since the present baseline study involves neither training nor parameter tuning on STAMM3D, we have retained all 19 pairs for evaluation rather than introducing a development subset.
Comment 7: Fairness of the baseline comparison needs greater attention.
Response 7: We thank the reviewers for drawing attention to this limitation of the initialized comparison. The initialization procedures are described in Section 4.2, and the possible effect of pre-resampling on SIFT3D and LBCV is discussed in Section 6. For SITK and NiftyReg, the initial transformation is supplied through the available initialization interfaces. The SIFT3D and LBCV implementations used in this study do not expose an equivalent interface; the initial alignment is therefore introduced by resampling the moving volume before registration.
We interpret the suggestion to run these methods directly on transformation matrices as supplying the image volumes together with an initial matrix while avoiding pre-resampling. The evaluated SIFT3D and LBCV implementations do not support this form of initialization. Incorporating the initial matrix into their internal estimation procedures would require modifying the implementations, which would extend beyond the present comparison of the selected baselines.
We agree that pre-resampling may affect the image information used by these methods. Accordingly, the initialized results should be interpreted as comparisons of the implemented registration pipelines, including their initialization procedures. The current experiment does not isolate the contribution of interpolation from that of the registration algorithm, so differences in this setting cannot be attributed to the algorithms alone. We have retained both initialized and uninitialized evaluations to make this limitation explicit and to show performance under both conditions.
Comment 8: TRE alone may not provide a complete account of registration quality.
Response 8: We thank the reviewer for the comment. Since STAMM3D objects are rigid and the transforms are non-deformable, only global misalignment is possible (no local mismatches can occur). We have added a paragraph to Section 3.2 and a theorem with proof in Appendix A, explaining that BBRE, based on TRE, aims to provide a complete account. They show that the residual (deformation) transform is affine, so the largest misalignment within the volume is attained at a corner and, hence, is captured by BBRE. BBRE is bounded from both sides by this maximum and vanishes only for exact registration. STAMM3D objects are rigid and only global misalignment is possible; thus, BBRE reflects the alignment of the entire volume rather than of the evaluation points only.
Comment 9: The benchmark should include explicit reproducibility and maintenance provisions. These should cover data licenses, anonymization, metadata standards, download integrity, containerized baseline implementations, deterministic evaluation scripts, versioning, submission rules, and procedures for adding future modalities and datasets.
Response 9: We thank the reviewer for highlighting the reproducibility and maintenance aspects of the benchmark. The Data Availability Statement identifies the public repositories and licenses: the benchmark and analysis code are available on GitHub under the MIT License, and the dataset is available on Zenodo under CC BY 4.0. The statement also specifies the code version used for the reported experiments, namely the v1.0 tag.
The study does not involve human participants, human data, or human tissue, and the dataset contains no identifiable personal information. Human-data anonymization is therefore not applicable. The volumes are distributed as TIFF slice sequences. Their acquisition conditions, dimensions, voxel sizes, fields of view, bit depth, and registration-pair definitions are documented in Section 3.1 and Tables 1 and 2. This provides the descriptive information needed to interpret the data, although the release does not introduce a separate standardized machine-readable metadata schema.
For a fixed set of estimated and reference transformations, BBRE and the reported summary measures are deterministic. The evaluation and analysis code is provided in the public repository. This should be distinguished from bitwise reproducibility of complete registration runs, which may also depend on the baseline implementation and execution environment. The Zenodo record identifies the archived dataset used in the study. We acknowledge the value of checksum-based download verification and containerized baseline distributions. Implementing and validating these additional provisions, however, would extend beyond the present dataset and baseline study.
STAMM3D is currently an author-maintained dataset and benchmark rather than a hosted submission service. We therefore do not introduce leaderboard submission rules or an automated dataset-ingestion procedure. Future code updates and extensions with additional modalities or datasets will be curated by the authors as resources permit. Researchers wishing to suggest or contribute an extension may contact the corresponding authors at the addresses provided in the manuscript.
Comment 10: To broaden the discussion of volumetric image analysis, medical-data processing, material characterization, and trustworthy AI evaluation, the authors may consider citing the following relevant work “Leveraging Explainable Artificial Intelligence for Transparent and Trustworthy Cancer Detection Systems”.
Response 10: We thank the reviewer for the suggestion. We have reviewed the proposed work and now cite it in the Introduction in the context of applications related to multimodal 3D registration.
Comment 11: Do not leave any section (heading) blank.
Response 11: We thank the reviewer for this remark. We have revised the manuscript and checked that all sections are complete and that none of the headings are left blank.
The following is a list of changes made during the revision process:
Lines 102-104. We have expanded the literature review by incorporating the references suggested by the reviewers, which demonstrate that accurate multimodal data alignment is a prerequisite for related tasks, such as computer-aided diagnosis and 3D reconstruction in remote sensing.
Lines 323-342. We have clarified the definition of a similarity transform, provided a rigorous mathematical definition, and explained why this particular class of transformations is considered in the benchmark.
Lines 366-368. We have added a clarification that the present work focuses on registration accuracy and robustness, while additional performance data (for future research) are provided in Appendix C.
Lines 390-394. We have added several sentences to clarify the dataset structure and its specific characteristics. The modality pairs and their characteristics, which form the basis of the benchmark, are summarized in Table 1 (which we have revised to improve clarity) and in a newly added Table 2.
Lines 484-494. We have clarified the procedure used to create the ground-truth transformations.
Lines 531 - 540. We have added explanations of the TRE and BBRE metrics, highlighting their differences and the technical details of their computation.
Lines 542-550. We have clarified the rationale behind the BBRE metric: it aggregates the globally maximal deviations from the ground truth, which are measured locally using TRE.
Lines 559-566. A qualitative analysis of the BBRE metric values has been added.
Lines 575-582. An explanation of the rationale behind selecting the clipping thresholds for MRE and Stability computation has been added.
Lines 613-615. An introductory sentence has been added to explain the selection of state-of-the-art classical non-learning-based registration algorithms and their evaluation in an out-of-the-box setting.
Lines 626-631. A note has been added clarifying that no baseline parameters were tuned on the STAMM3D evaluation pairs.
Lines 649-657. We clarified the design of the BBRE metric: it can be equally applicable to both forward- and reverse-direction registration.
Lines 665-670. A clarification has been added that reversing the registration direction from forward to reverse does not allow all algorithm-specific registration effects to be fully investigated within the benchmark. In particular, this comparison is limited because some algorithms employ parameter search grids whose configuration depends on the size of the initial or target volume, which becomes relevant when the registration direction is reversed.
Lines 723-726. An explanation of the statistical significance analysis of the results has been added.
Lines 771-776, 778. We have further clarified the motivation behind the benchmark. It introduces a novel approach to assessing registration universality by considering unprecedented modality and structural diversity compared to existing public 3D multimodal registration benchmarks and datasets, which typically focus on a single modality pair or a narrow object domain. We also acknowledge that the benchmark currently serves as a foundational proof-of-concept rather than providing exhaustive coverage of all possible 3D imaging scenarios.
Line 809. The statement has been clarified to explicitly refer to classical non-learning-based registration algorithms.
Lines 825-833. The Funding section has been updated.
Line 844. We have added the Informed Consent Statement.
Lines 849-853. Additional clarification regarding the versioning and update protocol has been provided.
Lines 858-935 (Appendix A). We have added a new Appendix dedicated to the theoretical properties of the BBRE metric, providing a theoretical justification for our choice of clipping parameters for MRE and Stability.
Lines 936-1002 (Appendix B). The new Appendix is dedicated to a statistical analysis of the results, with a particular focus on assessing the statistical significance of the findings.
Lines 1003-1011 (Appendix C). Additional material has been included, presenting an analysis of the execution time and peak memory consumption of the algorithms considered in the paper.
Reviewer 2 Report
Comments and Suggestions for AuthorsThis manuscript presents STAMM3D, an open benchmark for non-deformable multimodal 3D image registration, including a heterogeneous dataset, ground-truth transformations, a geometry-based evaluation framework, and baseline evaluations of several open-source registration methods. The benchmark addresses an important gap in multimodal 3D registration and provides useful resources for reproducible evaluation. However, because the primary contribution is a benchmark, the reliability of the reference transformations, robustness of the evaluation metrics, and fairness of the benchmarking protocol are particularly important. Several aspects of the current experimental design and analysis require further clarification and validation. The following issues should be addressed:
- The ground-truth construction relies on five independent expert registrations followed by iterative SimpleITK-based refinement with modality-specific metric selection and object-specific parameter tuning, while SimpleITK is also evaluated as one of the benchmarked registration frameworks. Although the refined transformations are subsequently ranked by additional experts, the uncertainty of the reference transformations is not quantitatively characterized. Please report inter-expert variation, residual landmark errors, or other measures of reference uncertainty, and verify whether the main benchmark conclusions remain stable under alternative expert-derived or landmark-based reference transformations.
- The manuscript makes relatively broad claims regarding universality and generalization, whereas STAMM3D contains 22 volumes forming 19 registration cases, with several cases constructed from repeated imaging of the same limited set of physical objects. The current dataset therefore appears more suitable for evaluating robustness across selected modality and acquisition changes than for establishing broad cross-object or cross-domain generalization. Please clarify the intended meaning of “universality,” provide more detailed per-modality or per-object analyses, or moderate the related claims accordingly.
- The manuscript adopts BBRE = 0.144 as the criterion for unsuccessful registration and also uses the same value to truncate large errors when computing MRE. Although the authors provide a geometric interpretation for this threshold, only a single failure cutoff is used in the current evaluation. Since both Stability and MRE directly depend on this value, the reported performance differences and method rankings may be sensitive to the threshold selection. Please provide a systematic sensitivity analysis using several reasonable BBRE failure thresholds and report the corresponding changes in Stability, MRE, and method ranking. In addition, the original unclipped BBRE distributions, medians, or percentile statistics should be provided to demonstrate that the main conclusions are not an artifact of the selected cutoff. The distinct role of the BBRE ≤ 0.005 criterion for visually indistinguishable alignment should also be clarified.
- The initialization protocol is not fully method-neutral. For SITK and NiftyReg, the initial transformation is incorporated using built-in functionality, whereas for SIFT3D and LBCV the initial transform is applied directly to the moving volume before registration. The manuscript itself notes that this pre-resampling can introduce interpolation artifacts and degrade the local features used by these methods. Therefore, part of the observed performance difference under initialization may arise from the initialization implementation rather than from the registration algorithms alone. Please adopt a more consistent initialization strategy where feasible, or explicitly separate this protocol-dependent effect when interpreting initialized cross-method comparisons.
- The current baseline evaluation mainly includes SIFT3D, SimpleITK, NiftyReg, and linearBCV (LBCV) from the deedsBCV package. These are useful classical and open-source baselines, but they do not fully represent contemporary learning-based or learning-assisted 3D registration approaches. The authors are encouraged either to include a suitable representative contemporary non-deformable registration method, if a fair and reproducible implementation is feasible, or to more precisely describe the current experiments as classical/open-source baselines rather than a comprehensive evaluation of state-of-the-art registration methods. In addition, since execution time and peak memory consumption are already recorded by the benchmark, reporting these efficiency results would improve the practical value of the comparison.
- The interpretation of the forward–reverse registration comparison requires further clarification. In the current protocol, all image pairs are constructed so that the moving image has equal or higher resolution than the fixed image, and directional sensitivity is evaluated by swapping the fixed and moving volumes. Therefore, the forward–reverse comparison simultaneously changes the registration direction, the resampling direction, and the reference geometry used by BBRE, since BBRE is computed from the corners and diagonal length of the fixed volume. The manuscript mainly attributes the observed asymmetry, particularly for initialized SITK, to the different effects of downscaling and upscaling on the optimization landscape. However, the current experiment does not fully isolate this factor from the other changes introduced by swapping fixed and moving images. Please clarify this issue and, if possible, provide an additional control using a common physical reference geometry, a symmetric error measure, or another experiment that separates the resampling-direction effect from the fixed–moving role reversal.
- Several internal-consistency and presentation issues should be carefully corrected. In particular, the CAD object used in the VXCT experiment is described inconsistently: the dataset description and abbreviation list identify it as an NVIDIA 4090 graphics-card CAD model, whereas Section 3.1.6 refers to an NVIDIA 4060 CAD model. Since these descriptions appear to refer to the same VXCT–CAD test object, the authors should verify the actual model used and ensure that the terminology is consistent throughout the manuscript and associated resources. In addition, several typographical and language errors are present, such as “mutimodal” and “wih.” A thorough proofreading of the manuscript is recommended to improve clarity and presentation quality.
- The manuscript would also benefit from a broader discussion of recent studies on geometric consistency, multi-view correspondence, and robust 3D reconstruction, which are closely related to reliable spatial correspondence estimation even though they address different tasks from multimodal 3D registration. The authors are encouraged to discuss relevant works such as “Visual Consistency Enhancement for Multi-view Stereo Reconstruction in Remote Sensing,” IEEE T-GRS, 2024; “Refined Cascade Cost Volume for Multi-View Remote Sensing Image Reconstruction,” IEEE T-GRS, 2025; and “Confident Multi-View Stereo,” IEEE T-MM, 2025. These studies may help strengthen the discussion of robust geometric correspondence and consistency modeling.
Author Response
We thank the reviewer for the detailed assessment and constructive recommendations concerning the reliability and interpretation of the benchmark. The revised manuscript provides a fuller description of reference-transform construction and distinguishes this procedure from baseline evaluation with fixed parameters. We have added threshold-sensitivity analyses, unclipped error summaries, and statistical comparisons in Appendix B, together with execution-time and peak-memory measurements in Appendix C.
We have also clarified that the experiments concern selected classical registration implementations and that the dataset provides limited coverage of possible objects and modality combinations. A broader comparison including learning-based methods is reserved for future work.
The point-by-point responses below describe the changes and explain the scope and limitations of the present evaluation. We believe these additions make the reported results and the basis for their interpretation clearer.
Comment 1: The ground-truth construction relies on five independent expert registrations followed by iterative SimpleITK-based refinement with modality-specific metric selection and object-specific parameter tuning, while SimpleITK is also evaluated as one of the benchmarked registration frameworks. Although the refined transformations are subsequently ranked by additional experts, the uncertainty of the reference transformations is not quantitatively characterized. Please report inter-expert variation, residual landmark errors, or other measures of reference uncertainty, and verify whether the main benchmark conclusions remain stable under alternative expert-derived or landmark-based reference transformations.
Response 1: We thank the reviewer for this remark. The description of the Ground Truth construction in Section 3.1.7 and the evaluated baseline description in Section 4 have been extended to differentiate the SITK-based optimisation used for Ground Truth construction and evaluation of baseline registration methods. Optimisation-based registration tools is SimpleITK library are highly customisable by design: optimized metric, image preprocessing and masking, optimizer strategy can be selected from a wide range of available options as well as a substantial amount of parameters to tune manually for each option. Therefore, during the Ground Truth construction, an optimization-based registration method was manually constructed via option and parameter selection for each pair separately to achieve the best result. On the other hand, in the registration baseline evaluation, methods were evaluated in the “out-of-the-box” usage scenario on all STAMM3D data pairs without any pair-specific tuning. For this purpose, two ready-made option and parameter sets (OrigSITK and EnhSITK) were adopted from prior publications. Created for GT construction optimization registration only uses SimpleITK library for convenience and the way of manual registration fine-tuning for each pair does not suggest an evaluation bias in favour of separate “out-of-the-box” registration.
Comment 2: The manuscript makes relatively broad claims regarding universality and generalization, whereas STAMM3D contains 22 volumes forming 19 registration cases, with several cases constructed from repeated imaging of the same limited set of physical objects. The current dataset therefore appears more suitable for evaluating robustness across selected modality and acquisition changes than for establishing broad cross-object or cross-domain generalization. Please clarify the intended meaning of “universality,” provide more detailed per-modality or per-object analyses, or moderate the related claims accordingly.
Response 2: As pointed out, STAMM3D consists of 22 high-resolution 3D images, each in the form of a large 2D image sequence. Most of the image acquisitions required time-consuming scanning, varying from hours to whole days, and involved sophisticated imaging setups with busy schedules, which caused the main difficulty for further expansion and is generally evident in the scarcity of real 3D image public datasets. As pointed out in Sec. 1, the majority of the datasets in the field of multimodal registration remain private, and existing public datasets mostly focus on a single modality pair and a specific domain of objects. Nevertheless, we were able to achieve a high level of data diversity in different modalities, covering many object domains: rat leg is related to the biomedical cancer research field, porous composites were studied for bio-resorbable implant usage, nutmeg and SD-card internal structure were examined for biology and microelectronics-related reasons, respectively, fossil brachiopod belongs to the paleontology field, and validation with a CAD model is a common manufacturing task. In addition, as pointed out in Sec. 3, the STAMM3D repository includes data augmentation utilities, which provide infrastructure to create custom benchmark setups from the provided 22 3D images. Therefore, we declare the novel universality of STAMM3D in comparison to existing datasets and benchmarks, which provides meaningful insight into registration algorithms' performance in case of modality- and object-diverse evaluation. We extended the discussion in Section 6 to clarify universality claims and address limitations arising from the number of data packages.
Comment 3: The manuscript adopts BBRE = 0.144 as the criterion for unsuccessful registration and also uses the same value to truncate large errors when computing MRE. Although the authors provide a geometric interpretation for this threshold, only a single failure cutoff is used in the current evaluation. Since both Stability and MRE directly depend on this value, the reported performance differences and method rankings may be sensitive to the threshold selection. Please provide a systematic sensitivity analysis using several reasonable BBRE failure thresholds and report the corresponding changes in Stability, MRE, and method ranking. In addition, the original unclipped BBRE distributions, medians, or percentile statistics should be provided to demonstrate that the main conclusions are not an artifact of the selected cutoff. The distinct role of the BBRE ≤ 0.005 criterion for visually indistinguishable alignment should also be clarified.
Response 3: We thank the reviewer. The two thresholds play different roles, which is now stated explicitly in Section 3.2. The value 0.005 is the resolution limit of the ground truth established in [69], and Remark 1 (Appendix A) gives its geometric meaning. The value 0.144 is the failure threshold, and Remark 1 shows that each failure condition alone yields nearly the same BBRE (0.142-0.15). As requested, we performed a systematic sensitivity analysis (new Appendix B). Stability, MRE, and the method ranking were recomputed for failure thresholds of 0.05, 0.1, 0.144, 0.2, and 0.3, with MRE clipped at the same value (Tables A6-A7 in Appendix B). Kendall’s rank correlation with the results at 0.144 is at least 0.86 for Stability and 0.95 for MRE in both directions, and the four initialized SITK setups rank first by MRE at every threshold. We also report the unclipped BBRE statistics: medians, quartiles, 90th percentiles, the number of registrations with BBRE <= 0.005, and the number of complete failures (Table A8). These data show that the observed BBRE values form two well-separated groups, successful registrations (BBRE <= 0.05) and failures (BBRE > 0.3), and only 2-8% of the registrations fall between 0.1 and 0.3. The failure threshold lies in this sparsely populated range, which confirms that both the threshold and the clipping value are appropriate and that the main conclusions are not an artifact of the selected thresholds.
Comment 4: The initialization protocol is not fully method-neutral. For SITK and NiftyReg, the initial transformation is incorporated using built-in functionality, whereas for SIFT3D and LBCV the initial transform is applied directly to the moving volume before registration. The manuscript itself notes that this pre-resampling can introduce interpolation artifacts and degrade the local features used by these methods. Therefore, part of the observed performance difference under initialization may arise from the initialization implementation rather than from the registration algorithms alone. Please adopt a more consistent initialization strategy where feasible, or explicitly separate this protocol-dependent effect when interpreting initialized cross-method comparisons.
Response 4: We thank the reviewers for drawing attention to this limitation of the initialized comparison. The initialization procedures are described in Section 4.2, and the possible effect of pre-resampling on SIFT3D and LBCV is discussed in Section 6. For SITK and NiftyReg, the initial transformation is supplied through the available initialization interfaces. The SIFT3D and LBCV implementations used in this study do not expose an equivalent interface; the initial alignment is therefore introduced by resampling the moving volume before registration.
We interpret the suggestion to run these methods directly on transformation matrices as supplying the image volumes together with an initial matrix while avoiding pre-resampling. The evaluated SIFT3D and LBCV implementations do not support this form of initialization. Incorporating the initial matrix into their internal estimation procedures would require modifying the implementations, which would extend beyond the present comparison of the selected baselines.
We agree that pre-resampling may affect the image information used by these methods. Accordingly, the initialized results should be interpreted as comparisons of the implemented registration pipelines, including their initialization procedures. The current experiment does not isolate the contribution of interpolation from that of the registration algorithm, so differences in this setting cannot be attributed to the algorithms alone. We have retained both initialized and uninitialized evaluations to make this limitation explicit and to show performance under both conditions.
Comment 5: The current baseline evaluation mainly includes SIFT3D, SimpleITK, NiftyReg, and linearBCV (LBCV) from the deedsBCV package. These are useful classical and open-source baselines, but they do not fully represent contemporary learning-based or learning-assisted 3D registration approaches. The authors are encouraged either to include a suitable representative contemporary non-deformable registration method, if a fair and reproducible implementation is feasible, or to more precisely describe the current experiments as classical/open-source baselines rather than a comprehensive evaluation of state-of-the-art registration methods. In addition, since execution time and peak memory consumption are already recorded by the benchmark, reporting these efficiency results would improve the practical value of the comparison.
Response 5: We appreciate the concern for the learning-based registration methods, which are an interesting subject for a STAMM3D benchmark study. But currently, due to the diversity of existing learning-based multimodal registration methods and the lack of necessary resources for a complete, fair, and reproducible learning-based multimodal registration benchmarking, this topic is planned for separate further research. As per your request, we have provided Table A11 and Table A12 in Appendix C with recorded execution time and peak RAM consumption, respectively. But due to the focus on registration quality and baseline expansion, which led to usage of two different workstations in the benchmarking process, we deferred from the performance analysis.
Comment 6: The interpretation of the forward-reverse registration comparison requires further clarification. In the current protocol, all image pairs are constructed so that the moving image has equal or higher resolution than the fixed image, and directional sensitivity is evaluated by swapping the fixed and moving volumes. Therefore, the forward-reverse comparison simultaneously changes the registration direction, the resampling direction, and the reference geometry used by BBRE, since BBRE is computed from the corners and diagonal length of the fixed volume. The manuscript mainly attributes the observed asymmetry, particularly for initialized SITK, to the different effects of downscaling and upscaling on the optimization landscape. However, the current experiment does not fully isolate this factor from the other changes introduced by swapping fixed and moving images. Please clarify this issue and, if possible, provide an additional control using a common physical reference geometry, a symmetric error measure, or another experiment that separates the resampling-direction effect from the fixed-moving role reversal.
Response 6: We thank the reviewer for this insightful comment. BBRE is normalized by the diagonal of the fixed volume and is therefore independent of the voxel size, so the different resolutions of the two images do not by themselves affect the metric. Moreover, Remark 2 (Appendix A) shows that, for a consistent algorithm and volumes covering the same object region, the forward and reverse BBRE values coincide to first order in the registration error. Hence, substantial forward–reverse differences reflect the behaviour of the algorithm rather than an asymmetry of the metric or of the reference geometry. We also tested the forward-reverse differences statistically for each setup (Appendix B, Table A9). For the initialized SITK setups, the forward direction yields lower BBRE for most image pairs, but with 19 pairs this difference is not significant after correction for multiple comparisons. We agree that the resampling direction and the fixed-moving role reversal cannot be separated in the current protocol, since in SITK the fixed image also defines the sampling grid. We have therefore revised the interpretation: we tried to correct so that the manuscript no longer claims a consistent advantage of the forward direction; the asymmetry is described as consistent with the different effects of downscaling and upscaling, and isolating these factors is left for future work.
Comment 7: Several internal-consistency and presentation issues should be carefully corrected. In particular, the CAD object used in the VXCT experiment is described inconsistently: the dataset description and abbreviation list identify it as an NVIDIA 4090 graphics-card CAD model, whereas Section 3.1.6 refers to an NVIDIA 4060 CAD model. Since these descriptions appear to refer to the same VXCT-CAD test object, the authors should verify the actual model used and ensure that the terminology is consistent throughout the manuscript and associated resources. In addition, several typographical and language errors are present, such as “mutimodal” and “wih.” A thorough proofreading of the manuscript is recommended to improve clarity and presentation quality.
Response 7: We thank the reviewer for pointing this out. The CAD model 4060 has been verified, and the terminology is now consistent throughout the manuscript and the associated resources. We have also thoroughly proofread the manuscript and corrected the typographical and language errors, including those mentioned.
Comment 8: The manuscript would also benefit from a broader discussion of recent studies on geometric consistency, multi-view correspondence, and robust 3D reconstruction, which are closely related to reliable spatial correspondence estimation even though they address different tasks from multimodal 3D registration. The authors are encouraged to discuss relevant works such as “Visual Consistency Enhancement for Multi-view Stereo Reconstruction in Remote Sensing,” IEEE T-GRS, 2024; “Refined Cascade Cost Volume for Multi-View Remote Sensing Image Reconstruction,” IEEE T-GRS, 2025; and “Confident Multi-View Stereo,” IEEE T-MM, 2025. These studies may help strengthen the discussion of robust geometric correspondence and consistency modeling.
Response 8: We thank the reviewer for the suggestion. We have reviewed the proposed works and now cite them in the Introduction as examples of related tasks that rely on accurate spatial correspondence, such as 3D reconstruction in remote sensing.
The following is a list of changes made during the revision process:
Lines 102-104. We have expanded the literature review by incorporating the references suggested by the reviewers, which demonstrate that accurate multimodal data alignment is a prerequisite for related tasks, such as computer-aided diagnosis and 3D reconstruction in remote sensing.
Lines 323-342. We have clarified the definition of a similarity transform, provided a rigorous mathematical definition, and explained why this particular class of transformations is considered in the benchmark.
Lines 366-368. We have added a clarification that the present work focuses on registration accuracy and robustness, while additional performance data (for future research) are provided in Appendix C.
Lines 390-394. We have added several sentences to clarify the dataset structure and its specific characteristics. The modality pairs and their characteristics, which form the basis of the benchmark, are summarized in Table 1 (which we have revised to improve clarity) and in a newly added Table 2.
Lines 484-494. We have clarified the procedure used to create the ground-truth transformations.
Lines 531 - 540. We have added explanations of the TRE and BBRE metrics, highlighting their differences and the technical details of their computation.
Lines 542-550. We have clarified the rationale behind the BBRE metric: it aggregates the globally maximal deviations from the ground truth, which are measured locally using TRE.
Lines 559-566. A qualitative analysis of the BBRE metric values has been added.
Lines 575-582. An explanation of the rationale behind selecting the clipping thresholds for MRE and Stability computation has been added.
Lines 613-615. An introductory sentence has been added to explain the selection of state-of-the-art classical non-learning-based registration algorithms and their evaluation in an out-of-the-box setting.
Lines 626-631. A note has been added clarifying that no baseline parameters were tuned on the STAMM3D evaluation pairs.
Lines 649-657. We clarified the design of the BBRE metric: it can be equally applicable to both forward- and reverse-direction registration.
Lines 665-670. A clarification has been added that reversing the registration direction from forward to reverse does not allow all algorithm-specific registration effects to be fully investigated within the benchmark. In particular, this comparison is limited because some algorithms employ parameter search grids whose configuration depends on the size of the initial or target volume, which becomes relevant when the registration direction is reversed.
Lines 723-726. An explanation of the statistical significance analysis of the results has been added.
Lines 771-776, 778. We have further clarified the motivation behind the benchmark. It introduces a novel approach to assessing registration universality by considering unprecedented modality and structural diversity compared to existing public 3D multimodal registration benchmarks and datasets, which typically focus on a single modality pair or a narrow object domain. We also acknowledge that the benchmark currently serves as a foundational proof-of-concept rather than providing exhaustive coverage of all possible 3D imaging scenarios.
Line 809. The statement has been clarified to explicitly refer to classical non-learning-based registration algorithms.
Lines 825-833. The Funding section has been updated.
Line 844. We have added the Informed Consent Statement.
Lines 849-853. Additional clarification regarding the versioning and update protocol has been provided.
Lines 858-935 (Appendix A). We have added a new Appendix dedicated to the theoretical properties of the BBRE metric, providing a theoretical justification for our choice of clipping parameters for MRE and Stability.
Lines 936-1002 (Appendix B). The new Appendix is dedicated to a statistical analysis of the results, with a particular focus on assessing the statistical significance of the findings.
Lines 1003-1011 (Appendix C). Additional material has been included, presenting an analysis of the execution time and peak memory consumption of the algorithms considered in the paper.
Reviewer 3 Report
Comments and Suggestions for AuthorsDear Editor,
This study (STAMM3D) presents a heterogeneous dataset consisting of 22 different 3D volumes (19 registration scenarios) and an automated benchmarking framework based on the Bounding Box Registration Error (BBRE) metric. The study offers an open-access and reproducible framework aimed at addressing the lack of publicly available data and standard evaluation infrastructure in the field of undeformed multimodal 3D image registration. However, the exclusion of state-of-the-art deep learning models from the comparison as a control group, the evaluation bias that the SimpleITK-based optimization used to generate the ground truth could introduce in favor of the compared methods, the degradation of feature quality caused by pre-resampling operations applied to some algorithms, inconsistencies in parameter configurations, and the failure to present the execution time and memory usage data mentioned in the methodology in the results—I recommend that the paper be returned to the authors for revision.
- Although the introduction of the article mentions current deep learning-based methods such as VoxelMorph, DeepReg, and TransMorph, the comparison section includes only traditional/classical algorithms. Although the examination of deep learning methods has been deferred to a future study, the failure to include at least a few deep learning models based on rigid or affine transformations as a control group in a current comparison article constitutes a significant shortcoming.
- While ground truth transformation matrices were obtained from real data, a SimpleITK-based optimization algorithm was used following manual annotation by experts. The inclusion of SimpleITK among the compared algorithms could lead to a potential evaluation bias in favor of SimpleITK-based optimization strategies. The authors should discuss how the ground truth optimized with SimpleITK affects the fair evaluation of other algorithms.
- In scenarios where manual initialization is applied, it is noted that the moving volume is pre-resampled prior to alignment for the SIFT3D and LBCV algorithms. Running methods such as SIFT3D and LBCV directly on transformation matrices, rather than applying them to a distorted voxel grid, would provide a more fair evaluation.
- The text states that the STAMM3D framework records the execution time and peak memory consumption for each alignment run. However, no measurements regarding these data are presented in the Results and Table 3 sections, and the analysis has been left to future work. In a benchmark paper, it is expected that time and memory complexity analyses—which determine the practical usability of algorithms—be included in the study.
- It is stated that for failed or unpredictable registrations, the BBRE value was artificially set to 100. It is noted that errors were clipped to 0.144 when calculating the MRE (Mean Registration Error). No statistical significance of performance differences between algorithms (e.g., p-values, Wilcoxon test) was reported.
- Nineteen registration pairs were generated from the 22 volumes in the dataset. Although various imaging modalities were covered, a dataset of 19 cases is numerically limited for making universal generalization claims. This should be more clearly emphasized among the paper’s limitations.
- In Section 3.1, Table 1, and the Abbreviations list, the graphics card used for the CAD test is specified as the NVIDIA 4090, whereas in Section 3.1.6, the same model is listed as the NVIDIA 4060.
Author Response
We thank the reviewer for the careful assessment and helpful suggestions regarding the baseline comparison and the presentation of the results. In the revised manuscript, we have expanded the description of reference-transform construction, added statistical comparisons and measurements of execution time and peak memory, and corrected the CAD-model designation. We have also made the limited dataset size and the scope of the classical baseline study more explicit.
The main contribution of STAMM3D is the combination of a heterogeneous public dataset and a common framework for evaluating non-deformable multimodal registration. The baseline experiments characterize the selected implementations under the reported conditions; they do not establish exhaustive generalization or cover all contemporary registration approaches.
Below, we provide point-by-point responses explaining the revisions and the choices retained in the current study. We believe the revised manuscript presents the contribution and its practical limitations more clearly.
Comment 1: Although the introduction of the article mentions current deep learning-based methods such as VoxelMorph, DeepReg, and TransMorph, the comparison section includes only traditional/classical algorithms. Although the examination of deep learning methods has been deferred to a future study, the failure to include at least a few deep learning models based on rigid or affine transformations as a control group in a current comparison article constitutes a significant shortcoming.
Response 1: We appreciate the concern for the learning-based registration methods, which are an interesting subject for a STAMM3D benchmark study. But currently, due to the diversity of existing learning-based multimodal registration methods and the lack of necessary resources for a complete, fair, and reproducible learning-based multimodal registration benchmark, this topic is planned for separate further research.
Comment 2: While ground truth transformation matrices were obtained from real data, a SimpleITK-based optimization algorithm was used following manual annotation by experts. The inclusion of SimpleITK among the compared algorithms could lead to a potential evaluation bias in favor of SimpleITK-based optimization strategies. The authors should discuss how the ground truth optimized with SimpleITK affects the fair evaluation of other algorithms.
Response 2: We thank the reviewer for this remark. The description of the Ground Truth construction in Section 3.1.7 and the evaluated baseline description in Section 4 have been extended to differentiate the SITK-based optimisation used for Ground Truth construction and evaluation of baseline registration methods. Optimization-based registration tools is SimpleITK library are highly customisable by design: optimized metric, image preprocessing and masking, optimizer strategy can be selected from a wide range of available options as well as a substantial amount of parameters to tune manually for each option. Therefore, during the Ground Truth construction, an optimization-based registration method was manually constructed via option and parameter selection for each pair separately to achieve the best result. On the other hand, in the registration baseline evaluation, methods were evaluated in the “out-of-the-box” usage scenario on all STAMM3D data pairs without any pair-specific tuning. For this purpose, two ready-made option and parameter sets (OrigSITK and EnhSITK) were adopted from prior publications. Created for GT construction optimization registration only uses SimpleITK library for convenience and the way of manual registration fine-tuning for each pair does not suggest an evaluation bias in favour of separate “out-of-the-box” registration.
Comment 3: In scenarios where manual initialization is applied, it is noted that the moving volume is pre-resampled prior to alignment for the SIFT3D and LBCV algorithms. Running methods such as SIFT3D and LBCV directly on transformation matrices, rather than applying them to a distorted voxel grid, would provide a fairer evaluation.
Response 3: We thank the reviewers for drawing attention to this limitation of the initialized comparison. The initialization procedures are described in Section 4.2, and the possible effect of pre-resampling on SIFT3D and LBCV is discussed in Section 6. For SITK and NiftyReg, the initial transformation is supplied through the available initialization interfaces. The SIFT3D and LBCV implementations used in this study do not expose an equivalent interface; the initial alignment is therefore introduced by resampling the moving volume before registration.
We interpret the suggestion to run these methods directly on transformation matrices as supplying the image volumes together with an initial matrix while avoiding pre-resampling. The evaluated SIFT3D and LBCV implementations do not support this form of initialization. Incorporating the initial matrix into their internal estimation procedures would require modifying the implementations, which would extend beyond the present comparison of the selected baselines.
We agree that pre-resampling may affect the image information used by these methods. Accordingly, the initialized results should be interpreted as comparisons of the implemented registration pipelines, including their initialization procedures. The current experiment does not isolate the contribution of interpolation from that of the registration algorithm, so differences in this setting cannot be attributed to the algorithms alone. We have retained both initialized and uninitialized evaluations to make this limitation explicit and to show performance under both conditions.
Comment 4: The text states that the STAMM3D framework records the execution time and peak memory consumption for each alignment run. However, no measurements regarding these data are presented in the Results and Table 3 sections, and the analysis has been left to future work. In a benchmark paper, it is expected that time and memory complexity analyses—which determine the practical usability of algorithms—be included in the study.
Response 4: As per your request, we have provided Table A11 and Table A12 in Appendix C with recorded execution time and peak RAM consumption, respectively. However, due to the focus on registration quality and baseline expansion, which led to the use of two different workstations in the benchmarking process, we deferred the performance analysis.
Comment 5: It is stated that for failed or unpredictable registrations, the BBRE value was artificially set to 100. It is noted that errors were clipped to 0.144 when calculating the MRE (Mean Registration Error). No statistical significance of performance differences between algorithms (e.g., p-values, Wilcoxon test) was reported.
Response 5: We thank the reviewer. The value of 100 serves only as a marker of a complete failure and does not affect the quantitative results: such cases are counted as failures in Stability and clipped to 0.144 in MRE, like all other failed registrations. This is now stated explicitly in Section 3.2. Following the suggestion, we added statistical tests on the paired per-image-pair results (Appendix B). We used the Friedman test over all setups, and pairwise the Wilcoxon signed-rank test for BBRE and the exact McNemar test for success, with Holm correction for multiple comparisons. BBRE values below the ground-truth resolution limit of 0.005 were treated as ties, and the rank-based tests are insensitive to the marker value. The differences between the setups are significant (Friedman test, p < 10^-30). Each of the four initialized SITK setups yields significantly lower BBRE than every other setup (adjusted p <= 0.025 in both directions), while the differences among these four setups are not significant.
Comment 6: Nineteen registration pairs were generated from the 22 volumes in the dataset. Although various imaging modalities were covered, a dataset of 19 cases is numerically limited for making universal generalization claims. This should be more clearly emphasized among the paper’s limitations.
Response 6: As pointed out, STAMM3D consists of 22 high-resolution 3D images, each in the form of a large 2D image sequence. Most of the image acquisitions required time-consuming scanning, varying from hours to whole days, and involved sophisticated imaging setups with busy schedules, which caused the main difficulty for further expansion and is generally evident in the scarcity of real 3D image public datasets. As pointed out in Sec. 1, the majority of the datasets in the field of multimodal registration remain private, and existing public datasets mostly focus on a single modality pair and a specific domain of objects. Nevertheless, we were able to achieve a high level of data diversity in different modalities, covering many object domains: rat leg is related to the biomedical cancer research field, porous composites were studied for bio-resorbable implant usage, nutmeg and SD-card internal structure were examined for biology and microelectronics-related reasons, respectively, fossil brachiopod belongs to the paleontology field, and validation with a CAD model is a common manufacturing task. In addition, as pointed out in Sec. 3, the STAMM3D repository includes data augmentation utilities, which provide infrastructure to create custom benchmark setups from the provided 22 3D images. Therefore, we declare the novel universality of STAMM3D in comparison to existing datasets and benchmarks, which provides meaningful insight into registration algorithms' performance in case of modality- and object-diverse evaluation. We extended the discussion in Section 6 to clarify universality claims and address limitations arising from the number of data packages.
Comment 7: In Section 3.1, Table 1, and the Abbreviations list, the graphics card used for the CAD test is specified as the NVIDIA 4090, whereas in Section 3.1.6, the same model is listed as the NVIDIA 4060.
Response 7: We thank the reviewer for pointing this out. The CAD model 4060 has been verified, and the terminology is now consistent throughout the manuscript and the associated resources. We have also thoroughly proofread the manuscript and corrected the typographical and language errors, including those mentioned.
The following is a list of changes made during the revision process:
Lines 102-104. We have expanded the literature review by incorporating the references suggested by the reviewers, which demonstrate that accurate multimodal data alignment is a prerequisite for related tasks, such as computer-aided diagnosis and 3D reconstruction in remote sensing.
Lines 323-342. We have clarified the definition of a similarity transform, provided a rigorous mathematical definition, and explained why this particular class of transformations is considered in the benchmark.
Lines 366-368. We have added a clarification that the present work focuses on registration accuracy and robustness, while additional performance data (for future research) are provided in Appendix C.
Lines 390-394. We have added several sentences to clarify the dataset structure and its specific characteristics. The modality pairs and their characteristics, which form the basis of the benchmark, are summarized in Table 1 (which we have revised to improve clarity) and in a newly added Table 2.
Lines 484-494. We have clarified the procedure used to create the ground-truth transformations.
Lines 531 - 540. We have added explanations of the TRE and BBRE metrics, highlighting their differences and the technical details of their computation.
Lines 542-550. We have clarified the rationale behind the BBRE metric: it aggregates the globally maximal deviations from the ground truth, which are measured locally using TRE.
Lines 559-566. A qualitative analysis of the BBRE metric values has been added.
Lines 575-582. An explanation of the rationale behind selecting the clipping thresholds for MRE and Stability computation has been added.
Lines 613-615. An introductory sentence has been added to explain the selection of state-of-the-art classical non-learning-based registration algorithms and their evaluation in an out-of-the-box setting.
Lines 626-631. A note has been added clarifying that no baseline parameters were tuned on the STAMM3D evaluation pairs.
Lines 649-657. We clarified the design of the BBRE metric: it can be equally applicable to both forward- and reverse-direction registration.
Lines 665-670. A clarification has been added that reversing the registration direction from forward to reverse does not allow all algorithm-specific registration effects to be fully investigated within the benchmark. In particular, this comparison is limited because some algorithms employ parameter search grids whose configuration depends on the size of the initial or target volume, which becomes relevant when the registration direction is reversed.
Lines 723-726. An explanation of the statistical significance analysis of the results has been added.
Lines 771-776, 778. We have further clarified the motivation behind the benchmark. It introduces a novel approach to assessing registration universality by considering unprecedented modality and structural diversity compared to existing public 3D multimodal registration benchmarks and datasets, which typically focus on a single modality pair or a narrow object domain. We also acknowledge that the benchmark currently serves as a foundational proof-of-concept rather than providing exhaustive coverage of all possible 3D imaging scenarios.
Line 809. The statement has been clarified to explicitly refer to classical non-learning-based registration algorithms.
Lines 825-833. The Funding section has been updated.
Line 844. We have added the Informed Consent Statement.
Lines 849-853. Additional clarification regarding the versioning and update protocol has been provided.
Lines 858-935 (Appendix A). We have added a new Appendix dedicated to the theoretical properties of the BBRE metric, providing a theoretical justification for our choice of clipping parameters for MRE and Stability.
Lines 936-1002 (Appendix B). The new Appendix is dedicated to a statistical analysis of the results, with a particular focus on assessing the statistical significance of the findings.
Lines 1003-1011 (Appendix C). Additional material has been included, presenting an analysis of the execution time and peak memory consumption of the algorithms considered in the paper.
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsWell revised.
Comments on the Quality of English LanguageNA
Reviewer 2 Report
Comments and Suggestions for AuthorsThe authors have satisfactorily addressed the major concerns raised in the previous review. The revised manuscript provides clearer explanations of ground-truth construction, registration evaluation metrics, and experimental protocols. Additional analyses of threshold sensitivity, statistical significance, and computational performance further strengthen the reliability of the benchmark. The limitations of the dataset and experimental scope are also appropriately acknowledged.