Review Reports
- Liang Li 1,†,
- Chen-Yi Jiang 2,† and
- Jian Zhuo 2,*
- et al.
Reviewer 1: Anonymous Reviewer 2: Anonymous Reviewer 3: Anonymous
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThis interesting study should include a Section 2 focused on "Related Works" to situate this research in the global state-of-the-art context.
Figure 1: Stage 4 should be contrasted and improved. The caption is shallow.
Patients' data should be shared in a repository.
Figure 2 should be better described in the caption and in the full-length text.
All equations must be double-checked. Besides, each equation should be described carefully as well as all included parameters.
Figure 3 should include subfigures for proper comparison.
The style and typography across all images are inconsistent; please revise them to ensure homogeneity (cf. Figure 5).
It is recommended to include the Geometric mean and the Jaccard index in the evaluation metrics.
Table 2 presents the ablation study. Only a slight margin of superiority of the proposed method over the literature is detected. Explain the real advantages, innovations, and contributions of the proposed method. Besides, the gains using the attention mechanisms in Table 3 also show modest competitiveness.
This research should include the hyperparameters selection strategy and the training and testing curves (FLOPS, number of repetitions, variances, error analysis, etc.).
Databases, images, and code should be publicly available for replication.
The conclusion should be revised to be supported by the experimental data.
Comments on the Quality of English Language
The manuscript requires deep proofreading.
Author Response
|
Comments 1: This interesting study should include a Section 2 focused on "Related Works" to situate this research in the global state-of-the-art context. The conclusion should be revised to be supported by the experimental data. |
|
Response 1: Thank you very much for your suggestion. Regarding the limited references listed in lines 77–80 on object detection in ureteroscopic images, it was not our intention to overlook relevant research in this field. Rather, for the specific application scenario of ureteral orifice detection, related work and efforts have indeed not attracted widespread attention or dedicated efforts from the research community. We have further analyzed the cited work by our peers and made additional supplementary additions after the original text. |
|
Comments 2: Figure 1: Stage 4 should be contrasted and improved. The caption is shallow. |
|
Response 2: The image was subsequently refined to illustrate, following the acquisition of the initial model, a comparative assessment and enhancement against similar work, along with an added subsection covering ongoing model refinement and iteration. Refer to Fig. 1 for more information. Comments 3:Figure 2 should be better described in the caption and in the full-length text. All equations must be double-checked. Besides, each equation should be described carefully as well as all included parameters.Figure 3 should include subfigures for proper comparison. The style and typography across all images are inconsistent; please revise them to ensure homogeneity (cf. Figure 5). Response 3: We have taken note of the issue you raised concerning the layout of the figures. After attempting to present all substructures as subfigures, we found that the resulting comprehensive diagram became excessively large, making it difficult to discern clear structural details directly from the figure and potentially leading to confusion over the hierarchical organization. Consequently, we revised the styling of all subfigures, adopting the style guidelines of Figure 5 as you suggested. Comments 4:It is recommended to include the Geometric mean and the Jaccard index in the evaluation metrics. Response 4: Thank you for your suggestion. Regarding the G-mean metric, for a single-class object – specifically the ureteral orifice in this paper – we have already presented the results in terms of TP and FP in the table. Since TN corresponds to the background and does not require separate detection, it has been omitted from the results. Regarding the Jaccard coefficient, we have reported the mAP@50 scores in the table, and in the subsequent supplementary experiments, we further explain why we did not perform a strict comparison using the 50–95 interval average precision. Comments 5:Table 2 presents the ablation study. Only a slight margin of superiority of the proposed method over the literature is detected. Explain the real advantages, innovations, and contributions of the proposed method. Besides, the gains using the attention mechanisms in Table 3 also show modest competitiveness.This research should include the hyperparameters selection strategy and the training and testing curves (FLOPS, number of repetitions, variances, error analysis, etc.). Response 5: We agree that there is no significant performance improvement compared with existing literature, and we greatly appreciate your raising this important point, as it relates directly to the rationale behind our experiments. From the results, we have achieved a substantial improvement over our original baseline; the combination of different components simply makes this improvement appear more incremental. Regarding the attention mechanism, its benefits are twofold. First, it yields improved performance as shown in Table 3. Second, it demonstrates better cumulative performance during dataset accumulation, enabling faster and more effective training on small datasets. Additionally, we have conducted a horizontal comparison with mainstream YOLO-series object detection algorithms and, following your suggestion, have provided a detailed description of the training strategies and relevant parameters. We hope this helps to better convey the advantages and improvements of our algorithm. Comments 6:Databases, images, and code should be publicly available for replication. Patients' data should be shared in a repository. Response 6: Your demand for data openness has been duly acknowledged, as it would substantially aid the paper’s citation and reproducibility. Nevertheless, since the images stem from clinical recordings and are integral to a funded project with a pending patent application, we sought retrospective approval from the ethics board, a step that has inevitably curtailed the dissemination of the dataset and the algorithm’s source code. We feel great regret over this constraint. Additionally, we have released the contact information of the corresponding author, and we invite interested institutions to formally reach out for further discussion and data requests regarding this study. |
Author Response File:
Author Response.docx
Reviewer 2 Report
Comments and Suggestions for AuthorsIntroduction
(1) The hypothesis that an artificial intelligence detection model is necessary, given the anatomical characteristics of the ureteral orifice and the difficulties of securing a clear surgical field (e.g., shadows from blood vessels, illumination issues), is logical and well-founded.
(2) However, to clarify the clinical superiority of this study in the introduction, it is necessary to describe in greater depth the limitations of existing object detection algorithms (such as the SSD-based study by Peng et al. and the tracking system-combined study by Liu et al.) and the technical/clinical differentiation of your approach.
Materials and Methods
(3) The data extraction and model construction processes are reasonably designed. However, there is a prominent issue of dataset bias that significantly limits clinical applicability. According to the exclusion criteria, patients with stent implantation (acting as exogenous markers) or those whose physiological contours were compromised due to surgical damage were excluded. In actual ureteroscopy settings, AI assistance is most desperately needed in these complex cases accompanied by anatomical variations or lesions. To prove the robustness of the algorithm, you must perform external validation or statistically analyze the model's response (e.g., detection failure rate) on a dataset of ureteral orifices with abnormal morphologies, even if the proportion is small.
(4) Evaluating the model's performance using mAP, Precision, and Recall metrics is objective and appropriate. Nevertheless, to prove its superiority as a clinical tool, the value of the paper would be greatly enhanced if clinical metrics comparing the model's performance with the detection accuracy and time required by urologists (both novices and experts) are supplemented, rather than merely comparing numerical values among different algorithms.
Results
(5) In the results section, it is excellent that the optimization of the model was statistically proven through ablation experiments and attention mechanism replacement experiments.
(6) However, there is a lack of qualitative consideration for the false positive (FP) and false negative (FN) cases where the model failed to predict. You should transparently disclose the limitations and suggest future improvement directions by visually analyzing the clinical conditions where detection failures primarily occurred (e.g., insufficient illumination, severe bladder wall contraction, etc.).
Discussion and References
(7) There are several obvious errors in the reference citations, which significantly affect the academic credibility of the manuscript. For instance, in the text, the study by Lazo et al. is cited as [15] for lumen segmentation in ureteroscopic images, but reference [15] in the bibliography is an entirely different paper on MR-TRUS prostate registration by Fu Y et al.
(8) Furthermore, the text cites the AiFURS system study by Liang H et al. as [16], but the authors of reference [16] in the bibliography are listed as Wang C et al. A comprehensive review and correction are required to ensure that all text citations perfectly match the reference list.
(9) The authors themselves acknowledge the lack of images with lesions or surgical scars as a limitation in the discussion. Considering this, please detail a realistic roadmap in the discussion on whether the proposed model is intended solely as an initial screening or educational tool, or if it has the potential to be expanded to cover complex surgical scenarios in the future.
Author Response
|
Comments 1: (1) The hypothesis that an artificial intelligence detection model is necessary, given the anatomical characteristics of the ureteral orifice and the difficulties of securing a clear surgical field (e.g., shadows from blood vessels, illumination issues), is logical and well-founded. (2) However, to clarify the clinical superiority of this study in the introduction, it is necessary to describe in greater depth the limitations of existing object detection algorithms (such as the SSD-based study by Peng et al. and the tracking system-combined study by Liu et al.) and the technical/clinical differentiation of your approach.
|
|
Response 1: In the revised manuscript, we have added further explanations regarding the limitations of existing studies in lines 84–94. The novelty of our work is elaborated in detail later, as presented in lines 121–129 of the Introduction section.
|
|
Comments 2: (3) The data extraction and model construction processes are reasonably designed. However, there is a prominent issue of dataset bias that significantly limits clinical applicability. According to the exclusion criteria, patients with stent implantation (acting as exogenous markers) or those whose physiological contours were compromised due to surgical damage were excluded. In actual ureteroscopy settings, AI assistance is most desperately needed in these complex cases accompanied by anatomical variations or lesions. To prove the robustness of the algorithm, you must perform external validation or statistically analyze the model's response (e.g., detection failure rate) on a dataset of ureteral orifices with abnormal morphologies, even if the proportion is small. (9) The authors themselves acknowledge the lack of images with lesions or surgical scars as a limitation in the discussion. Considering this, please detail a realistic roadmap in the discussion on whether the proposed model is intended solely as an initial screening or educational tool, or if it has the potential to be expanded to cover complex surgical scenarios in the future. |
|
Response 2: We appreciate your concern regarding the exclusion criteria, which we fully understand is quite justified. Our work is primarily intended to demonstrate a feasible vision in which the ureteral orifice can be recognized in real time from low‑resolution dynamic images, regardless of its morphological variations. On one hand, cases that have already undergone pigtail stent placement do not require identification because the stent itself serves as a clear marker. On the other hand, for cases presenting with cauliflower‑like hyperplastic lesions or erosive hyperemia, physicians typically initiate medical treatment to reduce inflammation before proceeding with surgery, thereby avoiding high‑risk procedures during an “uncontrolled acute phase of urinary tract infection.” In more urgent situations, puncture rather than retrograde ureteroscopic surgery via the natural tract is preferred. Due to the anatomical characteristics of the natural tract, the loss of a complete contour in the exclusion criteria occurs only during stent placement or, as mentioned above, in acute pathological conditions. Once the stent has fulfilled its function and is removed, the tract regains a complete contour (otherwise the pathological state persists). Therefore, this robustness experiment, while extending beyond real clinical scenarios, remains reasonable within the framework of digital image object detection. We acknowledge that higher requirements would demand more resources and pose greater challenges during implementation. We have addressed this point in the revised manuscript after line 435, where we further clarify some foreseeable challenges that may arise if the method is to be deployed for clinical assistance in the future. Comments 3:(4) Evaluating the model's performance using mAP, Precision, and Recall metrics is objective and appropriate. Nevertheless, to prove its superiority as a clinical tool, the value of the paper would be greatly enhanced if clinical metrics comparing the model's performance with the detection accuracy and time required by urologists (both novices and experts) are supplemented, rather than merely comparing numerical values among different algorithms. Response 3: Since the dataset was annotated by physicians, the model's accuracy under unlimited time conditions essentially reflects its accuracy relative to that of experienced physicians. Furthermore, analyzing how physicians of different skill levels respond to the target would be highly valuable for the practical application of the model. However, the required human resources and time are currently beyond our means to realize this vision. Therefore, we have added this discussion to the relevant section as a potential direction for future development of our experiments. Comments 4:(5) In the results section, it is excellent that the optimization of the model was statistically proven through ablation experiments and attention mechanism replacement experiments. (6) However, there is a lack of qualitative consideration for the false positive (FP) and false negative (FN) cases where the model failed to predict. You should transparently disclose the limitations and suggest future improvement directions by visually analyzing the clinical conditions where detection failures primarily occurred (e.g., insufficient illumination, severe bladder wall contraction, etc.). Response 4: Visual analysis of failed detection cases is clearly very important. Previously, we only analyzed the failures theoretically and statistically. Therefore, we have added this content in the Discussion section. In lines 426–434 of the fifth section of the revised manuscript, you will find our newly added analysis and corresponding visualization figures. Comments 5:(7) There are several obvious errors in the reference citations, which significantly affect the academic credibility of the manuscript. For instance, in the text, the study by Lazo et al. is cited as [15] for lumen segmentation in ureteroscopic images, but reference [15] in the bibliography is an entirely different paper on MR-TRUS prostate registration by Fu Y et al. (8) Furthermore, the text cites the AiFURS system study by Liang H et al. as [16], but the authors of reference [16] in the bibliography are listed as Wang C et al. A comprehensive review and correction are required to ensure that all text citations perfectly match the reference list. Response 5: Regarding reference 15, it is a clear mistake; we appreciate your pointing it out. The citation has been updated to the correct source in the relevant spot of the manuscript. Reference 16’s citation is correct, but we inadvertently cited the second author instead of the first in the paper. To avoid misunderstanding, we have restored the author’s name to the first author; we thank you for your careful observation. |
Author Response File:
Author Response.docx
Reviewer 3 Report
Comments and Suggestions for AuthorsA fresh look at the work shows researchers blending a big- filter ConvNeXt model with attention layers to spot ureteral openings. Yet medical relevance aside, flaws creep into how models were trained - processes lack clarity. Validation methods fall short, raising doubts about real-world usefulness. Hardware assumptions feel stretched beyond actual clinic setups. Metrics chosen do not back up statements on reliability or broad function. At this point, the approach cannot confidently claim stability across settings or practical use in hospitals.
Major Comments:
- One big problem hides in how the model learned. Starting fresh without prior training might sound clean - yet here it backfires. A network packed with complex layers meets just over fourteen hundred pictures. That mismatch raises red flags right away. Normally, models like this borrow knowledge from vast image sets before focusing on smaller tasks. Skipping that step forces the system to guess patterns it cannot truly grasp. Memorization becomes more likely than real understanding. Evidence should show if losses dropped evenly across learning stages. Curves revealing performance gaps between training and testing would expose hidden flaws. Claiming solid results means proving stability beyond the initial data pool. Just saying it works does not remove the need for deeper explanation. Why avoid widely accepted starting points needs clear reasoning - not assumptions. Numbers alone do not fix shaky foundations.
- The study pulled information from two hospital locations, saying it would support broader applicability. Yet combining both sets into one pool breaks the point of using multiple centers at all. For real-world reliability, training should happen on records from Site A, then tested separately on untouched cases from Site B. Right now, splitting everything randomly seven to three just checks consistency within the same mix. That approach confirms nothing beyond its own sample.
- The paper states it reaches 50 frames per second -roughly 20 milliseconds each. Yet deep inside Section 3, around lines 261 to 262, there's a catch. That speed runs solely on an NVIDIA RTX 4090. This chip? A beast meant for bulky workstations, far from what fits in hospital surgery spaces. Medical setups rarely pack such hardware into endoscopy units. So hitting just 50 FPS even with this powerhouse hints at serious processing demands. Performance like that won’t transfer smoothly into real clinical environments.
- Got to share how complex the model is - number of parameters, FLOPs, MACs. Speed matters just as much, so test it where it counts. Run inference on gear like low-end medical tools actually use, say edge-AI chips or basic GPUs. Results from strong lab machines won’t cut it. What runs fast there might crawl elsewhere. Real-world performance shapes trust. Skip the high-powered shortcuts, show how it works on modest hardware instead. Details matter most when lives are involved.
- Wrong score choice skews outcomes. Heavy dependence on mAP at 0.5 shows only part of the picture. Today’s detection tools need more than just half-overlap checks. Medical uses demand sharper box accuracy, yet this method skips finer alignment needs. Tight spaces call for tighter predictions - something this metric misses completely.
- A closer look shows what’s missing - full COCO metrics need to appear, especially mAP across thresholds from 0.5 to 0.95. Since the paper already points out how hard it is to spot tiny objects, results broken down by size matter too. Not just overall scores, but separate numbers for small, medium, and large items should show up clearly. Without these, the picture stays incomplete, even if earlier sections raise the right concerns.
- The evaluation misses key recent models, focusing only on older ones like ConvNeXt. Instead of just those, it should have been tested alongside leading real-time systems now common in endoscopy work - say, YOLOv7 or RT-DETR. Though the method outperforms its chosen baselines, skipping newer rivals weakens the impact. Comparisons matter most when they reflect what's actually being used today. Leaving them out makes the advance seem isolated, less grounded in current practice.
Author Response
|
Comments 1: One big problem hides in how the model learned. Starting fresh without prior training might sound clean - yet here it backfires. A network packed with complex layers meets just over fourteen hundred pictures. That mismatch raises red flags right away. Normally, models like this borrow knowledge from vast image sets before focusing on smaller tasks. Skipping that step forces the system to guess patterns it cannot truly grasp. Memorization becomes more likely than real understanding. Evidence should show if losses dropped evenly across learning stages. Curves revealing performance gaps between training and testing would expose hidden flaws. Claiming solid results means proving stability beyond the initial data pool. Just saying it works does not remove the need for deeper explanation. Why avoid widely accepted starting points needs clear reasoning - not assumptions. Numbers alone do not fix shaky foundations. |
|
Response 1: You raised a very profound question concerning the training methodology of the model. First, the network's complex architecture can exhibit hallucinations when confronted with limited data, in response to this concern, you further asked why we eschewed a generic pre training strategy in favor of training from scratch a seemingly large model, It is likely apparent that our survey of prior work on ureteral orifice detection in the background introduction is rather limited, this is because such work is uncommon in this domain, leaving no extensive pre training database from which we could perform transfer learning, while clinical images of colorectal polyps, tumors, and gastric ulcers have been widely investigated, their anatomical characteristics diverge fundamentally from those relevant to our study, early experimental results confirmed that pre training on these datasets did not significantly aid our later training processes. Your suggestion regarding loss curve analysis is exceptionally astute, such analysis will support the evaluation of the training procedure's efficacy and the dataset's scientific rigor, accordingly, we have made the corresponding revisions in the paper. |
|
Comments 2: The study pulled information from two hospital locations, saying it would support broader applicability. Yet combining both sets into one pool breaks the point of using multiple centers at all. For real-world reliability, training should happen on records from Site A, then tested separately on untouched cases from Site B. Right now, splitting everything randomly seven to three just checks consistency within the same mix. That approach confirms nothing beyond its own sample. |
|
Response 2: We apologize that our vague description may have misled your understanding. This was a narrative error caused by an overly brief and imprecise account of the dataset construction process. In the revised manuscript, we have provided further detailed clarification regarding this ambiguous part. In the updated version, you will find a clear description in lines 165–169, which reads as follows: Data partitioning was performed based on different hospital campuses. Specifically, the videos from the Songjiang District campus of Shanghai General Hospital, which were more abundant, were selected as the training set, while the dataset from the Hongkou District campus, following data cleaning and classification, was designated as the validation set, yielding an overall approximate ratio of 7:3. Comments 3:The paper states it reaches 50 frames per second -roughly 20 milliseconds each. Yet deep inside Section 3, around lines 261 to 262, there's a catch. That speed runs solely on an NVIDIA RTX 4090. This chip? A beast meant for bulky workstations, far from what fits in hospital surgery spaces. Medical setups rarely pack such hardware into endoscopy units. So hitting just 50 FPS even with this powerhouse hints at serious processing demands. Performance like that won’t transfer smoothly into real clinical environments. Got to share how complex the model is - number of parameters, FLOPs, MACs. Speed matters just as much, so test it where it counts. Run inference on gear like low-end medical tools actually use, say edge-AI chips or basic GPUs. Results from strong lab machines won’t cut it. What runs fast there might crawl elsewhere. Real-world performance shapes trust. Skip the high-powered shortcuts, show how it works on modest hardware instead. Details matter most when lives are involved. Response 3: The complexity and number of parameters of a model can have a profound impact on the subsequent development of this task. In the further revision of our paper, we have compared the detailed parameters of our model with those of similar models and their performance. This process will help readers gain a deeper understanding of the usability of our work. You will find the parameters of our model as well as those of other mainstream object detection models in Table 5 of the revised manuscript. Furthermore, the deployment of the model on edge devices, as you suggested, is a highly meaningful research direction. However, given that our work is still at a preliminary stage, the corresponding hardware experiments and optimizations are, for us, a longer‑term issue that requires further consideration. We have added this discussion to the relevant section to indicate potential future directions for extending our research. Comments 4:Wrong score choice skews outcomes. Heavy dependence on mAP at 0.5 shows only part of the picture. Today’s detection tools need more than just half-overlap checks. Medical uses demand sharper box accuracy, yet this method skips finer alignment needs. Tight spaces call for tighter predictions - something this metric misses completely. A closer look shows what’s missing - full COCO metrics need to appear, especially mAP across thresholds from 0.5 to 0.95. Since the paper already points out how hard it is to spot tiny objects, results broken down by size matter too. Not just overall scores, but separate numbers for small, medium, and large items should show up clearly. Without these, the picture stays incomplete, even if earlier sections raise the right concerns. Response 4: Regarding the classification of small vs. large objects, we acknowledge that we did not deliberately distinguish between them in the description of our experimental dataset. Thank you for pointing this out. First, all images used were derived from ureteroscopic videos. After cropping out redundant areas, the typical image size is only about 100 pixels in both width and height. This means that when the target occupies less than one‑third of the entire image, it fully qualifies as a small object under the COCO dataset definition. This situation can be clearly observed in Figure 10; thus, the entire dataset consists of small objects. Furthermore, as shown in Figure 10, the sizes of predicted bounding boxes vary considerably across different algorithms and results. This is because the effective and complete detection of the target has an inherently ambiguous boundary. Unlike polyps or tumors, the ureteral orifice, due to factors such as orientation and morphology, does not have a well‑defined boundary. Although we manually set a box‑selection standard during dataset construction, it was more a convenience for physician annotation than an absolute physiological or anatomical ground truth. Consequently, physicians tended to enlarge the bounding boxes to make the annotation more inclusive, which is an important factor affecting the high‑precision range. In our newly added experiments, we have included relevant data on this aspect and provided corresponding explanations of the results. Comments 5:The evaluation misses key recent models, focusing only on older ones like ConvNeXt. Instead of just those, it should have been tested alongside leading real-time systems now common in endoscopy work - say, YOLOv7 or RT-DETR. Though the method outperforms its chosen baselines, skipping newer rivals weakens the impact. Comparisons matter most when they reflect what's actually being used today. Leaving them out makes the advance seem isolated, less grounded in current practice. Response 5: Your suggestion to perform a lateral comparison among similar models is very valuable. Accordingly, we trained the YOLO series and RT DETR networks from scratch without any pre training, and we compiled the complete COCO dataset along with the models’ performance parameters into a new supplementary table 5 added to the manuscript to address the missing work. This also addresses the concern you raised earlier about the missing key parameters.
|
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThe authors have satisfactorily addressed all my previous concerns.
Reviewer 2 Report
Comments and Suggestions for AuthorsI have read the revised manuscript and the rebuttal letter. Thank you for your careful revisions. Your clinical justification for the dataset's exclusion criteria makes a lot of sense. It is true that patients who already have a pigtail stent do not require AI assistance for orifice localization, and those with acute inflammation usually receive medical treatment prior to surgery. This clinical context adequately defends the scope of your training data.
Furthermore, I appreciate that you explicitly acknowledged the lack of a direct comparison with urologists as a study limitation rather than overstating the model's current capabilities. Adding Figure 12 to discuss false-positive and false-negative cases—particularly noting how blurred textures and closed orifices affect the bounding boxes—greatly improves the transparency of your results. Finally, the citation errors regarding references 15 and 16 have been properly fixed. The manuscript has improved significantly, clearly states its practical constraints, and is now suitable for publication.
Reviewer 3 Report
Comments and Suggestions for AuthorsThanks for the very important changes . Now it has all requirements to be published.