Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper presents RaTrack, an innovative approach to improving rare disease detection in chest X-rays by addressing the long-tail distribution problem. The authors propose using a diffusion model with a novel calibration technique to generate high-quality synthetic images of rare diseases, effectively increasing both the quantity and diversity of training samples for tail classes. The key contributions include a head-to-tail alignment strategy that transfers knowledge from common to rare diseases during generation, and a two-stage filtering mechanism using both detection scores and CLIP scores to ensure the quality of generated images. The method shows promising results on two datasets (VinDr-CXR and RSNA), with significant improvements in detection performance for rare diseases - for example, improving the mean Average Precision (mAP) from 18.7% to 20.8% on VinDr-CXR, with particularly strong gains for rare classes (mAPr increasing from 10.1% to 17.0%). While the paper demonstrates clear effectiveness, it could benefit from more detailed ablation studies on the calibration parameter choices and a deeper analysis of failure cases.    If authors could apply some multimodal foundation models with the proposed methods to check the superiority with the proposed method like RAYDINO, it would strengthen the paper's contribution.Several interesting citations related to long-tail classification problems are missing from this paper.
1. Dam, Tanmoy, Md Meftahul Ferdaus, Mahardhika Pratama, Sreenatha G. Anavatti, Senthilnath Jayavelu, and Hussein Abbass. "Latent preserving generative adversarial network for imbalance classification." In 2022 IEEE International Conference on Image Processing (ICIP), pp. 3712-3716. IEEE, 2022.

2. S. S. Mullick, S. Datta, and S. Das, “Generative adversarial minority oversampling,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1695–1704.

3. X. Wang, Y. Lyu, and L. Jing, “Deep generative model for robust imbalance classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 124–14 133

Comments on the Quality of English Language

NA

Author Response

1. While the paper demonstrates clear effectiveness, it could benefit from more detailed ablation studies on the calibration parameter choices and a deeper analysis of failure cases.    
Response:
Thank you for your valuable suggestions. We have added results from using different thresholds in the filtering stage and provided a discussion on these results, as illustrated in Section 4.2 "filtering hyperparameters"(Page 9 Line 271). We have observed that overly strict filtering conditions limit the diversity of generated data, which in turn restricts the model's capability. Conversely, overly lenient filtering introduces noise, which also leads to a decline in model performance.
Since medical diagnosis is a highly specialized field, many conditions are challenging to distinguish visually, even for experts. Therefore, we can only draw conclusions based on the detection performance.

2. If authors could apply some multimodal foundation models with the proposed methods to check the superiority of the proposed method like RAYDINO, it would strengthen the paper's contribution.
Response:
We respectfully agree with this. Since the RAYDINO is not open-sourced, we have utilized another multimodal foundation model for chest X-rays EVA-X[1] for the generation filtering, the results are almost the same:
| Method | EVA-X | MedKLIP |
| ------ | ----- | ------- |
| mAP    | 20.7  | 20.8    |
| mAPr   | 17.2  | 17.0    |

Since our diffusion model is trained on a limited dataset, the maximum diversity it can introduce to tail data is inherently restricted. As a result, even with more strong filtering models, the overall impact on final performance remains limited.

[1] Yao, Jingfeng, et al. "EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learning." arXiv preprint arXiv:2405.05237 (2024).

3. Several interesting citations related to long-tail classification problems are missing from this paper.
Response:
Sorry for missing these related papers, we have added in the newly added "related work" part.

Reviewer 2 Report

Comments and Suggestions for Authors Paper title: Track the rareness: Improving Long Tail Medical Detection with Calibrated Diffusion Model   The paper structure should be revised for better organization. Namely, the “Introduction” section should be separated into two sections, placing the literature review in another section (for example, in a section titled “Related work”, or something similar). Additionally, more recent and relevant works should be analyzed/discussed, which will contribute to a better understanding of the contributions of the work presented by the authors, compared to the most recent approaches proposed by other authors. Only then will the degree of innovation of the work proposed in this paper be clear.   The text (and images) explaining the results achieved should not be placed in the “Introduction” section, but rather in a “Results” or “Discussion of results” section.   Figures must be placed after (and close to) the text where they are referred. For example, figure 2 is placed before the “Methods” section, where it is referred to. Furthermore, figure 2, namely its workflow, should be explained in more detail. Figures 3 and 4 are placed well before the text where they are mentioned. When the reader views them, he cannot perceive their meaning or analyze their content.   Although the authors compare their model with others, there is not enough information about the models considered (which models are called A, B, C, ...?), nor about the training process of those models. Were they fit on the same data set? No statistical significance test is performed to support the claim that the proposed model outperforms others.   The conclusion is brief and does not adequately summarize the findings or their implications. More importantly, the limitations of the study are not recognized and suggestions for future research are absent.   There are several grammatical errors throughout the text. Some sentences are not written clearly and concisely, making the manuscript difficult to read. Comments on the Quality of English Language

There are several grammatical errors throughout the text. Some sentences are not written clearly and concisely, making the manuscript difficult to read.

Author Response

1. The paper structure should be revised for better organization. Namely, the “Introduction” section should be separated into two sections, placing the literature review in another section (for example, in a section titled “Related work”, or something similar). 
Response: 
We agree. We have added a new section named Related Work introducing the long tail generative models and long-tail discriminative tasks, which enhances the overall structure and completeness of the paper.(Page3 line 94)

2. Additionally, more recent and relevant works should be analyzed/discussed, which will contribute to a better understanding of the contributions of the work presented by the authors, compared to the most recent approaches proposed by other authors. Only then will the degree of innovation of the work proposed in this paper be clear.   
Response: 
We agree. In the Discussion section,  we introduce three other methodologies to address the long-tail problem. We discuss their limitations and provide comparisons with our method.(Page 12 line 287-308)

3. The text (and images) explaining the results achieved should not be placed in the “Introduction” section, but rather in a “Results” or “Discussion of results” section.   
Response: 
We agree. We have moved the discussion on the TP score and feature distribution, along with the corresponding figure, to the Discussion part.

4. Figures must be placed after (and close to) the text where they are referred. For example, figure 2 is placed before the “Methods” section, where it is referred to. Furthermore, figure 2, namely its workflow, should be explained in more detail. Figures 3 and 4 are placed well before the text where they are mentioned. When the reader views them, he cannot perceive their meaning or analyze their content.
Response:
We agree. We have adjusted the order of these figures: the framework diagram of Ratrack(Figure 2) is now placed near its method section, and the filtering strategy diagram(Figure 3) is located alongside the relevant text. A clip score distribution figure(Figure 5) has been moved to the experiments section. To facilitate a better understanding of our method, we have added a block chart, like Figure 1, to reduce the difficulty for readers in comprehending our approach.

5. Although the authors compare their model with others, there is not enough information about the models considered (which models are called A, B, C, ...?), nor about the training process of those models. Were they fit on the same data set?
Response:
We apologize for the misleading presentation here. Models A, B, C, etc., refer to models trained using different strategies, and we have added explanations for these in section 4.2 (Page 8 261-267). They were indeed trained and tested on the same dataset. 

6. No statistical significance test is performed to support the claim that the proposed model outperforms others.   
Response:
We conducted multiple runs for all experiments in Table 2 and calculated the variance across these runs. Assuming a normal distribution, we performed t-tests for all methods compared to ours, and the p-values for all other methods were less than 0.05, indicating statistically significant differences.

6. The conclusion is brief and does not adequately summarize the findings or their implications. More importantly, the limitations of the study are not recognized and suggestions for future research are absent.   
Response:
Agreed. We have expanded the Conclusion section to include additional insights on our findings and to address the limitations of our approach. Finally, we have outlined a potential direction for future work.(Page 11 line 348)

7. There are several grammatical errors throughout the text. Some sentences are not written clearly and concisely, making the manuscript difficult to read.
Response:
Thank you for the advice. We thoroughly reviewed all sentences and corrected any grammatical errors. Additionally, we revised some sentences that were unclear or difficult to understand for improved clarity.

Reviewer 3 Report

Comments and Suggestions for Authors

The authors should address the following concerns:

1. The abstract should be revised to briefly include motivation, objective, methodology, and results.

2. The introduction section should include relevant statistical data to support the proposed methodology.

3. At the end of the introduction, the contributions made by the authors should be highlighted.

4. There should be a separate literature review section following the introduction.

5. The authors should present a block diagram of the proposed methodology.

6. It is recommended to include training curves and present assessment metrics in a separate subsection.

7. In the discussion section, the authors should compare their work with state-of-the-art techniques and include more recent articles.

Author Response

1. The abstract should be revised to briefly include motivation, objective, methodology, and results.

Response: 
Thank you for the suggestion. We have slightly modified the abstract and transferred it into the desired format(modified in the Abstract). Under the format, the structure becomes more clearer.

2. The introduction section should include relevant statistical data to support the proposed methodology.
Response: 
Thank you for the suggestion. We first add the long tail statistic for illustrating the head-to-tail bias (Added in page 1 line 22-23) for better support the standing point of our research. Furthermore, the experimental results (page 3 line 77-79) is also included to support the effectiveness of the proposed methods. 

3. At the end of the introduction, the contributions made by the authors should be highlighted.
Response:
We agree. The summarized contribution is listed in page 3 line 84-91.

4. There should be a separate literature review section following the introduction.
Response: 
Thank you for your advice. We have added a new section named Related Work introducing the long-tail generative models and long-tail discriminative tasks, which enhances the overall structure and completeness of the paper.

5. The authors should present a block diagram of the proposed methodology.
Response: 
 We agree. Indeed, designing a block chart to illustrate our overall workflow can make the paper clearer and easier to understand. We provided a block chart in Figure 1 to illustrate the workflow of our method.

6. It is recommended to include training curves and present assessment metrics in a separate subsection.
Response: 
We agree. Based on your suggestion, we have added a "Training Curves and Assessment Metrics" subsection in Section 4.1(page 7 line 230-244). This subsection discusses the fitting target shows the fit loss of our model and then explains the evaluation metrics used. 

7. In the discussion section, the authors should compare their work with state-of-the-art techniques and include more recent articles.
Response: 
We agree. In the Discussion section, we introduce three other methodologies to address the long-tail problem. We discuss their limitations and provide comparisons with our method.(Page 12 line 287-308)

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have addressed my concern. 

Reviewer 3 Report

Comments and Suggestions for Authors

Authors have made all the required modifications, can be accepted for publication.