Optimal Transport-Based Heterogeneous Federated Learning for Chest X-Rays

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. Figure 1, “The Framework of Federated Learning,” only presents the overall architecture of federated learning, while the manuscript lacks a detailed explanation of the components shown in the figure and their corresponding workflow.
2. Some abbreviations are not defined in full when they first appear, such as CNN, VGG11, OHFL, and SGD. The authors are advised to carefully check the entire manuscript and ensure consistent capitalization and notation throughout.
3. The authors state in Section 2.2 that the Sinkhorn algorithm can effectively reduce the computational complexity of optimal transport. However, the implementation details do not clarify whether the Sinkhorn algorithm is actually adopted, and no relevant parameters are provided. The authors are advised to supplement the corresponding implementation details and parameter settings.
4. It is recommended to add an illustrative figure in Section 3.1, “Layer Alignment,” to improve the readability and comprehensibility of the proposed method. In addition, the manuscript only states that the cost matrix is constructed by calculating the distance between pre-activation matrices, but it does not specify the distance metric or provide the corresponding calculation formula.
5. Section 4.1 only states that the dataset is divided into training and test sets at a ratio of 9:1, but it does not clarify whether an independent validation set is used. This weakens the rigor of the experimental results. The authors are advised to provide a complete description of the data-splitting protocol. In addition, the experiments are conducted on only one chest X-ray dataset, which is insufficient to fully demonstrate the generalization capability of the proposed method. It is recommended to include additional publicly available medical imaging datasets for further validation.
6. Section 4.2, “Model and Baseline Methods,” does not clearly specify the number of training epochs, batch size, random seed, number of repeated runs, or the hardware and memory configurations used in the experiments. The authors are advised to provide complete training parameters and hardware information to improve the reproducibility of the study.
7. The manuscript claims three main contributions, but no corresponding component-wise ablation experiments are provided. It is therefore unclear whether all three components contribute effectively to the final performance. The authors are advised to conduct additional ablation studies to evaluate the individual contribution of each component.
8. The discussion of research limitations and future work in the conclusion is relatively brief, and some related studies and references are outdated. The authors are advised to further elaborate on the limitations and future research directions and to include representative high-quality publications from the past five years.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript addresses an important topic by proposing an optimal transport-based heterogeneous federated learning framework for chest X-ray classification. The proposed approach appears promising; however, several important issues should be addressed before the manuscript can be considered for publication. The novelty of the proposed study should be more clearly demonstrated. The authors are encouraged to strengthen the literature review by incorporating more recent and state-of-the-art references, particularly those published within the last two to three years. A clearer discussion highlighting the differences and advantages of the proposed approach over existing methods is necessary. The manuscript should include a comprehensive comparison table summarizing the performance of existing methods that have been evaluated on the same or a similar chest X-ray dataset. This comparison would allow readers to better understand the contribution and competitiveness of the proposed method with respect to the current literature. The proposed framework is evaluated using a single dataset. To demonstrate the robustness and generalization capability of the proposed method, the authors should validate the model on an additional publicly available dataset with similar characteristics. The experimental evaluation should include comparisons with well-established classical machine learning and deep learning approaches commonly used for chest X-ray classification. Such comparisons are necessary to provide a fair and comprehensive assessment of the proposed method. The manuscript does not clearly explain how the training, validation, and test sets were generated. The authors should explicitly describe the data partitioning strategy, including whether patient-wise separation was employed, how the validation set was selected, and whether cross-validation was considered. Since federated learning is inherently affected by data heterogeneity, the manuscript should provide additional experiments or discussions investigating the impact of different data distributions (e.g., varying degrees of non-IID data) on the performance of the proposed framework. Such analyses would provide valuable insights into the robustness and practical applicability of the proposed method.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

In this paper, the authors propose a new federated learning framework for chest X-ray analysis. The motivation for the study and its contributions are clearly explained in the introduction section. However, I would recommend adding a short paragraph describing the paper's content at the end of the introduction section. The study is interesting and relevant, but lacks sufficient statistical validation and methodology description, so I recommend major revision.

1. There is something strange in the abstract sentence „…reduces aggregation time by approximately77…“, so please check this and correct it accordingly.

2. Training and tuning details are limited, and it should be explained in more detail how the hyperparameter search was performed, whether any standard optimizers were used or a basic grid search was applied. Also, the reproducibility of the study should be ensured by adding details on the neural network models used. The authors mention using a simple CNN, but without describing the architecture and how it was designed. Also, it would be useful to explain what the reason was for using VGG11 specifically. Why is it special compared to other state-of-the-art deep networks? I am also missing the information on whether the architectures were pretrained or trained from scratch.

3. The dataset contains around 5K images in total. Were there any data augmentation techniques performed on the dataset before training the models to improve the training stability? What was the train/val/test split in the experiment? Were there any overfitting issues, and how were they solved?

4. It was not mentioned in the methodology whether the dataset is balanced. If an imbalanced dataset was used, how did that affect the experiment, and was it specifically handled?

5. No robustness statistics are provided, as there is no information about multi-seed runs, using k-fold cross-validation, etc. There are no CI/SD reported, and no significance tests applied when comparing the performance of the proposed method and the existing ones (Table 2 and Table 3). Despite very small deltas, repeat experiments and report uncertainty before making strong ranking claims.

6. The complexity evaluation of the proposed method is not measured against other methods. It would be useful to see whether the proposed method increases complexity in addition to the improved performance metrics.

7. An ablation study would be very useful, especially on the mentioned „impact of data heterogeneity on model performance“. Also, examine which parts of the proposed framework have the most influence on the obtained performance results.

There are some other (minor) points that I consider important to address:

1. I would like to kindly ask the authors to address the limitations of their work in the Conclusion section and to list the possible shortcomings of their methodology, if they exist. Also, future work should be extended.
2. The references in the paper are rather old; there are only 9 out of 26 references from the year 2021 onwards. I believe that adding more recent references would increase the importance and the novelty of the study.
3. The full name of each abbreviation should be given only once (when first mentioned). First, the abbreviations IID and OHFL are used without providing their full names. Next, there is an abbreviation whose full name is given more than once in the main paper text (OT). It would be useful to define the abbreviations CDD and CIA before using them in Table 3.
4. It would be useful to mark the best result in Tables 2 and 3 in bold.
5. Figure 1 is not referenced in the text.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have revised the manuscript according to my previous comments. The quality of the revised manuscript has improved, and I have no further comments at this time.

Comments on the Quality of English Language

The authors have revised the manuscript according to my previous comments. The quality of the revised manuscript has improved, and I have no further comments at this time.

Reviewer 2 Report

Comments and Suggestions for Authors

The required revisions have been made, and the manuscript can now be considered acceptable for publication.

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have addressed all my comments.