FedZaCt: Federated Learning with Z Average and Cross-Teaching on Image Segmentation
Round 1
Reviewer 1 Report
This approach is smart and seems efficient, I recommend its publication in tis journal.
Author Response
Please see the attachment
Author Response File: 
Author Response.pdf
Reviewer 2 Report
I have attached my comments.
Comments for author File: Comments.pdf
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The paper under review proposes a federated learning framework, called "federated learning with z average and cross teaching". The proposed novelty and main difference to the common federated average lies in "z-average models have n models not one global model"; "each client downloads all the Z-average models instead of the corresponding model" and a combination of mechanisms on top to average. (lines 137++).
Unfortunately, there are several crucial parts in the paper that would need substantial improvement to reach an expected research standard in federated machine learning:
* the "z-averaging" and "cross-teaching" are somehow advocated as an entirely novel framework and paradigm for federated learning. In reality, z-averaging seems to be nothing more but one of the most popular techniques in all of machine learning (independently of the exact application), namely model ensembles. Similarly, what is then explained with somewhat ambiguous descriptions in "cross-teaching" seems to essentially correspond to accounting for the uncertainty between models as auxiliary information (each model finds slightly different solutions). I am not providing references here because the idea of ensembles and their benefits can be found in most machine learning books and has been around for various decades.
* From the federated learning side, the analysis, both empirically or theoretically is sub-par to typical federated learning publications. Federated learning is not only about a single accuracy metric, as used in the paper, but a genuinely important aspect is both the memory and even more importantly the communication cost. There is no analysis or mention of either in the paper, but it is clear that the idea of n-models comes with significant disadvantages in terms of computation and communication overheads. In contrast to such an increase, the shown accuracy improvements seem marginal at best.
* Independently of the above lack of a detailed analysis, there seems to be a flaw in the way that the existing experimental comparison is set-up. In short, it unfortunately is rather one sided and not fair in the way it compares algorithms. Primarily, this is due to the fact that communication is not taken into account and what is typically treated as important hyper-parameters, namely individual epochs and rounds of communication (leading to the question of communication frequency) is fixed. However, communication 5 models every 5 epochs is not as expensive (neither computationally nor in terms of actual sending/receiving) as communicating a single model. The shown comparison is thus inadequate as a fair number would communicate 5 models every 5 epochs and 1 model once per epoch to keep the cost the same in the comparison. I highly suspect that a single model per client that is communicated more frequently also suffers from less disparate in slightly asynchronous updates and would potentially even beat the less often communicated n-models approach that is proposed.
Overall, the idea of leveraging ensembles and their uncertainty in federated learning is an interesting one. The present execution and detail of the paper at hand will however require many improvements before being acceptable for publication. Some minor aspects such as language and presentation quality (e.g. some figures are referenced at intuitive places, tables 1+2 are pretty redundant with 3+4) should also be improved, but are less important in the whole picture of necessary improvements.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
