CrowdAttention: An Attention Based Framework to Classify Crowdsourced Data in Medical Scenarios

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper presents a novel method for Learning from Crowds (LFC), based on two modules trained end-to-end: (1) a classifier, and (2) a crowd attention module. The key innovation lies in the crowd attention module, which employs cross-attention to dynamically aggregate the crowdsourced labels based on the learned classifier. A major advantage of this approach is that it does not rely on per-annotator parameterization, which can be expensive to scale.

I would like to highlight the overall quality of the manuscript. The abstract, introduction, and related work sections are well-written and provide valuable context for the problem. The references used are relevant and up-to-date. The proposed methodology is original and addresses a gap in the literature, particularly in terms of deepening the understanding of crowdsourcing, which has often been overlooked in prior work. The experimental section is comprehensive, including both synthetic and real-world datasets, multiple baselines, and an insightful discussion.

I would like to pose the following questions and comments to further enhance the quality of the paper:

1. Figure 1 should be self-contained. I suggest including definitions of the notations either within the figure itself or in the caption.

2. Regarding the crowd attention module, does it involve any learnable parameters? What is the computational cost associated with this module?

3. The impact of the results would be more robust if the authors conducted statistical significance tests on the performance metrics.

4. While the paper emphasizes the predictive performance of the proposed method, I would be interested in learning more about its other characteristics, advantages, or limitations:

   • How much data and how many annotators are needed for the method to perform well?

   • Are the attention scores interpretable, and if so, what insights can they offer?

   • How well does the method scale computationally?

   Including additional experiments or discussion around these points would strengthen the paper.

Author Response

We sincerely thank you for your constructive and valuable comments, which have helped us to significantly improve our manuscript. Please find our detailed, point-by-point responses in the attached PDF file.

Sincerely,
Julián Gil, David Cárdenas, Álvaro Orozco, Germán Castellanos and Andrés Álvarez.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Please PFA and carefully check the highlighted part.

Comments for author File: Comments.pdf

Author Response

We sincerely thank you for your constructive and valuable comments, which have helped us to significantly improve our manuscript. Please find our detailed, point-by-point responses in the attached PDF file.

 

Sincerely,
Julián Gil, David Cárdenas, Álvaro Orozco, Germán Castellanos and Andrés Álvarez.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I believe that the quality of the paper has been improved with this revised version. So I recommend it for publication. I just only want to point out that the references are missing in the revised version.

Author Response

Dear Reviewers,

We would like to sincerely thank you for your valuable comments and suggestions, which have helped us improve the quality and clarity of our manuscript.

In the attached document, you will find our detailed responses to each of your observations, along with the corresponding changes made in the revised version of the paper.

We appreciate your time and consideration.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for addressing all my comment: Although manuscript is updated, yet there are few issues, must be addressed before final consideration.

Citations and References: We will thoroughly review the manuscript to ensure all in-text citations are correctly formatted and that the reference section is complete. This will enhance the clarity and credibility of our work.

Figure 4 Improvements: We will enhance Figure 4 by adjusting the DFLC-MW trend line for better visibility. Additionally, we will resize the figure to make it full-page and increase the x-axis dimensions. The legend will be repositioned to ensure it fits well within the visual without compromising clarity.

Contributions in Introduction: We will explicitly outline our contributions in the Introduction section, clearly stating, for example, “We propose CrowdAttention, which...” This will help readers understand the significance of our work from the outset.

Author Response

We would like to sincerely thank you for your valuable comments and suggestions, which have helped us improve the quality and clarity of our manuscript.

In the attached document, you will find our detailed responses to each of your observations, along with the corresponding changes made in the revised version of the paper.

We appreciate your time and consideration.

Author Response File: Author Response.pdf