The Level of Agreement between Self-Assessments and Examiner Assessments of Melanocytic Nevus Counts: Findings from an Evaluation of 4548 Double Assessments

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This is an interesting blinded researcher and participant study of the agreement on the nevus counts on the arms in 4548 University students.

A major limitation of the study is that the researchers performing the nevus counts included only 1 dermatologist, and the other researchers were 3 academic researchers and 2 physicians of other disciplines. It is possible that the level of knowledge of the academic researchers and of the two physicians of other disciplines on nevi recognition was similar to the one of the participant- meaning of novice level. So, this study does not investigate the agreement between physicians knowledgeable on nevi-expert level and participants -novice level, but rather researchers-novice level with participants-novice level.

This limitation should be added in a Limitations section in Discussion.

Also, throughout the manuscript it would be accurate to mention “researcher”-assessed nevus counts instead of “expert”.

MAJOR COMMENTS

1. In Methods, could it be clarified if in regard to the research questions of the validity of self-assessment of nevus counts, this is a prospective study, that was designed in 2006 to study self-assessment of nevi?

2. the raw agreement is not needed in the text of results, the kappa agreement is sufficient.

3. Also, in section 3.3 , instead of “chance-corrected” agreement, it is suggested to refer to the term the kappa coefficient.

4. In Table 1, with baseline information, could you add the nevus counts as assessed by the researchers, in nevus categories?

5. could you clarify the cut-offs in nevus categories, so that there is no overlap? For example in 5-10, and 10-16 categories, in which category are 10 nevi classified?

6. could you add an explanation for your results showing that self-assessed nevus counts were higher than the researcher-assessed counts? This is in disagreement with the literature showing that individuals underestimate their own nevus counts (Buettner, Garbe et al. Am J Epidemiology 2000, Cust et al. Cancer Epidemiol Biomarkers Prev 2015).

7. as discussed in the article, previous studies have assessed inter-rater agreement on nevus counts. Could you highlight the novelty of your study?

8. Why is the dermatologist who had a central role in the study, e.g. performing the nevus counts and educating the other researchers, not included in the authors group?

OTHER COMMENTS

1.Introduction, the statement that valid self-assessment of nevus counts is a prerequisite for using the number of nevi for targeted screening needs revision to show that self-assessment may facilitate targeted screening, but is not a prerequisite- physicians too could also assess the high-risk individuals in a baseline screening and select only those at risk for further screenings thus saving costs.

2. the study period seems long- why was it decided to prolong the study from 2006 until 2019?

3. in methods, please mention if photos of seborrheic keratoses and solar lentigos were also provided?

4. also detail some information on what the written instructions given to participants were about.

5. Table 2, could be revised for clarity, instead of joint distribution you mean Results. Also in the Table it would be nice to clarify the headings of the rows for “Self-assessment of nevus counts on the arms” and of the columns as “Researcher assessment of nevus counts on the arms”, as they were not experts.

6. could you mention the disciplines of the two physicians performing the nevus counts?

7. in Discussion, first sentence, this is not “controlled” study.

8. in Conclusions the last sentence should be revised, as the results of this study cannot be generalized for information for targeted screening programs.

9. in Discussion, line 209, the comment on studies with up to 1772 participants not having precision is not needed.

10. report the kappa-statistic with two decimal points

11. instead of reporting the 95% CI for the kappa statistic, it is proposed to report its  p-value of significance.

12. the confidence intervals for the raw agreement in Table 3 are not needed.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Thanks to the Editors for allowing me to review this interesting article with inspiring results. I recommend the acceptance after minor revisions:

-The authors affirm that "...a greater than average level of agreement in participants with 200 dark brown eyes and a lower than average level of agreement in those with dark blue 201 eyes. Participants’ eye color was the only factor that had a significant impact on the level 202 of agreement in our study". Is it possible to provide a (supposed) justification for this phenomenon?

-The authors should mention among the limitations of the study that they did not consider the final histologic diagnosis as an influencing factor in the self-assessment of lesions. For example, it is possible that spitz nevi (common in this age group and often clinically "deceptive" because of the rosy flesh color) are more diffcult to self-evaluate than common or blue nevi (bluish color suspicuious for melanocytic nature).

-Could you add to the discussion a brief point about which lesions (solar lentigo? angiomas? nonspecific skin changes? others) most frequently led to differences in self-assessment?

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The authors sought to assess the reliability of self-assessment of nevus counts. They collected a very large and reasonable homogeneous panel of mostly University students and trained observers to rigorously count nevi.

The extensive training and tools provided to the population seem far more extensive than most clinicians could provide in a clinical setting. Nevertheless, the studied population was not particularly good at counting. Indeed, in true clinical settings with patients of varied education levels, varied eyesight, and more variable skin types, we would expect the reliability would be far worse.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

The authors submitted a manuscript investigating the concordance of self-assessment and expert assessment of melanocytic nevus counts in a large cohort of 4548 participants. Given the rising prevalence of melanoma and the need for risk stratification, the authors report on a very relevant topic.

The introduction provides sufficient background information on melanoma and current risk stratification. The chosen experimental techniques are appropriate. The presented results and the discussion are reported in a clear and interpretable manner. However, I would recommend to add a paragraph on the future prospective.

Over the last years the integration of artificial intelligence (AI) in dermatology holds significant promise, particularly in the realm of risk stratification for nevus counts. New AI algorithms (even in mobile phone apps) can analyze skin images to identify and classify nevi with a high degree of accuracy. By automating the assessment, AI can assist patients and/or dermatologists in identifying potential indicators of melanoma risk more efficiently and more objectively (not only number of nevus but also shape and color). This technology may further enhance precision in identifying high-risk individuals, allowing for personalized monitoring and intervention strategies.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review the revised manuscript. Here are some points for further clarification:

1. A main limitation of this study is that some of the termed “experts” that assessed nevus counts were not dermatologists or physicians.

The term “expert” should not be used for nevus counting, as this not a procedure that if known makes someone an expert.  It is proposed to replace with “examiner” or “researcher”, or “trained”, throughout and in the title. 

2. In methods, you mention that “common acquired “nevi were counted. Could you add how were dysplastic acquired or congenital nevi diagnosed and excluded?

3. The paragraph on AI apps in page 9 should be revised. These apps have not been validated and “self-assessment of skin lesions without dermatological expertise” can not be advised.

4. These results cannot be generalized on the level of agreement between dermatologists/physicians and self-assessed nevus counts.

Dermoscopy is now an integral method of examining pigmented lesions by physicians. As dermoscopy was not used by the researchers to confirm if lesions were nevi, this should be mentioned in limitations, and the discussion points and conclusions reported in this context.

OTHER POINTS

-the term “blinded” in Methods. Ine 90, needs revision, and also in the title. Do you mean that counts were performed independently?

-In Table 1, instead of “nevus score” it would be clearer to say “Nevus counts on arms”

-In Discussion, other studies with less participants than 4000 like in this study, do not necessarily have limited precision. Please revise.

-Under-estimation of nevus counts is understandable. How can you explain the result of over-estimation of nevus counts by the participants you are reporting? as participants overestimated their nevus counts, what lesions do you suppose they counted instead of nevi?  Solar lentigos?

-please add the affiliation of Wolfgang Uter to be in Dermatology as mentioned in the Reply Letter (Department, Hospital).

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Perfect for me 

Author Response

Nothing to reply

Reviewer 4 Report

Comments and Suggestions for Authors

Accepted in present form

Author Response

Nothing to reply.

Round 3

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript was improved during the second round of revision, however some major points have not been addressed and are essential for the clarity and soundness of the manuscript.

First, the title of the article is misleading and the authors have not revised according to the point on the use of the word “expert” in the R1 round of revision, that was again asked to revise in the R2 round of revision and the point was again dismissed as the authors disagree. It is misleading to state that there was “expert” assessment in the nevus counts. Assessing nevus counts is not a procedure that makes one an expert, even if he/she is very well trained in the procedure. In other published studies, trained nurses performed the nevus counts and they would not be mentioned as experts. The term “expert” shows some substantial and widely recognized expertise in the discipline, and could have been used if the researchers were for example dermatologists with recognized expertise in the research of pigmented lesions, with dermoscopy or other significant published work. The same wording has been used throughout the manuscript.

Second, the authors were asked to discuss their main finding of the overestimation of nevus counts by the self-assessment and they disagree. It is necessary to offer some possible explanations for the study’s main finding.

Third, using the wording “blinded” in the title and text is misleading for the readers, pointing to a blinded study for one that does not have knowledge of study designs. In order to perform inter-rater agreement, it is a pre-requisite that the assessments were independent and one rater did not know the score of another.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf