Inter-Item Differences in Metacognitive Judgments: Insights into the Collective Wisdomware Underlying These Judgments

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

28-May-2026

 

Review of ‘Inter-item differences in metacognitive judgments: Insights into the collective wisdomware underlying these judgments’ by A. Koriat & N. Yehudai

 

SUMMARY: In a re-analysis of previously published data, the authors examine the consensuality and the replicability effects in metacognitive judgments across various types of judgments and various types of materials. The consensuality effect is a finding of greater strength of metacognitive judgments for responses of the same category as those made by the majority of other participants. The replicability effect is a finding of a correlation between the strength of one’s judgment for a particular item and the probability of other participants (or the same participant on a different occasion) making the same category (high/low) of judgment. The authors report reliable consensuality and replicability effects across different judgments and different materials, albeit with some interesting exceptions: for JOLs, the effects were not reliable across participants when unrelated pairs were studied, even though the replicability effects were significant when computed for the same participants tested on different occasions. The conclusion is that judgments for materials with some type of a semantic relationship across constituent words are based on a set of cues shared across participants, but judgments lacking those relationships are based on consistent yet idiosyncratic cues.

 

EVALUATION: I need to note that I am no expert on the correlation analyses of the type chiefly presented in the manuscript, so some of the discussed findings were not entirely clear to me. I’ll note the most prominent example in the comments section, not because I imply that anything is wrong here, but just to highlight that some writing could use unpacking for less knowledgeable readers. Besides the issue of clarity of some conclusions, I have no substantial problems with the manuscript. The addressed questions are interesting and the findings of consistent differences across various types of materials in terms of relying on a common set of cues are important. I thus recommend the publication of this manuscript, perhaps after minor rewrites to make it a tad more accessible.

 

COMMENTS:

• The study looks at the effect of consensuality, which is perhaps easy enough to understand, but also at various correlations or correlations across correlations, at which point the writing becomes a little hard-going. I’ll try to unpack one set of these analyses that is repeated across studies so that the authors could assess if my thinking goes in the right direction – if it does not, then perhaps this will give the authors clues as to what could be clarified in the revision.

 

On page 4, the authors look at the replicability effect on FOK judgments. They compute an FOK/Rep correlation, which for every participant and every item measures the extent to which the strength of this participant’s FOK judgment for a particular item is related to the likelihood of other participants making the same binarized FOK judgment for this item. These correlations are then averaged across items and the resulting correlations of .22 is found to be significantly different from zero. Then participants are divided into two random groups and these correlations are computed once again, but now separately for each group. The resulting correlations were then again averaged for each item and the subsequently resulting values were correlated across groups, which returned a significant correlation of .54.

 

If I understand correctly what has been done here, it is now important to understand how the authors’ conclusions follow. The authors write that ‘these results suggest that although FOK generally rely on collectively shared cues, there are significant inter-item differences in that reliance’. I wonder here about those ‘inter-item differences’ (which are crucial given the topic and the title of the study). Is this conclusion warranted because the computed correlations seem to be across items? I guess if correlations were computed across participants, a significant correlation would directly testify to what the authors refer to as collective wisdom – the strength of my judgments correlates with the direction of other people’s judgments because we rely on the same cues to make these judgments. The magnitude of the correlation would testify to the extent to which judgments of different people are based on common cues. But the magnitude of correlations across items testifies to the extent to which judgments for different items are based on common cues across participants. In which case, the question arises why this correlation, albeit significant, is not very high. Is it because all judgments to all items are only partially based on the same cues or is it because although some items are based on common cues to a large extent, for some items this is not the case? I think the authors imply the latter when they note that correlations ‘ranged from -.41 to +.90 across items’. But in this case, why are the conclusions derived from the mean value of the correlation rather than its variance? Does the reliability analysis, with its division of participants into two groups, pertain to this conclusion and if so, how? Again, if am not sure about my description of these results, but it could point the authors into considering where the readers’ understanding may stray and thus where additional clarifications could be useful.

 

• One additional comment concerns a comparison between within-participants and across-participants replicability effects. Although not the central finding of the study, there is a repeat mention that cross-individual correlations (reflecting the replicability effect) are consistently higher than within-individual correlations. This is really surprising as it would suggest that the sets of cues on which metacognitive judgments are based are more consistent across participants than for the same participants across different occasions. I wonder if this difference could rather be an artifact of a reduced range for within-participants correlations. While different participants are likely to vary widely in terms of the magnitude of metacognitive judgments they make, participants are likely to be more consistent, with relatively small variations across the means, resulting in a reduced range across which correlations could be computed and thus relatively low values of these correlations.

Author Response

We thank both reviewers for their comments. We did our best to respond to these comments and we think that the revised version benefitted from these comments and suggestions. In what follows we list point-by-point our responses to these comments:

Response 1: 

We thank the reviewer for asking for clarification. The mean correlation reported (.22) confirms the overall presence of a collective cue effect across the task. However, the wide variance (from -.41 to +.90) could simply be statistical noise. Therefore, the split-half reliability correlation (r =.53) demonstrates that this variance is an intrinsic property of the items: The tendency to pull from the collective wisdomware varies reliably between items. we added a paragraph to clarify this conclusion.

Response 2:

Thank you for this comment. We clarified that these results are likely driven by variance restriction (reduced range) within individuals, who naturally exhibit higher internal consistency across repeated presentations compared to the variance observed across a broader group. We revised the manuscript in multiple sections to adopt your explanation.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript is based on the reanalysis of previously published datasets. Although secondary analyses can provide valuable theoretical insights, several aspects of the manuscript require clarification and revision.

The abstract should be revised to better align with a conventional structured format, including the following components: (1) Background, (2) Methods, (3) Results, and (4) Conclusion.

Several formatting inconsistencies were identified throughout the manuscript. Some lines appear to be misaligned or improperly indented. This issue should be carefully reviewed and corrected. For example, lines 32–33 and 42–43.

The literature review appears to rely on a limited range of references. In some sections, supporting references are lacking, while in others the same sources are cited repeatedly (e.g., Koriat, 1995; Koriat & Bjork, 2005; Koriat et al., 2006; Koriat, 2011; Koriat, 2025; Koriat, 2026). The manuscript would benefit from incorporating a broader range of relevant literature to provide a more balanced and comprehensive theoretical foundation

Koriat, who is frequently cited throughout the manuscript, is also considered a co-author of the studies on which the present manuscript is based. While self-citation is appropriate, the literature review relies heavily on the authors’ own publications. As a result, references to broader research within the field appear limited. Expanding the literature review to incorporate a wider range of relevant studies would provide a more balanced theoretical foundation.

On page 2, lines 71–73, “In addition, differences in the FOK strength associated with a FOK-Choice response predicted differences in the likelihood that other participants will make the same FOK-Choice response” the wording appears overly causal. The reported findings seem to indicate a correlational association between FOK strength and consensually rather than a causal relationship. Clarification is recommended.

On the last paragraph of page 2, the overall purpose of the manuscript is briefly introduced; however, the specific research questions and/or hypotheses are not clearly articulated. Although the Results section is organized around a series of studies, it remains unclear which specific research question each study is intended to address.

The methodological section needs to be revised. While it is appropriate to refer readers to the original publications, the current manuscript should also provide a concise summary of participant characteristics, data collection procedures, instruments, analytical methods, and ethical considerations. Although some methodological information is provided in the Results section, key methodological details should be presented in the Method section. Including these details would allow readers to better understand the study without having to consult multiple previous publications.

Author Response

We thank both reviewers for their comments. We did our best to respond to these comments and we think that the revised version benefitted from these comments and suggestions. In what follows we list point-by-point our responses to these comments:

Response 1:

You asked that the abstract should be revised to better align with a conventional structured format, including the following components: (1) Background, (2) Methods, (3) Results, and (4) Conclusion. We completely agree. The abstract was fully restructured to follow the conventional format you suggested, ensuring a clearer and more structured overview of the study. Thus, the new abstract explicitly includes the headers: (1) Background, (2) Methods, (3) Results, and (4) Conclusions. We also expanded the Methods and Results sections within the abstract to provide a more comprehensive summary of the participant samples and the analytical approaches used.

Response 2:

You noted that several formatting inconsistencies throughout the manuscript. We apologize for these formatting oversights. We proofread the manuscript to fix all indentation errors, misaligned lines, and paragraph spacing issues, ensuring adherence to the journal's formatting guidelines.

Response 3:

You noted that the literature review relies on a limited range of references.You are of course correct. In the revised manuscript we expanded the literature review to include reference to other articles that include ideas that resonate with those suggested by the research from our lab. We tried to anchor the concept of "collective wisdomware" within broader theoretical frameworks in cognitive psychology. We have substantially expanded the literature review, particularly in the Discussion section. We added a section -- Section 4.1. The Concept of Collective Wisdomware and Broader Metacognitive Literature where we integrated several prominent theories that align with our findings.

Response 4:

You are correct: the wording appears overly causal. The term "predicted" in this context was intended in a statistical (regression) sense rather than a causal one. We reworded the conclusion to indicate a correlational association between FOK strength and consensually rather than a causal relationship.

Response 5:

You noted: “On the last paragraph of page 2, the overall purpose of the manuscript is briefly introduced; however, the specific research questions and/or hypotheses are not clearly articulated. Although the Results section is organized around a series of studies, it remains unclear which specific research question each study is intended to address.”

Thank you for noting this structural gap. We  added a new subsection at the end of the Introduction (Section 1.1. Research Questions and Hypotheses), which outlines the three primary questions driving our analyses (concerning the variance of reliance on collective wisdomware, the stability of this variance, and the specific item properties that predict this reliance) alongside our clear, a-priori hypotheses.

Response 6:

Thank you for your comment. Relying entirely on citations for methodological details places an unnecessary burden on the reader. We have thoroughly overhauled Section 2: Materials and Methods. Instead of a brief generalized paragraph, Section 2 is now divided into comprehensive subsections:

 

• 2.1. Participants (summarizing the aggregated demographic details).
• 2.2. Materials (detailing the paired associates, general-knowledge questions, and perceptual tasks used).
• 2.3. Procedure (explaining the computerized data collection process).
• 2.4. Analytical Methods (summarizing how continuous variables were dichotomized, how consensuality and replicability were indexed, and the split-half procedure).
• 2.5. Ethical Considerations (providing the necessary IRB and consent waiver statements).

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have addressed the reviewers’ comments well and made important improvements to the manuscript. The revised version is clearer and better organized. The abstract now follows a structured format, and the literature review has been expanded with relevant references and theoretical background. The research questions and hypotheses are now clearly presented. The Methods section has also been improved by providing more detailed information about the participants, materials, procedures, and data analysis. The revisions have strengthened the manuscript and addressed the main concerns raised during the review process.

Author Response

Dear Yoonhee Jang, Special Editor,

Thank you very much for your positive evaluation of our manuscript "Inter-item Differences in Metacognitive Judgments: Insights into the Collective Wisdomware Underlying These Judgments" (behavsci-4328032-v1) and for the constructive feedback. We are also grateful for the thorough reading and clear guidance on the required minor revisions.

We have carefully addressed all the points raised in your letter. Below is a detailed point-by-point response outlining the changes made in the revised manuscript:

• Abstract Structure: We have now removed all numbered headings from the structured abstract, presenting it as a single, cohesive paragraph in strict accordance with the journal's guidelines.
• APA Style, p-values, and Statistical Symbols:
  • We have revised our reporting of significance levels to use only the three conventional levels.
  • All exact p-values are now consistently reported with three decimal places, regardless of whether the finding is statistically significant.
  • We ensured that all statistical symbols are appropriately italicized throughout the text and tables.
• Table 2 Corrections: As requested, Table 2 now displays correlation values on only one diagonal side to avoid redundancy. Furthermore, we have removed the leading zero before the decimal point for all correlation coefficients.
• Variability Indexes and Effect Sizes: We have added variability indexes (Standard Deviation, SD) to all descriptive statistics across the manuscript's tables. Additionally, we have included Cohen's d as an effect size measure for all inferential statistics (e.g., t-tests), irrespective of their statistical significance.
• Typographical Errors: We have conducted a thorough review of the manuscript and corrected instances of missing spaces before decimal points.
• Formatting of Tables 4, 6, and 7: We have restructured Tables 4, 6, and 7 so that descriptive statistics (correlation coefficients, r) and inferential statistics (t and p values) are now presented in separate columns.
• In-text Citations: We have reviewed and updated all in-text citations to fully comply with the APA publication manual, ensuring that multiple-author citations are appropriately abbreviated using "et al.".

We believe these revisions fully address your concerns and have enhanced the clarity and rigor of our paper. We thank you again for your valuable time and guidance, and we look forward to the final acceptance of our manuscript.

Sincerely,

Asher Koriat & Noam Yehudai

Author Response File: Author Response.docx