Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

Thank you for the opportunity to review this manuscript.

I have carefully gone through the paper and believe that it addresses an important and timely topic: the reliability and bias of an LLM-based evaluator for assessing relational competencies in medical simulation. The manuscript has several strengths: it examines a real institutional system, ATENA, rather than a purely hypothetical tool; it combines perceived realism data from university hospital professors with agreement analysis between automated and human ECCS scores; and it engages critically with the risk of delegating evaluative authority to algorithmic systems. The focus on systematic overscoring, score compression, and underscoring of exemplary performances is especially relevant for medical education, where automated feedback may shape what students understand as “clinical empathy.”

However, the manuscript requires some revision before publication. The core idea is promising, but several aspects of the study design, statistical reporting, ethical documentation, and interpretation need more clarification:

First, either the title or abstract could be moderated. The manuscript refers to “systematic biases” and concludes that ATENA assesses a textual proxy of empathy rather than its paraverbal and contextual dimensions. This interpretation is plausible, but the evidence comes from one system, one first-generation prompt, GPT-3.5 Turbo, one institutional context, and one clinical scenario. The title and abstract should make this scope clearer. A more cautious formulation would indicate that the study identifies systematic bias patterns in this version of ATENA, rather than implying a general conclusion about all LLM evaluators in medical simulation.

Second, the manuscript needs to clarify the relationship between the ECCS scale used in the paper and the actual ATENA prompt. In the Introduction, the authors state that levels 0 to 5 were retained and that level 6 represented difficulties in appropriation in the educational context. However, Supplementary File 3 shows that the ATENA prompt defines ECCS levels 0 to 6 and explicitly allows non-integer scores. This creates an important inconsistency between the reported measurement model and the actual prompt used by the system. The authors should explain whether ATENA was prompted with a 0–6 scale but the analysis was restricted to 0–5, why this was done, and how this decision may have affected scoring behavior and score compression.

Third, the human reference standard needs to be strengthened. Component 2 is described as an agreement analysis between ATENA and “expert human assessment,” but the 129 transcripts were distributed between two human raters, with each coding approximately half independently. The manuscript acknowledges that this design does not permit calculation of interrater reliability across the full corpus. This is a serious limitation because the human score functions as the benchmark against which ATENA is judged. The four calibration transcripts and harmonization discussions are useful, but they do not substitute for double coding of at least a meaningful subset of the corpus. The authors should either provide double-coded reliability evidence for a subset or more explicitly frame the human score as a pragmatic expert reference rather than a validated gold standard.

Fourth, the statistical reporting contains some key inconsistencies that should be corrected. The Methods section states that the Bland–Altman vertical axis represents ATENA minus human. However, the Figure 1 caption states that the vertical axis is Human minus ATENA, while also describing the central line as −0.80 and interpreting this as systematic overscoring by ATENA of +0.80. The table reports mean bias as ATENA − human = +0.80. These signs and labels must be made consistent throughout the Methods, Results, figure caption, and table.

Fifth, the ICC model needs further justification. The authors use ICC(2,1) and state that raters can be considered representative of a larger pool of potential raters, including other versions of ATENA via the same prompt. This assumption is debatable because ATENA is not a randomly sampled rater but a specific algorithmic system and prompt. The authors also calculate ICC(3,1), which is useful. However, the manuscript should explain more carefully why ICC(2,1) is the primary metric rather than ICC(3,1), weighted kappa, Spearman correlation, or ordinal agreement statistics. The argument that ICC “exploits ordinal information” should also be phrased carefully because ECCS levels are ordered but not necessarily interval-equivalent.

Sixth, the realism-survey analysis should be clarified. Component 1 uses an ad hoc seven-item questionnaire completed by 21 PU-PHs. The authors calculate McDonald’s omega, but the seven items appear to represent distinct dimensions of realism rather than a single unidimensional construct. This is especially visible because removal of Q3 improves omega and the authors interpret Q3 as emotional realism rather than technical-conversational realism. The authors should either justify treating the seven items as one scale or present them as separate realism indicators without overemphasizing internal consistency. In Table 4, the heading says “One Sample T-Test,” even though the text reports Wilcoxon signed-rank tests; this should be corrected.

Seventh, the typology of bias is interesting but should be framed more cautiously. The manuscript identifies overscoring of weak performance, central tendency, and underscoring of exemplary performance. These patterns are supported by the confusion matrix and disagreement table, especially the concentration of ATENA scores around 4.2–4.3. However, the qualitative analysis of “surface empathy” markers was post hoc, AI-assisted, and based partly on a small random sample of seven transcripts from the overscoring subcorpus. The interpretation is valuable, but the authors should clearly distinguish between statistically observed disagreement patterns and exploratory qualitative explanations of why those patterns may occur.

Eighth, the authors should further explain the use of AI assistance in analysis and writing. The manuscript states that an LLM was used for rapid statistical exploration before manual validation in JASP, and later states that GPT-4 and Claude were used for writing assistance and verification of statistical calculations. It also notes that AI assistance was used in extracting and categorizing qualitative markers of surface empathy. This disclosure is appreciated, but the exact role of AI in the analysis pipeline should be specified more transparently. The authors should clarify which outputs were generated by AI, which were independently verified, whether AI-assisted categorizations were checked by both authors, and whether any AI-generated analytic categories were adopted.

Ninth, the ethics and data-protection section needs careful revision. The manuscript states that ethical approval was not required because the study was non-interventional and based on routinely collected anonymized educational data. However, Component 2 involved 129 student simulation transcripts processed through an LLM system, and the GDPR correspondence in Supplementary File 9 discusses OpenAI API transmission and institutional data-protection considerations. The authors should provide a clearer institutional statement explaining whether an ethics committee reviewed or exempted the study, whether the student data were collected before or after such confirmation, and whether students were informed that their simulation transcripts could be used for research and/or processed through OpenAI. The statement that informed consent was not required for student data should be supported with explicit institutional or legal justification.

Tenth, in this reviewer's humble opinion the conflict-of-interest statement should be moved from disclosure to interpretation. The authors appropriately state that both authors are designers of ATENA. This is important because the paper evaluates a system developed by the authors themselves. The manuscript should mention in the Methods or limitations how this potential conflict was managed analytically, for example through independent human raters, blinded scoring, external validation, or open data. The current disclosure is necessary but not sufficient for readers to assess possible bias in the evaluation.

Overall, this is a promising and potentially important manuscript. Its empirical finding—that ATENA may be credible as a simulator but unreliable as an evaluator—is valuable for medical education and for the broader debate on LLM-based assessment. However, the paper requires major revision to clarify the measurement framework, strengthen or qualify the human benchmark, correct statistical inconsistencies, improve ethics/GDPR reporting, and moderate claims about systematic bias and structural limitations.

Sincerely,
Reviewe

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This manuscript addresses a timely and important topic: the deployment of large language models for automated assessment of empathic communication in medical education. The authors present empirical evidence demonstrating that a GPT-3.5-Turbo-based virtual patient simulator (ATENA) achieves high perceived realism as a simulator while exhibiting poor evaluative agreement with human experts. The identified biases—score compression, overscoring of weak performance, and underscoring of exemplary performance—constitute valuable empirical contributions to the growing literature on algorithmic assessment in education. However, the manuscript suffers from significant structural, methodological, and conceptual weaknesses that substantially undermine its contribution. The presentation is disorganized, with erroneous section numbering and incomplete reporting of results. The methodological design contains a fundamental limitation—the absence of systematic double coding for human raters—that the authors acknowledge but insufficiently address. The theoretical framework, while ambitious, remains underdeveloped and poorly integrated with the empirical findings. The discussion section, though rich in interpretive claims, frequently overreaches beyond what the data can support. These issues require substantial revision before the manuscript can be considered for publication.

Abstract
The abstract provides a competent overview of the study's rationale, methods, and key findings. However, several improvements are necessary. First, the reported ICC of 0.146 and the Bland-Altman limits of agreement [-1.57; +3.16] are presented without sufficient contextual interpretation—readers are told these results "preclude any certificatory use" but the abstract would benefit from explicitly stating what constitutes acceptable agreement thresholds in this domain. Second, the phrase "systematic biases emerged: score compression (76% of scores within 4.2-4.3)" is misleading as written; the authors later clarify that this compression applies to ATENA scores specifically, not to all scores. The abstract should specify that this compression characterizes algorithmic rather than combined scoring. Third, the concluding assertion that ATENA "assesses a textual proxy of empathy rather than its paraverbal and contextual dimensions" is well-supported by the data but should be presented as a finding rather than a foregone conclusion. Finally, the abstract would benefit from a clearer statement of the sample size for Component 2 (n=129 simulations) and a more precise characterization of the study's scope—specifically, that these findings derive from a single clinical scenario and a first-generation prompt configuration. The keywords are appropriate but could be expanded to include "algorithmic bias" and "educational assessment" to better capture the manuscript's broader contributions.

Introduction
The introduction is comprehensive in scope but exhibits significant organizational and conceptual problems. The authors attempt to integrate multiple theoretical frameworks—sociotechnical theory, instrumental genesis, digital prudence, and algorithmic vigilance—without establishing clear relationships among them or demonstrating how they collectively inform the research questions. The transition from the literature review on LLM virtual patients (Section 1.4) to the theoretical positioning (Section 1.5) is abrupt and lacks a clear bridge explaining why the theoretical apparatus is necessary for the empirical investigation that follows.

The literature review in Sections 1.1-1.3 is generally thorough, though it would benefit from more critical engagement with the cited works. For instance, the authors cite Jamieson et al.'s finding of 89.7% agreement for factual verification tasks but do not adequately discuss why this finding has limited relevance to their own work on empathy assessment. The distinction between factual verification and relational competency assessment is crucial and deserves more prominent treatment. Similarly, the discussion of Ayers et al.'s finding that chatbot responses were preferred in 78.6% of cases is introduced but not sufficiently interrogated—this finding could be interpreted as supporting either the potential or the peril of LLM-based assessment, and the authors should clarify their position.

Section 1.5 on theoretical positioning is problematic. The authors invoke Latour's black box concept, Rabardel's instrumental genesis, Akrich's script theory, and Heiser et al.'s digital prudence typology, but these frameworks are presented as a list rather than an integrated analytical lens. The authors state that "Rabardel's theory of instrumental genesis offers a pertinent framework" but then fail to specify which aspects of this framework will be operationalized in the study. Similarly, the reference to Perrenoud's argument that "assessment is never an act of objective measurement" is potentially contradicted by the study's own reliance on quantitative agreement metrics; the manuscript would benefit from a more explicit discussion of how the authors reconcile their constructivist theoretical commitments with their positivist methodological approach.

The research questions (Section 1.6) are clearly stated and appropriately narrow in scope. However, the authors should specify more precisely what constitutes "acceptable" agreement for RQ2 and what criteria will be used to identify "recurrent" disagreements for RQ3. Without such specifications, the interpretation of results risks being post hoc and potentially circular. The introduction also lacks a clear statement of the study's novel contribution relative to the existing literature. While the authors note that ATENA "lies precisely at the intersection" of LLM virtual patients and automated assessment, they do not articulate what new knowledge their study generates beyond documenting what might be intuitively expected—that language models have limitations in assessing multimodal constructs. A clearer statement of the study's contribution to theory, methodology, or practice would strengthen the introduction considerably.

Methods
The methods section contains several substantive issues that require attention. The most significant concerns the design of Component 2, specifically the absence of systematic double coding for human raters. The authors acknowledge this limitation but do not adequately address its implications. With only two human raters coding approximately half the transcripts each, and ambiguous cases resolved through consensus, there is no way to quantify inter-rater reliability across the full corpus. This is particularly problematic because the study's central claim is that ATENA disagrees with human experts; if the human experts themselves would disagree substantially on the same transcripts, the interpretation of the disagreement metric changes fundamentally. The authors' mitigation measures—calibration training with Gregory Makoul, harmonization sessions, and consensus negotiation—are appropriate but insufficient to compensate for the absence of systematic reliability data. The manuscript would benefit from reporting inter-rater reliability on a subsample (e.g., 20% of transcripts coded by both raters) to establish a baseline against which ATENA-human agreement can be compared.

The statistical analyses are largely appropriate but require more detailed justification. The authors choose ICC(2,1) as the primary agreement metric, which is defensible, but they do not address the assumption that the two raters (ATENA and human) should be considered representative of a larger pool of potential raters. This assumption is questionable given that ATENA's scoring behavior is deterministic and non-random. The authors should either justify this choice more thoroughly or consider alternative metrics such as ICC(1,1) or weighted kappa. The Bland-Altman analysis is appropriately conducted, but the authors should provide more information about the heteroscedasticity checks performed, particularly given the observed compression of ATENA scores.

The description of the confusion matrix in Section 2.3.3 contains an important methodological detail that is insufficiently explained. The authors state that ATENA produced decimal scores (e.g., 4.3, 3.7) that were rounded to the nearest integer for agreement analyses. This rounding procedure could introduce systematic bias depending on the distribution of decimal values; for instance, if many values are in the 3.5-4.4 range, rounding would inflate the prevalence of score 4. The authors should provide the original decimal score distribution and demonstrate that rounding did not systematically alter the pattern of results. The confusion matrix in Table 7 retains decimal values, but the analysis of agreement patterns in Table 8 appears to use rounded scores for the "ATENA score (median)" column, creating a potential inconsistency.

The description of the qualitative analysis in Sections 4.2 and 4.3 would benefit from more methodological detail and should be moved or referenced from the methods section. The authors describe a "secondary post hoc analysis" of free-text commentaries but do not specify the coding procedures employed, the number of coders, or the measures taken to ensure analytical rigor. While the authors note that "extraction and categorization benefit from AI assistance," they do not specify which AI tool was used, how it was prompted, or how the outputs were validated. The use of AI assistance in qualitative analysis is not inherently problematic, but it requires transparent reporting. The authors should develop a dedicated qualitative methods subsection that addresses these issues and provides a clear rationale for the selection of the overscoring and underscoring subcorpora.

The participant descriptions are adequate, but the authors should provide more information about the student simulations in Component 2. The text states that students were "across all programmes and years" but does not specify the distribution of training levels, which could affect the range of performance scores and thus the agreement analyses. Additionally, the authors should clarify whether students were informed that their interactions would be used for research purposes and how anonymity was maintained in the transcripts.

Results
The results section is incomplete and contains several reporting deficiencies that must be addressed. Most notably, the authors have inserted numerous section headings (2.2.4 through 2.2.144) that appear to be erroneous placeholders from a template, suggesting that substantive content is missing. The results for Component 1 (realism survey) are reported in Sections 3.1, but the presentation is cursory and lacks important details. The descriptive statistics in Table 2 would benefit from inclusion of medians and interquartile ranges alongside means and standard deviations, given the ordinal nature of Likert-scale data. The authors report that Q3 (Expression of concern) was the only item whose removal improved Cronbach's omega, but they do not provide a substantive interpretation of this finding. The qualitative interpretation they offer in the discussion—that Q3 captures "emotional realism" while other items measure "technical-conversational realism"—would be better placed in the results section, as it constitutes an interpretation of the reliability analysis.

The results for Component 2 (evaluative agreement) are more complete but still require improvement. The ICC results in Table 5 are correctly presented, but the authors should provide the F-statistic and degrees of freedom for the ICC calculation, as these are standard reporting requirements. The Bland-Altman analysis in Figure 1 is appropriately presented, but the authors should explicitly state whether they tested for proportional bias (i.e., whether disagreement increases with the mean score). The wide limits of agreement (94.6% of the theoretical scale) are striking, but the authors should provide more context for what this means in practical terms—for instance, what proportion of simulations would receive a different classification (pass/fail, or level designation) under the two scoring systems?

The confusion matrix in Table 7 is difficult to interpret as presented. The column headers (2.5, 3.4, 3.5, etc.) are not in ascending order, making it challenging to discern patterns. More importantly, the authors report that "76% of ATENA's evaluations fall within this narrow band" of 4.2-4.3, but this calculation is not evident from the table. The authors should recalculate this percentage from the raw data and provide the precise count. The identification of three disagreement patterns in Table 8 is valuable, but the table does not report the actual scores for the "zone of agreement" (human score 4), only the mean difference. For consistency, the authors should report median ATENA scores and mean differences for all three patterns.

The qualitative results presented in Sections 4.2 and 4.3—the examples of overscored weak performances and underscored exemplary performances—are compelling but would benefit from more systematic presentation. The authors describe analyzing "a random sample of 7 transcripts" for the overscoring subcorpus, but they do not specify how randomness was achieved or justify why 7 transcripts is sufficient for a qualitative analysis. The examples provided are illustrative but do not demonstrate the prevalence of the identified patterns. The authors should provide more systematic evidence, such as a table categorizing the types of linguistic markers identified in the overscoring subcorpus and the frequencies with which they appeared. Similarly, the three types of paraverbal markers identified in the underscoring subcorpus (conversational progressiveness, tonal modulation, dynamic relational adjustment) would benefit from operational definitions and frequency counts.

Discussion
The discussion is ambitious and theoretically rich but suffers from overinterpretation and insufficient connection to the empirical data. Section 4.2's claim that ATENA "does not evaluate clinical empathy in the full sense but rather a textual proxy" is well-supported by the data and constitutes the study's most important contribution. However, the authors should more carefully distinguish between what the data demonstrate (that ATENA's scores diverge from human scores in systematic ways) and what they infer (that ATENA detects formal linguistic markers rather than contextual adequacy). While the qualitative examples provide support for this inference, the authors should acknowledge that their data do not directly test the hypothesis that ATENA is responding to specific linguistic features—that would require systematic manipulation of transcript content and examination of corresponding score changes.

Section 4.3's discussion of "surface empathy" and the comparison to Iyer et al.'s framework is appropriate but would benefit from more critical engagement. The authors present the concept of surface empathy as if it fully explains their findings, but they do not consider alternative interpretations. For instance, it is possible that ATENA's apparent overscoring of weak performances reflects a leniency bias characteristic of LLM evaluators more generally, rather than a specific misperception of empathy. The authors should engage with the literature on LLM evaluation biases more broadly and consider how their findings align with or differ from patterns observed in other domains.

The discussion of evaluative uncertainty in Section 4.4 is conceptually important but underdeveloped. The authors invoke the Carnegie Foundation's early work on marker variability to argue that disagreement is "a structural feature of evaluative judgment itself." While this is a valuable insight, the authors do not adequately address the implication for their study: if human experts would disagree substantially among themselves, then low ATENA-human agreement might reflect the inherent difficulty of the assessment task rather than a specific limitation of the algorithm. The authors should acknowledge that their design (with a single rater per transcript) cannot distinguish between these two sources of disagreement. The comparison of ICC(2,1) and ICC(3,1) in the results section provides some evidence that ranking disagreement persists even when level bias is controlled, but this does not address the fundamental issue of human unreliability.

Section 4.5 presents operational recommendations that are generally reasonable but insufficiently grounded in the study findings. The recommendation to include a "composite confidence indicator" calibrated on a benchmark dataset is sensible but does not follow directly from the results—the authors have not tested whether such an indicator would improve instructor interpretation. Similarly, the recommendations for "prompt optimization" (including few-shot examples, chain-of-thought reasoning, and ensemble averaging) are plausible but speculative, as the authors have not evaluated these interventions. The discussion would benefit from a more careful distinction between recommendations directly supported by the data and suggestions for future investigation.

The limitations section (4.6) acknowledges the major methodological limitations but frames them as "avenues" for future research rather than as constraints on the present study's conclusions. The authors should be more explicit about what their study can and cannot establish, given the design limitations. In particular, the single-scenario design and the GPT-3.5-Turbo configuration mean that the findings may not generalize to other clinical contexts or more recent models. The authors' citation of Ye et al. (2024) as evidence that "these biases persist, in attenuated forms, with more recent models" is somewhat misleading—Ye et al. studied bias in LLM-as-a-judge paradigms more broadly, not specifically in medical empathy assessment. The authors should qualify this claim and acknowledge that replication with newer models and different scenarios is needed.

Conclusion and References
The conclusion effectively summarizes the study's main findings and implications. However, the final paragraph's assertion that "what a language model can process and what an expert perceives in a situation do not overlap" is overly strong. The data show substantial disagreement, but they do not establish that the two assessments are entirely non-overlapping—there was a zone of agreement for mid-range performances. The conclusion would benefit from a more nuanced statement that acknowledges the partial agreement while emphasizing the systematic biases that prevent certificatory use.

The references are comprehensive and appropriately selected, but several issues require attention. Several references appear to have incomplete bibliographic information—for instance, the Cook et al. (2025) reference lists the journal but not the volume or page numbers. The authors should verify all references against the original sources and ensure complete citation data. Additionally, the authors cite several preprints (e.g., Ye et al., 2024; Iyer et al., 2026; Kyung et al., 2025) that have not undergone peer review. While preprint citations are increasingly common, the authors should consider whether the findings from these works have been subsequently published in peer-reviewed venues and update the references accordingly. The authors also cite Digital Promise (2024) and OECD (2025) for the AI literacy framework, but these references lack sufficient bibliographic detail—the Digital Promise reference includes a URL but no date of access, and the OECD reference lacks a report number.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

summary

This paper investigates an LLM-based system that simulates patients in medical consultations and evaluates clinical empathy, called ATENA. This It aims to address three RQs, regarding the credibility/perceived realism of the simulations (RQ1), the reliability of the empathy evaluations against expert human judgments (RQ2), and the ways the system evaluations (dis)agree with the human experts (RQ3). The purpose of the system is situated in educational contexts, whether and how much it can support making evaluations of students conducting medical consultations.

To collect the data for the realism assessment, 21 university hospital doctors in general medicine (expert humans) performed a simulated session with the LLM-based simulated patient. For RQ1, the same experts judged the realism of the interaction, according to an adhoc 7-point Likert scale questionnaire with 7 items, included: overall realism, patient reactions, patient expression of concern, patient understanding of clinical situation, sense of patient personality, relevant info exchange, and absence of dialogue blockages. Analyzing the ratings with wilcoxon signed-rank tests and McDonald’s omega coefficient, high internal consistency was identified for 6 of the 7 items interpretted by the authors as measures of technical-conversational realism, with slightly decreased internal consistency when incorporating the emotional realism item. This result was interpeted to show a limitation in the synthetic patient’s emotional expression convincingness, but credible textual simulation overall (RQ1).

To assess how the system evaluates clinical empathy, they used data from 129 simulated interactions from medical students, and 2 humans (a clinical psychologist/educational psych expert and a faculty in information and communication sciences) rated the them according to the previously established ECCS scale for clinical empathy, with free-text commentary on notable simulations. The raters did half of the samples, each, so each simulation was rated by one person. The ATENA system was prompted with instructions based on the ECCS scoring citeria. Results showed that ATENA vs human agreement is poor, both by ICC(3,1) which accounts for overscoring bias, and ICC(2,1) which considers agreement, and Bland-Altman analysis shows that disagreements can range by 4.7 points out of 5, thus ATENA system is not useful/valid replacements of human judgment (RQ2).

For RQ3 which pertains to how the model disagrees with humans systematically, the paper identified 3 trends: 1) ATENA overscores performances considered weak by humans (simulations scored ≤3 by human were scored ≥4 by ATENA in 65% of cases); 2) a central tendency effect in ATENA ratings where humans judged ≥4; 3) underscoring great performances (simulations scored =5 by human, tended around 4.3 by the system).

The paper provides a thorough discussion of these observations, interpreting the observed limitations, provided recommendations for developing such systems, and for addressing the study limitations in future work. Their main conclusion to the RQs is that this single system produces credible simulations but is unreliable as an evaluator; this observation goes against the theoretical conflation that often occurs (credibility of simulation and legitimacy of evaluation). Regarding ATENA’s empathy ratings, the discussion covers the limitations of the system of detecting surface-level linguistic markers of empathy, like politeness, reassurances, info structuring, yet it doesn’t account for context, how empathic it is with respect to the patient’s state. For recommended paths for development, the paper discusses: infocommunicational prudence (integrating functionality to provoke a critical interpretation of the feedback the tool produces, algorithmic literacy); individual prudences (digital literacy training on how LLMs work, examining ATENA errors in case studies); social prudences (cooperative engineering between instructors and designers, facilitated through institutional-level practices for transmitting feedback); and prompt optimization (improving rubic quality).

General comments

• The study was well-designed and the paper is a high-quality report with actionable insight for digital educational tools for medical communications.
• The data collection and analysis methods were specifically aligned and sound approaches for addressing each question.
• Generally, the limitations of the study are transparent, and paths for future work are provided to address them - mainly in the data collection/rating (having a single rater for the student simulations).
• The limitations in the system design have relatively little discussion compared to other aspects of the study design; though the design was not the focus of the study. However, prompts are provided.
• Weaknesses/areas to address are minor, according to my evaluation.

Weaknesses, areas to address

• The takeaway regarding the surface-level linguistic features of empathy is valid, and aligns with works from prior literature in the medical domain which the paper references. It also aligns with findings in the NLP literature (mainly from the ACL anthology) on empathy operationalizations, system designs, and evaluations. These observations pertain to the system design, I’d argue, which is textual only, ignoring audio and visual modalities. While in the context of this system, the humans are better able to identify paraverbal markers of empathy, it might be the case that a multi-modal system with audio and visual features could pick these up — this is mentioned briefly in the paper, but there is substantial literature in affective computing and multimodal systems that can support how the discussed limitations of ATENA can be better situated. I would recommend more engagement with the literature in NLP systems for evaluating empathy, especially in studies that pertain to textual modalities as these will be the most relevant for the study. Here are some starter recommendations:
  • Andrew Lee, Jonathan K. Kummerfeld, Larry An, and Rada Mihalcea. 2023. Empathy Identification Systems are not Accurately Accounting for Context. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1686–1695, Dubrovnik, Croatia. Association for Computational Linguistics.
  • Andrew Lee, Jonathan K. Kummerfeld, Larry Ann, and Rada Mihalcea. 2024. A Comparative Multidimensional Analysis of Empathetic Systems. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179–189, St. Julian’s, Malta. Association for Computational Linguistics.
  • Allison Lahnala, Charles Welch, David Jurgens, and Lucie Flek. 2022. A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2139–2158, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
• I suggest providing translations of French to English, at least on the figures in the paper.
• Explicit definitions of the 3 errors of the typology for RQ3 are needed (see specific comments)

Specific comments:

These are all minor points for minor edits, which mainly target clarity in the report. Based on my assessment, there is no need for major edits, such as those that require modified methodology or additional analyses.

• L37 to L45
  • The paragraph appears to hold some redundancies about the ECCS background. Ref 4 is cited when the validation of ECCS is discussed, saying that ~100 consultations were used in the validation. Ref 2 is cited as the foundational work of ECCS which assessed 249 empathic opportunities. The paragraph may just need clarification. I presume from how it’s written now that Ref 4 empirically validated the instrument while ref 2 introduced the instrument. It may help to introduce the foundational work first to describe the scheme, followed by the work that validated it (somewhat reversing the flow)
• L71 - constrains → constraints I think?
• L73 - “ECCS table into textual instructions without fine-tuning, making it a heuristic basis for documenting structural biases”
  • Link to a table in the paper and the exact textual instructions
  • Briefly define what “structural biases” refers to in this context, or exemplify with an in-text example if possible.
• L79 - “Med-PaLM reaching 92.6% versus 92.9% for clinicians [7]”
  • These values lack context so their purpose is unclear. They’re preceded by a mention of alignment “comparable to that of expert physicians”. If these values are a measure of “alignment”, define what alignment meant in that context and/or what the metric is and what it measures.
• L90 - “In the absence of a sufficiently structured prompt, models tend to evaluate toward the center of the scale [9]; this central tendency bias is compounded by a leniency bias whereby certain models systematically over-evaluate intermediate outputs relative to human raters [10].”
  • Which scale(s) in particular does this refer to? (not clear in this passage”
  • What does “over-evaluate” mean in this context, and what are “intermediate outputs”? It unclear if over-evaluate means rating empathy (or whatever is measured) higher/as better than human raters tend to. (This is clear later in the paper but making this clear early on will help)
• L100 - “LLMs appear structurally ill-suited to producing authentic empathy [13]”
  • What is the “structural” aspect being discussed here? Is it regarding the paradigm structure in that LLMs cannot produce authentic empathy, the dialogue interaction structure lacking coherent turn-taking skills, or something else?
• L88-190 - “a double-verification procedure was systematically applied — initial processing by an LLM for rapid exploration, followed by manual validation in JASP — and the results presented derive exclusively from JASP [see supplementary file 8].”
  • Saving for now, at this point I’m unsure what exactly was “processed” by an LLM and for what kind of exploration
• L202 - “yielding 154 assessments in total”
  • is 154 the number of dialogues in the dataset, or is this turn-level?
  • Unclear, as L205 says each of the 21 PU-PH completed a single simulated session
• L231 regarding RQ2 design - 129 digital simulations of clinical interviews
  • Clarification: Does the 129 refer to the number of dialogues? Is 129 also the number of students, so each student performed 1 simulation? Are the simulations from the professors excluded in this set?
• L250 “the disagreement patterns of Bias 3”
  • I don’t think Bias 3 had been described at all before this so I’m not sure what this refers to
• L280 - “in this graph” — can you add a reference to the graph in the paper?
• Supplementary material is provided in French, Figure 1 graph is annotated in French. I don’t know what the convention of this journal is, but it seems like perhaps they should be presented in English or paired with English translations
• L294 - Here the paper says that they exclude level 6 from their ratings, because of pedagogical applicability. Could you expand on the assessment that is lacks pedagogical applicability?
• L398 - minor rephrasing suggestion: “see Supplementary file 6 for full definitions of categories and examples in original language”
  • Can you provide english translations, at least of the definitions of categories, perhaps in the main paper itself?
• Section 4.3, paragraph starting at line 426 and paragraph starting at 434:
  • I find that the three paraverbal markers - mentioned as an advantage that humans have in identifying over LLMs - are not explicitly defined. I looked at Supplementary file 7 and these don’t seem apparent by the examples, nor do they in the interpretation paragraph with examples in the paragraph at L434. The most clear to me is conversational progressiveness, with the mention of the student doing re-engagement. The tonal modulation comment example mentions normalizing lymphoma whil keeping the importance of treatment adherence emphasized, with authors suggesting it highlights tonal modulation invisible in the transcript — what does tonal modulation actually mean here? To me the comment sounds like it remarks on balancing objectives, but I don’t see how the comment describes markers that are invisible in the transcript, perhaps with exception of the patient not panicking - is that where the tonal modulation occurs? And the 3rd category DRA is applied to a comment ‘transformation of negatives to dedramatize’, but I believe this could be observed in the transcript. I think this section of the discussion needs further clarity.
  • It also seems clear here, and later in the discussion around L513, that these might only pertain to systems that are truly a singular modality (text), while audio and visual modalities could be processed to address this (investigations needed). These other modalities are discussed as limitations to address, around L561, however.
• L579 - I’m having trouble interpreting what the authors mean in saying ATENA is currently appropriate for indicative formative feedback, not for certification. What is indicative formative feedback? Is that not part of evaluation, which the study determines that the current state is inadequate as an evaluator?

< !-- notionvc: 83d59f19-ac2d-487f-b339-3cfa70eaa5c0 -->

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have addressed every point of the corrections indicated in the last revision.... For my part, I recommend approval of the manuscript