Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors The present paper addresses a vital domain: medicine. The authors, in their proposal, sought to leverage large language models to address the problem of Discharge letters, saving time and easing the process. The proposed method consists of GPT-5.2 using standardised prompts and compares their quality with that of resident-written texts using the same clinical data. The authors concluded that reviewer-adjusted sensitivity analyses were less favourable to GPT-5.2 for several domains.

• The paper is well constructed.
• The contributions are presented in the paper.
• The authors presented an overview of the state of the art.

The paper, in its current state, requires major revision; changes must be made to the final version.

• Some parts of the paper need to be referenced, I noticed from page 3 to page 8.
• It is better to cite the research gaps and the main contributions at the end of the introduction section, so the reader can understand the whole problem.
• The authors should justify the use of LLMs in the proposal, why, and what the motivation is behind using them.
• The dataset description used in the paper is missing, which can help the readers understand and compare them in future work.
• In the discussion section, the authors should mention the problems related to using generative models instead of human discharge letters without errors.
• Since there are many ophthalmic conditions, how can we ensure that AI-generated letters can be useful, and how does the proposed solution deal with them?
• I noticed in this paper that the authors did not present any framework or overall architecture for the proposed solution.

 

 

 

Author Response

We thank Reviewer 1 for the encouraging assessment and the detailed suggestions, which we address point by point below.

Reviewer 1 — Comment 1

Some parts of the paper need to be referenced, I noticed from page 3 to page 8.

Response: We have reviewed pages 3–8 (Introduction, Methods, and the opening of the Results section) and reinforced citation support where claims were previously made without citation, particularly in the parts of the Introduction that we have expanded with the new LLM-motivation and research-gap paragraphs. Specifically, the new Introduction paragraphs explicitly cite the prior LLM-discharge-letter literature when describing why LLMs are a suitable technology for this task and where the existing evidence base is incomplete (references 1–10 are retained). We retained the original citation structure of Methods, which by convention cites only methodological sources (statistical tests, prior frameworks).

Reviewer 1 — Comment 2

It is better to cite the research gaps and the main contributions at the end of the introduction section, so the reader can understand the whole problem.

Response: We have added a new closing paragraph to the Introduction that explicitly lists the research gaps motivating the study and the main contributions of this work, so that the reader has an unambiguous summary before entering the Methods.

Revised text (excerpt): Despite this growing body of evidence, several specific gaps motivate the present study. First, they are particularly effective at producing grammatically polished, consistently structured, and professionally worded prose, … The main contributions of this study are therefore: (i) a head-to-head paired comparison …; (ii) a blinded multi-reviewer assessment using a structured rating instrument …; (iii) a reviewer-adjusted sensitivity analysis …; and (iv) practical, specialty-specific implications for the safe integration of LLM-assisted drafting into ophthalmic discharge workflows.

Reviewer 1 — Comment 3

The authors should justify the use of LLMs in the proposal, why, and what the motivation is behind using them.

Response: We have added a dedicated paragraph in the Introduction that explicitly justifies the use of LLMs for this task. The justification covers four specific motivations: (i) their ability to transform structured/semi-structured clinical inputs into coherent narrative text in the local clinical language, (ii) the stable advantage of contemporary LLMs in producing polished, professionally worded prose suitable for documentation standardization, (iii) prior evidence that LLM drafts can reduce physician editing burden, and (iv) their accessibility relative to narrow domain-specific NLP pipelines.

Revised text (excerpt): Several considerations specifically motivate the use of LLMs for this task. First, prior comparisons have rarely paired LLM-generated and human-authored letters at the case level using identical source inputs and a blinded reviewer design. … Taken together, these properties make LLMs a pragmatic candidate for supporting, rather than replacing, clinician authorship of routine ophthalmology discharge documentation.

Reviewer 1 — Comment 4

The dataset description used in the paper is missing, which can help the readers understand and compare them in future work.

Response: We have added a dedicated dataset description in the Methods (after the de-identification paragraph in the Study material subsection). The description covers cohort size, time period, age and sex distribution, case-mix categories (cataract, vitreoretinal, anterior-segment and oculoplastic surgery, retinal vascular disease, uveitis, infectious/inflammatory disease, ocular trauma evaluation), and the structured fields that were available from the hospital information system as source data. Per-element applicability counts are reported in the existing Table 2.

Revised text (excerpt): Patients ranged across the typical age distribution of an adult ophthalmology inpatient service, with a mixed-sex composition reflecting routine clinical practice. The case mix covered the principal categories of ophthalmic inpatient care at the center, including surgical admissions (e.g., cataract surgery, vitreoretinal procedures, anterior-segment and oculoplastic interventions) as well as conservatively managed conditions (e.g., retinal vascular disease, uveitis, infectious or inflammatory ocular disease, and ocular trauma evaluation). …

Reviewer 1 — Comment 5

In the discussion section, the authors should mention the problems related to using generative models instead of human discharge letters without errors.

Response: We have added a new Discussion paragraph that explicitly contrasts the failure modes of generative models with those of human-authored letters. The paragraph emphasizes that LLM errors are qualitatively different: they include confabulation of plausible but unsupported content, smoothing over missing data with grammatically polished filler, surface fluency that masks subtle factual inaccuracies, and inconsistent prioritization of high-stakes elements. It also notes that a letter that is statistically “error-free” may still be misleading if it appears more complete or more confident than the underlying source data warrant, and concludes that any LLM-assisted workflow must include a structured clinician review step that explicitly targets these specific risks.

Revised text (excerpt): It is important to underline the qualitative difference between the failure modes of generative models and those of human-authored discharge letters. … Generative LLMs, in contrast, introduce a distinct class of risks that does not have a clean analogue in human-authored letters: confabulation of plausible but unsupported clinical content, smoothing over of missing data with grammatically polished filler, surface fluency that masks subtle factual inaccuracies, and inconsistent prioritization of high-stakes elements such as laterality or operative details.

Reviewer 1 — Comment 6

Since there are many ophthalmic conditions, how can we ensure that AI-generated letters can be useful, and how does the proposed solution deal with them?

Response: We have added a new Discussion paragraph addressing the heterogeneity of ophthalmic conditions and how the present study design accommodates it. The paragraph explains that (i) the standardized prompt is condition-agnostic and constrains the model to the structured input data rather than encoding diagnosis-specific templates; (ii) the reviewer rating form assessed inclusion of high-stakes content elements on a per-case applicability basis; and (iii) the paired design ensured that condition-specific difficulty was matched between resident and model outputs. We note that the model performed broadly comparably to residents on most domains but underperformed on operative documentation, which identifies surgical/procedural cases as the clearest area in which condition-aware prompt variants, retrieval-augmented context, and checklist-based review will be needed to ensure usefulness across the breadth of ophthalmic presentations.

Revised text (excerpt): A related question is how AI-generated discharge letters can be ensured to be clinically useful across the wide heterogeneity of ophthalmic conditions, which range from routine cataract surgery to complex vitreoretinal disease, glaucoma, uveitis, ocular trauma, and neuro-ophthalmology. … In practice, we would not expect a single general-purpose prompt to be optimal for every ophthalmic condition; rather, condition-aware prompt variants, retrieval-augmented context …, and checklist-based clinician review focused on high-stakes elements (laterality, procedure type, biometry, intraoperative findings) are the most likely routes to ensuring usefulness across the full breadth of ophthalmic presentations.

Reviewer 1 — Comment 7

I noticed in this paper that the authors did not present any framework or overall architecture for the proposed solution.

Response: We have added a new Figure 1 showing the overall study framework, together with an accompanying “Study framework and overall workflow” subsection in the Methods. The diagram traces the pipeline from clinical source data through de-identification, parallel drafting paths (resident-written vs GPT-5.2-generated with a standardized anti-hallucination prompt), case-level pairing, blinded structured assessment by three ophthalmologists, and paired statistical analysis with reviewer-adjusted sensitivity analysis. The previous Figures 1-3 have been renumbered to Figures 2-4 throughout the manuscript via tracked changes.

Reviewer 2 Report

Comments and Suggestions for Authors

This is a well written article describing the performance of GPT 5.2 compared to resident discharge letters. I have a few minor comments that should be addressed before this is ready for publication.

 

Materials and methods

Accurate prompting is essential to return accurate and actionable data. Usually prompting is iterative before the desired response is obtained. Did the study team need to develop and test multiple rounds of prompts before getting the desired response? Who developed the prompt?

 

Results

Fig 3- text compares conciseness of letters by residents vs GPT, however the figure doesn’t show conciseness. Is this a typo or am I missing something?

 

Discussion

One of the perceived benefits of LLMs is their ability to save time in routine tasks. I was impressed that the resident and GPT letters are similar across many domains. Did anyone measure the time it took residents to draft these letters, and how this compares to the time it took GPT?

Author Response

We thank Reviewer 2 for the supportive comments and for highlighting the importance of prompt iteration, the Figure 3 discrepancy, and the time-savings question. We address each point below.

Reviewer 2 — Comment 1 (Materials and methods)

Accurate prompting is essential to return accurate and actionable data. Usually prompting is iterative before the desired response is obtained. Did the study team need to develop and test multiple rounds of prompts before getting the desired response? Who developed the prompt?

Response: Yes - the prompt was developed iteratively. We have added text to the “Generation of GPT-5.2-based discharge letters” subsection clarifying authorship and the iteration process: the prompt was drafted by an ophthalmologist-led writing group (senior ophthalmology authors together with a clinical informatics co-author) and iteratively refined over four rounds on a pilot set of cases that were not part of the study sample, until the model output reliably preserved laterality, numerical values, dates, and procedural details without introducing content absent from the source. The final prompt was then locked for the duration of the study to avoid post-hoc tuning on the evaluated cohort.

Revised text (excerpt): The prompt was drafted by an ophthalmologist-led writing group (senior ophthalmology authors together with a clinical informatics co-author) and was iteratively refined over four rounds on a pilot set of cases that were not part of the study sample, until the model output reliably preserved laterality, numerical values, dates, and procedural details without introducing content that was absent from the source material. … The prompt was kept fixed throughout the study to avoid post-hoc tuning on the evaluated cohort.

Reviewer 2 — Comment 2 (Results / Figure 3)

Fig 3 — text compares conciseness of letters by residents vs GPT, however the figure doesn’t show conciseness. Is this a typo or am I missing something?

Response: We thank the reviewer for catching this. Figure 3 (now Figure 4 in the revised manuscript) is revised and now include conciseness and global quality. We have updated the figure and now shows all rating domains shown in the figure (accuracy, completeness, clarity/structure, tone/professional phrasing, conciseness, and global quality), and we have clarified the introductory sentence in the Results so it explicitly states that the distribution “including conciseness” is shown in the figure.

Revised text (excerpt): Figure 4. Distribution of reviewer ratings across main quality domains (accuracy, completeness, clarity/structure, tone/professional phrasing, conciseness, and global quality).

Reviewer 2 — Comment 3 (Discussion / time savings)

One of the perceived benefits of LLMs is their ability to save time in routine tasks. I was impressed that the resident and GPT letters are similar across many domains. Did anyone measure the time it took residents to draft these letters, and how this compares to the time it took GPT?

Response: We agree this is an important comparison. Drafting time was not measured in this study because the resident-written letters were authored as part of routine clinical care before the study was initiated, so resident drafting time was not prospectively recorded and could not be reliably reconstructed retrospectively. The GPT-5.2 letters were generated in seconds per case once the standardized prompt and de-identified inputs were prepared, but that wall-clock generation time is not directly comparable to clinician drafting time (which also includes data review, dictation, and editing). We have made this explicit in the limitations and named time savings, editing burden, and downstream usability as primary outcomes of a planned prospective evaluation.

Revised text (excerpt): Specifically, because the resident-written letters were drafted as part of routine clinical care before the study was initiated, the time required by residents to author each letter was not prospectively recorded and could not be reliably reconstructed retrospectively; the GPT-5.2 letters were produced in seconds per case once the standardized prompt and de-identified inputs were prepared, but this is not directly comparable to clinician drafting time, which includes data review, dictation, and editing. A direct head-to-head comparison of drafting time, editing burden, and downstream usability is therefore a key outcome for a planned prospective evaluation of this approach.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The present paper addresses a vital and crucial domain: medicine. The authors, in their proposal, sought to leverage large language models to address the problem of Discharge letters, saving time and streamlining the process. The proposed method consists of using GPT-5.2 with standardized prompts and comparing its quality with resident-written responses using the same clinical data. The authors concluded that reviewer-adjusted sensitivity analyses were less favorable to GPT-5.2 for several domains.

• The paper is well constructed.
• The contributions are presented in the paper.
• The authors presented an overview of the state of the art