Span-Reference and Grounding Reliability in Generative Spanish Clinical Named Entity Recognition: A Validation Study

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors proposed a methodologically robust way in both "diagnostic framework construction" and "attribution analysis," clearly revealing the key insight that "structural validity does not imply usable spans." It establishes a four-stage diagnostic framework in order: syntax validation, parsing , source grounding, strict matching, offering a novel perspective for the evaluation paradigm of generative NER. This technical approach is insightful and holds potential for adaptation by other researchers. The study proposed a reliable alternative of using "occurrence-index references" instead of "absolute character offsets," providing reusable empirical evidence and diagnostic tools for the output interface design of future clinical information extraction systems. The use of Qwen and Llama also represents a practical and generalisable approach. In the experimental section, the study provides detailed and comprehensive comparative analyses in terms of error metrics, significance levels, and comparisons with representative prior work, thereby offering a practical foundation and empirical guidance for similar research.

      The main limitations included are required to revise:

1)The generalisability of the findings and the stringency of the evaluation criteria may not align with clinical compromise and real-world practice; therefore, the recommended approaches may not necessarily be the most suitable choices in practical deployment scenarios.

2)The study lacks zero-shot/few-shot comparisons. It focuses primarily on "fine-tuning" but does not compare with current mainstream clinical information extraction paradigms for large language models (e.g., in-context learning / few-shot prompting), which limits the applicability of its conclusions in the era of general-purpose large models.

3)The primary experiments are based on Spanish clinical case reports, introducing dataset homogeneity and language bias. The test set does not comprise independent external datasets from different hospitals, regions, or time periods.

4)The format effectiveness of this study is questionable for long clinical notes, discharge summaries, or imaging reports (which contain extensive unstructured narrative), particularly with respect to long-span entities or nested entities.

5) Although their proposed "N-th occurrence" indexing as a replacement for absolute offsets performs well on standard datasets, this counting strategy may be highly susceptible to minor string variations in real-world electronic health record systems, leading to complete failure (poor robustness).

6) Minor issues: a. The layout of the graphical modules in Figure 1 does not comply with scientific paper formatting standards. Connections between modules should use strictly horizontal and vertical lines, without arbitrary angled lines. Additionally, the order of the last row could be reversed from left to right to avoid excessively long connecting lines.  b. It is also recommended to add an overall flowchart that clearly illustrates each step and its corresponding functional modules.

Author Response

Comment 1: The generalisability of the findings and the stringency of the evaluation criteria may not align with clinical compromise and real-world practice; therefore, the recommended approaches may not necessarily be the most suitable choices in practical deployment scenarios.
Response 1: We agree that performance under strict-span evaluation does not by itself establish clinical utility. We retained strict-span F1 because it is the official evaluation metric for all three datasets and directly measures whether a predicted entity matches the annotated source span. We revised the Discussion, Limitations, and Conclusion to make clear that these benchmark results do not establish clinical utility or deployment readiness. Our practical recommendations are now limited to workflows that require exact source spans; approximate matching and downstream utility require separate evaluation (pp. 22-25, lines 795-817, 839-878, and 896-913).

Comment 2: The study lacks zero-shot/few-shot comparisons. It focuses primarily on "fine-tuning" but does not compare with current mainstream clinical information extraction paradigms for large language models (e.g., in-context learning / few-shot prompting), which limits the applicability of its conclusions in the era of general-purpose large models.
Response 2: We agree that the fine-tuned comparison alone should not be generalized across adaptation regimes. We therefore evaluated the same two unadapted model checkpoints and four output pipelines (inline XML, mention-list JSON, a tab-separated mention list, and direct-offset JSON) on the official test sets under zero-shot and three-shot prompting. The task-and-format query, generation limits, parsers, grounding rules, and scorers were the same as in the fine-tuned evaluation. Zero-shot inference used no training examples and therefore produced one deterministic result for each task-model-format combination. For three-shot inference, we evaluated three fixed sets, each containing three examples selected from the training split, and report variability across these sets separately from variability across fine-tuning seeds. Zero-shot strict F1 was at most 0.131 and three-shot F1 at most 0.386. Adding three examples improved the mention-list JSON and tab-separated pipelines relative to zero-shot prompting. It produced little benefit for inline XML, whose three-shot F1 ranged from 0 to 0.028 across the task-model conditions, while direct-offset JSON remained at zero. Every descriptive paired bootstrap interval for prompted minus fine-tuned F1 lay below zero. This is a matched comparison of two compact 3B checkpoints under fixed prompts; it does not show that prompting in general is inferior. The design and analysis are reported in Methods (pp. 7 and 10-12, lines 235-243, 389-398, and 461-469), the complete results and Table 9 in Results (pp. 18-19, lines 639-666), and the interpretation and limitations in Discussion (pp. 21 and 23, lines 729-739 and 843-850).

Comment 3: The primary experiments are based on Spanish clinical case reports, introducing dataset homogeneity and language bias. The test set does not comprise independent external datasets from different hospitals, regions, or time periods.
Response 3: We agree and added an external CARMEN-I evaluation of 458 Spanish, clinically annotated sections from Hospital Clínic de Barcelona. We applied the previously trained primary and diagnostic models without further training, prompt changes, retries, or output repair. CARMEN-I was released as an annotated corpus rather than as a shared-task benchmark. We therefore calculated precision, recall, and F1 with a deterministic strict-span evaluator. After duplicate removal, a true positive required an identical document ID, span boundaries, and mapped label; corpus micro-F1 was calculated from pooled counts. This matches the strict span-and-label criterion used by the task-specific scorers for the original test sets. The string-grounded primary pipelines achieved mean F1 of 0.610-0.659, occurrence-index models achieved 0.610-0.652, and direct-offset performance remained at or below 0.002. Inline XML fell to F1 of 0.113-0.177 because most accepted candidates could not be grounded to a source position. The external CARMEN-I evaluation therefore supports the direct-offset finding while showing that other representations do not transfer uniformly. The Methods describe the controlled-access subset and its scope (p. 8, lines 273-292) and the CARMEN-I evaluator (p. 11, lines 433-438). Results report the primary and diagnostic evaluations separately (pp. 19-20, lines 667-700; Table 10), and Discussion and Limitations restrict the evidence to one additional hospital and period (pp. 21 and 24, lines 740-748 and 867-875). The study does not establish multi-hospital, cross-regional, or cross-language generalization.

Comment 4: The format effectiveness of this study is questionable for long clinical notes, discharge summaries, or imaging reports (which contain extensive unstructured narrative), particularly with respect to long-span entities or nested entities.
Response 4: We agree that the original clinical case reports do not represent the full range of long hospital documents or richly nested extraction tasks. The scope of this study is Spanish clinical text, and we do not claim that the findings extend to other languages. We now report the observed document-length, span-length, and topology ranges and state explicitly that results within the original collection do not establish effectiveness on complete discharge summaries, imaging reports, or longitudinal records. The CARMEN-I evaluation adds predominantly discharge-report sections, with a median of 228 words, a 99th percentile of 1,021 words, and a maximum of 1,441 words, but only five radiology reports and four progress notes. We therefore describe it as section-level rather than full-note validation and report that up to 10.0% of completions in one CARMEN-I condition reached the unchanged generation limit (Methods, pp. 7-8, lines 267-292; Results, pp. 19-20, lines 667-700; Limitations, pp. 23-24, lines 829-835 and 867-875).

Comment 5: Although their proposed "N-th occurrence" indexing as a replacement for absolute offsets performs well on standard datasets, this counting strategy may be highly susceptible to minor string variations in real-world electronic health record systems, leading to complete failure (poor robustness).
Response 5: We agree and have narrowed the corresponding claim. Occurrence indexing requires the generated mention to match the source string exactly under the tested grounding rule; spelling, whitespace, abbreviation, or normalization differences can therefore prevent grounding before the occurrence index is applied. The Abstract, Discussion, and Conclusion now restrict the finding to exact matching under the tested conditions and identify fuzzy or normalized matching as a different pipeline requiring separate validation. Without further training, occurrence indexing achieved F1 of 0.610-0.652 on CARMEN-I and improved over original direct offsets by 0.608-0.650 in all six task-model comparisons. This supports transfer to one additional exact-matching setting, not robustness to arbitrary string variation (Abstract, p. 1, lines 14-23; Methods, p. 9, lines 338-345; Results, p. 20, lines 691-700; Discussion and Limitations, pp. 22-24, lines 777-784 and 851-857; Conclusion, pp. 24-25, lines 896-907).

Comment 6a: The layout of the graphical modules in Figure 1 does not comply with scientific paper formatting standards. Connections between modules should use strictly horizontal and vertical lines, without arbitrary angled lines. Additionally, the order of the last row could be reversed from left to right to avoid excessively long connecting lines.
Response 6a: We revised Figure 1 so that every connector is horizontal or vertical and reversed the evaluation row, eliminating the long diagonal transition in the submitted figure. We also redistributed and resized the modules for legibility at the final manuscript width (Figure 1 and caption, p. 6).

Comment 6b: It is also recommended to add an overall flowchart that clearly illustrates each step and its corresponding functional modules.
Response 6b: We expanded Figure 1 into a single overall study flowchart rather than adding a redundant second overview. It distinguishes the fine-tuned primary comparison, the prompted comparison, the position-reference diagnostic, inference on their respective evaluation sets, and the common syntax-to-scoring cascade. The caption explains that CARMEN-I is a separate inference-only evaluation and that the analysis settings remain separate (Figure 1 and caption, p. 6).

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

1. Eliminate the textual repetition in the Introduction (lines 100–115). Clarify the methodological scope by clearly distinguishing the main protocol conducted on the official datasets from the diagnostic study based on the 75-document subset.
2. Extend the literature review to constrained decoding techniques (Constrained Decoding / Grammars). Contextualize the findings by discussing whether the observed failures in numerical indexing in clinical Spanish also occur in other languages.
3. Retain the cascading demonstrative structure (overall performance → pipeline analysis → fine-grained diagnosis), as this structure rigorously demonstrates that the failure originates from source anchoring and character counting.
4. Emphasize the three major contributions of the study:
   • demonstrating that syntactic compliance (JSON) does not guarantee exact source anchoring;
   • validating the occurrence index as an alternative to character offsets;
   • providing concrete recommendations for the deployment of RAG-based and clinical information extraction systems.

Author Response

Comment 1: Eliminate the textual repetition in the Introduction (lines 100-115). Clarify the methodological scope by clearly distinguishing the main protocol conducted on the official datasets from the diagnostic study based on the 75-document subset.
Response 1: We removed the repeated background and clarified the two protocols in the Introduction, Methods overview, and revised Figure 1. The primary comparison uses 72 adapters trained on 675 documents and evaluated once on the corresponding official test sets. The position-reference diagnostic uses 54 separately trained adapters under the 600/75/75 training-derived split and does not use the official test set (Introduction, pp. 2-3, lines 58-108; Methods and Figure 1, pp. 5-7, lines 203-257).

Comment 2: Extend the literature review to constrained decoding techniques (Constrained Decoding / Grammars). Contextualize the findings by discussing whether the observed failures in numerical indexing in clinical Spanish also occur in other languages.
Response 2: We extended Related Work and Discussion with verified primary sources on grammar-constrained and source-conditioned generation. The revision distinguishes guaranteeing surface syntax from constraining input-dependent span values. We also add direct evidence from English NER, where an Entity+Position representation substantially underperformed inline copy-and-mark output, and Arabic span annotation, where generated indices disagreed with the extracted text. A recent multilingual aggregate comparison provides broader context, while we state explicitly that these studies do not establish a uniform effect size across languages, genres, tokenizers, or systems. These additions appear in Related Work (p. 4, lines 147-176) and Discussion (pp. 21-23, lines 749-757 and 804-809).

Comment 3: Retain the cascading demonstrative structure (overall performance -> pipeline analysis -> fine-grained diagnosis), as this structure rigorously demonstrates that the failure originates from source anchoring and character counting.
Response 3: We retained this structure. The Results continue from target-conversion checks and overall strict-span performance through syntax, parsing, and grounding to the fine-grained position-reference diagnosis and robustness checks. Any external validation is placed only after that original cascade and is not pooled with it. The original cascade occupies Sections 4.1-4.5 (pp. 12-18, lines 499-638), followed by separate prompted and CARMEN-I evaluations in Sections 4.6-4.7 (pp. 18-20, lines 639-700).

Comment 4: Emphasize the three major contributions of the study: demonstrating that syntactic compliance (JSON) does not guarantee exact source anchoring; validating the occurrence index as an alternative to character offsets; and providing concrete recommendations for the deployment of RAG-based and clinical information extraction systems.
Response 4: We strengthened the distinction between syntax and grounding and limited the occurrence-index conclusion to exact strings. For workflows requiring exact source provenance, we recommend validating generated mentions against the source, retaining only uniquely resolved spans when required, and logging each evaluation stage separately. We do not claim evaluated RAG deployment because the study includes no retrieval component, retrieval endpoint, clinician evaluation, or deployment study; the Discussion therefore frames relevance to retrieval-augmented workflows conditionally (Abstract, p. 1, lines 1-23; Introduction, p. 3, lines 96-108; Discussion, pp. 20-23, lines 701-817; Conclusion, pp. 24-25, lines 879-913).

Author Response File: Author Response.pdf