Span-Reference and Grounding Reliability in Generative Spanish Clinical Named Entity Recognition: A Validation Study

Abstract

Generative named entity recognition (NER) systems must identify clinical concepts and map them to exact source spans. We investigated how span-reference design affects this mapping and whether failures arise from recognition or source localisation. Four representation-and-grounding pipelines—inline XML, mention-list JSON, a tab-separated mention list, and direct-offset JSON—were compared for disease, procedure, and symptom recognition in a public Spanish clinical case-report collection. Two instruction-tuned 3-billion-parameter models were fine-tuned for each task, and outputs were evaluated for syntactic validity, parsing, grounding, and official strict-span performance. Inline XML, mention-list JSON, and the tab-separated format achieved F₁ scores of 0.637–0.736, whereas direct-offset JSON achieved 0.001–0.005. With the same unadapted checkpoints, zero-shot F₁ was at most 0.131 and three-shot F₁ at most 0.386; three-shot prompting improved the mention-list outputs, while inline XML and direct offsets remained near zero. Direct-offset outputs usually contained relevant mention text but incorrect character positions. On a separate 75-document confirmation partition, grounding the emitted strings recovered F₁ of 0.603–0.701, while replacing absolute offsets with mention-occurrence numbers achieved 0.652–0.740. Without further training, the external CARMEN-I evaluation on 458 hospital-record sections yielded F₁ of 0.610–0.659 for string-grounded or occurrence-index outputs, compared with 0.113–0.177 for inline XML and at most 0.002 for direct offsets. These results show that output validity alone does not establish usable span extraction. Separating recognition from source localisation identifies where an otherwise valid generation fails, while occurrence-based references provide a more reliable alternative to absolute character offsets under exact matching in the tested settings.

1. Introduction

Named entity recognition (NER) identifies mentions of concepts in text and assigns them to predefined categories. In clinical natural language processing, these mentions may describe diseases, procedures, symptoms, or other information needed for tasks such as terminology normalisation and clinical phenotyping. Evaluation commonly requires an exact match between the predicted and annotated character spans. A system must therefore recognise the correct concept and locate the precise characters that express it.

This requirement is not straightforward for generative large language models (LLMs). Unlike sequence-labelling models, which assign a label to each input token, a generative model returns a new sequence of text. Its predictions must be interpreted and connected to the source document before they can be evaluated as spans. A model can identify the right clinical phrase but still produce malformed structure, copy the phrase incorrectly, select the wrong occurrence of a repeated phrase, or generate inaccurate character indices. These are distinct errors, but an aggregate NER score does not show where they arise.

The chosen output format determines both what the model must generate and how its predictions can be located. Inline tags preserve position by placing boundaries in a copy of the source. A list of mention strings is shorter, but a later grounding step must search for each mention and resolve repeated occurrences. Numerical-offset formats state a position directly, but require the model to count characters accurately. These alternatives are consequently more than cosmetic serialisations: they distribute the localisation task differently between the model and the deterministic processing that follows it.

Previous studies show that output structure can affect information extraction and other generation tasks, although the magnitude varies with the model, task, and constraint [1,2,3,4,5]. Generative NER has consequently used several span-reference designs, including inline tags, mention strings, occurrence references, and numerical indices  [6,7,8]. Clinical studies have also examined prompt-based and instruction-tuned extraction [9,10], while work on Spanish clinical text shows that output structure and subsequent source matching can affect extraction quality [11]. The remaining question for fine-tuned Spanish clinical NER is not which serialisation is intrinsically best, but whether each complete representation-and-grounding pipeline converts generated entities into exact source spans and, when it does not, where the prediction is lost.

The primary comparison fine-tunes 72 adapters: four output formats, two instruction-tuned 3-billion-parameter models, three training seeds, and disease, procedure, and symptom annotation layers from one public Spanish clinical-case-report collection. The four formats are inline XML (iXML), mention-list JSON (mJSON), a tab-separated mention list (TSV), and direct-offset JSON (oJSON). They cover tag-based, string-match, and numerical-index span references and are evaluated on the corresponding official test sets. Documents, models, training procedure, and decoding controls are shared across formats. Because positional information, targets, parsers, and grounding rules differ, the estimand is the complete pipeline; mJSON–TSV is the narrower serialisation contrast because both contain the same information and use the same grounding rule.

Separately, a 54-adapter position–reference diagnostic uses a 600-document training partition and evaluates the original offset instruction, an explicit coordinate definition, and occurrence-index JSON on internal 75-document development and confirmation partitions. It does not add conditions to the official test comparison. This separation lets the primary experiment establish the performance contrast and the diagnostic experiment test whether missing mention text, ambiguous instructions, or absolute numerical localisation accounts for the direct-offset result. To examine transfer beyond the case-report collection, we evaluate the previously trained primary and diagnostic pipelines without further training or prompt adjustment on de-identified Spanish hospital-record sections from CARMEN-I. This CARMEN-I evaluation assesses the scope of the pipeline and diagnostic findings rather than adding a new primary research question or pooling evidence across the three evaluation settings. We also evaluate the unadapted checkpoints under zero-shot and three-shot prompting on the official test sets. This secondary comparison retains the same final task and format instruction as the fine-tuned systems; the three-shot condition prepends fixed training-derived examples. It therefore tests the four pipelines without task-specific weight updates while treating sensitivity to example selection separately from variability across fine-tuning seeds.

The study addresses five research questions:

• Do the four representation-and-grounding pipelines differ in strict-span NER performance?
• At which stage do their errors arise: syntax, parsing, source grounding, or entity recognition?
• Why does direct generation of character offsets succeed or fail relative to tag- and string-based span references?
• Are the conclusions stable when inference is performed concurrently or one document at a time?
• How do the same pipelines behave under zero-shot and three-shot prompting with the unadapted checkpoints?

The paper makes three main contributions. First, it provides a controlled comparison of four complete generative NER pipelines under official strict-span evaluation, showing how alternative span-reference designs affect reliability across three clinical entity types and two model families. Second, it makes the final score interpretable by separating failures of syntax, parsing, source grounding, and entity correctness; this distinguishes well-formed output from entities that can actually be located. Third, a matched position–reference experiment shows that the direct-offset collapse is associated with numerical localisation rather than an absence of relevant mention text, and evaluates occurrence references as an alternative. Together, these contributions provide an evidence-based way to select, diagnose, and report output interfaces for generative clinical NER. The separate CARMEN-I evaluation tests how far these contributions transfer to the audited CARMEN-I setting while retaining the original primary and diagnostic analyses. The prompted comparison further bounds the adaptation regime to which the controlled fine-tuning results apply.

The remainder of the article is organised as follows. Section 2 relates the study to prior work on generative NER, output formats and span-reference strategies, and clinical extraction evaluation. Section 3 describes the data and methods, and Section 4 reports the results. Section 5 discusses their implications and limitations, before Section 6 presents the conclusions.

2. Related Work

Three bodies of work define the methodological context: generative NER, output and span-reference constraints, and evaluation of clinical extraction pipelines. Their common issue is the conversion of generated text into scored source spans.

2.1. Generative Named Entity Recognition

NER has conventionally been treated as sequence labelling, using models such as conditional random fields and Bidirectional Encoder Representations from Transformers (BERT) to assign a tag to each token. Generative NER instead asks a decoder to describe the entities in its output. GPT-NER, for example, reproduces the input with markers around entity mentions [6]. Other work shows that instruction-tuned generative models can approach encoder-based biomedical NER systems after training on reformatted annotations [10] and can transfer to new entity-recognition tasks through prompts and examples [9,12].

The generated answer nevertheless requires interpretation before standard NER metrics can be calculated. Labrak et al. [13] describe task-specific processing needed to recover entities from variable model outputs, whilst Averly and Ning [14] use decomposition and filtering to improve the retrieval of clinical mentions. In Spanish clinical NER, del Moral-González et al. [11] compare prose, dictionary-like, and JSON-like outputs and find that both output complexity and subsequent source matching affect extraction quality. Pan et al. [15] compare prompting, QLoRA fine-tuning, and encoder-only fine-tuning on four Spanish medical NER datasets, including SympTEMIST; their results provide relevant model- and training-strategy context rather than a direct comparison of span-reference representations. Few-shot results for toxic-habit extraction likewise illustrate the continuing difficulty of fine-grained entity recognition in Spanish clinical text [16].

2.2. Output Formats and Span Reference

Output format is increasingly recognised as a source of variation in LLM evaluation. Ravikumar et al. [1] show that changing the requested structure can substantially alter information-extraction performance. Related studies identify model preferences for particular structures and measurable costs from format restrictions even when the requested content is unchanged [2,3]. These effects are not uniform. Causal comparisons by Yuan et al. [4] find little evidence of a format effect in most tested scenarios involving larger frontier models, while Fan [5] reports that losses from complex schemas vary across model-capacity conditions. Model capacity and task therefore need to be considered when format results are interpreted.

Constrained decoding can reduce one class of output failure. Formal grammars can guarantee that a decoded response follows a declared structure, and an input-dependent grammar can additionally restrict the permitted output using the current input  [17]. Such constraints must be aligned with the model’s subword vocabulary to avoid unnecessary loss of task accuracy [18]. These guarantees do not collapse syntax and grounding into one problem. A JSON grammar may require integer-valued start and end fields while still permitting coordinates that select the wrong source characters; constraining a field to a valid input span requires an input-conditioned value constraint rather than syntax alone.

For generative NER, span reference is a particularly important component of the output. Lu et al. [19] distinguish augmented language, in which annotations are inserted into a copy of the input, from separate structured annotations that must later be aligned with it. Semin et al. [8] further classify span references as tags, indices, or string matches. Tag methods retain position through the placement of boundaries; index methods emit numerical locations; and string-match methods search the source for generated mention text. On a 13-language NER benchmark, their aggregate results show that numerical indices are unreliable when positions are not visible in the input. Their occurrence field distinguishes repeated exact strings, and their source-conditioned LogitMatch decoder prevents the mention field from leaving the input. Because the NER results are pooled across languages, they do not establish the same effect size for every language.

Direct evidence from English and Arabic shows that the numerical-position problem is not confined to Spanish. In an English NER ablation, GPT-NER’s Entity+Position representation achieved an F₁ of 38.73, compared with 92.68 for inline copy-and-mark output under the same CoNLL-2003 setup; the error analysis identified confusion about position-index semantics [6]. In prompted Arabic propaganda-span annotation, GPT-4 frequently returned start and end indices that disagreed with otherwise extracted span text; replacing those indices with the first exact source occurrence materially changed the score [20]. More generally, character-level benchmarks show that LLMs may encode the spelling of their subword tokens without using that knowledge reliably for text manipulation [21]. This is compatible with difficulty generating absolute offsets, but it does not establish the mechanism behind errors in clinical NER.

Zhan et al. [7] directly compare inline, grouped, occurrence-based, and offset-based outputs across flat and nested NER datasets, including the biomedical GENIA corpus, and several models. Their occurrence-based representation retains positional information without absolute character counting, whereas direct offsets perform poorly in many conditions. Dagdelen et al. [22] similarly show, for scientific information extraction, that structured output affects accuracy as well as the ease with which predictions can be normalised for downstream use. These findings motivate treating the representation, parser, and source-grounding rule as a connected pipeline. They also motivate our diagnostic comparison of absolute offsets, occurrence references, and text-based grounding.

2.3. Evaluation of Clinical Extraction Pipelines

Strict-span NER conflates several possible causes of an incorrect prediction. Even in non-generative systems, aggregate F₁ can conceal boundary errors, type confusion, and omitted mentions [23]. Generative systems add failures of syntax, schema, copying, and source alignment. A valid JSON object, for example, may still contain a phrase that is absent from the source or offsets that select different characters. Reporting only the final score obscures this distinction.

Clinical LLM reporting guidance emphasises transparent, task-specific reporting [24]. The present study applies that principle to the path from generated output to evaluated span by assessing syntax, parsing, grounding, and strict-span correctness separately.

Table 1. Comparison with closely related generative information-extraction studies.

Study	Setting	Span Reference	Evaluation	What This Study Adds
Ravikumar et al. [1]	Multiple information-extraction tasks and models	Equivalent output structures	Task performance across formats	Clinical strict-span evaluation and separate analysis of syntax, parsing, grounding, and entity correctness
Zhan et al. [7]	Four NER datasets and eight models	Tags, strings, occurrences, and offsets	Format-specific span conversion	Supervised Spanish clinical NER, official task-specific scorers, and stage-specific error analysis
Semin et al. [8]	Four span-labelling tasks; one fixed seed	Tag, index, and string-match taxonomy	Constrained and unconstrained generation	Evaluation of span-reference strategies in a no-repair, fine-tuned clinical NER pipeline with multiple training seeds
del Moral et al. [11]	Spanish clinical collection translated to English; zero-shot models	Several generated structures	Candidate normalisation and grounding	Experiments on the original Spanish task data using supervised fine-tuning and exact scoring with the official task-specific scorers
Present study	Three Spanish clinical annotation layers; fine-tuned and prompted 3B models	Tags, exact strings, offsets, and occurrence indices	Syntax, parsing, grounding, and strict-span scoring	An integrated comparison of complete pipelines, adaptation regimes, stage-specific diagnostics, and paired document-level uncertainty estimates

summarises the closest studies. Our contribution is not the already reported sensitivity to output structure, difficulty of latent numerical indexing, or utility of occurrence references. It is a standardised end-to-end pipeline comparison across three annotation layers of Spanish clinical case reports, using supervised fine-tuning and matched zero-/three-shot prompting of two 3-billion-parameter model families, official strict-span scorers, no-repair candidate routing, diagnostics that distinguish parsing, grounding, and scoring failures, and paired document-level uncertainty estimates.

3. Materials and Methods

The aim of this study is to evaluate how the representation of generated entities and their grounding in the source text relate to strict-span performance and to identify where conversion failures arise. We address this aim through a primary fine-tuned comparison of four representation-and-grounding pipelines, a prompted comparison using the unadapted checkpoints, a position–reference comparison of three ways to refer to an entity’s position using training-derived data, and a separate external CARMEN-I evaluation of the fine-tuned pipelines on hospital-record text (Figure 1). Within each task and model, the primary comparison keeps the documents, training procedure, and decoding settings unchanged. The prompted comparison retains the final task and format instruction while removing task-specific weight adaptation and, in the three-shot condition, adding fixed training-derived examples. The position–reference comparison tests whether direct-offset errors reflect unclear counting instructions, missing mention text, or difficulty generating absolute character counts. CARMEN-I inference changes neither model weights nor format-specific prompts. All four analyses evaluate syntax, parsing, source grounding, and strict-span correctness separately, while their results remain distinct.

Section 3.1 and Section 3.2 define the data partitions, output representations, and position references. Section 3.3 and Section 3.4 describe training and inference, Section 3.5 and Section 3.6 define evaluation and statistical analysis, and Section 3.7 covers reproducibility.

3.1. Data and Splits

We use three public Spanish clinical named entity recognition (NER) tasks: DisTEMIST for diseases [25], MedProcNER for medical procedures [26], and SympTEMIST for symptoms, signs, and clinical findings [27]. The tasks annotate different entity types in the same collection of clinical case reports and provide official training and test sets. We retain the original task labels: ENFERMEDAD (disease), PROCEDIMIENTO (procedure), and SINTOMA (symptom), respectively. The official strict-span scorers are used for evaluation.

For each task, the 750 documents in the official training set are divided into 675 training documents and 75 development documents using the same fixed document split across the three annotation layers (

Table 2. Task statistics. Entity counts refer to gold annotations; Docs denotes documents. Topology columns report the percentage of entities participating in at least one overlap relation in the combined training and development partitions.

Task	Entity Type	Train		Development		Test		Pre-Test Topology (%)
		Docs	Entities	Docs	Entities	Docs	Entities	Any Overlap	Nested	Crossing
DisTEMIST	Disease	675	7193	75	872	250	2598	4.0	3.6	0.4
SympTEMIST	Symptom/sign	675	8119	75	973	250	3104	0.6	0.2	0.5
MedProcNER	Procedure	675	9875	75	1190	250	3619	3.8	3.1	0.7

). The development set is used to inspect target lengths, set generation limits, and test the parsers and grounding procedures. The official test set is used only for final inference and scoring. It is not used for training, prompt development, or selection of analysis settings.

The prompted comparison also uses the official test sets. Zero-shot inference uses no training examples and therefore has no example-selection seed. For three-shot inference, all examples come from the 675-document training partition. For each task, documents with at least one annotated entity are sorted by identifier and shuffled independently with seeds 13, 17, and 23; the first three documents define each ordered set of examples. The same set is used across both model families and all four formats within a task, with only the assistant target serialisation changing by format. Development and test documents are excluded from these sets. The selection procedure, example order, prompts, generation limits, and analysis were specified before the prompted test inference.

The diagnostic comparison examines three ways to identify an entity’s position: the original direct-offset instruction, an instruction that explicitly defines how offsets are counted, and an occurrence index attached to the entity text. It uses only documents from the official training distribution. We reserve 75 documents from the primary 675-document training subset as a confirmation partition, train all 54 diagnostic adapters on the remaining 600 documents, and retain the original 75-document development partition. Each condition is checked on development data before evaluation on the confirmation partition; the official test set is not used.

All three position references were first compared for DisTEMIST. We then applied the unchanged occurrence-index condition to MedProcNER and SympTEMIST to determine whether the same pattern appeared in the other annotation layers. Because those two comparisons were specified after the DisTEMIST result was available, they describe the extension of the pattern rather than provide independent tests specified in advance. The confirmation documents were excluded from diagnostic training but come from the same clinical-case- report collection.

In each task, some documents contain the same mention text more than once. Because mJSON and TSV return the mention string without its position; the string alone cannot always identify which occurrence is annotated. Under the count-matched grounding rule defined below, the maximum recovery of gold spans from perfect mention strings is 95.1% for DisTEMIST, 96.1% for MedProcNER, and 96.7% for SympTEMIST. The annotations also contain nested and crossing spans, as quantified in Table 2. Complete split-specific span-topology statistics are reported in the Supplementary Material S1 and accompanying Supplementary Data S2.

Each document is processed as a single example. The 99th percentile of document length is approximately 1660 tokens for both model tokenisers, below the 8192-token sequence limit. For each task, model, and format, the generation limit is the 99th percentile of development-set gold target length plus a 20% margin to accommodate ordinary variation above the percentile estimate. The resulting limits range from 2768 to 4529 tokens for iXML, 603 to 1295 for mJSON, 458 to 990 for TSV, and 938 to 2162 for oJSON.

External Validation Data

CARMEN-I version 1.0.1 is a controlled-access collection of de-identified electronic health record text from Hospital Clínic de Barcelona, collected between March 2020 and March 2022 [28]. We selected the 458 released files marked as Spanish, annotated for clinical entities, and distributed with protected elements replaced by synthetic equivalents. The same 458 files are used for each task, including files with no gold entity of the corresponding type. The compatible CARMEN-I labels ENFERMEDAD (disease), PROCEDIMIENTO (procedure), and SINTOMA (symptom) define the three task endpoints, yielding 4911, 5556, and 7116 gold annotations, respectively.

The evaluation units are the sections distributed with the corpus; discharge and transfer reports that were segmented during corpus construction are not reconstructed into full notes. The subset contains 430 discharge-report sections, 14 transfer-report sections, five death reports, five radiology reports, and four progress notes. Its median document length is 228 words, the 95th and 99th percentiles are 729 and 1021, and the maximum is 1441. Across the three analysed labels, 95% of spans contain at most 52 characters and the maximum is 197 characters; within individual task layers, 0.9–2.1% of entities participate in an overlap relation. Thus, the subset changes institution, period, and source genre, but provides sparse evidence for radiology and progress notes and does not constitute full-note or richly nested validation. CARMEN-I is not used for training, prompt design, generation-budget selection, model selection, or output repair.

3.2. Output Formats and Position References

The formats represent the three span-reference families described by Semin et al. [8]: tags placed in a copy of the source (iXML), mention strings that must be located in the source (mJSON and TSV), and numerical character indices (oJSON). Here, a representation-and-grounding pipeline comprises the format-specific instruction, target representation and schema, together with the deterministic parser, source-grounding rule, and policy for rejected candidates. The mJSON–TSV comparison is therefore the closest test of serialisation structure because both carry mention strings and labels and use the same grounding rule. The other contrasts compare complete pipelines that also differ in retained positional information, target length, parser, and grounding route. For fine-tuning, a deterministic conversion rule transforms each gold annotation into the expected output for its format. Only the format-specific instruction and this expected output differ between conditions. The input, entity definition, model, fine-tuning settings, and decoding procedure remain the same.

Table 3. Information and source-grounding procedure for each output format. Target-token ranges are means across task–model combinations.

Component	iXML	mJSON	TSV	oJSON
Span reference	Tag boundaries	Exact mention string	Exact mention string	Character indices
Emitted information	Source copy, tags, label	Mention, label	Mention, label	Start, end, mention, label
Parser unit	Tagged document	JSON object	TSV line	JSON object
Grounding	Tag positions after exact source reconstruction	Count-matched source search	Count-matched source search	Offset–text equality
Repeated mention	Position retained	Occurrence inferred	Occurrence inferred	Position requested
Main failure routes	Copy, tag, or parse error	JSON, copy, or occurrence ambiguity	Line, copy, or occurrence ambiguity	Index, range, or text mismatch
Mean target tokens	1122–1379	202–296	151–227	308–503

summarises the information carried by each format and the corresponding grounding rule.

Table 4. Illustrative outputs for the source phrase dolor óseo (bone pain), annotated as SINTOMA over characters 0–10.

Format	Expected Output
iXML	<SINTOMA data-span=“s0000” data-boundary=“start”/>dolor óseo<SINTOMA data-span=“s0000” data-boundary=“end”/>
mJSON	{“entities”:[{“text”:“dolor óseo”,“label”:“SINTOMA”}]}
TSV	dolor óseo<TAB>SINTOMA
oJSON	{“entities”:[{“start”:0,“end”:10,“text”:“dolor óseo”,“label”:“SINTOMA”}]}

shows the four targets for the same simple annotation. In iXML, each entity receives an identifier such as s0000. Its start and end markers carry the same data-span identifier, allowing the parser to pair the two boundaries. The identifier is only a pairing label and does not encode a character position; data-boundary distinguishes the start marker from the end marker. Further examples, including repeated and overlapping mentions, are provided in the Supplementary Material S1.

• iXML—Inline XML (tag family).

The model reproduces the source and places paired boundary tags around each entity. The shared data-span value tells the parser which start and end markers belong to the same entity. Removing all tags must reconstruct the source exactly; the tag positions then determine the entity offsets. Because boundaries are represented independently, the format can express the nested and crossing spans present in the annotation layers.

• mJSON—Mention-list JSON (string-match family).

The model returns a JSON object whose entities array contains objects with text and label fields. Offsets are recovered by an exact search in the source. If the model emits the same string more than once, its first prediction is assigned to the first source occurrence, its second prediction to the second occurrence, and so forth. This rule cannot identify a later annotated occurrence when an earlier identical string is not annotated. For example, if dolor (pain) occurs twice but only the second occurrence is an entity, the mention string alone does not reveal which occurrence was intended.

• TSV—Tab-separated mention list (string-match family).

Each line contains a mention and label separated by a tab. TSV carries the same entity information and uses the same count-matched grounding rule as mJSON, but avoids the JSON object and array structure. The mJSON–TSV contrast therefore provides the closest comparison of serialisation syntax in the study.

• oJSON—Direct-offset JSON (index family).

Each JSON object contains start, end, text, and label. A candidate is grounded only when its indices are within range and source[start:end] equals the emitted text. Thus, oJSON asks the model to generate an absolute position, whereas iXML encodes position through tag placement.

The diagnostic position–reference comparison uses three matched conditions: the original oJSON instruction; an instruction that explicitly defines zero-based Unicode indices, an inclusive start, and an exclusive end; and occurrence-index JSON. The third condition emits a mention, label, and zero-based occurrence number among exact, non-overlapping appearances of that mention from left to right. For example, it can refer to the second occurrence of dolor without counting every preceding character. This retains a positional choice without requiring an absolute character index. The input itself contains no visible character coordinates.

As a diagnostic for oJSON, we also ignore the generated numbers and ground its emitted mention strings with the count-matched rule. This text-based rescue estimates how much performance is lost specifically through incorrect indices. It is not used for the primary score and does not repair predictions.

3.3. Models and Fine-Tuning

We evaluate Qwen2.5-3B-Instruct [29] and Llama-3.2-3B-Instruct [30], two instruction-tuned model families at the same 3-billion-parameter scale. We selected this compact scale to hold the model size constant across families and to make the complete 72-adapter comparison with three training seeds computationally feasible. This is also a scientifically relevant setting because previous work indicates that format sensitivity depends on model capacity: little evidence of format effects has been found in most tested scenarios involving frontier models, whereas greater degradation has been observed under lower-capacity conditions [4,5]. Our design therefore evaluates format reliability in compact models rather than the effect of model scaling; whether the findings extend to larger models remains an open question. The native context limits are 32,768 and 131,072 tokens, respectively; training uses a common maximum sequence length of 8192 tokens. Exact model snapshots, tokeniser and chat-template checksums are provided in the versioned public repository.

Each model is fine-tuned using quantised low-rank adaptation (QLoRA) [31,32]. All primary task–model–format combinations use the same five-epoch, completion-only training procedure with three seeds (13, 17, and 19). The LoRA rank is 64, and the effective batch size is eight. Training and inference used NVIDIA A100-SXM4-40GB GPUs (NVIDIA Corporation, Santa Clara, CA, USA). The Supplementary Material S1 summarises the remaining optimisation, quantisation, and prompt-template settings; the versioned public repository provides their machine-readable definitions and environment records. Targets differ substantially in length, so identical numbers of examples and epochs do not imply equal token exposure or computation. Exact target-token and runtime summaries are reported in the Supplementary Data S2. The external CARMEN-I evaluation reuses the previously trained primary and diagnostic adapters without updating their weights. Results from these two adapter matrices remain separate because they were trained on 675 and 600 documents, respectively. The prompted comparison instead loads the same two instruction-tuned checkpoints without adapters or additional weight updates. It therefore compares adaptation regimes only for these compact model snapshots, rather than representing prompting by larger general-purpose models.

3.4. Inference

Inference uses vLLM (version 0.9.2) [33] with the corresponding adapter and the same chat template used during fine-tuning. Decoding is greedy, and the generation limits are those derived from development targets. The primary configuration permits up to 32 concurrent sequences. To assess whether concurrency changes the conclusions, the official test predictions are also generated with one sequence at a time. Each run records output validity, parsing and grounding outcomes, generation length, latency, and the software and hardware environment. To check whether the main pattern depends on prompt wording, we also train one seed for each model and format using two alternative Spanish phrasings of the DisTEMIST instruction and evaluate them on the development set. The entity definition and output structure are unchanged.

For zero-shot inference, the unadapted checkpoint receives the exact primary task-and-format prompt used as the fine-tuned system’s final user query, with no system message or labelled example. For three-shot inference, three user/assistant exchanges are prepended: each user turn applies that same prompt to a selected training document, and the assistant turn contains its deterministic gold target in the evaluated format. The final test query is unchanged. Both regimes use the tokeniser-provided chat template, greedy decoding, the concurrent serving configuration, and the same development-derived task–model–format generation limit as the fine-tuned condition. We measured prompt lengths with each model’s tokeniser and confirmed that every prompt, together with its generation limit, fit within the configured context window.

CARMEN-I inference uses the same model snapshots, adapters, chat templates, greedy decoding, format-specific prompts, and generation limits as the corresponding original condition, with the concurrent serving configuration only. We measured every complete CARMEN-I prompt with each model’s tokeniser and confirmed that the prompt, together with its generation limit, fit within the model’s context window. Each released section is processed as one example; source-bearing prompts and generations remain in protected storage.

3.5. Evaluation

Evaluation is separated into four stages. First, an output is syntactically valid when its tags are balanced, its JSON is well formed, or each non-empty TSV line contains two fields. Second, a format-specific parser checks the expected fields, types, and task label and extracts candidate entities. Third, candidates are grounded to the source using tag positions, exact offsets, or count-matched string search, as defined above. Finally, grounded candidates are compared with the gold annotations using the corresponding task-specific official scorer. A strict true positive requires an exact match of start offset, end offset, and label. At the document level, we record whether the output is syntactically valid, whether the complete output is parsed, and whether an imperfect output still yields accepted candidates. This keeps valid empty outputs distinct from wholly rejected outputs.

Primary scoring applies no fuzzy matching, format repair, or generation retry. Candidates extracted from an imperfect document continue through grounding and scoring, whereas malformed items are rejected. Each accepted candidate follows one mutually exclusive route: source or position failure, duplicate, strict true positive, boundary or occurrence error, or grounded spurious prediction. Unmatched gold entities are omissions. Repeated parsed rows are retained because identical strings may refer to different source occurrences, while duplicate grounded span–label predictions are removed before official scoring. The Supplementary Material S1 provides the operational evaluation definitions and states whether each percentage is calculated over all document outputs, parser-accepted entity candidates, or gold entities. Complete routing counts are provided in the Supplementary Data S2.

Strict metrics across all documents remain primary. One-to-one exact-or-overlap scores serve only as boundary diagnostics. First-occurrence and all-occurrences grounding for mJSON and TSV are reported only as sensitivity analyses because they deliberately alter how an emitted string is assigned to the source. First-occurrence matching can miss repeated entities, whereas all-occurrences matching can assign a mention to locations that the model did not intend.

Because CARMEN-I was released as an annotated corpus rather than as a shared-task benchmark, we calculate its precision, recall, and F₁ with a deterministic strict-span evaluator. It requires exact agreement in start offset, end offset, and task label, matching the criterion used by the task-specific scorers for the original test sets. The same syntax, parsing, grounding, and mutually exclusive candidate-routing definitions are used. No fuzzy matching, normalisation, output repair, or retry is used.

3.6. Statistical Analysis

The statistical analysis asks whether differences between conditions are larger than the variation associated with the evaluated documents and training seeds. Because each condition is evaluated on the same documents, comparisons are paired by document. Within each seed, true-positive, false-positive, and false-negative counts are summed across the 250 official test documents before micro-F₁ is calculated. We obtain 95% confidence intervals for differences between conditions from 2000 paired bootstrap samples of the documents [34] and average each difference across the three seeds.

Formal hypothesis tests are limited to two planned contrasts for each annotation layer and model: mJSON versus TSV, which compares two serialisations carrying the same information and using the same grounding rule, and oJSON versus iXML, which compares the two position-carrying pipelines. We test both contrasts for each of the three annotation layers and two models, giving $3\times 2\times 2=12$ tests. Each test uses 10,000 paired randomisations. Because testing several contrasts increases the chance of a false positive, p-values are adjusted with the Holm procedure [35] to control that probability across the family of 12 tests. Other format contrasts are descriptive. Before examining the test results, we selected an absolute F₁ difference of 0.02—two points on the 0–1 F₁ scale—as the smallest effect of practical interest for this engineering comparison. This pragmatic threshold is used to interpret effect sizes and confidence intervals; it is neither a test of statistical significance nor a clinically validated criterion. A second bootstrap resamples both documents and training seeds as a sensitivity check, although three seeds provide only a limited estimate of training variation.

The prompted comparison is descriptive and does not enter the 12-test Holm family. Zero-shot provides one deterministic estimate per task–model–format combination because no training or example-selection seed is present. Three-shot repeats each combination with three fixed example sets; results are reported for each set and as their mean, standard deviation, and range. For comparison with supervised fine-tuning, paired bootstrap samples of the 250 test documents contrast the zero-shot estimate or the mean over three example sets with the mean over three fine-tuning seeds. Fine-tuning seeds describe optimisation variability, whereas the three example-selection seeds describe sensitivity to example choice; they are not treated as interchangeable replicates.

The diagnostic position–reference comparison is analysed separately on the 75 development and 75 confirmation documents using the same three-seed aggregation and paired bootstrap. The original and fully specified offset instructions are considered practically equivalent only when the complete 95% confidence interval for their difference lies within $[-0.02,+0.02]$; intervals crossing either boundary are inconclusive. The official test comparisons and statistical analyses were defined before the test results were examined. We applied the same analysis procedure to the 75-document confirmation partition. As described in Section 3.1, the MedProcNER and SympTEMIST occurrence-reference comparisons were added after the DisTEMIST result was examined and are therefore interpreted descriptively.

The external CARMEN-I evaluation is descriptive and remains separate from the original hypothesis-test family. For each task, model, and condition, we report strict micro-precision, recall, and F₁ by seed and as the mean and standard deviation over three seeds. Selected within-matrix contrasts use 2000 paired document-bootstrap samples. The primary and diagnostic adapter matrices are reported in separate column groups; they are not pooled or compared as though their training data were controlled.

3.7. Reproducibility

The study materials retain the random seeds, training-example identifiers and order, exact model snapshots, tokeniser and chat-template checksums, prompts, package versions, GPU model, dataset and canonical-file checksums, scorer revisions, and analysis settings. The Supplementary Material S1 contains the complete prompt specifications, output examples, executed settings, and evaluation definitions; the accompanying Supplementary Data S2 provides detailed source-free results. Machine-readable configurations, environment records, and all reproduction code are in the versioned public repository. The public task releases are available from their version-specific records, linked in the Data Availability Statement.

4. Results

Unless stated otherwise, results refer to concurrent inference and are averaged over three training seeds.

4.1. Target-Conversion Check

Converting gold annotations to each target format and parsing them back recovered all gold mentions in every training and development split. Exact positions were also fully recovered for iXML and oJSON. For mJSON and TSV, count-matched string search recovered 94.9–96.7% of gold spans. This range is the maximum strict-span score attainable by these two pipelines under the specified grounding rule, even if a model emits every correct mention string. The only pre-model loss found by this check was ambiguity when a string appeared at several source positions but the output did not identify the annotated occurrence.

4.2. Strict-Span Performance

iXML, mJSON, and TSV produced similar point estimates across tasks and models (

Table 5. Strict-span precision, recall, and F₁ by format, task, and model (mean ± SD over three seeds) under concurrent vLLM serving. Bold F₁ marks the oJSON member of each Holm-significant oJSON–iXML contrast and the TSV member of the significant SympTEMIST/Qwen TSV–mJSON contrast (Figure 2). ^† oJSON precision uses fewer than 10 grounded spans per run (grounding <0.003) and is not meaningfully interpretable. Published-best rows show the top official shared-task submissions for context.

Task	Model	Format	Precision	Recall	F1
DisTEMIST	Qwen	iXML	0.772 ± 0.008	0.663 ± 0.023	0.713 ± 0.010
		mJSON	0.713 ± 0.006	0.685 ± 0.009	0.698 ± 0.007
		TSV	0.711 ± 0.006	0.703 ± 0.002	0.707 ± 0.003
		oJSON	0.804  † ± 0.120	0.002 ± 0.001	0.004 ± 0.002
	Llama	iXML	0.763 ± 0.003	0.634 ± 0.019	0.692 ± 0.011
		mJSON	0.728 ± 0.001	0.676 ± 0.008	0.701 ± 0.004
		TSV	0.716 ± 0.007	0.681 ± 0.004	0.698 ± 0.003
		oJSON	0.958 † ± 0.072	0.002 ± 0.000	0.005 ± 0.001
	Published best [25]	Reference	0.792	0.763	0.777
MedProcNER	Qwen	iXML	0.805 ± 0.007	0.671 ± 0.017	0.732 ± 0.007
		mJSON	0.741 ± 0.004	0.721 ± 0.005	0.731 ± 0.002
		TSV	0.732 ± 0.004	0.740 ± 0.002	0.736 ± 0.003
		oJSON	0.600 † ± 0.529	0.001 ± 0.001	0.002 ± 0.002
	Llama	iXML	0.770 ± 0.005	0.678 ± 0.009	0.721 ± 0.004
		mJSON	0.738 ± 0.006	0.713 ± 0.007	0.725 ± 0.004
		TSV	0.733 ± 0.008	0.719 ± 0.005	0.726 ± 0.005
		oJSON	0.822 † ± 0.168	0.001 ± 0.000	0.002 ± 0.001
	Published best [26]	Reference	0.810	0.788	0.799
SympTEMIST	Qwen	iXML	0.722 ± 0.011	0.581 ± 0.020	0.644 ± 0.016
		mJSON	0.632 ± 0.005	0.642 ± 0.006	0.637 ± 0.003
		TSV	0.636 ± 0.004	0.666 ± 0.011	0.651± 0.007
		oJSON	0.750 † ± 0.250	0.001 ± 0.000	0.001 ± 0.001
	Llama	iXML	0.707 ± 0.003	0.614 ± 0.015	0.657 ± 0.008
		mJSON	0.650 ± 0.004	0.647 ± 0.005	0.649 ± 0.004
		TSV	0.656 ± 0.009	0.651 ± 0.009	0.654 ± 0.003
		oJSON	1.000 † ± 0.000	0.001 ± 0.001	0.002 ± 0.001
	Published best [27]	Reference	0.804	0.699	0.748

). Their F₁ scores were 0.692–0.713 for DisTEMIST, 0.721–0.736 for MedProcNER, and 0.637–0.657 for SympTEMIST. We use the 0.02 practical threshold defined before the test results were examined to distinguish small estimated effects from differences of practical interest. None of the observed differences among these three pipelines reached that threshold, although the confidence intervals reported below do not establish practical equivalence.

Paired confidence intervals quantify uncertainty in differences evaluated on the same test documents (Figure 2). mJSON and TSV provide the closest comparison of serialisation because they emit the same entity information and use the same grounding procedure. Their F₁ differences were close to zero in five of the six task–model combinations. For SympTEMIST with Qwen2.5-3B-Instruct, TSV was 0.014 higher than mJSON (95% CI [0.004, 0.023]; Holm-adjusted $p=0.039$). This result supports a positive difference in that combination, although the estimated effect is smaller than the 0.02 practical threshold and its confidence interval includes values on both sides of that threshold.

Comparisons of mJSON and TSV with iXML describe differences between complete pipelines rather than serialisation alone, and were not included in the family of formal hypothesis tests. Every mJSON–iXML and TSV–iXML interval included zero, so no difference was established in those comparisons. Some intervals nevertheless extended beyond $-0.02$ or $+0.02$; consequently, the data are not precise enough to conclude that the three pipelines are practically equivalent.

Direct-offset JSON performed markedly worse, with F₁ between 0.001 and 0.005 in all six task–model combinations. Its differences from iXML ranged from $-0.730$ to $-0.643$; every confidence interval excluded zero, and every Holm-adjusted randomisation test was significant (adjusted $p=0.0012$). The apparently high oJSON precision values in Table 5 are based on fewer than ten grounded spans per run. With such small denominators, these estimates are unstable and should not be interpreted as evidence that oJSON produces precise predictions; its recall below 0.003 and F₁ of 0.001–0.005 describe its effective performance more meaningfully. The diagnostic analysis below locates the immediate failure during source grounding: 99.8–99.9% of parser-accepted oJSON candidates contained an invalid or source-inconsistent position. Because mJSON also uses JSON, syntax alone cannot explain the result. Section 4.4 examines whether the failure reflects unclear offset instructions, missing entity text, or difficulty generating absolute character positions.

The widest range among the three seed-specific F₁ values in any primary condition was 0.030. Resampling documents and seeds did not alter the main pattern: the six oJSON–iXML intervals remained far below zero, whereas mJSON–TSV differences remained small. Per-seed results and all pairwise intervals are reported in the Supplementary Data S2.

For context, the highest F₁ values in this study were 0.713, 0.736, and 0.657 for DisTEMIST, MedProcNER, and SympTEMIST, respectively. The best published shared-task scores were 0.777, 0.799, and 0.748 [25,26,27]. These values are not controlled baselines because the systems used different models, resources, and training procedures.

4.3. Syntax, Parsing, and Grounding

We examine the pipeline in three steps: can the output be read, can a candidate entity be located in the source, and does that grounded span match the gold annotation? All four formats usually passed the first step. Syntax-valid rates were at least 97.5% for iXML, 98.4% for mJSON, 99.2% for TSV, and 96.4% for oJSON, with similarly high complete-parser acceptance (

Table 6. Output-format reliability diagnostics pooled over three seeds under concurrent serving. Values are percentages. Syntax valid and fully parsed are calculated over all documents. Grounded, strict true positive (TP), and boundary error are calculated over parser-accepted candidates.

Task	Model	Format	Syntax Valid	Fully Parsed	Grounded	Strict TP	Boundary Error
DisTEMIST	Qwen	iXML	99.2	98.9	88.7	68.4	12.4
		mJSON	98.9	98.9	98.7	70.4	12.5
		TSV	99.6	99.6	98.6	70.0	12.3
		oJSON	99.1	99.1	0.2	0.2	0.0
	Llama	iXML	98.8	98.8	86.0	65.6	12.5
		mJSON	99.3	99.3	98.9	71.9	13.0
		TSV	99.2	98.9	97.3	69.7	13.0
		oJSON	96.4	96.4	0.2	0.2	0.0
MedProcNER	Qwen	iXML	98.9	98.9	86.2	69.3	11.4
		mJSON	99.1	99.1	99.1	73.4	13.3
		TSV	99.7	99.6	99.1	72.5	13.2
		oJSON	99.2	99.2	0.1	0.1	0.0
	Llama	iXML	97.5	97.5	89.7	69.0	13.4
		mJSON	99.1	99.1	99.2	73.2	13.8
		TSV	99.9	99.6	99.0	72.5	14.0
		oJSON	98.7	98.4	0.1	0.1	0.0
SympTEMIST	Qwen	iXML	99.6	99.6	86.8	62.7	13.2
		mJSON	99.1	99.1	98.3	62.0	14.9
		TSV	99.6	99.5	96.6	61.3	13.9
		oJSON	99.3	99.3	0.1	0.1	0.0
	Llama	iXML	98.0	98.0	89.6	63.3	14.3
		mJSON	98.4	98.4	99.2	64.4	14.9
		TSV	99.7	99.6	94.6	61.9	14.0
		oJSON	99.6	99.6	0.1	0.1	0.0

). Structural validity therefore explains little of the performance difference. Panel (a) of Figure 3 pools these document-level checks across tasks, models, and seeds. Syntax, schema acceptance, and full parsing are successive checks, while partial acceptance and absence of candidates are additional flags; they are not five outcomes that should sum to 100%.

The main separation occurred during source grounding. The proportion of accepted candidates that could not be located was 10.3–14.0% for iXML, 0.8–1.7% for mJSON, 0.9–5.4% for TSV, and 99.8–99.9% for oJSON. iXML failures mainly occurred when the generated document copy differed from the source. mJSON and TSV usually found the emitted strings, although boundary and occurrence errors remained. In oJSON, almost every parser-accepted candidate contained an invalid or source-inconsistent numerical reference. Thus, valid JSON did not imply a usable entity span. Complete candidate routes, duplicate handling, relaxed boundary diagnostics, and task–model–seed strata are reported in the Supplementary Data S2.

Completion truncation did not explain the direct-offset result: only 94 of 18,000 primary completions (0.52%) reached their generation limit, and none approached the model context limit. The formats nevertheless differed in output length and supervised token exposure, particularly because iXML reproduces the source. The comparison therefore evaluates complete practical pipelines rather than equal-compute conditions. Detailed length, token-exposure, and runtime summaries are reported in the Supplementary Data S2.

4.4. Why Did Direct Character Offsets Fail?

Direct-offset JSON (oJSON) was usually readable and contained the expected fields, yet almost none of its predicted spans could be accepted. We tested three explanations: whether the counting instructions were ambiguous, whether the models failed to recognise the entity text, and whether absolute character counting was the main obstacle.

First, we tested whether the counting instructions were ambiguous. The fully specified instruction tells the model to count Unicode characters from zero, with an inclusive start and an exclusive end. Mean strict-span F₁ nevertheless remained between 0 and 0.0077 across annotation layers and models (

Table 7. Confirmation results for direct-offset prompting, text-based grounding, and occurrence-index JSON. Values are mean strict-span F₁ over three training seeds on 75 documents. $\Delta$ is explicit minus original direct-offset F₁; brackets give the paired document-bootstrap 95% confidence interval. Text grounding ignores generated indices and count-matches the exact emitted mention. The MedProcNER and SympTEMIST occurrence-reference results are descriptive comparisons, as explained in Section 3.1.

Task	Model	Original Offset	Explicit Offset	Occurrence Index	Δ Offset 95% CI	Text Grounding Original	Text Grounding Explicit
DisTEMIST	Qwen	0.0009	0.0077	0.7051	0.0069 [0.0009, 0.0147]	0.6818	0.6810
DisTEMIST	Llama	0.0052	0.0026	0.7097	−0.0026 [−0.0091, 0.0035]	0.6642	0.6637
MedProcNER	Qwen	0.0026	0.0040	0.7400	0.0013 [−0.0027, 0.0063]	0.7008	0.6888
MedProcNER	Llama	0.0000	0.0000	0.7118	0.0000 [0.0000, 0.0000]	0.6876	0.6865
SympTEMIST	Qwen	0.0007	0.0000	0.6520	−0.0007 [−0.0023, 0.0000]	0.6136	0.6185
SympTEMIST	Llama	0.0000	0.0000	0.6622	0.0000 [0.0000, 0.0000]	0.6110	0.6034

). Differences from the original instruction ranged from $-0.0026$ to 0.0069, and every 95% confidence interval lay within the predefined $\pm 0.02$ range. Clearer counting instructions therefore did not materially improve direct-offset performance.

Next, we discarded the generated start and end numbers and located each emitted mention by exact count-matched search. F₁ increased to 0.603–0.701. Fewer than 0.8% of parsed strings were absent from the source, whereas more than 99% had an invalid or source-inconsistent position (

Table 8. Candidate-error composition on the confirmation partition for direct-offset and occurrence-index conditions. Counts aggregate 75 confirmation documents, two models, and three seeds. Candidate percentages use parsed candidates; omission uses gold annotations. Boundary combines left-, right-, and both-boundary errors.

Task	Condition	Parsed	Absent	Invalid Reference	Strict TP	Boundary	Wrong Occurrence	Spurious	Omission
DisTEMIST	Original offset	4804	0.5%	99.3%	0.1%	<0.1%	<0.1%	0.0%	99.8%
DisTEMIST	Explicit offset	4797	0.6%	99.1%	0.3%	<0.1%	0.0%	<0.1%	99.7%
DisTEMIST	Occurrence index	4703	0.8%	0.6%	69.5%	13.5%	1.7%	13.9%	15.8%
MedProcNER	Original offset	6089	0.7%	99.2%	0.1%	<0.1%	0.0%	0.0%	99.9%
MedProcNER	Explicit offset	6118	0.5%	99.3%	0.1%	<0.1%	0.0%	0.0%	99.9%
MedProcNER	Occurrence index	6082	0.4%	0.8%	71.9%	15.8%	1.1%	9.9%	12.3%
SympTEMIST	Original offset	5940	0.8%	99.1%	<0.1%	<0.1%	0.0%	0.0%	100.0%
SympTEMIST	Explicit offset	5872	0.8%	99.2%	0.0%	0.0%	0.0%	0.0%	100.0%
SympTEMIST	Occurrence index	5922	0.8%	0.7%	62.7%	15.0%	0.7%	19.9%	16.7%

). The errors were therefore not primarily missing entity phrases or simple one-character boundary shifts: the generated numbers usually selected different source characters.

Finally, we replaced absolute character counts with occurrence numbers. Occurrence-index JSON asks the model to identify, for example, the second appearance of a mention rather than its absolute character position. It achieved confirmation F₁ of 0.705–0.710 for DisTEMIST, 0.712–0.740 for MedProcNER, and 0.652–0.662 for SympTEMIST. Improvements over the matched original-offset conditions ranged from 0.651 to 0.737, with every paired confidence interval excluding zero. Development results showed the same contrast, with occurrence-index F₁ of 0.660–0.747. Candidate-level results show that the remaining occurrence-reference errors were mainly boundary or spurious predictions rather than invalid positions.

Together, these analyses show that the models often recognised the entity phrase but could not reliably translate it into absolute start and end positions under the tested conditions. Retaining the predicted mention text while ignoring or replacing absolute character indices recovered most of the lost performance, whereas clarifying the offset convention did not. Selecting an occurrence of the phrase was substantially more reliable than counting characters.

4.5. Robustness Checks

We examined whether the main findings changed under alternative grounding rules, prompt wording, or inference configuration, and whether they were concentrated in particular types of documents.

Changing how repeated mentions were grounded affected mJSON and TSV. Always selecting the first source occurrence reduced F₁ by 0.023–0.034 relative to count-matched grounding. Selecting every occurrence increased F₁ by 0.001–0.007. Neither alternative changed the broad ordering of formats. These results show that the grounding policy matters for repeated mentions and that count-matched search occupies a conservative middle position.

To test prompt wording, two Spanish paraphrases of the DisTEMIST instruction were evaluated on the development set for both models, all formats, and one seed. Across the three wordings, iXML, mJSON, and TSV achieved F₁ of 0.698–0.753, whereas oJSON remained at 0.002–0.007. The ordering of the first three formats varied, but the direct-offset result did not.

Changing the inference configuration also had little effect on aggregate scores. Serial and concurrent inference differed by at most 0.0025 in seed-averaged strict F₁ across the 24 primary conditions (mean absolute difference 0.0008). Although only 71.5% of raw generations were identical, 92.8% of grounded spans and 93.8% of document-level strict scoring counts matched. Many raw differences, particularly alternative invalid oJSON offsets, therefore disappeared during grounding. These data support the stability of the aggregate conclusions under the tested serving settings, not byte-identical generation. Complete results are provided in the Supplementary Data S2.

Finally, post hoc descriptive stratification by document-length and entity-density quartiles, repeated source mentions, and flat versus nested or crossing annotations did not reverse the principal contrast: direct-offset F₁ remained at most 0.012 in these strata. Generation-limit strata contained only 7–41 document–run outputs per format and are reported without inferential claims. Full strict and one-to-one relaxed results for all strata, together with task–model summaries, are provided in the Supplementary Data S2. None of these checks rescued direct offsets; grounding-rule changes nevertheless shifted mJSON and TSV scores.

4.6. Zero- and Three-Shot Prompting

The 96 prompted experiments used the same official test documents, final task-and-format queries, generation limits, parsers, grounding rules, and scorers as the corresponding fine-tuned conditions. Zero-shot inference produced one deterministic estimate per task–model–format combination; three-shot results were averaged over three fixed sets of training examples (

Table 9. Official-test strict-span F₁ under supervised fine-tuning, zero-shot prompting, and three-shot prompting. Fine-tuned values are means (SD) over training seeds 13, 17, and 19. Three-shot values are means (SD) over three fixed sets, each containing three training examples selected with seeds 13, 17, and 23. Zero-shot uses no selected examples and is therefore one deterministic estimate without an SD. Training seeds and example-selection seeds represent different sources of variation.

Task	Model	Pipeline	Fine-Tuned	Zero-Shot	Three-Shot
DisTEMIST	Qwen	iXML	0.713 (0.010)	0.000	0.028 (0.012)
		mJSON	0.698 (0.007)	0.118	0.165 (0.072)
		TSV	0.707 (0.003)	0.006	0.203 (0.040)
		oJSON	0.004 (0.002)	0.000	0.000 (0.000)
	Llama	iXML	0.692 (0.011)	0.000	0.000 (0.000)
		mJSON	0.701 (0.004)	0.000	0.326 (0.001)
		TSV	0.698 (0.003)	0.000	0.352 (0.003)
		oJSON	0.005 (0.001)	0.000	0.000 (0.000)
MedProcNER	Qwen	iXML	0.732 (0.007)	0.000	0.009 (0.004)
		mJSON	0.731 (0.002)	0.099	0.230 (0.023)
		TSV	0.736 (0.003)	0.002	0.262 (0.031)
		oJSON	0.002 (0.002)	0.000	0.000 (0.000)
	Llama	iXML	0.721 (0.004)	0.000	0.000 (0.000)
		mJSON	0.725 (0.004)	0.000	0.348 (0.020)
		TSV	0.726 (0.005)	0.000	0.386 (0.019)
		oJSON	0.002 (0.001)	0.000	0.000 (0.000)
SympTEMIST	Qwen	iXML	0.644 (0.016)	0.000	0.011 (0.007)
		mJSON	0.637 (0.003)	0.131	0.189 (0.017)
		TSV	0.651 (0.007)	0.024	0.245 (0.016)
		oJSON	0.001 (0.001)	0.000	0.000 (0.000)
	Llama	iXML	0.657 (0.008)	0.000	0.000 (0.000)
		mJSON	0.649 (0.004)	0.000	0.241 (0.014)
		TSV	0.654 (0.003)	0.000	0.271 (0.003)
		oJSON	0.002 (0.001)	0.000	0.000 (0.000)

).

In zero-shot inference, strict F₁ was zero in all 12 Llama experiments. Qwen zero-shot mJSON achieved 0.099–0.131 across tasks, and zero-shot TSV achieved 0.002–0.024; Qwen iXML and oJSON remained at zero. Providing three examples raised mJSON to 0.165–0.348 and TSV to 0.203–0.386. The corresponding iXML values were 0–0.028, and every prompted oJSON condition remained at zero. Variation across example sets was small for most Llama experiments but larger for Qwen mJSON and TSV; for example, DisTEMIST Qwen mJSON ranged from 0.093 to 0.238.

The descriptive paired comparisons used the same bootstrap document samples while averaging fine-tuned systems over training seeds and three-shot systems over the three example sets. Every interval for prompted minus fine-tuned F₁ was below zero. For iXML, mJSON, and TSV, three-shot mean F₁ was 0.340–0.723 lower than the corresponding fine-tuned mean. Direct offsets also favoured fine-tuning, but both regimes were effectively non-functional: fine-tuned F₁ was 0.001–0.005 and prompted F₁ was zero.

Providing examples changed where outputs failed. In zero-shot Llama inference, no mJSON or TSV document passed the declared syntax, whereas three-shot complete-parser acceptance was 0.772–0.941 for those pipelines. Three-shot iXML was also often parseable, but its candidate grounding rate was only 0–0.142; oJSON grounding remained at or below 0.0001 despite complete-parser acceptance of 0.501–0.835. Generation limits were reached by 956 of 6000 zero-shot outputs (15.9%) and 2096 of 18,000 three-shot outputs (11.6%). The prompted comparison therefore shows that three-shot prompting improved schema adherence and mention-list extraction, but did not reproduce the fine-tuned models’ source-copy or positional behaviour under the fixed prompts and budgets.

4.7. External Validation on CARMEN-I

All 126 adapter conditions produced outputs for each of the 458 CARMEN-I sections. Every complete prompt plus its unchanged generation allowance fitted the corresponding context window. Across individual task–model–condition runs, 0.2–10.0% of completions reached the configured generation limit; no retry, repair, segmentation, or budget adjustment was applied. The primary and position–reference matrices are reported separately in

Table 10. Strict-span performance on the external CARMEN-I subset. Values are mean F₁ (SD) over three training seeds. The first four columns report primary adapters trained on 675 documents; the final three report separately trained diagnostic adapters using 600 documents. The two column groups are not a controlled seven-condition comparison.

Task	Model	Primary Adapters (675 Training Documents)				Diagnostic Adapters (600 Training Documents)
		iXML	mJSON	TSV	oJSON	Original Offset	Explicit Offset	Occurrence
DisTEMIST	Qwen	0.147 (0.013)	0.610 (0.004)	0.659 (0.007)	0.001 (0.001)	0.002 (0.001)	0.002 (0.001)	0.610 (0.009)
	Llama	0.135 (0.014)	0.618 (0.011)	0.653 (0.003)	0.001 (0.001)	0.001 (0.000)	0.001 (0.000)	0.610 (0.007)
MedProcNER	Qwen	0.177 (0.009)	0.652 (0.004)	0.654 (0.009)	0.002 (0.000)	0.001 (0.001)	0.001 (0.001)	0.652 (0.014)
	Llama	0.157 (0.008)	0.639 (0.000)	0.642 (0.002)	0.002 (0.001)	0.002 (0.000)	0.002 (0.000)	0.639 (0.002)
SympTEMIST	Qwen	0.160 (0.003)	0.633 (0.015)	0.646 (0.004)	0.001 (0.001)	0.001 (0.001)	0.001 (0.000)	0.627 (0.010)
	Llama	0.113 (0.007)	0.631 (0.009)	0.636 (0.003)	0.002 (0.002)	0.001 (0.001)	0.001 (0.000)	0.613 (0.005)

because their adapters were trained on different partitions.

In the primary matrix, the two string-grounded pipelines retained substantially more strict-span performance than the position-carrying outputs. Mean F₁ was 0.610–0.653 for mJSON and 0.636–0.659 for TSV. iXML instead achieved 0.113–0.177, and oJSON remained at 0.001–0.002. The paired difference between mJSON and TSV (mJSON minus TSV) excluded zero for both DisTEMIST models (differences of $-0.049$ and $-0.035$), favouring TSV; the other four intervals included zero. All six comparisons of oJSON with iXML (oJSON minus iXML) were below zero, with differences from $-0.175$ to $-0.112$. These CARMEN-I comparisons are descriptive and do not enter the original Holm-adjusted hypothesis family.

Parsing success was similar across formats, but grounding failures differed sharply. Pooled over tasks, models, and seeds, 96.1–97.5% of outputs passed complete parsing across the four primary formats. However, 86.9% of accepted iXML candidates had an unusable positional reference, chiefly because the generated source copy did not reconstruct the CARMEN-I section exactly; the corresponding proportion was 97.9% for oJSON. For mJSON and TSV, only 2.6% and 5.9% of candidates, respectively, failed source grounding. Thus, the CARMEN-I data preserved the direct-offset grounding failure but did not preserve the official-test similarity between iXML and the string-grounded pipelines.

The separately trained position–reference adapters showed the same distinction more directly. Occurrence-index JSON achieved a mean F₁ of 0.610–0.652, whereas both the original direct-offset condition and the condition with fully specified coordinates remained at or below 0.002. The F₁ differences between occurrence indexing and original offsets ranged from 0.608 to 0.650, and all six paired 95% bootstrap intervals excluded zero. Intervals comparing the explicit and original offset instructions included zero in five task–model comparisons; the remaining SympTEMIST–Llama interval was slightly negative. Hence, changing the institution and document setting did not rescue absolute character generation, while references to exact-string occurrences retained most of the string-grounded performance.

5. Discussion

5.1. Principal Findings

On the official shared-task test sets, iXML, mJSON, and TSV achieved similar practical performance across annotation layers and models, while direct-offset JSON failed under strict-span scoring despite usually producing valid JSON. None of the observed differences among the first three pipelines reached the predefined two-point threshold, although the uncertainty intervals are too wide to establish equivalence. The contrast is therefore not between JSON and non-JSON output. It concerns how each complete pipeline represents and recovers an entity’s source position. This similarity depended on supervised adaptation: with the unadapted checkpoints, zero-shot F₁ was at most 0.131 and three-shot F₁ at most 0.386.

Separating parsing, grounding, and scoring locates the main loss. Most oJSON outputs had valid syntax and the expected fields, but almost none of their numerical positions selected the emitted mention in the source. iXML instead failed mainly when the generated copy differed from the original document, while mJSON and TSV grounded most mention strings through exact search. Structural compliance and reliable source localisation are thus different properties: reporting only whether an output is valid conceals the principal failure observed here.

The diagnostic comparison further separates recognition from position generation. Telling the models explicitly to count Unicode characters from zero, with an inclusive start and exclusive end, did not materially improve direct-offset performance. The generated mention strings nevertheless recovered F₁ of 0.603–0.701 when their numbers were ignored, and occurrence references achieved 0.652–0.740. Under the tested conditions, the models often recognised the relevant phrase and could select its relative occurrence, but did not reliably translate that choice into absolute character indices. The data establish this immediate failure route; difficulty with latent numerical indexing is an interpretation rather than a directly observed model mechanism, and offsets may behave differently in larger models or when positions are visible in the input.

The prompted comparison shows that representation reliability also depends on the adaptation regime. Providing three examples substantially improved mJSON and TSV over zero-shot inference, especially for Llama, by teaching the expected answer structure and supplying task examples. They did not make the compact base models copy and tag the source reliably or generate usable absolute positions. Consequently, the near-parity of iXML, mJSON, and TSV after fine-tuning should not be extrapolated to prompted use of the same checkpoints. Conversely, these results do not show that prompting in general is inferior: the final instruction was held fixed for comparability, only three training-derived example sets were evaluated, and no prompt search, larger model, or constrained decoder was tested. The evidence is therefore a matched comparison for two 3B checkpoints rather than a benchmark of general-purpose prompting strategies.

The external CARMEN-I evaluation both supports and qualifies this account. Absolute offsets again failed at grounding, and occurrence references retained F₁ of 0.610–0.652 without further training. The official-test similarity among iXML, mJSON, and TSV did not transfer: iXML fell to 0.113–0.177 because most accepted candidates did not reconstruct a usable position in the CARMEN-I source, whereas mJSON and TSV remained at 0.610–0.659. Occurrence indexing therefore showed the strongest transfer among the tested position–reference strategies. This evidence comes from one additional hospital and document setting; it establishes neither general robustness nor a model-intrinsic ordering of output formats.

These results extend earlier work on numerical span references to fine-tuned Spanish clinical NER [7,8]. The same broad failure is not confined to Spanish: an English GPT-NER ablation found a large disadvantage for Entity+Position relative to inline copy-and-mark output [6], and prompted Arabic span annotation has produced indices that disagree with otherwise extracted span text [20]. Semin et al. additionally report weak unexposed indexing on a multilingual NER benchmark. Their result is aggregated over 13 languages, so these sources do not establish a language-independent effect size or shared causal mechanism. Character-level manipulation results provide compatible context [21], but do not by themselves explain the oJSON errors here. Under official strict-span scoring, our analysis shows that valid output syntax can conceal an almost complete grounding failure and that examining each processing step distinguishes lost entity recognition from lost localisation. Alternative prompt wording and serial rather than concurrent inference did not explain the direct-offset result, although those checks were deliberately limited in scope.

5.2. Implications for Generative Clinical Extraction

An output format should be considered together with the parser and grounding rule that turn it into source spans. A predicted clinical entity may need to remain linked to the exact passage from which it was extracted for strict evaluation, provenance, annotation review, or later processing. Valid JSON or XML does not provide that link when the reported source position is wrong. The four evaluation stages used here distinguish failures in syntax validation, parsing, source grounding, and strict entity matching, thereby revealing structurally valid but unusable predictions. Generative NER studies should therefore describe the emitted information, handling of repeated mentions, candidate rejection policy, grounding failures, and the population used to calculate each metric. This pipeline-level account is consistent with clinical LLM reporting guidance [24] and with taxonomies of tag-, string-match-, and index-based span reference [8].

Each viable format also presents a practical trade-off. iXML preserves the source position and accommodates overlapping spans, but copying the document makes it long and creates a failure route that became pronounced in the CARMEN-I evaluation. mJSON is readily validated, although repeated strings require a declared grounding policy. TSV carries the same information more compactly, and its similar performance offers no evidence that JSON field names improve recognition in this setting. Direct offsets are convenient for downstream software only when their coordinates are reliable. Occurrence indices provide a useful alternative because they express a positional choice without character arithmetic. Their performance transferred to CARMEN-I, although they still require exact mention strings, and evidence from one additional hospital does not establish wider generality.

The prompted results add a further operational distinction. Examples can improve surface-format adherence without ensuring that the resulting candidates are grounded correctly. In particular, three-shot oJSON was often parseable but still produced no strict-span performance, while three-shot iXML frequently parsed without reconstructing the source. Prompted extraction should therefore be audited through the same staged syntax, parsing, grounding, and scoring sequence rather than evaluated only for whether an answer resembles the requested schema.

The length differences also qualify comparisons of efficiency. Equal examples, epochs, and optimisation steps do not yield equal token exposure when one format copies the source. The present results therefore compare complete practical pipelines, not equal-compute representations. Likewise, the choice of string-grounding rule affects the score: first-occurrence matching misses repeated entities, whereas all-occurrences matching can introduce unsupported spans. Count-matched grounding is conservative, but cannot resolve every annotated occurrence from the mention text alone.

Span-reference design is therefore a reliability concern for clinical extraction pipelines, not merely an output-format preference. When exact source provenance is required, generated entities should be verified against the intended source spans before they are used for annotation review, normalisation, or downstream analysis. Structural validity alone is insufficient for that purpose.

Grammar-constrained decoding could prevent malformed JSON or enforce a declared field schema [17,18], but a grammar that permits arbitrary integers cannot ensure that those integers select the emitted mention. Input-dependent constraints can instead limit values to the current source, and source-conditioned decoding can require a generated mention to be an input span. These are promising alternatives to post-generation rejection, but they define different systems from the unconstrained, no-repair pipelines evaluated here.

The practical recommendation is therefore conditional on the downstream requirement. When exact source provenance is required, a clinical extraction or retrieval-augmented pipeline should validate every generated entity against the source and retain only a verified span reference before later processing. Syntax, parsing, grounding, and task correctness should be logged separately so that a structurally valid but ungrounded entity is not treated as supported evidence. Workflows that tolerate approximate boundaries may reasonably choose a different metric or review policy; this study did not evaluate retrieval quality, clinician utility, or deployment outcomes.

5.3. Limitations

The study covers two 3-billion-parameter model families, one fine-tuning configuration, and three single-entity tasks drawn from the same Spanish clinical document collection. The three annotation layers are not independent corpora, so consistency across them is not external replication. We followed the official definition and scorer for each task and trained each annotation layer separately, which keeps the entity type fixed when the output pipelines are compared. Simultaneous multi-type extraction was not evaluated because it would require a merged target and a joint evaluation protocol, introducing another design factor beyond the representation-and-grounding comparison. The effect sizes may also differ for larger models, other languages, or independently collected clinical text. The shared-task reference scores are contextual rather than controlled baselines.

The original collection consists of clinical case reports rather than complete longitudinal records, full discharge summaries, or imaging reports. Each document was processed as one example, and the 99th percentile was approximately 1660 tokens, so context-window fit in this study is not evidence of reliable extraction from much longer notes. Overlapping annotations were present, but involved a small minority of entities, and the canonical task schema contains only contiguous spans. Performance may therefore differ with longer narratives, longer entity spans, denser overlap, or document-level section dependencies.

Three training seeds provide limited information about optimisation variability. The primary bootstrap intervals mainly describe variation across the 250 test documents conditional on the fitted adapters, and seed resampling cannot estimate training variance precisely. The prompt-wording sensitivity uses only DisTEMIST and one seed. In addition, the formats differ in retained positional information and target length. Only mJSON and TSV isolate serialisation structure closely; other contrasts apply to complete representation-and-grounding pipelines.

The adaptation-regime comparison is likewise narrow. It uses the same two 3B instruction-tuned checkpoints and unchanged final instructions, rather than prompt-optimised or larger general-purpose models. Three fixed example sets provide only a limited view of sensitivity to example selection, and their seeds are not equivalent to fine-tuning seeds. The generation limits were inherited from fine-tuned development targets; 15.9% of zero-shot and 11.6% of three-shot outputs reached a limit. Alternative example counts, selection policies, prompts, budgets, sampling, or constrained decoding could produce different results and require separate evaluation.

Repeated, nested, and crossing mentions create different opportunities for error across formats. String-only outputs cannot always distinguish an annotated occurrence from an identical unannotated one, whereas copied-source tagging must reproduce the document. Occurrence indexing also depends on an exact emitted string: a spelling, whitespace, abbreviation, or normalisation difference can prevent all occurrences from being found. Fuzzy or normalised matching could reduce that failure but would change the evaluated pipeline and can introduce unsupported alignments, so it requires its own validation. The canonical data contain contiguous spans, so the study cannot determine whether discontinuous annotations were lost in earlier corpus preparation.

The 75-document confirmation partition is internal to the same collection and is not an external validation set. It was excluded from all diagnostic-comparison training, but had belonged to the training material used in the earlier primary experiment. Strict no-repair scoring was chosen to measure the reliability of the generated output itself; deterministic correction or alternative generation methods could improve a format but would constitute a different system. Calibration, clinician usefulness, workflow integration, subgroup bias, downstream safety, and deployment readiness were not evaluated.

CARMEN-I broadens the document and institutional setting but remains a single-hospital collection from one period, dominated by released discharge-report sections. The selected disease, procedure, and symptom labels are compatible task endpoints, but differences in annotation guidance, case mix, and document distribution cannot be separated from representation robustness. The generation limits were derived from the original development data; up to 10.0% of completions for one CARMEN-I condition reached a limit, and changing the budgets after observing CARMEN-I would have defined a different evaluation. The CARMEN-I evaluator applies the same strict span-and-label criterion as the official task scorers used for the original test sets. Because the task releases are public, their presence in unknown pretraining mixtures cannot be excluded. Future work should test these span-reference strategies across model scales, languages, joint multi-type extraction settings, entity inventories, and independent clinical collections.

6. Conclusions

This study shows that output representation is part of the evaluated generative NER pipeline rather than a cosmetic choice made after entity recognition.

First, a controlled comparison under the corresponding official task scorers found that inline XML, mention-list JSON, and TSV achieved strict-span F₁ of 0.637–0.736 across two fine-tuned 3-billion-parameter models and three Spanish clinical annotation layers. Direct-offset JSON instead achieved 0.001–0.005. The span-reference pipeline can therefore determine whether recognised entity text becomes an accepted source span.

Second, the matched prompted comparison bounded this result to supervised adaptation. With the unadapted checkpoints, zero-shot F₁ was at most 0.131 and three-shot F₁ at most 0.386. Providing three examples improved mention-list JSON and TSV, but iXML and direct offsets remained near zero. Fine-tuned format behaviour should therefore not be assumed for prompted use of the same compact models.

Third, separating syntax, parsing, source grounding, and entity correctness located the direct-offset failure during localisation: the outputs were usually well formed but could not be grounded. Valid structure alone is therefore insufficient evidence of reliable span extraction.

Fourth, the matched position–reference comparison showed that direct-offset outputs usually retained relevant mention text, while clearer counting instructions did not materially improve their numerical positions. Occurrence references instead achieved F₁ of 0.652–0.740 on the 75-document confirmation partition. Under these conditions and with exact emitted strings, referring to an occurrence was more reliable than asking the model to count characters.

Finally, the external CARMEN-I evaluation, conducted without further training on 458 hospital-record sections, preserved the direct-offset failure and yielded occurrence-reference F₁ of 0.610–0.652. It also exposed a boundary of the official-test result: iXML fell to 0.113–0.177, while the string-grounded pipelines retained 0.610–0.659. The CARMEN-I evidence therefore supports occurrence-based localisation in one additional setting, but does not establish robustness across institutions, languages, or clinical tasks.

Together, these contributions provide a framework for selecting and evaluating output interfaces: report the representation, parser, grounding rule, and treatment of failures, and diagnose each stage before interpreting the final NER score. Further work should test these findings across model scales, languages, entity inventories, additional independent clinical collections, and workflows that evaluate approximate matching or downstream utility rather than strict source spans.