Valid but Not Always Runnable: An Open, Reproducible Benchmark of Large Language Models Drafting Gherkin Scenarios

Abstract

Behaviour-Driven Development (BDD) encodes acceptance criteria in Gherkin, but hand-authoring is laborious, and it is unclear which large language model (LLM) drafts it best. We benchmark eight LLMs generating Gherkin from three requirement corpora (requirement lists, user stories, RFP excerpts) over 2960 generations, scoring validity, runner acceptance, judged coverage and quality, similarity to gold standard, stability, and cost. Validity is near the ceiling, yet only 78% of outputs load in the Cucumber runner: a fifth emits several Feature blocks per file. Two student annotators (a small, non-expert panel) calibrate the judge on 72 blinded generations. Human score levels are matched (error 0.33 versus 0.35 between the humans) but outputs are ordered far less reliably (ICC 0.47 versus 0.64; coverage 0.21 versus 0.76): magnitudes hold, but fine rankings do not. Against that gold standard, models span 66–107% of the human–human ceiling, ordering differently again. Pareto analysis leaves three of eight models non-dominated: cost varies 157×, judged quality 0.36 points. Per-model prompt tuning yields no cross-validated gain; a restrictive token budget truncates verbose models. Two newer models displace the low-cost front: tier-level findings transfer, and model names are dated quickly. We release the corpora, gold standard, and prototype. Model choice should weigh cost and runner acceptance over judged quality.

1. Introduction

Behaviour-Driven Development (BDD) [1,2,3] encodes the expected behaviour of a system as concrete, example-based acceptance scenarios written in Gherkin, a structured natural-language notation whose Feature/Scenario/Given/When/Then keywords are simultaneously readable by non-technical stakeholders and executable, once bound to step definitions, by test runners such as Cucumber. This dual role is BDD’s central appeal and its central cost. High-quality Gherkin must faithfully capture every requirement while remaining declarative and unambiguous, yet producing it by hand for a non-trivial specification is slow and error-prone.

Large language models (LLMs) have made the automatic drafting of such scenarios plausible, and a small but growing body of work reports that frontier models can produce usable Gherkin from user stories and requirement descriptions [4,5,6]. Yet the evidence base remains fragmented. Existing studies typically fix a single input format and report a narrow slice of quality signals, and most evaluate on data that is not released, which makes their results difficult to compare or reproduce [7,8]. There are exceptions: Rathnayake et al. [4] release both their dataset and their code, and we position our contribution against theirs on scope and measurement rather than on openness (Section 2). Two recent secondary studies reach the same conclusion independently, identifying evaluation thoroughness, traceability, and  reproducibility as the field’s most pressing gaps [9,10].

A question of direct practical relevance is therefore left open: does the type of input artefact, whether a terse Requirement List, an agile user story, or the prose of a formal Request for Proposal (RFP), change how well, how completely, and how cheaply an LLM can produce Gherkin, and do different models excel on different input types? Answering this requires holding the task fixed while systematically varying the model, the input type, and the measurement lens, and doing so on data and code that others can inspect and rerun.

This paper contributes such a benchmark, with four elements. First, we define a controlled experiment that crosses eight LLMs with three requirement corpora: Requirement Lists, User Stories, and constructed RFP-style excerpts. The last has not, to our knowledge, previously been used as an input to Gherkin generation (the closest precedent generates Gherkin from regulatory prose [11]). We introduce it only as an author-constructed, exploratory probe, not as a validated new artefact type, and because each corpus comes from a different source, we report the corpus comparison as such rather than as an effect of artefact format (Section 3.1). Second, we assemble a metric suite deliberately spanning instruments with different failure modes, running from the objective parser and runner checks through an LLM judge with its robustness controls to similarity against a hand-authored gold standard (Section 3). Third, we release an open, resumable Python prototype, the full dataset, and the hand-authored gold standard, openly archived on Zenodo [12], so that the entire pipeline is reproducible. Fourth, we contribute two methodological demonstrations: that the output-token budget is a first-class, easily overlooked confound in generative benchmarking (Section 4.6) and that a tolerant parse harness can overstate real runner acceptance by twenty points (Section 4.12).

We flag at the outset which findings are firm and which are provisional, since the two rest on different evidence. The findings that carry the paper are judge-independent, resting on the official Cucumber parser and runner, on measured cost, and on an output-token budget we control. The  judged-quality ordering, the coverage-based model separation, and any single-model steer that derives from them are scored by an LLM judge whose agreement with human raters we now measure directly (Section 4.9). It reproduces human score magnitudes about as closely as a second human does but agrees on ordering well below the human–human ceiling, so we report them as provisional throughout. Section 5, including its practitioner guide, keeps the two classes apart, and  Section 6 states each finding in full.

2. Related Work

2.1. LLM Generation of Gherkin and Acceptance Tests

The closest work to ours is the multi-model study by Rathnayake et al. [4], who benchmark GPT-4, Claude 3, and Gemini on generating Given–When–Then scenarios from user stories and fuller requirement descriptions, using text-similarity, semantic-similarity, LLM-as-judge, and human-expert evaluation. They report that Claude 3 is preferred by humans while GPT-4 scores higher on similarity metrics, that the best prompting strategy is model-specific, and, most relevant to our design, that detailed requirement descriptions yield markedly better scenarios than terse user stories. They evaluate three input configurations (user story only, requirement description only, and both together) drawn from two underlying artefact genres, and they release their dataset of 500 user-story/BDD-scenario pairs together with their source code under an MIT licence. The artefacts originate from four proprietary products of a single company, obtained under a data-sharing agreement, so the corpus is industrial in provenance but nonetheless openly available.

Our study is therefore not distinguished from theirs by openness, and we do not claim it is. It differs in scope and in measurement. We evaluate eight models rather than three, spanning a frontier, an efficient, and an open-weight tier; we draw inputs from two independent public corpora (PURE and Dalpiaz) rather than one company’s product suite, which trades industrial realism for cross-organisational breadth; we add a third artifact genre, discursive procurement-style prose, rather than a third combination of the same two genres; and we add objective structural, anti-pattern, stability, and cost instrumentation alongside the judged scores, together with the output-token-budget analysis of Section 4.6. Their inclusion of a human-expert evaluation alongside the automated scores is the component our own measurement design most directly builds on (Section 4.11).

Two further multi-model studies bracket ours on measurement strategy. Karpurapu et al. [13] evaluate several LLMs formulating BDD acceptance tests, establishing the multi-model framing early but resting on syntactic validation of a modest user-story sample, which is the narrow-metric limitation our suite is built to widen. On the other hand, Siddeeq et al. [14] compare epic-organised against requirement-aligned Gherkin on 107 requirements drawn from the same PURE corpus we sample, and combine lexical (TF–IDF) coverage, dense-embedding semantic coverage, and a pre-registered blinded expert evaluation by four researchers. Two of their findings bear directly on our design. Their lexical coverage measure scored the stronger system lower while the semantic measure scored it higher, mirroring the paraphrase-sensitivity we document for our own lexical coverage check (Section 4.10). And their expert panel produced a near-zero Fleiss’ $\kappa$ (roughly $-0.08$ to $0.03$) driven by systematic scale-usage differences between raters, even though the directional ranking held in fifteen of sixteen rater–document pairs. That combination, unreliable absolute agreement alongside a stable ordering, is the outcome any human-rating study of Gherkin quality must be prepared to report. Our own calibration study (Section 4.9) reproduces the shape of it while landing on firmer numbers: our two raters agree substantially rather than near-zero, yet the judge still resolves close orderings poorly.

Several industrial case studies demonstrate feasibility in production settings. Ferreira et al. [5] chain GPT-4 Turbo to turn user stories into Gherkin and then into executable Cypress tests, reporting that practitioners found 95% of generated scenarios helpful. Fonseca et al. [6] present AToMIC, which generates Gherkin, page objects, and UI test scripts for a large commercial mobile application. They report 93.3% Gherkin syntactic correctness with substantial developer time savings. These studies establish practical value, but each fixes one model and one input type and evaluates on a single company’s proprietary data.

Two comparative studies most resemble a benchmark, and 

Table 1. This study compared against the three closest comparative studies. “Not reported” means the dimension is not described in the cited work rather than that it was omitted. Model counts are as evaluated in each paper.

Study	Models/Corpora	Automated Evaluation	Human Evaluation	Public Release
Rathnayake et al. [4]	3 models; 500 user-story/BDD pairs from four proprietary products of one company	Text similarity, semantic similarity, LLM-as-judge	Yes: human-expert preference	Yes: dataset and code (MIT)
Fernandes et al. [7]	7 models; free-form test descriptions	METEOR; repeated-measures variability	Not reported	No dataset or prototype
Dos Santos et al. [8]	4 commercial assistants; user stories with Gherkin acceptance criteria	Similarity, coverage, accuracy, efficiency	Not reported	No dataset or prototype
This study	10 models (8 primary, 2 refresh); 74 artifacts from PURE, Dalpiaz, and constructed RFP excerpts; 3700 generations	Parser validity raw and normalised, Cucumber runner acceptance, primary plus three off-panel LLM judges, chrF against human gold standard, anti-patterns, stability, cost and Pareto	Yes: two blinded raters on 72 items, reported against a measured human–human ceiling (Section 4.9); 30-item hand-authored gold standard with an external coverage audit	Yes: corpora, gold standard, prototype, and analysis scripts

sets out their designs. Fernandes et al. [7] find that zero-shot prompting is as effective as few-shot prompting and that Gemini offers the best accuracy–stability balance; Dos Santos et al. [8] rank four commercial assistants across similarity, coverage, accuracy, and  efficiency. Both are valuable but evaluate on small samples with a narrow metric set and release no reusable dataset or prototype. Hassani et al. [11] take a human-centred view, having participants rate Gherkin generated from food-safety regulations in terms of relevance, clarity, completeness, singularity, and time savings. Their five-dimensional rubric informs our quality assessment, and their documented failure modes (omissions and hallucinations) motivate our grounding and coverage metrics.

Table 1 sets our design against the three closest comparative studies on the dimensions that decide whether a benchmark result can be trusted and reused. The pattern it shows is that breadth of models, breadth of measurement, human validation, and public release rarely co-occur: Rathnayake et al. pair human evaluation with a full release but on three models and one company’s data; the two larger-panel comparisons release neither data nor code and rest on a single automated metric family. Our contribution is the combination rather than any one column.

2.2. Requirements-to-Test Generation and Classical Precursors

The broader problem of generating tests from natural-language requirements is more mature. Korraprolu et al. [15] compare six LLMs generating test cases from requirements, using coverage against a Simulink oracle as an objective metric, while Alagarsamy et al. [16] fine-tune models for text-to-testcase generation and report a clean triad of syntactic correctness, requirement alignment, and code coverage that we adapt to the Gherkin setting. Before LLMs, rule-based and classical NLP pipelines addressed adjacent problems: Bob and Storer [17] generated step-definition code from existing Gherkin and evaluate it with mutation testing, and  Gröpler et al. [18] formalise requirements with NLP for downstream test generation. Recent systematic reviews confirm that BDD/Gherkin generation remains a thin sub-slice of a literature dominated by unit-test generation, and that standardised, reproducible evaluation is largely absent [9,10].

2.3. Data Scarcity and Evaluation Methods

A recurring obstacle is the scarcity of paired natural-language-to-Gherkin data with trustworthy references. Most of the datasets used above are proprietary. Galloy et al. [19] responded by synthesising a BDD dataset with a self-instruct procedure, one of the few open efforts. On the measurement side, the official Cucumber parser provides an objective validity signal. Reference-based text metrics such as BLEU/chrF and BERTScore [20] are applicable when a gold standard reference exists, though their known bias towards surface overlap is pronounced for Gherkin’s fixed keyword scaffolding. Reproducible-evaluation guidance in turn recommends reporting scores as intervals over repeated runs [21]. Where no gold standard reference exists, we adopt the LLM-as-a-judge methodology [22] but heed its documented failure modes, namely judges favouring their own family’s generations [23] and rewarding longer outputs [24], which directly motivate our off-panel and length-controlled robustness checks. Closest to our own instrument-validation problem, Huang et al. [25] evaluate LLM judges specifically for Gherkin test-coverage assessment against a set of expert-annotated scripts, scoring twenty judge configurations by absolute error against the expert ground truth rather than by inter-rater agreement and report that a small, inexpensive judge matched the best frontier configuration at a fraction of the cost. Their design also illustrates a circularity worth naming: the scoring rubric was embedded in the judge prompt, so the judge was aligned to the same rubric that generated the reference scores. Our calibration protocol (Section 4.9) keeps the human rubric and the judge prompt textually parallel for comparability but derives the human scores independently, and we report agreement statistics rather than error against a rubric the judge was shown. A weaker form of the same circularity survives in our design, and we treat it as a limitation (Section 5.2): a rubric that misframes Gherkin quality would mislead our human raters and the judge alike. On the quality side, the BDD-specification-quality literature (characterising specification quality [26] and detecting duplicate examples in BDD specifications [27]) grounds our anti-pattern and Gherkin-construct metrics, and recent surveys situate LLM-based Gherkin generation within the broader landscape of software testing with LLMs [28]. Our benchmark integrates these threads into a single, openly archived pipeline.

2.4. Positioning

Taken together, prior work establishes that LLMs can generate Gherkin and that the model, prompt, and input detail all matter, but no single study is simultaneously multi-model, multi-input-type (spanning enumerated requirements, user stories, and more discursive prose requirements), multi-metric, and reproducible. The RFP-style arm is best understood as a third, more discursive form of prose requirements rather than a categorically new artefact type. Because it is authored rather than field-sampled, we include it only as an exploratory probe and rest no headline claim on it (Section 5.1).

3. Materials and Methods

3.1. Research Questions

RQ1. How do LLMs differ in syntactic validity, requirements coverage, and  scenario quality when generating Gherkin? RQ2 (descriptive, corpus-level). Do the three requirement corpora (Requirement Lists, User Stories, constructed RFP excerpts) differ in the quality and coverage of the Gherkin they elicit, and does the corpus effect differ by model? RQ2 asks about these three specific corpora and not about requirement formats; the design cannot answer a format question even in principle, for the reason set out below, and we draw no format claim from it. RQ3. How do models compare on generation time, token usage, and cost, and where is the quality–cost Pareto front? RQ4. How stable is each model’s output across repeated runs?

RQ2 is deliberately posed over corpora rather than over artefact formats. Each of our three input types is drawn from a different source (PURE for Requirement Lists, Dalpiaz for User Stories, and author-constructed text for the RFP excerpts), so format is completely confounded with source, subject matter, granularity, and intrinsic difficulty. No statistical adjustment can recover a format effect from this design, because the design contains no item that appears in more than one format. Isolating a format effect would require expressing the same underlying requirements in all three formats and comparing within items, which our corpus does not support and which we identify as the natural next experiment (Section 5.2). We therefore report RQ2 as a three-corpus comparison throughout and phrase every conclusion drawn from it in those terms.

The requirements-coverage and scenario-quality components of RQ1 and RQ2 are measured by an LLM judge calibrated against two blinded human raters in Section 4.9, where it is found to agree with them only moderately and below the ceiling those raters set between themselves. We therefore treat those sub-questions as instrument-limited and report their answers as provisional throughout, foregrounding the objective sub-questions as the ones the study answers firmly. The  answers grounded in objective metrics (syntactic validity, well-formedness, stability, time, and cost) do not carry this caveat.

3.2. Experimental Design

We use a factorial design crossing model, input type (three levels), and  source artifact, with five repetitions per cell to quantify stochasticity. Eight models were evaluated in the primary run, spanning proprietary frontier systems, an efficient tier, and  open-weight models: Claude Opus 4.8, Claude Sonnet 5, GPT-5.5, GPT-4o, GPT-4o-mini, Gemini 2.5 Pro, Llama-3.1-70B-Instruct, and DeepSeek-V3 [29]. OpenAI and Anthropic models were called through their native APIs; Gemini, Llama, and DeepSeek through OpenRouter. Two further models, Gemini 3.5 Flash and DeepSeek-V4-Flash, were added afterwards as a panel-refresh batch under an identical protocol; because they were generated later, they are reported separately in Section 4.15 rather than pooled with the primary run. All models received an identical system and user prompt and the same evaluation pipeline, with a maximum of 6000 output tokens in the primary run. An initial batch used a 1500-token limit. Because  that ceiling truncated a substantial fraction of outputs from the most verbose models and distorted their measured performance (Section 4.6), the primary run raises the limit to 6000, and the 1500-token batch is retained and reported as a budget-sensitivity comparison. Decoding temperature was fixed at 0 for every model that accepts it. Three models (GPT-5.5, Claude Opus 4.8, Claude Sonnet 5) rejected an explicit temperature and ran with their provider-default adaptive reasoning, which we record per generation and treat as a source of the stability differences examined in RQ4. Each generation logs full provenance (timestamp, run label, and the exact model snapshot the provider reports), so the dataset is auditable and can be extended or replicated as a distinct batch.

3.3. Inputs

The 74 source artefacts comprise 25 Requirement Lists, 25 User Stories, and 24 RFP excerpts. Requirement Lists are drawn from the PURE corpus of public software requirements documents [30], extracting requirement-bearing (“shall”/“must”/“should”) statements grouped into short lists. That extraction segments each document at terminal punctuation, a heuristic with one known failure mode: it carries no abbreviation guard, so a statement containing “i.e.” or “e.g.” is cut at the abbreviation and its remainder discarded. Four of the 146 requirement statements in this arm are truncated this way (2.7%, in 3 of the 25 artifacts), two of them in the 30-item gold standard subset of Section 4.11. We disclose the defect rather than repair it, because these files are the exact inputs behind every generation reported here: regenerating them would invalidate the collected data and break reproducibility of the released corpus. The truncation is applied identically to every model, and the User Stories and RFP arms do not pass through this splitter and are unaffected. User Stories are drawn from the Dalpiaz public user-story dataset [31], whose atomicity and completeness criteria are formalised in the Quality User Story framework [32] that our singularity and completeness rubric dimensions echo. No RFP-to-Gherkin corpus exists, so the RFP arm is constructed for this study as functional-requirements excerpts modelled on public-sector procurement documents, spanning 24 application domains. This is disclosed as a construct-validity limitation (Section 5.1). Every artefact’s provenance (source dataset, file, and licence) is recorded in a released manifest.

3.4. Metrics

Correctness: syntactic validity via the official Cucumber Gherkin parser [33], applied after the two normalisation steps of Appendix C. An output is valid when every block parses, and well-formed when, in addition, every scenario carries steps. The  well-formedness check does not verify that a Scenario Outline has a populated Examples table, so an Outline with placeholder steps but no Examples would count as well-formed while expanding to no executable scenarios. This is a known proxy limitation. Structure: scenario counts (with step counts used within the length analysis of Section 4.7), Given/When/Then completeness, and the use of first-class Gherkin constructs (Scenario Outline/Examples tables and Background blocks), all derived from the parsed syntax tree. A Scenario Outline counts as one scenario (its syntax-tree node), not as its expanded Examples rows. Coverage and quality: scored by an LLM judge, Claude Sonnet 5 in the primary run, returning per-artifact requirements coverage and a four-dimension quality rubric (relevance, clarity, completeness, and  singularity). Because the rubric’s completeness dimension and the separate coverage metric both, through the same judge, assess whether all required behaviours are captured, the two are correlated judge-scored signals rather than orthogonal measures. Being judge-scored, both share the judge’s limitations. The judge is itself one of the evaluated models, so its potential self-preference and verbosity bias are quantified, disclosed as the primary threats to validity (Section 5.1), and tested by re-scoring with three off-panel judges (Section 4.8) and controlling for output length (Section 4.7). Because the judge scores exactly the two behaviours the generation prompt commands (“cover every requirement; invent nothing extra”; Appendix A), absolute coverage and invented-requirement levels reflect instruction compliance as well as capability, so only the relative differences across models are informative. Both signals also measure requirement presence (whether each requirement is addressed), not behavioural test adequacy. Neither captures negative-path, boundary, or error scenarios, so even a maximally “covered” output is not thereby an adequate test suite. By the same token, the “invent nothing extra” instruction and the invented-requirement count jointly target faithful transcription of the source, not test-design quality: a model that derives legitimate negative or boundary scenarios beyond the stated requirements would be counted against on the hallucination signal. This is intrinsic to the draftability (not adequacy) scope of the study. The objective metrics do not depend on the judge. Anti-patterns: a UI/technical-vocabulary leakage signal and a step-phrasing repetition signal derived from the syntax tree, with the exact detection rules in Appendix D and the implementation in the released prototype [12]. The declarative-versus-imperative ideal these probe is the one characterised for BDD by Binamungu et al. [26], but the leakage signal is a lexical vocabulary filter and does not by itself measure imperative style. The repetition signal is likewise a phrasing heuristic and is reported descriptively, since repeated steps are as often legitimate step reuse (the basis of shared step definitions) as a smell, and it is distinct from the whole-scenario duplicate-example detection of Binamungu et al. [27]. These heuristic detectors are not validated against human-labelled anti-patterns, so the signals are reported as indicative rather than definitive. Moreover, because the system prompt explicitly prescribes declarative, non-UI steps (Appendix A), the low measured leakage rate reflects instruction compliance under that prompt rather than an intrinsic model tendency. Efficiency: generation latency, prompt and completion tokens read from provider usage fields, and monetary cost from a logged per-model pricing snapshot. Cost reflects generation only, excluding evaluation overhead (judge and off-panel re-scoring costs are additional). Stability: the within-artifact standard deviation of scenario count across the five repetitions (output-volume stability), complemented by a content-level measure, the mean pairwise Jaccard overlap of the step sets across the five repetitions (Section 4.4). Runner acceptance: whether the official Cucumber runner (cucumber-js) loads the generated file in dry-run mode, and the number of distinct step definitions it then demands (Section 4.12). This is stricter than parser validity and fully objective. Reference-based similarity: chrF against Gherkin hand-authored by two independent annotators on a 30-item stratified subset, reported against a measured human–human agreement ceiling and a same-genre floor (Section 4.11). BERTScore is not reported: on this corpus, the surface metric is already at the limit of what a reference-based instrument can resolve, for the reasons Section 4.11 sets out. Human calibration: two annotators independently rated a blinded, stratified sample of 72 generations on the judge’s own rubric, giving both a judge–human agreement figure and the human–human ceiling that makes it interpretable (Section 4.9).

3.5. Prototype and Reproducibility

All measurements are collected by an open, configuration-driven Python prototype released with the paper. Generation is idempotent and cached by a content hash, so runs are resumable and never re-spend on completed work. The same mechanism lets new models or replication batches be added and merged without recomputation. Judge results are cached separately, so the (rate-limited) evaluation phase is likewise resumable. Syntactic validity is decided by the official parser rather than heuristics, and the objective measurement core is covered by an automated test suite. Two senses of reproducibility should be distinguished. The pipeline and dataset are exactly reproducible: the content-hash cache makes every generation, metric, and judge call re-derivable, and the released artifact reproduces all tables and figures. The model outputs, by contrast, are not deterministic even at temperature 0. The temperature-0 open-weight and non-reasoning models still vary run-to-run (RQ4 reports non-zero scenario-count SDs and low step-set overlap for them), and the three frontier models that reject an explicit temperature and run with provider-default adaptive reasoning vary more. All outputs are therefore reported with recorded model-snapshot provenance rather than as reproducible constants, and it is the pipeline, not the model outputs, that the artefact makes bit-exactly reproducible.

3.6. Analysis

Model comparisons on the same items use non-parametric tests (Friedman with post-hoc Wilcoxon and Holm correction) with Kendall’s W effect sizes. Input-type effects (RQ2) use linear mixed-effects models with crossed random effects for model and source item, which correctly handle the nesting of source items within input type. The model × input-type interaction is tested with cluster-robust standard errors clustered on a source item (74 clusters). Because inference clustered on model instead rests on only eight clusters (too few for the usual cluster-robust variance estimator, which is then anti-conservative), the one model-clustered slope we report (the length effect of Section 4.7) is additionally verified with a wild cluster bootstrap over the model clusters. The model × input-type interaction is clustered on the 74 source items rather than on the model,  so it is not subject to the few-cluster limitation. Per-model estimates are reported with 95% confidence intervals bootstrapped over source items, and the judge-scored quality ranking is re-estimated after controlling for output length (Section 4.7). Because the 74 items were selected deterministically rather than sampled, these intervals quantify item heterogeneity conditional on this fixed corpus, not sampling error towards the PURE or Dalpiaz populations. Judge test–retest reliability is estimated by re-scoring a subsample (Section 4.8). We report per-test p-values without a family-wise correction across the full battery of analyses, so the conclusions rest on the direction and size of effects rather than on any single marginal p-value.

4. Results

The primary run produced 2960 generations (8 models × 74 source artifacts × 5 repetitions) at a 6000-token output budget with no generation failures and no truncated outputs.

Table 2. Benchmark results per model, pooled across Requirement Lists, User Stories, and  RFPs (74 source artifacts, 5 repetitions each). Parse and well-formed are the fraction of outputs accepted by the official Cucumber parser and, additionally, in which every scenario carries at least one step. Both parse columns are reported because the difference between them is itself a finding. Raw is what a naive single-Feature parser accepts from the unmodified model output (77.9% overall); norm. is measured after the normalisation of Appendix C (fence-stripping and multi-Feature splitting, 98.6% overall). Normalisation rescues 20.7% of generations, almost all of them multi-Feature files, and it reverses the ordering: the four models with the lowest raw rates are among the highest once normalised (Section 4.1). Raw parse is close to, but not the same as, runner acceptance (Section 4.12), which additionally requires the file to load in Cucumber. Quality and coverage are LLM-judge scores, each with a bootstrap 95% CI over the 74 source items. Stability is the mean within-artifact standard deviation of scenario count across the five repetitions (lower is more deterministic). Rows are ordered by the primary judge’s mean quality for reference only. Because that ordering is judge-scored and within re-scoring noise (Section 4.7 and Section 4.8), it is not an endorsed ranking.

Model	Parse	Parse	Well-	Cover-	Quality	Scen.	Latency	Cost	Stab.
	Raw (%)	Norm. (%)	Formed (%)	Age (%)	(1–5)	/Item	(s)	($/gen)	(SD)
Claude Sonnet 5	72.7	99.7	94.9	99.9 [99.7, 100]	4.68 [4.62, 4.73]	7.2	7.4	0.0136	0.78
Claude Opus 4.8	40.8	99.5	99.5	99.8 [99.4, 100]	4.66 [4.60, 4.71]	7.5	8.9	0.0209	0.74
GPT-5.5	99.7	100.0	100.0	99.1 [97.6, 100]	4.65 [4.59, 4.71]	6.3	13.5	0.0342	0.58
Gemini 2.5 Pro	50.8	98.4	98.4	91.3 [87.8, 94.5]	4.52 [4.44, 4.60]	5.9	17.5	0.0195	0.86
DeepSeek-V3	74.6	96.5	95.7	98.7 [97.7, 99.4]	4.38 [4.32, 4.44]	5.3	9.4	0.0003	0.59
GPT-4o	98.6	98.9	98.9	98.8 [97.6, 99.6]	4.37 [4.31, 4.42]	5.3	3.0	0.0036	0.16
GPT-4o-mini	94.9	98.9	98.9	98.2 [97.2, 99.1]	4.35 [4.29, 4.41]	5.4	5.6	0.0002	0.22
Llama-3.1-70B	91.1	97.3	97.3	96.5 [94.9, 97.8]	4.32 [4.25, 4.39]	5.2	8.9	0.0003	0.50

reports each model’s performance pooled across the three input types, ordered by judged quality. The four research questions are addressed in turn below, followed by analyses of fidelity and Gherkin constructs (Section 4.5); a budget-sensitivity analysis (Section 4.6) that motivates the 6000-token choice; a judge-free coverage check on the Requirement List arm (Section 4.10); and length-control, judge-robustness, per-dimension, and prompt-sensitivity analyses that stress-test the judged metrics. Syntactic validity, well-formedness, scenario counts, latency, and cost are objective and independent of any judge. Requirements coverage and scenario quality are scored by the LLM judge (Claude Sonnet 5, itself an evaluated model) and are therefore the provisional signals flagged in Section 3 and Section 5.1.

4.1. RQ1: Differences Across Models

Models differed significantly on every quality dimension, though by very different margins. Friedman tests with the 74 source artifacts as blocks rejected the null of equal performance for syntactic validity (${\chi }^2\left(7\right)=23.6$, $p=0.001$, Kendall’s $W=0.05$), scenario quality (${\chi }^2\left(7\right)=222.0$, $p<0.001$, $W=0.43$), and requirements coverage (${\chi }^2\left(7\right)=106.0$, $p<0.001$, $W=0.20$). The effect is moderate-to-large for quality, small-to-moderate for coverage, and negligible for syntactic validity, which was uniformly high. Post-hoc Wilcoxon signed-rank tests with Holm correction found 19 of 28 model pairs differing on quality.

At the corrected 6000-token budget, every model produced Gherkin with uniformly high validity rates. Parse rates ran from 96.5% to 100% and well-formedness from 94.9% to 100%, so no model failed structurally (Figure 1). This uniform validity depends materially on the two normalisation steps of the measurement protocol (stripping Markdown code fences and splitting multi-Feature output; Appendix C). A naive single-Feature parser applied to the raw model output would accept only 77.9% of generations. The 20.7% that normalisation rescues are almost entirely multi-Feature files (20.9% of outputs emit more than one Feature block, whereas Markdown fences affected only 0.1%). The rescue is concentrated on the four models that emit substantial multi-Feature output (Claude Opus 4.8, Gemini 2.5 Pro, Claude Sonnet 5, and  DeepSeek-V3; Figure 2), whose raw parse rates fall to 40.8–74.6% while the single-Feature models need almost none. The high, uniform validity is thus a property of the model output under a documented normalisation layer, not of the raw text. A  benchmark that parsed raw single-Feature output would have wrongly ranked the frontier models as the least valid, another instance of the measurement-design sensitivity that Section 4.6 documents for the token budget. Splitting multi-Feature output is a deliberate scoring choice, and Section 4.12 measures what a strict single-file toolchain does with the same output instead.

On judged quality, however, the panel is tightly compressed into a 0.36-point range on a five-point scale, a pronounced ceiling. Excluding the exploratory RFP arm leaves this ceiling essentially unchanged (a 0.36-point range of 4.22–4.58 over the Requirement List and User Story inputs alone), and the frontier-versus-cheap band-mean gap is likewise stable (0.31 pooled versus 0.30 without the RFP arm). Neither the ceiling nor the cost argument that follows from it (RQ3 below) therefore depends on the constructed arm. The frontier systems sit at the top (Claude Sonnet 5 4.68, 95% CI [4.62, 4.73]; Claude Opus 4.8 4.66, [4.60, 4.71]; GPT-5.5 4.65, [4.59, 4.71]), but their confidence intervals (over source items) overlap completely, so the fine ordering within the top three is not statistically distinguishable. What is distinguishable is the gap between this frontier band and the low-cost/open band (DeepSeek-V3 4.38, [4.32, 4.44]; GPT-4o 4.37; GPT-4o-mini 4.35; Llama-3.1-70B 4.32, [4.25, 4.39]), whose intervals do not overlap with the leaders’.

Two facts temper even the band-level quality reading. First, the judge is Claude Sonnet 5, and the two highest-scoring models are from its own family (the Claude family averages 4.67, ahead of Google 4.52, OpenAI 4.46, DeepSeek 4.38, Meta 4.32). Second, the frontier models’ one objective distinction is verbosity. They emit more scenarios per artefact (Claude Opus 7.5, Sonnet 7.2, GPT-5.5 6.3) than the rest (5.2–5.9), and the judge demonstrably rewards length (Section 4.7). We treat the quality axis as nearly non-discriminating at the top and locate the panel’s real model separation in coverage, stability, and cost, of which only cost is fully objective. Coverage, moreover, does not so much grade the panel as isolate a single outlier. Seven of the eight models exceed 96% and are mutually indistinguishable, and the entire spread comes from Gemini 2.5 Pro at 91.3% (95% CI [87.8, 94.5]; Figure 1). This one-outlier reading also resolves an apparent conflict with the effect sizes (quality’s $W=0.43$ exceeds coverage’s $W=0.20$). Kendall’s W measures the consistency with which items rank the models, not the magnitude of the differences, so quality is highly concordant yet compressed into a 0.36-point band near the ceiling, whereas coverage is less concordant but, through the single Gemini outlier, spans a practically wider range. By “discriminating” we mean practical separation, not rank concordance, and even that separation is one low-coverage model rather than a graded field.

4.2. RQ2: Differences Across the Three Requirement Corpora

The three corpora differed in the coverage and quality they elicited, but not in syntactic validity. We report these as corpus differences and not as effects of artifact format, for the reason set out in Section 3.1. Across models, Requirement Lists yielded the lowest coverage (0.95) and the constructed RFP excerpts the highest (1.00), with User Stories in between (0.99). A Friedman test across the three input types with models as blocks was significant for coverage (${\chi }^2\left(2\right)=8.9$, $p=0.012$, $W=0.56$) and, more strongly, for quality (${\chi }^2\left(2\right)=12.3$, $p=0.002$, $W=0.77$; input-type quality means 4.39/4.40/4.69 for Requirement Lists/User Stories/RFPs). Syntactic validity did not differ by input type (${\chi }^2\left(2\right)=5.4$, $p=0.069$). Figure 3 shows the coverage pattern. Requirement Lists are systematically the hardest arm, and Gemini 2.5 Pro’s coverage falls most sharply there (0.81 versus 0.99 on RFPs). Rather than depressing performance, the constructed RFP arm was the easiest input for both coverage and quality. This is most likely a property of how it was built rather than of procurement text as such. Real RFPs are typically among the hardest requirement sources (boilerplate, legalese, non-functional and contractual clauses, cross-references, internal contradictions), whereas our excerpts are clean, self-contained functional-requirement statements, so the “easiest arm” result should not be read as evidence that LLMs handle genuine procurement prose easily (Section 5.1).

These effects survive a more appropriate analysis. Because each source item belongs to exactly one input type (items are nested within type, not crossed with it), the model-blocked Friedman above is underpowered (eight blocks). Treating the 74 source items as the independent unit (item-level mean over models and repetitions), a Kruskal–Wallis test confirms the input-type effect for both quality ($H=38.1$, $p<{10}^{-8}$) and coverage ($H=33.9$, $p<{10}^{-7}$). The RFP arm scores higher than the other two arms combined on quality (item means 4.69 versus 4.40, Mann–Whitney $p<{10}^{-9}$) and coverage ($p<{10}^{-4}$). A linear mixed-effects model with crossed random intercepts for model and source item agrees in direction and magnitude (RFP quality $+0.29$ over Requirement Lists, coverage $+0.046$; User Stories indistinguishable from Requirement Lists on quality, $\beta =+0.003$, $p=0.77$), and a model × input-type interaction on quality is present ($F\left(14\right)=2.64$, $p=0.004$, cluster-robust on item), with Gemini 2.5 Pro showing the largest input-type swing. This Gaussian mixed model is a corroborating, not primary, analysis. Coverage is a proportion bounded at a ceiling and quality a capped rating, for which a Gaussian likelihood is only approximate, so we rely on the non-parametric item-level tests above for inference and read the mixed-model coefficients as directional confirmation rather than as calibrated effect sizes. A beta mixed model for the bounded coverage proportion, or a cumulative-link (ordinal) mixed model for the 1–5 quality rating, would respect these bounds more faithfully, and we leave those bounded-response models to future work. Two cautions apply, however. First, the three input types come from three different sources (PURE, Dalpiaz, and  author-constructed excerpts), so input type is confounded with input provenance. The  measured effect is equally a corpus effect, since each type is also a distinct corpus, and the two cannot be separated in this design. A clean type test would require the same source documents rendered into all three formats. Second, the RFP arm’s advantage in particular cannot be separated from its being hand-authored, clean, and self-contained (a construct-validity limitation; Section 5.1). The input-type effect, including the model × input-type interaction, should therefore be read as correlational rather than causal, and the RFP arm as exploratory.

4.3. RQ3: Cost, Tokens, and Cost-Efficiency

Cost per generation spanned more than two orders of magnitude, from $0.0002 (GPT-4o-mini) to $0.0342 (GPT-5.5), while judged quality spanned a narrow 0.36-point band (4.32–4.68). The full primary run of 2960 generations cost $34.27 across the eight models (judge and off-panel re-scoring excluded). Token usage underlies these costs and is itself informative. Prompt sizes were similar across models (339–550 tokens), but mean completion length varied nearly seven-fold, from about 275–300 tokens for the terse open models (GPT-4o, DeepSeek-V3, Llama-3.1-70B) to 1084 for GPT-5.5 and 1907 for Gemini 2.5 Pro, whose long completions are dominated by hidden reasoning tokens rather than by more scenarios (its scenario count is mid-pack). The full per-model token, latency, and cost figures are tabulated in Appendix B (

Table A2. Measured per-model efficiency on the primary run (means over all 74 inputs and 5 repetitions): mean prompt and completion tokens (from provider usage fields), generation latency, and cost per generation. Latency and cost reproduce Table 2 and are repeated here for a consolidated efficiency view. The prompt- and completion-token columns are the additional detail.

Model	Prompt Tok.	Completion Tok.	Latency (s)	Cost ($/gen)
Claude Sonnet 5	550	796	7.4	0.0136
Claude Opus 4.8	550	726	8.9	0.0209
GPT-5.5	343	1084	13.5	0.0342
Gemini 2.5 Pro	345	1907	17.5	0.0195
DeepSeek-V3	339	300	9.4	0.0003
GPT-4o	344	275	3.0	0.0036
GPT-4o-mini	344	276	5.6	0.0002
Llama-3.1-70B	347	300	8.9	0.0003

). Generation latency ranged from 3.0 s (GPT-4o) to 17.5 s (Gemini 2.5 Pro) and tracks completion length more closely than cost. The  result is a steep cost–quality frontier (Figure 4).

We deliberately do not summarise that frontier as a quality-per-dollar ratio. Judged quality is an ordinal five-point score with an arbitrary zero, so dividing it by a dollar cost does not yield a meaningful efficiency quantity, and because the quality scores are compressed into a 0.36-point band, such a ratio would in any case reproduce the inverse cost ranking almost exactly rather than expressing a trade-off. We instead report a Pareto analysis with uncertainty (

Table 3. Pareto analysis with uncertainty, replacing the quality-per-dollar ratio. Confidence intervals and front membership come from a cluster bootstrap over the 74 source items (10,000 replicates, the five repetitions averaged within each item first). “On front” marks the models that are non-dominated at the point estimate; $P(\mathrm{n}\mathrm{o}\mathrm{n}\text{-}\mathrm{d}\mathrm{o}\mathrm{m}.)$ is the fraction of bootstrap replicates in which the model is non-dominated, and is the quantity to read, since it carries the sampling uncertainty. Judged quality is an LLM-judge score that agrees with human raters only moderately and below the human–human ceiling (Section 4.9); cost is objective.

Model	Quality [95% CI]	Cost $/gen [95% CI]	On Front	P(non-dom.)
GPT-4o-mini	4.35 [4.29, 4.41]	0.00022 [0.00021, 0.00022]	yes	1.00
Llama-3.1-70B	4.32 [4.24, 4.39]	0.00026 [0.00025, 0.00027]	–	0.17
DeepSeek-V3	4.38 [4.32, 4.43]	0.00031 [0.00030, 0.00032]	yes	0.79
GPT-4o	4.37 [4.31, 4.42]	0.00362 [0.00350, 0.00373]	–	0.29
Claude Sonnet 5	4.68 [4.62, 4.73]	0.01359 [0.01277, 0.01446]	yes	1.00
Gemini 2.5 Pro	4.52 [4.44, 4.60]	0.01950 [0.01870, 0.02028]	–	0.00
Claude Opus 4.8	4.66 [4.61, 4.71]	0.02089 [0.01960, 0.02214]	–	0.10
GPT-5.5	4.65 [4.59, 4.71]	0.03423 [0.03228, 0.03631]	–	0.12

, Figure 4), which asks the following well-posed question: Which models are not beaten on both axes at once? Resampling the 74 source items (cluster bootstrap, 10,000 replicates, repetitions averaged within item first) gives a 95% confidence interval for each model’s mean quality and mean cost, and the fraction of replicates in which a model lies on the Pareto front.

Only three of the eight models are non-dominated. GPT-4o-mini is on the front in every replicate (cheapest at $0.00022 per generation, quality 4.35), as is Claude Sonnet 5 (highest quality at 4.68, $0.0136); DeepSeek-V3 holds an intermediate position in 79% of replicates. Every other model is beaten on both axes by one of those three in the large majority of replicates. The  sharpest case is Gemini 2.5 Pro, which is dominated in every replicate: Claude Sonnet 5 delivers higher judged quality at 70% of its cost with probability 1.00. Within the frontier tier, Claude Sonnet 5 also dominates Claude Opus 4.8 in 90% of replicates and GPT-5.5 in 87%, so the two most expensive models in the panel buy no quality advantage over a cheaper stablemate. In the low-cost tier, GPT-4o-mini dominates Llama-3.1-70B in 83% of replicates. Costs span 157× ($0.00022 to $0.03423) against a judged-quality span of 0.36 points on a five-point scale, which is why the front is so sparse and why cost, not judged quality, does the discriminating work.

GPT-4o sits between the tiers, mid-band on quality (4.37) but the fastest model in the panel (3.0 s) and inexpensive ($0.0036), pairing its low latency with the lowest hallucination rate in the panel (judge-scored; grounding, a copy-sensitive lexical floor, is not read as a merit here). It is nonetheless dominated in 71% of replicates, chiefly by DeepSeek-V3.

Two caveats bound the practical reading of this ranking. First, per-generation API cost is only one component of total cost of ownership. Since the Gherkin is not executed here and needs human review, at realistic labour rates, the review cost dwarfs the sub-cent API cost, so the cost gap matters chiefly in high-volume, low-review regimes. Second, for the open-weight models, the decisive advantage is often self-hosting (keeping proprietary requirements and RFP text off third-party APIs and avoiding vendor lock-in), a  data-governance dimension our cost metric misses. This ranking is also the most time-perishable result in the study, resting on a dated per-model pricing snapshot (Section 3) and excluding evaluation overhead.

4.4. RQ4: Output Stability

Determinism varied roughly five-fold. Measured as the mean within-artifact standard deviation of scenario count across five repetitions, the two OpenAI non-reasoning models run at temperature 0 were the most deterministic (GPT-4o 0.16, GPT-4o-mini 0.22). The three models that reject an explicit temperature and run with provider-default adaptive reasoning were more variable, as expected (GPT-5.5 0.58, Claude Opus 4.8 0.74, Claude Sonnet 5 0.78), with  DeepSeek-V3 (0.59) and Llama-3.1-70B (0.50) in between. Gemini 2.5 Pro was the least stable of all (0.86) despite accepting a temperature of 0, indicating that its run-to-run variation is not temperature-controlled. Bootstrap 95% confidence intervals over the 74 source items confirm the extremes but not the middle. The temperature-0 non-reasoning models (GPT-4o [0.06, 0.28], GPT-4o-mini [0.12, 0.34]) are clearly the most deterministic, and Gemini 2.5 Pro ([0.66, 1.07]) and Claude Sonnet 5 ([0.61, 0.96]) are clearly the least, but 16 of the 28 pairwise intervals overlap, so the fine ordering (and any clean separation of the adaptive-reasoning trio from the temperature-0 open models) is under-powered at five repetitions. These estimates should therefore be read as output-volume stability under mixed decoding regimes rather than as a resolved per-model ranking. For a BDD toolchain in which reproducible test suites matter, this variability is a practical adoption concern that is independent of average quality and that the single-number quality rankings obscure.

Scenario-count stability is, however, a coarse proxy. Two runs can share a scenario count yet differ entirely in step content, and because a Scenario Outline is counted as one node, it also does not reflect run-to-run churn in Examples table rows for the Outline-heavy models. A content-level measure, the mean pairwise Jaccard overlap of the step sets across the five repetitions, is uniformly low, from 0.09 (GPT-5.5, DeepSeek-V3) to 0.50 (GPT-4o). Even the most count-stable model regenerates roughly half of its step text run to run. The two stability measures agree on the ordering only moderately (Spearman $\rho =-0.55$ between mean Jaccard and count-SD; GPT-4o is the most stable on both), which means scenario-count stability overstates true reproducibility. The RQ4 conclusion should accordingly be scoped to output-volume stability, with content-level reproducibility remaining low across the panel.

4.5. Anti-Patterns and Gherkin Constructs

Beyond validity and coverage, the judge and the parsed syntax trees expose fidelity, anti-pattern, and Gherkin-construct signals (

Table 4. Fidelity, anti-pattern, and Gherkin-construct signals per model (ordered by judged quality). Halluc. is the mean number of judge-flagged invented requirements per output and Ground. is the lexical overlap with the source (0–1)—both are fidelity signals; GWT is the fraction of scenarios with a full Given/When/Then, crediting shared Background Given steps; UI-voc is the mean fraction of steps using UI/technical vocabulary, and Repeat. is the mean fraction near-duplicating an earlier step’s phrasing; Outline and Backgr. are the fraction of a model’s outputs using Scenario Outline/Examples and Background at least once.

Model	Halluc.	Ground.	GWT (%)	UI-voc (%)	Repeat. (%)	Outline (%)	Backgr. (%)
Claude Sonnet 5	0.28	0.44	98.0	0.2	11.3	8.9	67.8
Claude Opus 4.8	0.18	0.50	97.3	0.2	14.0	13.2	26.2
GPT-5.5	0.11	0.48	97.9	0.1	17.7	38.9	25.9
Gemini 2.5 Pro	0.26	0.34	97.6	0.4	9.5	18.1	60.8
DeepSeek-V3	0.16	0.49	91.5	0.3	8.9	0.0	0.0
GPT-4o	0.02	0.64	95.3	0.3	8.7	0.0	0.0
GPT-4o-mini	0.06	0.59	96.0	0.2	7.7	0.0	72.2
Llama-3.1-70B	0.11	0.54	91.5	0.3	10.2	0.0	84.6

). On fidelity, the  judge counts invented (“hallucinated”) requirements and a lexical score measures grounding in the source. Because grounding has lexical overlap with the source (Appendix D), it rewards reuse of source vocabulary and may under-credit sound declarative paraphrase. Since verbatim source-copying maximises grounding while producing less declarative Gherkin, grounding is a lower-bound signal. Low grounding flags possible invention, but high grounding does not certify fidelity. Both differ significantly across models (Friedman with the 74 items as blocks: hallucination ${\chi }^2\left(7\right)=71.8$, $p<{10}^{-12}$, $W=0.14$; grounding ${\chi }^2\left(7\right)=312.0$, $p<{10}^{-60}$, $W=0.60$). The verbose frontier models hallucinate the most. Claude Sonnet 5 invents 0.28 requirements per output (95% CI [0.20, 0.38]) and Gemini 2.5 Pro 0.26, while GPT-4o hallucinates the least (0.02 [0.00, 0.05]) and is the best grounded (0.64 [0.61, 0.67] versus 0.34 [0.32, 0.36] for Gemini). Because GPT-4o is only mid-length (5.3 scenarios per artefact, terser than the frontier but not the tersest model in the panel), its low hallucination is not merely an artefact of brevity. Normalised per scenario, it remains the lowest (0.004 invented requirements per scenario, against 0.031 for the similarly terse DeepSeek-V3 and 0.042 for Gemini 2.5 Pro), so its fidelity edge survives accounting for output volume. It is this per-scenario hallucination count, rather than the grounding score, that carries the fidelity reading, and GPT-4o’s panel-high grounding co-occurs with its panel-low clarity (Section 4.13). The grounding differences are large and cleanly separated, whereas the hallucination ordering is coarser, since the within-frontier gaps partly overlap in their confidence intervals (Claude Sonnet 5’s overlaps Claude Opus 4.8’s). The robust statement is that the low-hallucination, well-grounded cluster (GPT-4o, GPT-4o-mini) is distinct from the verbose frontier models, rather than a fine per-model rank among the frontier. Part of the frontier models’ extra output is thus invention rather than genuine coverage, a pattern the judge’s quality score does not visibly penalise, which reinforces the length-bias caution of Section 4.7. Unlike quality and coverage, the hallucination count carries no test–retest or cross-judge robustness check, so it is the least-validated of the judge signals that feed the practitioner guide (Section 5) and should be read accordingly.

Given/When/Then completeness, computed so that a Background’s shared Given steps count towards each scenario it precedes (Appendix D), is uniformly high across the panel (91.5–98.0%). Under a stricter per-scenario rule that ignores the Background, GPT-4o-mini and Llama-3.1-70B appear to drop sharply (to 68% and 85%), but this is an artefact of their idiomatically factoring shared Given steps into a Background (used by 72.2% and 84.6% of their outputs, respectively). A scenario then legitimately contains only When and Then, and crediting the Background removes the apparent deficit (GPT-4o-mini rises to 96.0%). Structural completeness accordingly does not discriminate among these models and is not a defect axis. GWT completeness is a heuristic structural signal, not a validity requirement. A  legitimately precondition-free scenario, such as a pure state assertion, can be idiomatic yet score below 100%.

Anti-pattern signals were rare across the panel. UI/technical-vocabulary leakage, the use of implementation terms such as click or css that declarative Gherkin avoids, averaged only 0.25% of steps (highest for Gemini 2.5 Pro at 0.4%, lowest for GPT-5.5 at 0.1%), consistent with the system prompt’s instruction to avoid UI mechanics. This is a vocabulary filter, not a detector of imperative style, which a keyword rule cannot measure. Step-phrasing repetition, the fraction of steps that closely match an earlier step’s wording, averaged 11.0% (highest for GPT-5.5 at 17.7%). We report it descriptively rather than as a smell, since repeating a step such as “the user is logged in” across scenarios is the normal, encouraged step reuse of BDD rather than duplication to be avoided. Scenarios were also uniformly compact (mean 3.2–4.1 steps each, rarely exceeding six), so the over-long, incidental-detail scenario smell was essentially absent across the panel.

The sharper differences are in the use of first-class Gherkin constructs. Use of Scenario Outline/Examples tables, the idiomatic construct for data-driven scenarios [34], was confined almost entirely to the frontier models (GPT-5.5 38.9% of outputs, Gemini 18.1%, Claude Opus 13.2%, and Claude Sonnet 8.9%), whereas the four low-cost and open models never used it (0%). This gap is largely a prompt artefact rather than a capability difference. The base prompt (Appendix A) lists Feature, Background, and  Scenario blocks but does not mention Scenario Outline/Examples, so the cheaper models were never asked for the construct, while the frontier models volunteered one that was not requested. Re-generating the 24-artefact probe subset under a permissive prompt that explicitly invites a Scenario Outline for data-driven requirements, holding the system prompt and everything else fixed (Appendix A), reverses the picture. Three of the four low-cost models then use it (Llama-3.1-70B on 100% of outputs, DeepSeek-V3 29%, and GPT-4o 13%), and  only GPT-4o-mini still never does (0%), while all four frontier models rise to 67–79%. The  base-prompt 0% therefore reflects instruction-following, not inability, for every low-cost model except GPT-4o-mini, and it should not be read as a capability ceiling. Construct presence is not construct appropriateness, however. We do not assess whether each use was warranted by a genuinely data-driven requirement, and a near-universal rate such as Llama-3.1-70B’s 100% more plausibly reflects over-application than uniformly sound judgement. Background usage was common but idiosyncratic (0% for DeepSeek-V3 and GPT-4o, 84.6% for Llama-3.1-70B) and is likewise reported descriptively, since a Background whose steps are not relevant to every scenario it precedes is its own mild smell. These construct-level differences are invisible to the parse and quality metrics. A model can score identically on validity while producing a flat, non-idiomatic Gherkin. The pattern is not a clean “frontier is more idiomatic” story, however. A Scenario Outline compresses many similar scenarios into one table rather than inflating the scenario count, so it does not explain the frontier models’ higher counts, and Background usage was in fact highest for two open models (Llama-3.1-70B 84.6%, GPT-4o-mini 72.2%). The one robust construct-level split is that only the frontier models used Scenario Outline/Examples tables at all.

4.6. Budget Sensitivity: Output Token Limit as a Confound

The 6000-token primary run was prompted by a failure of an earlier 1500-token batch whose results, left uncorrected, would have produced badly misleading conclusions. At 1500 tokens, 342 of 2960 generations (11.6%) were cut off by the output-token ceiling (provider finish_reason = length). The truncation was highly concentrated: 303 of Gemini 2.5 Pro’s 370 generations (82%) and 36 of GPT-5.5’s (10%) were truncated, versus at most three for any other model (Figure 5). Because a truncated Gherkin file is typically unparseable or missing scenarios, this ceiling did not degrade all models equally. It selectively penalised the two most verbose or reasoning-heavy systems.

The effect on the objective, judge-independent metrics is unambiguous. Raising the limit to 6000 tokens eliminated truncation entirely (0 of 2960) and recovered exactly the two affected models. Gemini’s well-formedness rose from 64.9% to 98.4%, its mean scenario count from 2.2 to 5.9, and its parse rate from 92.4% to 98.4%, while GPT-5.5’s parse rate and well-formedness rose from 90.3% to 100%. In other words, the 1500-token artefact would have reported Gemini as producing roughly one third as many scenarios as the panel and GPT-5.5 as the least valid OpenAI model, both conclusions being artefacts of the budget rather than properties of the models. The apparent run-to-run instability of these models was similarly inflated, because  random truncation adds variance to the scenario count that stabilises once the budget is adequate. We base these claims on the objective metrics alone. The two batches also used different LLM judges (the 1500-token batch was scored by GPT-4o), so the judged-quality and coverage scores are not compared across budgets, and only the 6000-token batch is treated as primary.

The methodological lesson generalises beyond this study. The fact that an output token budget chosen without regard to model verbosity can penalise verbose or chain-of-thought models is known in principle. Our contribution is a concrete, quantified demonstration (finish-reason rates and an actual ranking inversion from a single mis-set parameter) that shows how easily it is overlooked. The practical prescription is simple. Benchmarks of generative output should report truncation (finish-reason) rates, verify that the budget is non-binding for every model, and treat the budget as a controlled experimental variable rather than an implementation detail.

4.7. Does the Quality Lead Survive a Length Control?

The frontier quality lead co-occurs with greater output length, and LLM judges are documented to reward verbosity [22,24], so we test how much of the ranking length explains. Two facts must be read together. Between models, mean scenario count and mean judged quality are strongly rank-correlated (Spearman $\rho =0.88$, $p=0.004$). Yet at the generation level, output length (scenario and step counts) explains only 9.6% of the pooled variance in judged quality (18.0% within items, once item difficulty is removed). The two reconcile through aggregation level. Length tracks the coarse between-model ordering well but accounts for little of the item-to-item spread. With item difficulty held fixed by item fixed effects, each additional scenario is associated with a $+0.086$ increase in judged quality, so the judge does reward length. Because this slope is estimated over generations clustered within only eight models, we do not rely on the naive pooled p-value. A  wild cluster bootstrap over the eight model clusters (Webb weights, null imposed, 9999 replicates) still rejects the null ($p=0.003$). Eight clusters strain even this remedy, so we read it as one leg of a triangulation with the item-clustered test and the between-model correlation, not as a standalone result. Clustering instead on the 74 source items gives a far smaller $p<{10}^{-26}$, but because within-item length variation is largely between models, that design overstates the model-level correlation, so we report it only as corroboration. The frontier ordering survives the adjustment. When quality is regressed on length and each model’s mean residual is compared, the length-adjusted top three are the same frontier set (Claude Sonnet 5, GPT-5.5, and Claude Opus 4.8, the latter two swapping places). The exact third member (Claude Opus 4.8 or Gemini 2.5 Pro) differs by 0.003 across residualisation methods and is within noise. The full inter-model quality range (best minus worst) shrinks from 0.36 to 0.22 points.

This adjustment must be read as an association, not as a decomposition, and we make no claim that the surviving 0.22 points represent genuine quality. Output length plausibly reflects two things at once: a judge preference for verbosity and genuine additional behavioural coverage. Regressing length out cannot separate them, because both explanations predict the same observed pattern. Residualising on length removes whatever quality the extra output legitimately earned along with whatever verbosity bias it attracted, so the procedure neither identifies the bias contribution nor bounds it in a direction we can sign. The honest statement is narrow: the frontier–rest ordering is not an artefact of length alone, since it persists after conditioning on length, and the magnitude of the residual gap is not interpretable as a bias-free quality difference. Only a human-rated comparison can attribute the gap, which is why we rest no ranking claim on this analysis. The adjustment is two-sided, since some of the frontier models’ extra output is genuine coverage and richer structure (Section 4.5) rather than padding, so a length control may over-correct. The conclusion is robust to the length proxy. We repeated the control on visible output length and the word count of the generated feature file, which is what the judge reads and which excludes the hidden reasoning tokens that inflate the completion-token count for GPT-5.5 and Gemini 2.5 Pro. This again leaves the length-adjusted top three of the same frontier set (GPT-5.5, Claude Sonnet 5, and Claude Opus 4.8) and shrinks the inter-model range to 0.18 points, close to the 0.22 of the count-based control. Together with the family self-preference (Section 4.8) and the ceiling effect, this is why we rest no strong claim on quality and direct the model comparison to coverage, stability, and cost.

4.8. Judge Robustness: Independent Off-Panel Judges

Because the primary judge (Claude Sonnet 5) is itself an evaluated model, we re-scored all 2960 generations with three additional judges drawn from vendors absent from the panel and sharing no lineage with any panel model: Mistral Large, Qwen3-235B, and Grok 4.3.

Table 5. Mean judged quality (1–5) per model under the primary judge (Claude Sonnet 5) and three independent off-panel judges, ordered by the primary judge. “Neutral” is the mean of the three off-panel judges. The Claude family ranks highest under every judge, and the top three models are invariant as a set, but the single highest-scoring model is GPT-5.5 under the neutral mean and under Qwen3-235B and Grok 4.3.

Model	Sonnet 5 (Primary)	Mistral Large	Qwen3-235B	Grok 4.3	Neutral Mean
Claude Sonnet 5	4.68	4.85	4.97	4.70	4.84
Claude Opus 4.8	4.66	4.82	4.96	4.67	4.82
GPT-5.5	4.65	4.85	4.98	4.72	4.85
Gemini 2.5 Pro	4.52	4.71	4.87	4.53	4.70
DeepSeek-V3	4.38	4.76	4.95	4.50	4.74
GPT-4o	4.37	4.75	4.93	4.44	4.71
GPT-4o-mini	4.35	4.70	4.89	4.37	4.65
Llama-3.1-70B	4.32	4.69	4.91	4.37	4.66

reports each model’s mean judged quality under all four judges.

Three results follow. First, the top of the ranking is judge-invariant at the family and top-three level, but not at the single top-one slot. The Claude family attains the highest mean quality under every judge, including all three off-panel ones, and the three frontier systems (Claude Sonnet 5, Claude Opus 4.8, GPT-5.5) constitute the top three under every judge. The single highest-scoring model, however, is GPT-5.5 under the neutral consensus and under two of the three off-panel judges (tied for first under the third). Only the primary Claude judge places a Claude model first, which is itself the direction a residual self-preference would take. Agreement is high whether or not the primary judge is included. Kendall’s $W=0.90$ among the three off-panel judges alone and $W=0.875$ across all four (${\chi }^2\left(7\right)=24.5$, $p<0.001$). The per-model rank correlations between the neutral consensus and the primary judge are positive but, computed over only eight models, imprecise (Spearman $\rho =0.83$ overall; individual judges $\rho =0.67$–$0.90$, not reliably distinguishable). This design therefore rules out only literal vendor self-recognition. Because all four judges are LLMs that share a documented verbosity preference [22,23], and the frontier models are the most verbose, off-panel agreement is equally consistent with a shared length bias as with shared perception of quality. Only the length control of Section 4.7 separates the two, and  it finds the frontier lead real but partly length-inflated. This bound applies to relative rankings, not only absolute levels. Because the judges score exactly the two behaviours the generation prompt commands (Section 3), a model whose instruction-following style matches the judges’ expectations can be favoured over its peers, a confound no combination of LLM judges can close. We thus claim only that the family-level lead is not an artefact of the primary judge recognising its own family, not that it is bias-free, and not that any single Claude model is the best under a neutral judge.

Second, the judges differ in leniency and in how they order the lower tier. The neutral judges score every model higher and compress the range (Qwen3-235B assigns 4.87–4.98 to all eight models), and under them, the low-cost models (DeepSeek-V3, GPT-4o) close much of the gap to the frontier that the stricter primary judge showed. The primary Claude Sonnet 5 judge is thus the harshest on non-Claude models, which means the cost-effectiveness conclusion of Section 4.3 is conservative under the primary judge and only strengthens under the neutral panel. Consequently, the fine-grained ordering within the closely spaced lower tier and  Gemini 2.5 Pro’s exact rank are judge-sensitive, whereas the top-tier set and the family-level result are not.

Third, the quality metric is noisy at the resolution the top ordering demands. The decisive evidence is the overlapping item-level confidence intervals of Section 4.1. Judge noise at the single-generation level is substantial (re-scoring a 64-generation subsample three times gives the primary judge a per-scoring quality SD of 0.096, bootstrap 95% CI [0.075, 0.119] over the 64 items, mean range 0.21), but model means over hundreds of generations are stable to within 0.03 (a same-judge re-scoring, Section 4.13). The top ordering is therefore fragile not because re-scoring moves the model means, but because the 0.03 gap among the top three is smaller than the item-to-item spread those means summarise. The temperature-0 neutral judges are far more repeatable (mean SD 0.026 for Mistral Large, 0.017 for Qwen3-235B) and Grok 4.3 the least (0.113). Qwen3-235B is so lenient, however, that it barely separates the eight models and contributes little discrimination, so the invariant top tier rests mainly on Mistral Large, the  most repeatable neutral judge, with Grok corroborating. Coverage behaves far better than quality on both axes of reliability. It is repeatable within every judge (SD $<0.01$) and robust across judges (the per-model coverage ranking agrees at Kendall’s $W=0.958$), and Gemini 2.5 Pro is the lowest-coverage model under every judge (89.6–93.9%). Its coverage deficit, the clearest judge-scored model separation, is therefore not an artefact of the primary Claude judge being harsh on non-Claude output. This cross-judge agreement establishes reliability, not validity, however. For the prose arms, the judges also infer the requirement inventory that forms the coverage denominator, so agreement can reflect a shared reading rather than external correctness. The coverage separation is accordingly firmest for Requirement Lists, whose denominator is explicitly enumerable. The human calibration of Section 4.9 bears directly on this: there, the human raters counted the requirement inventory themselves rather than inheriting the judge’s, and judge–human coverage agreement proved the weakest result in the study (ICC(2,1) $=+0.21$ against a human–human $+0.76$). Cross-judge reliability on coverage is therefore high while validity is not, and the coverage ordering beyond the single outlier should be read as provisional for all three arms.

4.9. Human Calibration of the Judge

Every judged number reported so far has been provisional because the judge had never been checked against human raters. This section supplies that check.

Two annotators independently rated 72 generations drawn from the primary run, stratified over the three input types and the eight models, with model identity and every judge score withheld. They scored the same four 1–5 quality dimensions the judge scores and, separately, counted the requirements the source contains, the requirements at least one scenario exercises, and any behaviours asserted without basis in the source, from which a coverage ratio was computed exactly as it was for the judge. The rubric they worked from is the one the judge is given (Appendix A), so both instruments answer an identical question. The annotators are the two who authored the gold standard of Section 4.11; eight of the 72 items draw on rfp_0007, for which they had themselves written reference Gherkin, and every figure below is therefore reported again with those eight excluded.

Judge–human agreement in isolation would be uninterpretable, for the same reason a raw chrF against 100 is uninterpretable. The question is not whether the judge matches a human exactly but whether it matches one about as closely as a second human does. Because both annotators rated all 72 items, that ceiling is measured on precisely the same items.

Table 6. Human calibration of the LLM judge on 72 blinded generations rated independently by two annotators. The upper block is the human–human agreement ceiling measured on the same items; the lower block compares the judge against the mean of the two humans. $\rho$ is Spearman’s rank correlation, ICC(2,1) is a two-way random-effects absolute-agreement coefficient, MAE is the mean absolute error (on the 1–5 scale for quality, on the 0–1 ratio for coverage, and in counts for hallucinations), and qw-$\kappa$ is a quadratic-weighted Cohen’s $\kappa$ on rounded quality scores. Values in parentheses exclude the eight items drawn from rfp_0007, for which the annotators had authored the gold standard.

Comparison	Dimension	$\mathit{\rho }$	ICC(2,1)	MAE	qw-$\mathit{\kappa }$
Human A vs. human B	Quality	+0.73 (+0.69)	+0.64 (+0.60)	0.354 (0.395)	+0.56 (+0.54)
(ceiling)	Coverage	+0.72 (+0.70)	+0.76 (+0.75)	0.033 (0.037)
	Hallucinations	+0.79 (+0.75)	+0.80 (+0.78)	0.250 (0.281)
Judge vs. human	Quality	+0.60 (+0.57)	+0.47 (+0.44)	0.330 (0.334)	+0.46 (+0.45)
consensus	Coverage	+0.53 (+0.56)	+0.21 (+0.21)	0.047 (0.049)
	Hallucinations	+0.43 (+0.42)	+0.45 (+0.46)	0.403 (0.375)

reports both.

The judge is moderately calibrated, and it sits below the human ceiling on every dimension. On quality, it reaches $\rho =+0.60$ against a human–human ceiling of $+0.73$, and ICC(2,1) $+0.47$ against $+0.64$. The gap is wider on the count-based dimensions: hallucinations $\rho =+0.43$ against $+0.79$, and coverage $\rho =+0.53$ against $+0.72$. Excluding the eight self-authored items moves nothing of substance (quality $\rho =+0.57$, ICC $+0.44$; coverage $\rho =+0.56$, ICC $+0.21$), so the annotators’ prior exposure to one source artefact is not driving the result.

Two readings of Table 6 pull in opposite directions, and both belong in the record. In absolute terms, the judge is close to human: its mean absolute error against the human consensus on quality, 0.330 points, is smaller than the error between the two humans themselves, 0.354. A judged quality of 4.5 therefore means approximately what a human rater would mean by it. In  ordering terms, it is materially worse than a human, and the coverage row shows the split at its sharpest: MAE is a negligible 0.047, while ICC(2,1) is only $+0.21$, against a human–human $+0.76$. That combination is the signature of a ceiling. Nearly every generation covers nearly everything, so the judge lands on the right level almost automatically while agreeing poorly about which generation covers more than which. This is the same pattern the judge-free checks reported independently: a compressed quality band (Section 4.1), a coverage outlier that is visible to every instrument while the remaining ordering is not (Section 4.10), and a chrF ordering that disagrees with the judge (Section 4.11).

The calibration therefore confirms the paper’s claims at exactly the resolution they were already made, and no further. Judged magnitudes are trustworthy, and so is the one large coverage separation. The fine ordering among closely spaced models is not: a judge that agrees with a human at ICC $+0.47$ on quality cannot adjudicate a 0.03-point gap among the top three, which is the same conclusion the overlapping confidence intervals of Section 4.1 and the re-scoring noise of Section 4.8 reached from other directions. Our result is a mild improvement on the nearest published precedent, where Siddeeq et al. [14] report a Fleiss $\kappa$ near zero among four blinded human raters on a similar task, while the model ordering stayed stable across 15 of 16 comparisons; here the humans agree substantially with each other (quadratic-weighted $\kappa =+0.56$ on quality) and the judge agrees with them moderately ($+0.46$). The shape of the finding is the same in both studies: aggregate orderings survive, and individual scores should not be over-read.

4.10. A Judge-Free Coverage Check on the Requirement List Arm

Because the Requirement List denominator is known, coverage on that one arm can be measured without the judge. We restrict this check to the Requirement List arm because only its enumerated requirements (R1, R2, …) expose an externally fixed denominator. User-story acceptance criteria are not uniformly itemised, and the RFP excerpts are prose. Each Requirement List generation was re-scored by matching every enumerated requirement to the generated scenarios with a transparent idf-weighted lexical rule (Appendix D), counting a requirement as covered when its best-matching scenario clears a similarity threshold.

This judge-free check agrees with the judge on the one thing already visible in Table 2. Gemini 2.5 Pro is the lowest-coverage model at every threshold tried (0.3–0.6), and its deficit widens as the threshold tightens (objective coverage 0.47 versus 0.78–0.93 for the rest at a middle threshold). The agreement does not reach beyond that outlier. The rank correlation with the judge over the eight models is only $\rho \approx 0.24$, and it is carried entirely by Gemini: excluding Gemini, the correlation turns slightly negative ($\rho \approx -0.13$), because the lexical rule rewards reuse of source wording and ranks the well-grounded GPT-4o and GPT-4o-mini highest while the judge ranks the Claude family highest. The two instruments are also not independent in the way orthogonal corroboration would require. A model that paraphrases the requirements more heavily, rather than covering fewer of them, would show the same lexical deficit, and Gemini also has the lowest lexical grounding in the panel (0.34, against 0.44–0.64 for the rest). The lexical measure therefore cannot separate lower coverage from heavier paraphrase, which is the very validity question the judge is meant to answer, and its absolute level is a lower bound because lexical matching misses declarative paraphrasing. On a conservative reading, two surface-sensitive instruments with different failure modes agree only that the single outlier is a genuine outlier. The near-ceiling ordering stays method-dependent, and the human calibration of Section 4.9 confirms rather than resolves this: judge and human agree on coverage at ICC(2,1) $=+0.21$, the weakest agreement measured anywhere in this study, so coverage validity beyond the outlier is not established by any instrument we have.

4.11. Similarity to Hand-Authored Gold Standard, Anchored to a Human–Human Ceiling

The checks so far are either judge-scored or lexical proxies computed against the source. A third instrument is available now that two annotators have hand-authored reference Gherkin. Two annotators independently authored a gold standard for a 30-item stratified subset (10 per input type) following the released annotation guide, without seeing any model output. All 60 gold standard files parse under the official Cucumber parser. This yields a similarity signal in which no LLM participates at any stage: the references are human, and chrF is deterministic.

A reference set is only as good as its coverage of the sources, so we audited that directly rather than assuming it. The authoring guide instructs annotators to exercise every requirement, and both annotators report having done so for all 156 source requirements in the subset. That self-report carries no information, since a compliant annotator can only answer one way, so each annotator additionally coded the other’s gold standard, requirement by requirement, judging whether some scenario in a file they had not written exercises it. An independent reader accepts 153 of 156 requirements as covered in each direction: 306 of 312 pooled, or 98.1% (95% Wilson CI 95.9–99.1%), against the authors’ own claim of 100%. The six gaps share no requirement in common between the two directions, so they are properties of a particular gold standard file rather than requirements that resist expression as a scenario. Coverage is weakest on User Stories (95.0%, five of the six gaps) and complete on the RFP arm (100%), which is a second and independent indication that the constructed procurement excerpts are the most regular of the three corpora, the  same conclusion the chrF ratio of 1.07 reaches below.

We report no inter-annotator $\kappa$ for gold standard coverage, and the reason is structural rather than an omission. Cohen’s $\kappa$ requires two raters labelling the same units; in the cross-coding design, each requirement is coded by exactly one reader, and pairing a reader against the author’s own coding cannot help because that coding has no variance. With two annotators and two reference sets, no configuration yields two non-authors coding the same file. Inter-rater reliability for the coverage construct is instead measured where it can be, on the 72 shared items of the calibration study, where the two annotators agree at $\rho =+0.72$ and ICC(2,1) $=+0.76$ (Section 4.9).

Reference-based similarity is only interpretable against two anchors, and we compute both. The  human–human ceiling is the chrF between annotator A’s and annotator B’s gold standard for the same item: 67.40 overall (65.28 on Requirement Lists, 68.44 on RFP excerpts, 68.47 on User Stories). Two competent annotators, working from identical sources under a shared style guide, agree only this much on the surface, because Gherkin admits many correct phrasings of the same behaviour. The  skeleton floor is the chrF between gold standard references for different, unrelated items, 35.07 over 870 pairs. That floor is high because every feature file shares the same Feature/Scenario/Given/When/Then scaffolding, and it means the entire usable dynamic range of the metric on this task is the 32.3 chrF points between 35.07 and 67.40. Reporting a raw chrF against 100 would be meaningless; we therefore report each model both raw and rescaled onto the floor-to-ceiling interval.

Table 7. Similarity of generated Gherkin to the hand-authored gold standard on the 30-item subset (1197 generations), with the two anchors that make it interpretable. “chrF (multi-ref)” scores against both annotators as multiple references; the 95% CI is a bootstrap over the 30 items. “% of range” rescales the mean single-reference chrF onto the floor-to-ceiling interval, so 0% is as similar to the gold standard as an unrelated feature file and 100% is as similar as a second human’s gold standard. No LLM is involved in any column.

Model	chrF (Multi-Ref)	[95% CI]	% of Range	Judged-Quality Rank
Claude Opus 4.8	72.16	[69.79, 74.56]	107	3
Claude Sonnet 5	69.01	[66.40, 71.65]	100	1
GPT-4o	68.35	[65.80, 70.86]	91	7
GPT-5.5	67.97	[65.57, 70.37]	97	2
Llama-3.1-70B	66.54	[64.04, 69.06]	87	8
GPT-4o-mini	66.12	[63.61, 68.63]	85	6
DeepSeek-V3	64.35	[61.80, 66.82]	80	5
Gemini 2.5 Pro	58.13	[55.46, 60.80]	66	4
Human–human ceiling	67.40		100
Skeleton floor (unrelated items)	35.07		0

gives the result. The models span 58.1 to 72.2 raw, or 66% to 107% of the floor-to-ceiling range, a far wider separation than the 0.36-point judged-quality band. Two models (Claude Opus 4.8 and Claude Sonnet 5) sit at or above the human–human ceiling. That does not mean they write better Gherkin than the annotators; it means their output is lexically closer to the gold standard pair than the two annotators are to each other, which is what one expects from models that track source wording more closely than a human deliberately writing declaratively.

The consequential finding is that this judge-free instrument disagrees with the LLM judge. Over the eight models, the rank correlation between chrF and judged quality on the same items is only $\rho =+0.36$; with judged coverage, it is $\rho =+0.67$. Two models move sharply. Gemini 2.5 Pro falls from fourth on judged quality to last on chrF, corroborating from a third independent direction the coverage outlier already visible to the judge (Table 2) and to the lexical check (Section 4.10). GPT-4o rises from seventh to third. We read this cautiously and in one direction only: it further undermines confidence in the fine judged-quality ordering, but it does not establish the chrF ordering as correct in its place. chrF is a surface metric that rewards lexical proximity to a particular reference, so a model that paraphrases correctly is penalised exactly as one that omits behaviour is, which is the same paraphrase-versus-omission ambiguity we document for the lexical coverage check (Section 4.10) and that Siddeeq et al. [14] report for TF–IDF coverage. Two surface-sensitive instruments disagreeing with a verbosity-sensitive one narrows the space of orderings that could be right; it does not identify which. The human rating study of Section 4.9 is the arbiter, and it does not award the decision to either instrument, which leaves the fine ranking unsettled by design rather than by omission.

One further reading bears on the RFP arm. It is the only input type on which the models exceed the human ceiling (mean chrF 73.21 against a ceiling of 68.44, a ratio of 1.07), against 0.98 for Requirement Lists and 0.92 for User Stories. Author-constructed procurement excerpts appear to be lexically easier to track than field-sampled artifacts, which supports treating that arm as an exploratory probe rather than as evidence about real RFPs (Section 5.1).

4.12. From Parsing to Running: A Cucumber Runner Probe

Every validity number reported so far comes from the official Gherkin parser library operating under our normalisation protocol. A reviewer of BDD tooling should ask a stricter question: would the actual test runner accept these files, and how much glue code would binding them require? We therefore re-ran all 2960 primary-run generations through cucumber-js 13.2.1 in dry-run mode with no step definitions loaded, recording whether the runner loaded the file at all and how many step definition snippets it demanded.

The first result is a substantial gap between parsing and loading, and it is a finding about our own metric as much as about the models. Under our protocol, 98.6% of outputs are parse-valid, but  only 78.0% are loadable by Cucumber. The cause is almost entirely file packaging: 618 generations (20.9%) emit more than one Feature block in a single response, which our normalisation layer splits and parses separately but which Cucumber rejects outright, since it permits exactly one Feature per file. The gap is strongly model-dependent, and it falls hardest on two models that the rest of the study scores well (

Table 8. Cucumber runner probe over all 2960 primary-run generations (cucumber-js 13.2.1, dry run, no step definitions). “Parse valid” is our own protocol; “runner loads” is the fraction the real runner accepts. The gap is multi-Feature packaging, which our normalisation layer tolerates and Cucumber does not. “Defs/scenario” is the number of distinct step definitions the runner demands, computed on the generations that loaded.

Model	Parse Valid (%)	Runner Loads (%)	Multi-Feature (%)	Defs/Scenario
GPT-5.5	100.0	99.7	0.3	3.74
GPT-4o	98.9	98.6	0.3	3.19
GPT-4o-mini	98.9	94.9	4.1	3.11
Llama-3.1-70B	97.3	91.1	6.2	3.26
DeepSeek-V3	96.5	74.6	23.2	3.40
Claude Sonnet 5	99.7	73.5	26.5	3.51
Gemini 2.5 Pro	98.4	50.8	47.6	3.89
Claude Opus 4.8	99.5	40.8	58.9	3.26
All	98.6	78.0	20.9	3.40
Human gold standard (A+B)	100.0	100.0	0.0	3.17

): Claude Opus 4.8 loads 40.8% of the time and Gemini 2.5 Pro 50.8%, against 99.7% for GPT-5.5 and 98.6% for GPT-4o. This is a mechanically repairable defect, and we deliberately do not credit models for the repair, because the reviewer’s question is what happens when the artefact is handed to the runner as produced. It is nonetheless a second instance of this paper’s methodological theme: as with the output-token budget (Section 4.6), a benchmarking convenience, here, a multi-block-tolerant parse harness, silently flatters a fifth of the corpus.

Conditional on loading, the second result is that the integration cost is essentially model-invariant. The runner demands 3.11 to 3.89 distinct step definitions per scenario across the eight models (coefficient of variation 0.081), and the human-authored gold standard sits inside that range at 3.165. A pure-Python replication of the same quantity by step-pattern normalisation tracks the runner at $\rho =+0.98$ over 2309 generations. Step-phrase reuse within a feature is low for every model (0.054–0.128) and lower still for the human annotators (0.039), so neither models nor humans factor a shared step vocabulary. Binding generated Gherkin to step definitions therefore costs about what binding human-authored Gherkin costs.

Anchoring bindability to the humans makes the same point with an explicit ceiling. Scoring each model’s step patterns against one annotator’s library at a time (the way the human–human ceiling itself is computed), two independent annotators reuse only 4.1% of each other’s step phrasings exactly, and 28.6% at a token-level Jaccard threshold of 0.6. Models reach 44–172% of the exact ceiling and 28–101% of the fuzzy ceiling. Model step vocabulary is, in other words, about as idiosyncratic as a second human’s.

We are explicit about what this probe does not show. No step definition was implemented and no system under test exists, so nothing here licenses a claim that any generated scenario would pass. What it measures is runner acceptance and the size of the glue-code obligation. On  those two axes, the conclusion is that the portability risk in LLM-drafted Gherkin lies in file packaging rather than in step-definition volume. None of the three measures correlates significantly with judged quality across the eight models (all $p>0.2$), which, with only eight clusters, is underpowered but is what the data show: these are orthogonal axes, not a proxy for the judge.

4.13. Which Rubric Dimensions Separate the Models?

The judged quality reported so far is the mean of four rubric dimensions (relevance, clarity, completeness, and singularity; Appendix A), and the primary judge cache retained only that mean. To see where the quality differences lie, we re-scored the rep-0 subsample (592 generations, all eight models over the 74 items) with the same Claude Sonnet 5 judge, retaining the four dimensions. The re-scoring reproduced each model’s pooled quality within 0.03 points (the largest per-model deviation, comfortably inside the judge’s test–retest noise of Section 4.8), so the dimensions decompose the same quality signal as Table 2.

Table 9. Mean per-dimension quality (1–5) per model from re-scoring the rep-0 subsample (592 generations) with the primary Claude Sonnet 5 judge, ordered by pooled judged quality (Table 2). Relevance is saturated across the panel. The frontier lead concentrates in clarity and completeness, and completeness is the lowest-scored dimension for every model.

Model	Relevance	Clarity	Completeness	Singularity
Claude Sonnet 5	4.89	4.64	4.43	4.72
Claude Opus 4.8	4.97	4.61	4.28	4.73
GPT-5.5	4.92	4.73	4.36	4.61
Gemini 2.5 Pro	4.74	4.78	4.04	4.65
DeepSeek-V3	4.81	4.26	4.08	4.45
GPT-4o	4.93	4.04	3.91	4.54
GPT-4o-mini	4.82	4.08	3.92	4.51
Llama-3.1-70B	4.74	4.09	3.81	4.49

gives the per-model dimension means.

The four dimensions are far from equally discriminating. Relevance is saturated across the panel (4.74–4.97), with every model producing on-topic scenarios. Singularity is nearly the same (4.45–4.73). The frontier lead instead concentrates in clarity (frontier mean 4.66 versus 4.12 for the low-cost tier, a 0.54-point gap) and completeness (4.36 versus 3.93, a  0.43-point gap), and completeness is also the lowest-scored dimension for every model (3.81–4.43), the one axis with real headroom. The overall-quality ceiling of Section 4.1 is thus partly an averaging artefact. Pooling a saturated dimension (relevance) with two that still separate the models compresses the visible spread, and a rubric weighted towards clarity and completeness would discriminate more. The mean also hides within-model structure. GPT-4o scores near the top on relevance (4.93) but near the bottom on clarity (4.04), and  Gemini 2.5 Pro has the highest clarity (4.78) yet completeness (4.04) as its own weakest dimension, consistent with its coverage deficit (Section 4.1). These dimension-level readings are, like the aggregate, judge-scored.

4.14. Prompt Sensitivity

Because a benchmark that reports a model ranking should know how far that ranking depends on the prompt, we re-ran a 24-artifact subset (eight per input type, one repetition, all eight models, and a primary judge) under two alternative prompt templates (a few-shot variant and a terse variant) and compared per-model judged quality against the base prompt on the same items. The  frontier models remained at or near the top under all three prompts. Claude Sonnet 5, GPT-5.5, and Claude Opus 4.8 form the top three under the base and terse prompts, and under the few-shot prompt, Gemini 2.5 Pro displaces Opus for third. The fine ordering, however, is prompt-sensitive. Agreement across the three prompts is only moderate (Kendall’s $W=0.67$; base–few-shot Spearman $\rho =0.85$, base–terse $\rho =0.52$, few-shot–terse $\rho =0.16$), with  most of the movement in the compressed mid and lower tier. Consistent with prior reports that the best prompt is model-specific [4,7], the per-model rankings should be read as prompt-conditional, while the coarse frontier-versus-rest structure is more stable than the exact order. Two caveats bound the probe. It is deliberately small (one prompt sweep on a subset), and with a single repetition per cell, the prompt effect is partly confounded with the run-to-run stochasticity documented in Section 4.4 (content stability is low), so the weakest correlation (few-shot–terse $\rho =0.16$) is likely inflated by noise rather than a pure prompt effect. The probe is intended to bound, not eliminate, prompt sensitivity.

A common-prompt versus model-specific condition. A shared prompt buys experimental control at the price of answering a narrower question, namely how the models compare under that prompt. Since the best prompt is known to be model-specific [4,7], a benchmark that reports a ranking should also report what happens when each model is allowed its own prompt under an equal tuning budget. The three templates above supply exactly that, at no additional generation cost: the budget is three candidate templates per model, identical for every model, with no per-model prompt engineering beyond the choice among them. We compare a common condition (every model under the base template) against a model-best condition (every model under whichever of its three templates scores highest). Because the repetition budget differs across templates, the primary comparison uses repetition 0 only for all three; the five-repetition base variant is a sensitivity check and agrees throughout.

Choosing the maximum of three noisy estimates and then reporting that maximum is upward-biased, so we report model-best twice. Selecting and evaluating on the same 24 items gives a naive mean gain of $+0.034$ quality points. Cross-fitting the selection (choose the template on a stratified half of the items, evaluate it on the held-out half, swap, average over 400 random splits) gives a mean realised gain of $-0.012$ points. The apparent benefit of per-model prompt selection is therefore entirely a winner’s curse: at this tuning budget, on this task, a  practitioner following the procedure would gain nothing, and five of the eight models nominally prefer a template other than the common one purely as an artefact of selection noise.

The ranking is correspondingly stable. Between the common and model-best conditions, Spearman $\rho =+0.95$ and Kendall $\tau =+0.86$ under cross-fitting ($\rho =+0.97$, $\tau =+0.91$ naive), and the top three are identical under both. We therefore retain the common-prompt protocol for the main analysis, now with evidence rather than assumption behind it, while noting the limitation the design cannot escape: a budget of three fixed templates is small, and genuine per-model prompt engineering with a larger search space might yet reorder the panel. What we can say is that the objection does not bite at the budget tested, and that reporting the naive maximum, as an unwary tuning study would, would have manufactured a gain that does not exist.

4.15. Panel Refresh: Two Current Efficient Models

A benchmark of a fast-moving field dates quickly, and a panel assembled at one moment can understate what the efficient tier offers. We therefore re-ran the full protocol for two models that were already available when the primary run was collected but were not in it: Gemini 3.5 Flash (generally available 19 May 2026) and DeepSeek-V4-Flash (released 24 April 2026; the provider slug resolves to that build). Both were run on the same 74 artifacts with the same base prompt, the same 6000-token budget, temperature 0, five repetitions, and the same primary judge, for 740 further generations with no API errors.

The two batches are not simultaneous, and we do not pool them silently. The primary run was collected on 3 July 2026 and the refresh on 22 August 2026, so provider snapshots may differ and the pricing snapshots carry separate dates (Appendix B). Any refresh-versus-primary difference below is therefore confounded with batch, and we report it as a scope check on the study’s conclusions rather than as a controlled model contrast.

Both new models are competitive, and one changes the picture materially (

Table 10. Panel refresh. The two models marked * were generated on 22 August 2026 under the primary protocol; the remaining eight are the primary run of 3 July 2026, repeated here for comparison. “Runner” is the fraction of raw outputs the Cucumber runner loads (Section 4.12); “Multi-F.” is the fraction emitting more than one Feature block. P(non-dom.) is Pareto non-domination probability recomputed over all ten models, so it is not comparable to the eight-model values in Table 3. Judged quality remains an LLM-judge score from an instrument calibrated on the primary run (Section 4.9), where it resolved close orderings poorly.

Model	Parse (%)	Runner (%)	Multi-F. (%)	Quality	Coverage	Cost ($/gen)	P(non-dom.)
* DeepSeek-V4-Flash	97.3	85.1	25.7	4.42	0.966	0.00014	1.00
GPT-4o-mini	98.9	94.9	5.4	4.35	0.982	0.00022	0.02
Llama-3.1-70B	97.3	91.1	10.8	4.32	0.965	0.00026	0.00
DeepSeek-V3	96.5	74.6	40.5	4.38	0.987	0.00031	0.07
GPT-4o	98.9	98.6	1.4	4.37	0.988	0.00362	0.02
Claude Sonnet 5	99.7	73.5	41.9	4.68	0.999	0.01359	1.00
* Gemini 3.5 Flash	97.3	97.3	0.0	4.53	0.983	0.01424	0.06
Gemini 2.5 Pro	98.4	50.8	59.5	4.52	0.913	0.01950	0.00
Claude Opus 4.8	99.5	40.8	64.9	4.66	0.998	0.02089	0.10
GPT-5.5	100.0	99.7	1.4	4.65	0.991	0.03423	0.12

). DeepSeek-V4-Flash displaces the entire low-cost tier. It is cheaper than the previously cheapest model ($0.00014 versus $0.00022 per generation) and scores higher on judged quality (4.42 versus 4.35), so it is non-dominated in every bootstrap replicate and dominates Llama-3.1-70B (probability 1.00), GPT-4o-mini (0.98), GPT-4o (0.97), and DeepSeek-V3 (0.93). Recomputing the Pareto front over all ten models shrinks it from three members to two: {DeepSeek-V4-Flash, Claude Sonnet 5}, with both GPT-4o-mini and DeepSeek-V3 dropping off. Gemini 3.5 Flash strictly improves on its predecessor on every axis we measure: judged quality 4.53 versus 4.52, coverage 0.98 versus 0.91, cost 27% lower, latency less than half (8.3 s versus 17.5 s), and runner acceptance 97.3% versus 50.8%. It nonetheless remains dominated by Claude Sonnet 5 in 94% of replicates, which is cheaper and higher-scoring still.

The refresh also sharpens the packaging finding of Section 4.12. The multi-Feature defect that made Gemini 2.5 Pro the second-worst model for runner acceptance is essentially absent in its successor: Gemini 2.5 Pro emitted several Feature blocks in 59.5% of generations, Gemini 3.5 Flash in 0.0%, and every parse-valid Gemini 3.5 Flash output loads in the runner. DeepSeek improves more modestly, from 40.5% to 25.7% (runner acceptance 74.6% to 85.1%). The  defect is therefore substantially a property of model vintage rather than an inherent feature of LLM-drafted Gherkin, which strengthens rather than weakens the methodological point: a parse harness that tolerates it will misreport whichever models happen to exhibit it.

One caution transfers directly from Section 4.6. Two DeepSeek-V4-Flash generations terminated at exactly the 6000-token ceiling with finish_reason “length” and returned no visible output at all, having spent the entire budget on hidden reasoning tokens. That is only two of 370 generations, but it shows the output-token budget that was generous for the July panel is not automatically generous for a newer reasoning-capable model, and  it is precisely the failure mode the 1500-token batch exhibited at scale.

The gold standard references of Section 4.11 also let us score the refresh panel with an instrument in which no LLM participates, which matters here because every quality and coverage column of Table 10 otherwise rests on the same judge (Section 4.9), which was calibrated on the primary run rather than on this batch. Both new models were run on the 30-item gold standard subset (150 generations each), so this check required no further generation. DeepSeek-V4-Flash reaches a multi-reference chrF of 68.41 (95% CI [66.18, 70.57]), or  92% of the floor-to-ceiling range, and Gemini 3.5 Flash 64.82 ([62.86, 66.67]), or 82%. Read against the eight-model values of Table 7, and subject to the batch caveat above since those were collected seven weeks earlier, DeepSeek-V4-Flash would sit behind only the two Claude models: the cheapest model in the study is also among the closest to human-authored gold standard. The generational improvement is visible on this axis too, with DeepSeek-V3 rising from 64.35 to 68.41.

The sharper result is a comparison that carries no batch confound at all, because both models come from the same 22 August batch and were scored on the same items under the same protocol. On those 30 items, the two instruments disagree, significantly and in opposite directions. chrF places DeepSeek-V4-Flash above Gemini 3.5 Flash by 3.59 points, higher on 23 of 30 items (Wilcoxon signed-rank, $p=0.0004$). The judge places it below, by 0.11 points on the five-point scale and higher on only eight of 30 items ($p=0.036$; judged means restricted to the gold standard subset are 4.39 and 4.50, against the 4.42 and 4.53 of the full 74-artefact panel in Table 10). The  disagreement is therefore not an artefact of comparing across batches, across item sets, or across aggregation choices. It is two instruments ordering the same generations oppositely, and the judge-favoured ordering is the weaker of the two effects.

This also reproduces, in an independent batch and with a newer model, the Gemini-specific pattern of Section 4.11, where the judge ranked Gemini 2.5 Pro fourth on quality while chrF ranked it last. Here the judge again prefers the Gemini output and chrF again does not. That the pattern survives a change in model generation suggests a systematic interaction between the judge and Gemini’s output style rather than a property of that one outlier. The caution of Section 4.11 applies unchanged. What this establishes is that for these two efficient models, the choice of instrument, not only the choice of model, decides which one a reader would pick.

What survives the refresh is the study’s structure, not its specific model names. Cost remains the axis that discriminates, judged quality remains compressed near its ceiling (the ten-model span is 4.32–4.68, unchanged), the frontier premium still buys no discriminable aggregate quality, and the low-cost tier remains the rational default for first-draft generation. What does not survive is any particular low-cost recommendation: within seven weeks, the cheapest non-dominated model changed. We therefore restrict every model-specific claim in this paper to the exact models, providers, snapshots, and dates recorded in Appendix B, and read the tier-level conclusions, not the per-model ordering, as the transferable result.

5. Discussion

Three findings stand out, and the last is methodological (the input-type effect of RQ2 and the stability findings of RQ4 are taken up in the model-choice and prior-work discussion that follows).

First, model price and recency do not buy discriminable Gherkin quality. The frontier systems did top the judged-quality ranking, but by a margin that sits inside the judge’s own re-scoring noise and inside the overlapping confidence intervals of Section 4.1. A length control further attributes a substantial part of it to the judges’ verbosity preference rather than to quality (Section 4.7). On the judge-independent axes of parse validity and well-formedness, all eight models cluster near the ceiling, though runner acceptance separates them sharply (Section 4.12), so “validity does not discriminate” is a statement about the parser, not about deployability. The practical implication is not that the frontier is worse but that aggregate judged quality is the wrong axis on which to choose a model for this task. The per-dimension decomposition qualifies this. The frontier does separate from the low-cost tier on clarity and completeness, the two non-saturated rubric dimensions (Section 4.13), a gap the blunt mean-of-four aggregate hides. That separation is judge-scored and still provisional, but it means the cost steer is firm only where a human reviewer absorbs any clarity or completeness shortfall. In high-volume, light- or no-review use, the provisional frontier edge re-enters. The axes that do separate objectively or near-objectively point away from the most expensive models, and they are not equally hard evidence. Cost is fully objective, whereas coverage is judge-inferred (Section 5.1), and once the judge is removed, the coverage separation collapses to the single Gemini outlier (Section 4.10). The  discriminating stability is likewise output-volume stability under mixed decoding regimes, with content-level reproducibility being low for every model in the panel (Section 4.4). We rest the model-choice argument most firmly on cost and treat coverage and stability as provisional. This tempers the assumption that adopting the latest, priciest flagship materially improves BDD generation.

Second, low-cost and open-weight models are strong, economical substitutes. DeepSeek-V3, Llama-3.1-70B, and GPT-4o-mini reached near-ceiling parse validity and judged quality within the band of the top models at a cost one to two orders of magnitude lower, and GPT-4o-mini is on the Pareto front in every bootstrap replicate (Table 3). They also need the least post-hoc repair, since far more of their raw output loads in the Cucumber runner than the frontier’s (Section 4.12). Because judged quality barely varies across the panel, that cost advantage is close to free for scale generation or a reproducible pipeline, and the near-ceiling objective metrics support it independently of the judge. The one construct the cheaper models rarely produced under the base prompt was the Scenario Outline/Examples table, but a permissive prompt elicited it from every low-cost model except GPT-4o-mini (Section 4.5), so this is a prompting gap rather than a capability one. On Background usage, the open models matched or exceeded the frontier. Teams wanting data-driven scenarios should simply prompt for the construct. Among the proprietary options, GPT-4o pairs the lowest latency and near-perfect count-determinism (both objective) with the lowest invented-requirement rate in the panel (Section 4.5), a reading that rests on the hallucination count rather than on the copy-sensitive grounding score. The hallucination signal is itself judge-derived, and the calibration of Section 4.9 finds it the dimension on which judge and human diverge most in rank ($\rho =+0.43$, against a human–human $+0.79$). We therefore report GPT-4o’s profile as a hypothesis, not a single-model recommendation, and  leave model choice to the priority-conditional guidance below.

Third, and most consequentially, an apparent model “failure” proved to be an artifact of measurement design rather than a property of the model. An initial run with a 1500-token output budget showed Gemini 2.5 Pro producing one third as many scenarios as the panel and covering fewer than half the requirements, and GPT-5.5 as the least valid OpenAI model. Both effects disappeared at a fair 6000-token budget (Section 4.6). Because  coverage and completeness are precisely the metrics a truncation confound corrupts, this is a cautionary result for LLM benchmarking generally. A benchmark that does not control the output budget can rank a capable model last, and no single benchmark number should be treated as a stable property of a model. The episode does caution against treating “produces valid Gherkin” as sufficient evidence of usefulness (a scenario file can parse cleanly while omitting most of the specified behaviour), but the sharper lesson is the converse. An  apparent completeness deficit may be an artefact of the measurement rather than the model.

What the benchmark establishes today. The durable findings above are judge-independent; the provisional ones are the judged-quality ordering, the coverage-based separation, and any single-model recommendation, which we do not make. The human calibration of Section 4.9 bounds that provisional set rather than lifting it.

Table 11. Practitioner model-choice guide by team priority, distilling the findings of Section 4. Rows differ in evidential status, and the difference is load-bearing. Runner acceptance, Pareto dominance, validity, latency, cost, and output-volume determinism are objective. Cells resting on judged quality, coverage, or hallucination are LLM-judge signals from an instrument that agrees with human raters only moderately, and well below the human–human ceiling, on all three (Section 4.9); they should be treated as provisional. Grounding is a deterministic but unvalidated lexical heuristic. Similarity to the hand-authored gold standard is judge-free but surface-sensitive (Section 4.11). In every row, a human should be kept in the review loop: the calibration licenses the judge’s score levels, not its fine orderings. The rows name the eight primary-run models; the panel refresh of Section 4.15 shows the tier-level guidance is stable but that individual names date fast, so the priority column should be read rather than the model column as the durable part.

Team Priority	Suggested Model(s)	Principal Caveat
General first-draft generation (default)	Any low-cost model: GPT-4o-mini, DeepSeek-V3, Llama-3.1-70B	All draft valid Gherkin at near-ceiling judged quality. The per-draft dollar difference among them is immaterial once a human reviews them, so the choice should be based on convenience or ecosystem fit. GPT-4o-mini and Llama also need file-splitting least often (Section 4.12).
Drop-in runner readiness (objective)	GPT-5.5 (99.7%), GPT-4o (98.6%), GPT-4o-mini (94.9%)	Fraction of raw outputs the Cucumber runner loads without repair. Claude Opus 4.8 (40.8%) and Gemini 2.5 Pro (50.8%) emit several Feature blocks in one file and need a splitting step before any runner accepts them. This is objective, mechanically repairable, and invisible to parse-validity alone.
Best fidelity signal (provisional)	GPT-4o, GPT-4o-mini (cluster)	Lowest-hallucination cluster (GPT-4o 0.02 invented per output), but provisional: hallucination is judge-scored, and it is the dimension on which the judge and human ranked outputs most differently ($\rho =+0.43$ against a human–human $+0.79$, Section 4.9). Grounding (GPT-4o 0.64) is a copy-sensitive lexical floor, not a quality merit.
Data-driven scenarios (Scenario Outline)	Any model except GPT-4o-mini, prompted permissively	Under the base prompt, only frontier models emit Scenario Outline, but a permissive prompt elicits it from most low-cost models too in a 24-item probe (Llama 100%, DeepSeek 29%, GPT-4o 13%; GPT-4o-mini still 0%): prompt for it explicitly.
On-prem/data governance	Open-weight: Llama-3.1-70B, DeepSeek-V3	Self-hostable, keeping requirements and RFP text off third-party APIs. Same Scenario Outline caveat as the data-driven row.
Minimal cost at scale	DeepSeek-V4-Flash if available (Section 4.15), else GPT-4o-mini, then DeepSeek-V3, Llama-3.1-70B	The dollar edge dominates only in high-volume, light- or no-review pipelines (unlike the default row, where review cost washes it out). With light review, the provisional frontier clarity/completeness edge (Section 4.13) re-enters. Same Scenario Outline caveat as the data-driven row.
Output-volume determinism	GPT-4o	Lowest scenario-count SD (0.16), though content-level reproducibility is low across the whole panel.
Low-latency/interactive use	GPT-4o (3.0 s), GPT-4o-mini (5.6 s)	Fastest in the panel, but latency is a single-session wall-clock through non-uniform API routes (Section 5.1), so it should be read as indicative. The reasoning-heavy GPT-5.5 (13.5 s) and Gemini 2.5 Pro (17.5 s) are the slowest.
Maximum quality, cost no object	Claude Sonnet 5	Of the frontier tier, it is the only Pareto-non-dominated member: it dominates Claude Opus 4.8 in 90% of bootstrap replicates and GPT-5.5 in 87%, so the two costlier models buy nothing measurable (Table 3). The quality lead itself is small, direction-only, within judge re-scoring noise, and not separable from length (Section 4.7), so the premium is not firmly established (judge-scored; the judge resolves gaps this small poorly, Section 4.9). It also needs file-splitting for 27% of outputs.
Not recommended on current evidence	Gemini 2.5 Pro	The only model dominated in every bootstrap replicate (Claude Sonnet 5 gives higher judged quality at 70% of the cost). It is also lowest on requirement coverage, lowest on similarity to hand-authored gold standard (66% of the human ceiling), the least stable, the slowest, and the second-worst on runner acceptance. Four instruments with different failure modes agree.

distils both into a priority-driven guide. GPT-4o appears in several rows because it leads on those axes, not as an endorsed default; it is kept out of the low-cost default only on price (about 17× GPT-4o-mini), so a team for which fidelity or latency dominates should prefer it. A different priority (data-driven scenarios, on-premises hosting, or minimal cost at scale) points elsewhere.

Bottom line for practitioners. What can be acted on rests on the judge-independent findings. Any of the eight models drafts a syntactically valid, well-formed Gherkin, so a low-cost or open-weight model (GPT-4o-mini, DeepSeek-V3, Llama-3.1-70B) is a rational default for first-draft generation, and in-house evaluations should control the output-token budget. This cost comparison is across models, not against a human writing Gherkin, which the study does not measure. What cannot be acted on is the fine quality or coverage ranking, or any claim that the drafts are good enough to reduce human review: those rest on a judge that resolves close orderings poorly (Section 4.9) and on a human-authoring comparison we do not make. A human should be kept in the review loop, which the calibration supports rather than merely defers.

Relation to prior work. Our design corroborates and extends the closest prior study, Rathnayake et al. [4], who benchmarked three frontier models on user stories and requirement descriptions and found the Claude family preferred by human and LLM judges while GPT-4 led on similarity metrics. We likewise place the Claude models at the top on judged quality. This is partial independent support for their result. Three off-panel judges reproduce the Claude-family lead (Section 4.8), which argues against literal self-recognition. The corroboration is only partial, however, because all four judges are LLMs that share a verbosity preference (Section 4.7) and Rathnayake et al.’s own signal is part human, part LLM. Our own human calibration (Section 4.9) supplies what was missing on our side without settling the comparison: it shows that an LLM judge can be trusted for score levels but not for the close orderings on which a top-tier claim of this kind depends. We extend the comparison to eight models, a third input type (constructed RFP-style excerpts), and a released dataset and prototype. Our input-type result refines theirs. Whereas they found richer requirement descriptions produced better scenarios than terse user stories, we find Requirement Lists to be the hardest arm for coverage, consistent with the view that enumerated atomic requirements raise the completeness bar (every item must be covered) more than narrative user stories or the more discursive RFP excerpts do. The broader requirement-quality literature and standards likewise link requirement phrasing and granularity to downstream defect risk and to the verifiability and singularity criteria our rubric echoes [35,36]. Our Gemini result is more nuanced than Fernandes et al. [7], who reported Gemini offering the best accuracy–stability balance on free-form test descriptions. On structured requirements to Gherkin, Gemini reached mid-pack judged quality but had the lowest coverage and the highest run-to-run variability in the panel, which suggests its relative standing is sensitive to both input format and the choice of evaluation metric.

5.1. Threats to Validity

The most important threat is judge validity, and three concerns compound. (i) The primary judge (Claude Sonnet 5) is itself an evaluated model whose own family scores highest. Re-scoring all generations with three off-panel judges rules out literal self-recognition, but not the shared verbosity bias common to LLM judges [22,23], for the reason set out in Section 4.8. (ii) A length control (Section 4.7) shows the frontier quality lead is real in direction but partly inflated by that verbosity preference, and the top ordering lies within the judge’s own re-scoring noise (the item-level confidence intervals of the top models overlap; Section 4.8). We therefore make no strong claim on the quality axis. (iii) The judge is now calibrated against two blinded human raters (Section 4.9), and the outcome bounds this concern rather than dispelling it. Judge–human agreement falls below the human–human ceiling on every dimension and is weakest of all on coverage. A judge-free similarity check against the hand-authored gold standard (Section 4.11) orders the models differently from the judge, which deepens this concern without resolving it, since chrF has its own surface bias. All three concerns bear only on the judge-scored metrics (coverage and quality) and not on syntactic validity, well-formedness, scenario count, stability, or cost, which are computed objectively and on which our strongest claims rest. Coverage, though also judge-scored, is far more repeatable than quality (test–retest SD $<0.01$ versus 0.096) and discriminates more sharply, but it too (being inferred by the judge, and for prose inputs inferred against the judge’s own reading of the requirement set) is precisely the dimension on which judge and human agreed least once measured (Section 4.9). The judged scores show a pronounced ceiling (quality 4.32–4.68, coverage 0.91–1.00), so top-of-table differences are suggestive, not decisive. This ceiling may in part reflect the limited resolving power of a five-point, mean-of-four quality rubric on a task all eight contemporary models handle competently, rather than genuine model parity. Decomposing the rubric (Section 4.13) supports this, since the ceiling is driven by a saturated relevance dimension while clarity and completeness still separate the models, so a finer or deliberately harder-calibrated rubric might separate models that appear tied here. We therefore treat “judged quality does not discriminate” as a joint statement about the models and the instrument, and flag this rubric-ceiling possibility as a limitation of the quality metric itself.

Further threats apply. The public input corpora may overlap with model pretraining data, which could inflate absolute scores. Because all models share identical inputs, this does not uniformly bias the relative comparison, but different models may have memorised different fractions, so a residual differential-contamination effect on the ranking cannot be excluded. Absolute numbers accordingly warrant caution, and the constructed RFP arm partially mitigates leakage for that input type. That RFP arm is authored for this study rather than sampled, so its external validity is limited. The fact that it was the easiest arm may partly reflect these excerpts being cleaner and more self-contained than sampled procurement documents, which would inflate their scores relative to field artifacts. Three frontier models (GPT-5.5, Claude Opus 4.8, Claude Sonnet 5) could not accept a fixed temperature and ran with provider-default adaptive reasoning, so their determinism is not directly comparable to the temperature-0 models. We report this asymmetry rather than force an unsupported setting, and it is itself an operational finding, though Gemini 2.5 Pro, which accepts a temperature of 0 yet was the least stable model, shows that temperature setting alone does not explain the stability differences. Generation latency is likewise only indicative rather than a clean model property: it is a single-session wall-clock measured through non-uniform API routes (OpenAI and Anthropic natively, the  others via an OpenRouter proxy), so it conflates model inference speed with routing, queueing, and  network variance, and the proxied models show heavy tails. We therefore read the latency ordering as indicative under our measurement conditions rather than as a benchmarked inference-speed ranking. The primary run used a single prompt template. A robustness probe with two alternative templates (Section 4.14) indicates that the coarse frontier-versus-rest structure is prompt-stable while the fine per-model ordering is not, so the rankings are prompt-conditional. Additionally, the 25 PURE and 25 Dalpiaz artefacts were selected deterministically (source documents and their requirement chunks sorted, then taken in a fixed round-robin across documents until the per-type target was reached) rather than randomly. This is reproducible and avoids arbitrary hand-picking, but it is not a probability sample. The item-level inference we report (bootstrap CIs over items, crossed item random effects, item-clustered tests) should therefore be read as conditional on these 74 artefacts, with the five repetitions as the stochastic element, rather than as licensing generalisation to the PURE or Dalpiaz populations. Representativeness of the input sets is not guaranteed, and the two batches (run1/run2) may differ in provider model snapshots as well as in token budget. The token-budget claim rests on the directly observed truncation mechanism rather than on a cross-batch score comparison. Finally, the study is English-only. Because Gherkin supports localised keywords and BDD is practised across languages, generalisation to non-English requirements and scenarios is untested.

5.2. Limitations and Future Work

The LLM judge is now calibrated, and the limitation is what the calibration found. Section 4.9 reports the blinded two-rater study earlier drafts listed as outstanding, and Section 5.1 sets out what it bounds and what it leaves standing.

The calibration has its own limits, and they bound how far the result travels. It rests on 72 generations rated by two annotators: enough to measure the human–human ceiling and locate the judge against it, but not to estimate calibration per model or per input type, which is where a reader most wants it. The annotators are the students who authored the gold standard rather than practising BDD engineers, and both scored against the same rubric the judge is given, so a rubric that systematically misframes Gherkin quality would bias human and judge alike, and this design could not detect it. A larger panel of practitioners rating against an independently developed rubric is the natural next step, and the released packet, rubric, and agreement scripts are built to be re-run that way. Our raters agree substantially with one another where the expert panel of Siddeeq et al. [14] did not (Section 2), yet the same caution survives in both studies. Aggregate orderings are more trustworthy than the individual scores beneath them.

A second gap is executability, which this revision narrows but does not close. We still implement no step definitions and run no system under test, so we cannot report whether scenarios pass. “Valid, well-formed, runner-loadable Gherkin” remains a proxy for a testable acceptance specification, not a guarantee of one. Closing the gap means binding steps and executing them against a real application, as in the industrial studies of Ferreira et al. [5] and Fonseca et al. [6].

A third gap is that RQ2 compares corpora, not formats, and no reanalysis of this design can change that. The decisive experiment is a within-item one: take a single set of underlying requirements, express each of them as a requirements-list entry, as a user story, and as procurement prose, and  compare the Gherkin each elicits for the same item. This design holds subject matter, granularity, and difficulty fixed and isolates the format effect our corpus cannot. We regard it as the most valuable single follow-up to this study, and it is the one we intend to run next. Two further extensions are natural: sampling genuine procurement prose to replace the constructed RFP arm (whose being author-written is its chief limitation, and which the similarity analysis of Section 4.11 shows to be lexically easier than field-sampled text), and  extending the corpus beyond English to test whether the model and corpus effects hold under Gherkin’s localised keywords. A complementary practical dimension, salient for the procurement setting the RFP arm evokes, is data governance. Open-weight models can be self-hosted so that proprietary requirement text need not leave the organisation, a criterion our cost metric does not capture but that often dominates real model-selection decisions. Because  the full reproduction package is openly archived on Zenodo [12], adding models, prompt variants, alternative judges, or the gold standard subset is a configuration change rather than a full re-run.

6. Conclusions

We presented an open, reproducible benchmark of large language models drafting Gherkin behavioural specifications from three requirement corpora, and released the prototype, the 74-artefact corpus, the hand-authored gold standard, and the resulting dataset of 3700 generations (2960 primary, 740 refresh).

Three findings are judge-independent and carry the paper. First, uniform syntactic validity is partly an artefact of measurement, because it holds under a documented normalisation layer while only 78% of the same outputs load in the actual Cucumber runner. A tolerant parse harness can therefore overstate real runner acceptance by twenty points. Second, cost is the one fully objective discriminator, spanning more than two orders of magnitude while judged quality spans 0.36 points, and framed as a Pareto problem, only three of the eight models are non-dominated. Third, an output-token limit set too tight manufactures spurious failures that vanish at a fair budget, so a reproducible benchmark must control the output budget as deliberately as the model and the prompt.

Two further methodological results concern the evaluation protocol. Granting every model its own best prompt under an equal three-candidate budget produced no cross-validated gain at all, so per-model prompt tuning reported without held-out evaluation manufactures a gain that does not exist. And a panel refresh (Section 4.15) found the structure holding while the model names did not, which is why we scope every model-specific claim to the snapshots and dates of Appendix B and offer the tier-level findings as the transferable result.

Judged quality did not discriminate. The models sat within a 0.36-point ceiling band, and the ordering fell within the judge’s own re-scoring noise. Two independent checks sharpen rather than resolve this. Scored against a Gherkin that was hand-authored by two annotators, the models order differently from the judge, and a blinded two-rater calibration (Section 4.9) places the judge close to human on score levels but well below the human ceiling on ordering. The judged metrics are therefore dependable for coarse comparisons and not for a ranking, which is the resolution at which we state every judged claim.

We report the corpus comparison as a comparison of three specific corpora and not of requirement formats, for the reason given in Section 3.1.

For teams adopting LLM-assisted BDD, these results argue for choosing models on cost and runner acceptance rather than on headline capability or judged quality. Runner acceptance deserves particular weight, because unlike judged quality, it directly determines whether a draft can be handed to a test runner without repair. We make no single-model recommendation. A low-cost or open-weight model is a rational default for first-draft use, and cost differences within that tier matter chiefly at a high volume with light review, since review labour otherwise dwarfs the sub-cent API cost.

Finally, the benchmark measures the draftability of Gherkin rather than its executability. Closing that gap and re-running the calibration with a larger and more expert rating panel are the two extensions the released infrastructure most directly supports.