Evaluating Generated Old English: A Dependency-Based Method with Pre-Trained Word Embeddings

Abstract

This paper raises a methodological question: How can we assess machine-made Old English when there is no parallel reference text and the standard metrics do not fit the task? We propose a pipeline with five measuring layers plus two compliance components, including lexical attestation with form linking, frequency-profile diagnostics, character-level comparison, word-embedding geometry under a verified mapping and dependency parsing, with bootstrap confidence intervals around the main contrasts. We apply the pipeline to a machine-made version of Gregory’s Dialogues: 4217 sentences, one for each sentence of the Old English original, generated under hard constraints. The unattested residue is two word types and 0.003% of tokens. The frequency profile diverges from the original by 0.008, less than the original diverges from the background corpus. At character level, in embedding space and in parsed syntax, the generated text stands at the same distance from the Dictionary of Old English Corpus as the original itself does. We propose an overall metric G, the geometric mean of seven bounded components, which scores the text at 0.991 with a confidence interval of [0.991, 0.992]. Two blind detection experiments with expert judges place the index externally: roughly two-thirds of generated sentences pass as authentic to specialists, so the divergence the pipeline measures is real at corpus scale but not available to sentence-by-sentence reading. The main contribution is a reusable evaluation method for historical language generation, together with a single interpretable score that subsumes the partial metrics without hiding them.

1. Introduction

This paper deals with generated historical text and, more specifically, it engages in the design of a method of evaluation suitable for under-resourced languages, as is the case of Old English. The evaluation method should provide valid evidence, be checkable by a philologist and serve as a template for any Old English generation system.

The problem behind the proposed pipeline is straightforward. A text can look like Old English and still be wrong in ways that no single score will catch. It can have perfect word order and invented vocabulary. It can have attested vocabulary and modern phrasing underneath. Natural Language Processing tends to reward whatever its chosen metric measures. Historical linguistics wants an account of the language itself: forms that are attested, spellings that a scribe could have written, clauses built the way the period built them [1]. These two sets of demands do not meet by default. If we want results that a historical linguist will accept, the evaluation has to look at the language layer by layer.

Such a close look needs richly annotated resources. Much multilingual work reaches scale with thin annotation, which serves engineering but not philology. What is needed here is an overall approach based on corpus data that gives fine detail on morphosyntax, attestation, spelling variation and structure [2,3]. Such an approach requires annotated tagsets, clear links between corpora and dictionaries, explicit normalisation steps and models trained for the language.

For machine-made Old English prose, the goal is imitation of style and structure, not translation. This is why the usual metrics are not valid. BLEU and ROUGE assume that the output should match a reference almost word for word, so they punish acceptable variation [4,5]. Character overlap in the chrF family fails for a different reason: Old English spelling varies from manuscript to manuscript and from scribe to scribe, so two good texts can differ a great deal at the surface [6]. Perplexity says nothing about whether the words are attested or the clauses well built. A single embedding cosine can be high even when much of the vocabulary is invented. We do not discard these measures absolutely, although we exclude them as the main criterion in a task with no parallel reference.

The design of an evaluation procedure for generated Old English text has two sides. On the descriptive side, we need to break quality into layers that a reader can interpret one at a time, rather than collapse everything into one number from the start. On the inferential side, we need to know whether the observed contrasts survive resampling. The pipeline that is proposed serves both sides: attestation checks anchor the vocabulary in a corpus index, frequency diagnostics compare the balance of common and rare words, character n-grams watch the spelling, embedding comparisons measure distributional closeness with the coverage stated, parser metrics cover clause structure, and a bootstrap layer puts intervals around the main contrasts.

The ultimate aim of the evaluation procedure is not a verdict of the pass-or-fail kind. We seek a more nuanced assessment: where the generated text agrees with the attested language and where it departs from it, as well as the types of departures that arise. The overall score quantifies convergence or divergence with the source language and provides a measure that can be interpreted by historical linguists and, moreover, can guide future improvements of the evaluation pipeline.

The contributions of this article are fourfold. First, it defines a reusable multi-layer evaluation pipeline for generated Old English that requires no parallel reference: lexical attestation with form linking, frequency-profile diagnostics, character-level comparison, word-embedding geometry under a verified mapping and dependency parsing, each layer reporting against both the source text and the language at large. Second, it introduces the reference behaviour criterion that turns the question ‘how close is close enough?’ into a measurement: a good imitation should stand at the same distance from the background corpus as its source does, layer by layer. Third, it proposes and applies an overall metric G, the geometric mean of seven bounded components with a bootstrap confidence interval, which summarises the profile without hiding it, and demonstrates the whole method on a 114,451-token generated version of Gregory’s Dialogues. Fourth, it validates the metric externally and delimits its scope: two blind detection experiments with expert judges of Old English establish that the divergence G measures is real at corpus scale yet largely inaccessible to sentence-by-sentence reading, which fixes the level at which the index should be applied and cited.

With this aim in mind, the remainder of the article is organised as follows. Section 2 reviews related work, including the resources of the Nerthus Project, the Universal Dependencies framework for historical languages, distributional semantics for Old English and the problem of evaluating generated historical text. Section 3 presents the data, the generation procedure and the two comparison axes that frame the study. Section 4 describes the evaluation procedure layer by layer, from lexical attestation to dependency parsing, together with the uncertainty quantification and the overall metric G. Section 5 reports the results, opening with a qualitative reading and then working through the quantitative layers in turn. Section 6 discusses the convergence between the layers, the construction and limits of the overall score, the linguistic interpretation of the findings and the limitations that remain, along with directions for future work. Section 7 concludes and Section 8 documents the reproducibility of the study and the availability of the data, scripts and evaluation outputs.

2. Related Work

This section reviews relevant work, including the textual and lexical resources of the Nerthus Project, the Universal Dependencies annotation framework for historical languages, the distributional semantics for Old English, and the problems that arise in the evaluation of generated historical text.

Most of the infrastructure for computational Old English has been built by the Nerthus Project at the University of La Rioja. The lexical foundation is the Nerthus database, an inventory of the Old English headword list with morphological, etymological and semantic information [7]. On top of it, the project produced ParCorOEv3, an open-access parallel Old English–Present-Day English corpus of 303,342 records of vernacular prose [8]. Its texts include Ælfric’s Catholic Homilies, the Anglo-Saxon Chronicle, the Old English Orosius, the Laws and the Gospel of Mark. ParCorOEv3 supplies aligned text but no syntactic annotation. The conversion to Universal Dependencies came in two stages. Domínguez Barragán [9] annotated a subset of about 25,000 words, and the OEDT release [10] then added a UD-adapted 30,000-word ParCor subset, which yields a 55,000-word treebank with fixed train, development and test splits. The OEDT is at present the only UD treebank for Old English. Its lemma layer is grounded in the Nerthus database, so the treebank, lexical resource and parser form one continuous set of resources.

The parsing work closest to ours is Martín Arista and Metola Rodríguez [11], who report a UD parser for Old English with 88.92% unlabelled attachment score (UAS) and 79.65% labelled attachment score (LAS) on held-out data. Their design combines a biaffine attention parser of the Dozat and Manning [12] family with character-level input and Old English word embeddings. It provides both the published baseline for the present parser and a template for its architecture.

Regarding the annotation framework, Universal Dependencies [13,14] offer one scheme for morphology and dependency syntax across languages. It has become the standard for historical languages, with treebanks for Ancient Greek, Latin, Old French, Old Church Slavonic, Gothic and Vedic Sanskrit. At training sets of the size of the OEDT, parsers reach the high 70 s to mid 80 s on LAS. Character-level input and biaffine attention are the design choices that most often reach that ceiling [15,16].

Regarding the distributional semantics and word embeddings for Old English, the Skip-gram with Negative Sampling variant of Word2Vec [17,18] predicts context words from a target word, with a few negative pairs as contrast. It produces spaces in which related words sit at small angles from each other [19,20]. For historical languages with little data, it remains competitive: its parameter count is small enough to train well on the corpus sizes available, and its outputs can be read along axes that philology has long cared about. For the sake of this study, we have trained a 200-dimensional model on the Dictionary of Old English Corpus (henceforth DOEC). Training keeps the vernacular characters þ, ð and æ, which carry phonological and morphological information. The embedding space used in the present pipeline has been retrieved from the Stanza ‘nerthus.pt’ pre-trained representation, whose row mapping we verify before use (see Section 4).

With respect to the generation of text in historical languages, whether by neural models, retrieval-and-edit pipelines or evolutionary methods [21,22], it raises an evaluation problem that modern languages do not. Surface-similarity metrics assume reference outputs. For generated Old English, there is no natural reference: the output is not a translation of a source with a known Old English rendering, and it is not the completion of a prompt with one right answer. Likelihood metrics do not help either. A perplexity score from a model trained on the same data that feeds the generator mixes two things that should stay apart: fidelity to the training distribution and the fit of the own model to that distribution. The method adopted here relies instead on structural comparison, in the tradition of corpus profiling found in Liu [23] on dependency-length minimisation and Futrell et al. [24] on dependency distance across languages.

3. Method

This section describes the data and presents the generation and corpus compilation procedures.

The evaluated corpus is a five-column file comprising sentence code, original Old English, translation of the original, generated Old English, and a literal translation of the generated line. The generated text holds 114,451 tokens in 4217 lines. The in-domain reference is the original Old English Gregory’s Dialogues in Wærferth’s translation [25], and the broad historical baseline is the DOEC [26]. The reference dictionary is the digital edition of Bosworth and Toller Anglo-Saxon Dictionary [27].

Lexical attestation control comes from the corpus index. Parser resources come from the OEDT 50K UD splits and the word-embedding space from ‘ang_embeddings.pt’, aligned to the Stanza ‘nerthus.pt’ representation [3].

Two dictionaries complete the resource set. The lemma-and-inflection dictionary lists 37,522 lemmas with 149,286 attested inflectional forms, compiled from the York-Toronto-Helsinki Parsed Corpus of Old English Prose [28] and lexicographical sources. The orthographic variant dictionary records 162,253 attested spelling variants organised by lemma and tag. Both were specifically built for a companion generation project. They serve as generation constraints and as linking layers in the attestation analysis (see Section 4).

Four features make this a densely annotated working environment. Morphosyntactic structure is represented through documented UD categories, which allows evaluation relation by relation. Lexical documentation is anchored to an external index rather than guessed from corpus frequency. Spelling variability is handled through explicit normalisation, so that lookup success and parser comparability become measurable. Last but not least, the environment keeps the resources needed for reuse: models, parses, embeddings and the workflows that link corpus tokens to dictionary entries.

Turning to the generation procedure, the text was produced as one generated sentence for each of the 4217 source sentences, imitating the style of the original without copying it, in such a way that each line is provided with its own literal Present-Day English translation. The instruction asked for an exercise of rewriting a text in the same language and register, sentence by sentence, the way a monastic pupil might paraphrase his master’s prose.

The choice of generator was driven by three practical requirements of the constrained procedure rather than by any claim of uniqueness. The batch protocol demands a context window large enough to hold sixty source sentences, their translations, the instruction file and the validator reports of successive repair rounds; the repair loop demands a model able to run the validator, read its output and correct only the flagged lines without disturbing the rest; and the lexical constraint demands reliable adherence to a closed list of admissible forms over thousands of lines. Pilot runs with GPT-5 showed higher rates of out-of-lexicon tokens. In this respect, we would like to emphasise that the evaluation pipeline, which is the contribution of this article, is independent of the generator: any system whose output is a line-aligned Old English text can be scored by the same seven components, and comparison across generators is precisely the use case for which G carries a confidence interval. The generator was Claude Fable 5 (Anthropic), model identifier claude-fable-5, reached through a third-party access point to the Anthropic Messages API (19 July 2026). Sampling ran at the documented server-side default: temperature 1.0, the top of the 0-to-1 range, with no nucleus or top-k truncation. Two further specifications matter for reproduction. Reasoning (‘adaptive thinking’) is always active on this model and cannot be turned off; the model decides how much to reason on each request, and the only external control is the effort parameter just mentioned. The context window is one million tokens by default, with up to 128,000 output tokens per request, far above the size of any batch used here.

The source text was divided into 71 batches of about sixty sentences, and each batch went to a fresh instance of the model together with a fixed instruction file. Each input line carries the sentence code, the original Old English and its Present-Day English translation; the output must carry one line per input line, with the generated Old English and a fresh literal translation of it. The model is told to convey the meaning of the original sentence, with the Present-Day English translation as a guide, and not to copy the original phrasing: swapping thorn for eth or adjusting a spelling does not count as variation. The instructions also fix the grammar of the target: late West Saxon prose, correct case government after prepositions, verb-second order after þa and þonne in main clauses, the verb-final tendency in subordinate ones, and the dialogue frame of Gregory and Peter with varied formulas. Three resources supported this work. The lemma-and-inflection dictionary and the orthographic variant dictionary supplied the stock of admissible forms, compiled into the lexicon described below. The original text itself, present line by line in the input, served as the model of content and style. And a validator script closed the loop: after writing a batch, the model had to run the validator, read its report, repair every flagged line, and run it again, until the batch came back clean. Some batches passed on the first attempt; the stubborn ones went through several rounds of repair, most often for a word that fell outside the lexicon.

Four constraints were enforced during generation, not checked afterwards. A lexicon of allowed surface forms was compiled first, from three sources: the attested and propagated inflections of the lemma-and-inflection dictionary, the orthographic variant dictionary, and the token list of the original Gregory itself, which covers the proper names. The list holds 227,899 forms. A validator then rejected any output line that contained a token outside this list, any line with a diacritic (macrons are a habit of dictionaries and teaching grammars, not of manuscripts), any line shorter than 0.8 or longer than 1.25 times its source, and any line too similar to its source (token-set Jaccard above 0.7). Generation and repair continued until every one of the 4217 lines passed all four checks.

The method uses two comparison axes. Generated against original Gregory tests fidelity to the source. Generated against DOEC tests compatibility with the language at large. Using only the first axis risks punishing acceptable variation; using only the second risks rewarding generic Old-English-like output that misses the Gregory style. The second axis also supplies the reference behaviour for the convergence analysis (see Section 6). The original itself stands at a measurable distance from the DOEC on every layer: it is one text, in one register, against a corpus that also holds glossaries, chronicles, scientific prose, laws and verse. A generated text that imitates the original well should stand at about the same distance from the DOEC and in the same direction. This empirical question is asked once per layer.

To finish up this section, a distinction is needed between reproducibility of the generation and reproducibility of the evaluation. The generation is not bit-reproducible: sampling ran at temperature 1.0 without a fixed seed, through an access point whose server-side configuration may change over time, and the provider does not guarantee identical outputs for identical requests. We therefore record everything that can be recorded (model identifier, access date, sampling parameters, instruction file, batch structure and validator version) and treat the generated corpus itself as the fixed object of study. That corpus is released in full, together with the lexica, the parses, the embedding space, the scripts and the bootstrap code, so every number reported in this article is exactly recomputable from the released materials. A regeneration under the same settings would constitute a new sample from the model, to be scored with intervals of its own, as noted in Section 5; the pipeline, not any single run, is what the release makes reproducible.

4. Evaluation Procedure

The procedure is an architecture rather than a list of calculations. Each component performs a different function, so that it can see something that the others cannot. Figure 1 presents the evaluation procedure.

The frequency-index component checks each word form against the DOEC index. It sorts the vocabulary into bins (‘very_high’, ‘high’, ‘mid’, ‘low’, ‘rare’, ‘hapax’, and ‘oov’) and compares bin distributions by Jensen–Shannon divergence (JSD). It covers the blind spot of geometry-only methods: a text can sit close in vector space and still overproduce forms that no scribe ever wrote or draw only on the most common words and starve the rare bins. This component exists to account for both.

The linking component asks a question that strict lookup cannot answer: Is a form that fails the index an invented word, or an attested word in different spelling or inflection? It applies the resources in order: the attested inflections of the lemma-and-inflection dictionary, its lemma headwords, and the orthographic variant dictionary. Only forms that fail every layer count as unattested. The component also flags defects of the measuring stick itself, such as index omissions and reports them separately from defects of the text.

The character track represents each corpus by its 3-, 4- and 5-g counts over the 12,000 character n-grams most frequent in the DOEC and compares profiles by frequency-profile cosine and by JSD. It watches what the word-level components cannot, such as spelling patterns, inflectional endings as letter sequences, and the surface texture of the prose. Old English orthography is where machine generation most often betrays itself, either by inventing letter sequences or by writing dictionary spellings that no scribe used.

The cut at 12,000 is a coverage decision rather than a tuned parameter. Ranked by corpus frequency, the top 12,000 n-grams cover 70.0% of the running character n-gram mass of the DOEC, and the last one admitted still occurs 660 times in the corpus. This ranking, therefore, retains the bulk of the information while excluding n-grams whose counts are too sparse to support stable frequency estimates. The cosine is taken between frequency profiles over this shared vocabulary. Being scale-invariant, it requires no normalisation for the differing lengths of the three corpora, and no inverse-document-frequency weight is applied, since across three corpora such a weight would be degenerate. The results do not depend on the exact cut: recomputing the track at 6000 and at 24,000 n-grams moves the generated–original cosine by less than 0.002 (+0.0013 and −0.0008 respectively) and leaves the ordering of the three corpus pairs unchanged on every measure.

The word-level component computes token coverage, centroid cosine, and shared-vocabulary JSD in the native ang_embeddings.pt space. Before use, the row mapping is verified: the matrix equals the Stanza nerthus.pt embedding table from row five onward, so row i corresponds to token i + 5 of the Stanza vocabulary. Without this check, a simple index shift could corrupt every result downstream. The component registers distributional closeness without string-level matching and keeps the coverage figure in view, so that a high cosine cannot be over-read when part of the vocabulary has no vector. One further guard against over-reading is definitional: this is a corpus-level instrument. It asks where the vocabulary of the whole text lives, not whether any single line means what its source line means; the sentence-level question has a natural design of its own in this corpus (see Section 5).

The parser is a joint tagger–lemmatiser–dependency parser of the Stanza family [3], with three connected parts. The tagger and the morphological-feature predictor share a bidirectional LSTM encoder over word and character representations; the word part starts from the pre-trained Old English embeddings. The lemmatiser is a sequence-to-sequence character model conditioned on the predicted tag [29]. The dependency parser is a deep biaffine attention model [12] that scores arcs and labels through biaffine products.

Training and evaluation use the OEDT [10]: 55,000 tokens in CoNLL-U format under UD v2 conventions. The 50K configuration used here merges training and development into one practical training set of about 50,000 tokens; the test split stays held out. On that split, the parser reaches 84.89 UAS and 76.71 LAS, below the Martín Arista and Metola Rodríguez [11] baseline of 88.92/79.65 but within an interpretable margin for treebanks of this size. Per-relation analysis shows strength on the frequent core relations (‘case’, ‘det’, ‘nsubj’, ‘root’) and weakness on the complex ones (‘ccomp’, ‘xcomp’, the ‘obl’ family). Structural conclusions are weighted accordingly.

Three families of structural metrics are computed over the parses. The verb-final ratio is the share of finite verbs that stand last among the non-punctuation tokens of their sentence; the alternation of verb-final, verb-second and verb-first orders is one of the most studied features of Old English syntax [30,31]. Dependency-length statistics include the mean dependency length and the share of dependencies longer than five tokens [23,24]. Divergences are computed between corpora on the part-of-speech distribution, the relation distribution, and the joint head–relation–dependent distribution. The generated text was parsed with the same pipeline, the same models and the same settings as the original and the DOEC line subset of the earlier runs, so the three parses are directly comparable.

The independence component works at four levels because a variation must not be a copy: exact line matches after normalisation (lowercasing, diacritic stripping, thorn–eth merging), token-set Jaccard similarity per line, positional n-gram overlap for n = 4, 6, 8 and maximal shared token spans of eight words or more, whose combined token share measures how much of the text sits inside passages lifted from the source.

Percentile bootstrap intervals are computed for the main contrasts [32]: 1000 resamples over the 4217 lines for the lexical, character and overlap measures and over the 4516 parsed sentences for the structural measures. The reference distributions stay fixed; what is resampled is the generated text. Intervals are recentred on the full-corpus point estimates. Point estimates alone overstate certainty when a statistic depends on a few heavy items, as span counts and rare-bin masses do.

Seven partial results, one per layer, are rescaled to the interval [0, 1] so that 1 means indistinguishable from the reference behaviour on that layer:

-

A (attestation) = 1 − unattested residual token mass after linking;

-

W (word geometry) = centroid cosine to the original in the native space;

-

F (frequency profile) = 1 − frequency-bin JSD to the original;

-

S (syntactic structure) = 1 − head–relation–dependent JSD to the original;

-

C (character surface) = character n-gram cosine to the original;

-

I (independence) = 1 − share of tokens inside shared spans of eight or more words;

-

L (length fidelity) = min(r, 1/r), where r is the token ratio of generated to original.

The overall score is the geometric mean:

$${\left(\mathrm{A}\cdot \mathrm{W}\cdot \mathrm{F}\cdot \mathrm{S}\cdot \mathrm{C}\cdot \mathrm{I}\cdot \mathrm{L}\right)}^{{\displaystyle \frac{1}{7}}}$$

The geometric mean is chosen over the arithmetic one. Under a geometric mean, a component near zero drags G toward zero no matter how good the rest are, so a text that invents half its vocabulary cannot ride on its word order. Under an arithmetic mean, it could.

Of the seven components, four (W, F, S, and C) are free measurements, while three (A, I, and L) verify compliance with constraints enforced at generation time; Section 6 discusses the consequences of this asymmetry for interpreting G.

Two design choices deserve further comment. First, every component is oriented toward the original, not toward the DOEC, because the task is imitation of a source; the DOEC enters the validation of the components (Section 6), not their definition. Second, the bootstrap presented in this section runs through the whole construction, so G carries a confidence interval like any other statistic.

To finish up this section, it is necessary to explain why certain metrics have been excluded. BLEU, ROUGE, and chrF are excluded because the task has no natural reference output; building artificial references would import the biases of whoever built them. Perplexity is excluded because it mixes vocabulary frequency with syntactic well-formedness in a way that cannot be untangled: a generator that strings frequent words together badly can outscore one that writes good prose with the marked vocabulary typical of the period. BERTScore-style metrics [33] are excluded because no adequate contextual model exists for Old English.

5. Results

This section gives a short qualitative reading of the results followed by a detailed quantitative analysis: independence and length first, then attestation, frequency, character, word geometry and structure. Then, the intervals are reported and the overall score is calculated. Each layer is documented with its own statistics in what follows: the generated text versus the original in

Table 1. The generated text against the original. The upper panel profiles the two corpora; the four constraints in the lower panel were enforced during generation by a validator, and every line had to pass all four before it was accepted.

Quantity	Generated Text	Original Gregory
Corpus profile
Lines (one per source sentence)	4217	4217
Running tokens	114,451	111,646
Distinct word types	8463	11,658
Type–token ratio	0.074	0.104
Token ratio to the original	1.025 [1.022, 1.028]	—
Generation constraints (enforced by the validator)
Allowed surface forms	lexicon of 227,899 attested forms	—
Diacritics	none permitted	none present
Line length	0.8–1.25 × source line	—
Similarity to the source line	token-set Jaccard ≤ 0.7	—

Brackets give the 95% percentile bootstrap interval over the 4217 lines. The generated text runs 2.5% longer than the original in running tokens while drawing on 27.4% fewer distinct types.

, the overlap and independence measures in

Table 2. Overlap between the generated text and the original at every level checked. The 30 shared spans of eight or more tokens are almost all lists and fixed parallel constructions.

Overlap Measure	Value
Lines identical to the original after normalisation	0 of 4217
Lines above the Jaccard similarity ceiling (0.7)	0 of 4217
4-g positions shared with the original	4.73% [4.54, 4.92]
6-g positions shared	0.38%
8-g positions shared	0.06%
Shared spans of eight or more tokens	30
Tokens inside such spans	0.24% [0.15, 0.33]

Brackets give 95% percentile bootstrap intervals over the 4217 lines. Normalisation lowercases, strips diacritics, and merges thorn with eth.

, the attestation decomposition in

Table 3. Attestation of the generated vocabulary against the DOE index, and the decomposition of the strict out-of-vocabulary mass of the generated text.

Quantity	Generated	Original	DOEC
Strict lookup against the index
Out-of-vocabulary types	35 (0.41%)	2 (0.02%)	1069 (0.57%)
Out-of-vocabulary token mass	2.27% a	1.27% a	3.00%
Decomposition of the generated out-of-vocabulary mass
The preposition on (index artefact)	2.23%	—	—
Matched as an attested inflection (16 types)	0.017%	—	—
Matched as a lemma headword (6 types)	0.009%	—	—
Matched as a recorded spelling variant (10 types)	0.010%	—	—
Unattested residual (2 types: mislicnes, stihtan)	0.0026% [0.0000, 0.0069]	—	—

^a The preposition on is absent from the DOE index itself, so the strict lookup counts it as out of vocabulary. It alone carries 2.23 of the 2.27 points in the generated text, and the 1.27% of the original has the same source. Type percentages are shares of the vocabulary of each corpus; mass percentages are shares of running tokens. Brackets give the 95% bootstrap interval. The residual of the two types is three tokens in 114,451.

, the frequency-bin profile in

Table 4. Share of running tokens per frequency bin of the DOE index. Percentages are shares of the running tokens of each corpus and sum to 100 down each column.

Frequency Bin (DOE index)	Generated (%)	Original (%)	DOEC (%)
Very high (>10,000)	32.69	34.38	26.23
High (1001–10,000)	27.35	24.47	23.43
Mid (101–1000)	23.09	20.40	21.17
Low (11–100)	11.13	12.19	15.00
Rare (2–10)	2.94	5.46	7.46
Hapax (1)	0.52	1.83	3.70
Out of vocabulary	2.27	1.27	3.00
Rare and hapax combined a	3.46	7.29	11.16

^a Computed from the unrounded bin shares, so it differs slightly from the sum of the two rounded rows above it. Jensen–Shannon divergence between the profiles: generated–original 0.0081 [0.0076, 0.0085]; generated–DOEC 0.0235; original–DOEC 0.0112. The generated text is therefore closer to its source than the source is to the language at large.

and Figure 2, the character, word and structure tracks with their DOEC baselines in

Table 5. The three representation tracks. The upper panels give pairwise comparisons between corpora; the lower panel gives values computed for each corpus separately, and its column headings are restated accordingly.

Measure	Generated–Original	Generated–DOEC	Original–DOEC
Character track (3/4/5-g over the top 12,000 DOEC n-grams)
Cosine	0.982 [0.981, 0.983]	0.898	0.893
Jensen–Shannon divergence	0.025 [0.024, 0.025]	0.086	0.081
Word track (native embedding space under the verified mapping)
Centroid cosine	0.9989 [0.9988, 0.9990]	0.9777	0.9784
Shared-vocabulary divergence	0.107	0.240	0.251
Structure track (Stanza ang parses) a
UPOS divergence	0.0005	0.0608	0.0631
Relation divergence	0.0022	0.1244	0.1276
Head–relation–dependent divergence	0.0081 [0.0076, 0.0086]	0.1641	0.1601
Per-corpus values	Generated	Original	DOEC
Embedding coverage	0.970	0.955	0.929
Verb-final ratio	0.104	0.073	n.a. b
Mean dependency length	3.49	3.41	2.23
Dependencies longer than five tokens	0.166	0.157	0.069

^a Sentences parsed: 4516 generated, 4293 original and 5022 in the DOEC line subset. The counts differ because the parser resegments long periods. Difference in mean dependency length, generated minus original: +0.088 [0.066, 0.110]. ^b The parser returns no finite verb in the DOEC line subset, so no verb-final ratio is defined for it. Brackets give 95% percentile bootstrap intervals. On every pairwise measure, the last two columns are to be read together: the generated text stands at the distance of the original from the DOEC.

, the cross-layer convergence in

Table 6. Convergence between the evaluation layers. Each row states what a layer measures, the value obtained for the generated text, and the reference behaviour of the original on the same measure.

Layer	What It Measures	Generated	Original (Reference Behaviour)
Attestation	documentary basis of each form	residual 0.003% of tokens	0% by definition
Frequency profile	balance of common and rare vocabulary	divergence to original 0.0081	divergence to DOEC 0.0112
Character surface	spelling and letter sequences	divergence to DOEC 0.086	divergence to DOEC 0.081
Word geometry	lexical-semantic neighbourhoods	cosine to DOEC 0.978	cosine to DOEC 0.978
Syntactic structure	clause building and dependencies	triple divergence to DOEC 0.164	triple divergence to DOEC 0.160
Independence	copying from the source	0.24% of tokens in shared spans	—
Length fidelity	token-by-token size match	ratio 1.025	—

All divergences are Jensen–Shannon. The first two layers refer to the original, and the next three to the DOEC, because the reference behaviour that each layer tests differs; the underlying values are those of Table 3, Table 4 and Table 5.

and the components of the overall score in

Table 7. Components of the overall metric G. Each component is scaled to [0, 1], where 1 means indistinguishable from the reference behaviour on that layer. G is their geometric mean, so one weak layer governs the total and cannot be offset by the others.

Component	Definition	Value
A—attestation	1 − unattested residual token mass	0.99997
W—word geometry	centroid cosine to the original in the native space	0.99889
F—frequency profile	1 − frequency-bin divergence to the original	0.99191
S—syntactic structure	1 − head–relation–dependent divergence to the original	0.99191
C—character surface	character n-gram cosine to the original	0.98189
I—independence	1 − share of tokens in shared spans of eight or more words	0.99761
L—length fidelity	min(r, 1/r) for token ratio r	0.97550
G—overall	geometric mean of the seven components	0.9911 [0.9906, 0.9916]

Brackets give the 95% percentile bootstrap interval. Components A, I and L verify compliance with constraints enforced at generation time, whereas W, F, S and C are free measurements. Restricted to the four free components, the geometric mean is 0.9911, so the three compliance components move G by 0.00007.

. All intervals are 95% percentile bootstrap intervals as defined in Section 4. Each layer is computed over a stated number of units, and these differ. The frequency, character and overlap layers run over 4217 aligned lines and 114,451 generated tokens against 111,646 original ones; the attestation layer over 8463 generated types against 11,658; the character layer over the 12,000 most frequent DOEC n-grams of orders three, four and five; the word layer over the 97.0% of generated tokens that carry a vector, against 95.5% of the original and 92.9% of the DOEC; and the structural layer over 4516 parsed generated sentences against 4293 original ones, the counts differing because the parser resegments long periods.

Examples (1)–(3) show what the variation does and where its limits lie.

(1) Original: Þa acsode se abbud hine þus cweþende: hwider wilt þu gangan? ‘Then the abbot asked him, saying thus: where do you want to go?’ Generated: Se abbud hine þa befran & cwæð: hwyder fundast þu nu to farenne? ‘The abbot then questioned him and said: where do you intend now to go?’

In example (1), the participial formula þus cweþende gives way to the coordinated befran & cwæð, and the modal wilt þu gangan becomes fundast þu to farenne, with a correctly inflected infinitive after fundian. Both constructions are attested in the prose of the period; the generated one is the plainer of the two.

(2) Original: Gregorius him andswarode: sum wæs swyþe arwyrðes lifes wer, þam wæs nama Bonefacius. ‘Gregory answered him: there was a man of very honourable life, whose name was Boniface.’ Generated: Gregorius andwyrde him: sum wer wæs swiðe halig on his life, se wæs Bonefacius gehaten. ‘Gregory answered him: there was a man very holy in his life, who was called Boniface.’

In example (2), the descriptive genitive arwyrðes lifes is turned into a predicative adjective with a prepositional phrase, and the dative naming idiom þam wæs nama is exchanged for the passive gehaten. The grammar is correct on both sides; what is lost is Wærferth’s preference for the marked construction.

(3) Original: Ne sæde ic hit ær, þæt he wære deofol nalles munuc? ‘Did I not say it before, that he was a devil, not a monk?’ Generated: Hu, ne cwæð ic eow ær, þæt he næs munuc, ac deofol wære? ‘How, did I not say to you before, that he was not a monk, but was a devil?’

In example (3), the generated line keeps the subjunctive wære in the reported clause and even ends it verb-finally, but the emphatic negator nalles is replaced with the commoner ne ... ac contrast, and an interjection and a pronoun of address are added.

The pattern illustrated by these examples repeats across the corpus. The generated line keeps the sense, the register and the approximate weight of its source, respects case government and clause order, and reaches for the unmarked, less complex and more frequent construction wherever the original allows a choice.

Overall, the generated text is not shorter than the original but slightly longer. It comprises 114,451 running tokens against 111,646, a ratio of 1.025 with interval [1.022, 1.028], and every line sits inside the 0.8–1.25 band by construction. What is smaller is the vocabulary rather than the text: 8463 distinct word types against 11,658, a fall of 27.4%, which lowers the type–token ratio from 0.104 to 0.074. The distinction governs everything that follows, because the two quantities point in opposite directions and only the second is a finding. Length was a constraint, enforced line by line and therefore verified rather than discovered; lexical range was never constrained, and it is the one measure on which the generated text falls clearly short of its model, as the frequency bins and the qualitative reading independently confirm. Table 1 shows the corpus, including the original and the generated Old English. The two texts match line for line. Its vocabulary is drawn entirely from a lexicon of attested forms, compiled before generation from the two dictionaries and the token list of the original.

Independence holds at every level checked, as can be seen in Table 2. No line of the 4217 matches a source line after normalisation, and no line exceeds the Jaccard ceiling. Positional overlap falls fast with n: 4.7% of 4-g positions, 0.38% of 6-g, 0.06% of 8-g. The shared spans of eight or more tokens number thirty and cover 0.24% of the text [0.15, 0.33], and nearly all of them are lists or fixed parallel constructions, like the catalogue of animal cries in Book 4 or the sentence that runs ‘the envious burn with the envious, the unbelieving with the unbelieving’, where the content itself fixes the words. A human imitator may share the same passages. As a general rule, the model reads each source line and its translation by design, so conceptual dependence is the point of the task, not a defect that the checks should remove. What the four levels rule out is reproduction. The text reformulates its source at the line, the n-gram, and the span and nowhere copies it.

The comparison against the DOE index leaves 35 of the 8463 generated types unmatched (0.41% of types, 2.27% of tokens). The decomposition can be seen in Table 3. One word, the preposition on, carries 2.23 of the 2.27 points: it is absent from the index altogether, which is a shortcoming of the index and not of the text, and the OOV (out-of-vocabulary) set of the original (1.27%) has the same source. Of the remaining 34 types, 16 are attested inflections in the lemma-and-inflection dictionary, 6 are lemma headwords, and 10 are recorded spelling variants. Two types survive every layer: mislicnes and stihtan, ordinary headwords in other reference works, together 0.0026% of tokens [0.0000, 0.0069]. For comparison, the token set of the DOEC itself in unindexed forms is 3.0%. Two boundaries of this layer are relevant for the interpretation of results. Because lexical admissibility was enforced during generation, the attestation result is best read as verification (proof that the constraint held and that the underlying resources give broad coverage) rather than as a discovery about the habits of the model. And attestation certifies existence, not aptness: that a form is documented does not make it grammatically or semantically right in its line for context dependency reasons, a question the remaining layers approach and philological reading must settle locally. Stated as a single figure, the proportion of new words in the generated text is three tokens in 114,451, or two types in 8463: 0.0026% of the running text and 0.024% of the vocabulary. Neologisms in the strict sense, that is, forms outside the pre-compiled lexicon of 227,899 attested spellings, number zero by construction, since no line containing one was ever accepted.

The frequency analysis considers the same vocabulary from a different perspective. This is shown in Table 4 and Figure 2. The generated text distributes its token mass across the frequency bins as the original does, bin by bin: heavier than the DOEC in the very-high and high bins, lighter in the rare and hapax bins, as narrative prose is against a corpus that contains various styles and registers. The JSD between generated and original profiles is 0.0081 [0.0076, 0.0085]. The distance between the original and the DOEC on the same measure is 0.0112. In a nutshell, the generated text is closer to its source than the source is to the language at large.

One number in this layer takes a different line, and a distinction is needed to interpret it. In running tokens, the generated text is not shorter but 2.5% longer than the original (114,451 against 111,646). What is lower is the number of distinct word types: 8463 against 11,658 at nearly equal length. The rare and hapax bins show where the difference lies: 3.5% of generated token mass against 7.3% of the original. An important lesson can be learned from these results. The constrained lexicon prevents invention. It does not create richness. The generator reached more often for the common word (more frequent) and the unmarked construction (less complex). This is what the frequency layer is for: it passes no judgement on any single choice, but it totals up a drift that no reading of individual lines would catch, the systematic pull toward frequent vocabulary and plain syntactic architecture. The decoding settings described in Section 3 may explain this. If the interface defaults applied, the run already sampled at the top of the temperature range, so the narrowness cannot be laid at the door of conservative decoding; its sources must then be lexical preferences of the model and the repair loop, which rewards the safe word and grammar every time a rare segment falls outside the lexicon. Only a regeneration under recorded settings can separate these.

At character level (Table 5), the generated text and the original are close neighbours: cosine 0.982 [0.981, 0.983], JSD 0.025 [0.024, 0.025] over the top 12,000 DOEC n-grams. The comparison against the background corpus illustrates this behaviour. The generated text stands at cosine 0.898 and JSD 0.086 from the DOEC; the original stands at 0.893 and 0.081. The two texts look out at the language from the same spot. A text with dictionary spellings, or with invented letter sequences, would stand somewhere else. What the track guarantees is texture, not correctness: profiles of this kind are corpus-wide averages and a handful of anomalous spellings would vanish into twelve thousand n-gram counts. The claim that the layer supports is that the surface pattern of the whole text is the surface pattern of its source, while the form-by-form ground stays with the attestation layer, which reads every token against the index. Table 5 gives the three representation tracks in one place. The last two columns of each pairwise row must be read together: on nearly every measure, the generated text stands at the same distance from the DOEC as the original itself does.

As can be seen in Table 5, in the native embedding space, coverage is high for all three corpora (97.0% of generated tokens have a vector, against 95.5% for the original and 92.9% for the DOEC), so the geometry rests on nearly the whole text rather than on a given fragment. The centroid cosine between generated and original is 0.9989 [0.9988, 0.9990]. The shared-vocabulary JSD is 0.107. Against the DOEC, the generated centroid stands at cosine 0.978; the original stands at 0.978 as well, to three decimals. The distributional distance from the DOEC (JSD 0.240 generated, 0.251 original) tells the same story: the generated text departs from the language at large a little less than its source does, in the same direction.

The centroid draws further attention because it is the statistic most easily over-read. A corpus-level centroid is a global diagnostic of lexical-semantic compatibility, not a sentence-level measure of meaning preservation: it shows that the generated work, taken as a whole, occupies the same broad semantic region as the original, but it cannot rule out compensating differences between lines, nor semantic errors inside them. The line-by-line design of the corpus addresses this question. With 4217 aligned pairs, a similarity can be computed for every pair separately, and the result is a distribution rather than a point: how many lines hold the meaning of their source, which drift, and in what ways. We therefore read the cosine of 0.9989 as evidence of semantic-domain consistency, not as proof that every generated line preserves the meaning of its source. Sentence-pair comparison, supplemented by bidirectional lexical alignment and by philological reading of the low tail, is the appropriate next step for local semantic fidelity.

The parses in Table 5 cover 4516 generated sentences against 4293 original ones; the sentence counts differ because the parser resegments long periods. The three distributional measures are very near zero: UPOS JSD 0.0005, relation JSD 0.0022, head–relation–dependent JSD 0.0081 [0.0076, 0.0086]. Against the DOEC subset, the generated text stands at 0.061, 0.124, and 0.164 on the same three measures; the original stands at 0.063, 0.128 and 0.160. Once again, the two texts keep the same distance from the background, on the layer that word-level methods cannot see.

The scalar metrics agree. Mean dependency length is 3.49 in the generated text against 3.41 in the original, a difference of +0.088 [0.066, 0.110]; the share of long dependencies is 0.166 against 0.157; the verb-final ratio is 0.104 against 0.073. The generated clauses are slightly heavier than those of the original, and the finite verb closes its sentence a little more often. Both differences are small compared to the distance that separates either text from the DOEC subset (mean dependency length 2.23), but they are stable and they underline syntactic variation. A word of caution is necessary at this point. Parsing every corpus with the same model makes the aggregate profiles comparable because systematic error strikes all sides alike. It does not guarantee that parsing errors distribute identically over historical prose and machine-made prose. The structural measures therefore evaluate corpora (dependency patterns, word order, and clause organisation in the mass). They do not assess any single generated sentence as grammatical. This can be done only by means of philological reading.

All intervals quoted above come from the bootstrap described in Section 4. Directional findings are strong. The frequency JSD against the original stays below the original-to-DOEC value across the whole interval. The character and word cosines move in the third decimal. The span share, the statistic most exposed to a few heavy items, keeps its interval inside [0.15%, 0.33%]. The structural delta of mean dependency length excludes zero, so the heavier clause weight is signal and not noise. The intervals answer one question and should not be generalised to another. Resampling lines shows that the corpus-level contrasts are not driven by a handful of influential lines; it quantifies variation within this one generated text. About variability across generations it says nothing, so that a new run, another prompt, a different model or temperature would need intervals of its own.

To provide an external check on the G-index, we ran two blind detection experiments on the generated text that the index scores. In the first, two experts in Old English judged 200 unlabelled sentences in shuffled order: 100 authentic sentences and 100 generations drawn from disjoint rows of the aligned file, so that no content appeared in both conditions. Sentence-length distributions were matched exactly, the class balance was not disclosed, and orthography and editorial apparatus, including macrons, brackets and quotation conventions, were normalised, since these features otherwise permit near-perfect classification on evidence that is editorial rather than linguistic. Accuracy was 62.5% and 61.0%. Recall on the generated class was 40% and 33%, so roughly two-thirds of generated sentences passed as authentic, while authentic sentences were accepted at 85% and 89%. The pattern indicates modest discriminability under a conservative criterion rather than an absence of discrimination. Detected generated sentences did not, however, score worse on the index than undetected ones ($p=0.158$ and $0.102$). The only reliable predictor of detection was sentence length. This proved to be a response bias, since the experts also flagged short authentic sentences (16.1 against 23.1 tokens, $p=0.002$). Once length was stratified, the index predicted nothing (all $p>0.21$). Given that randomly drawn generations cluster tightly on G, a second experiment contrasted the 50 lowest- and the 50 highest-scoring generations, paired within token-count strata so that a gap of 3.2 standard deviations in G was tested at constant length. Detection rates did not differ (34.0% against 39.0%, $p=0.557$), and after adjustment for length, the index again predicted nothing ($\beta =+0.158$, $p=0.49$; $\beta =+0.032$, $p=0.90$).

6. Discussion

This section discusses the results from the perspectives of the convergence between layers, the overall metric, and the linguistic assessment of textual plausibility. The limitations of the work and the pending research for future work are also presented in this section.

The five measuring layers were built to assess different aspects: the index sees documentation, the bins see the balance of common and rare, the n-grams see spelling, the embedding space sees lexical company, and the parser sees clause structure. They share no representation and no normalisation beyond tokenisation. If the generated text were good in one dimension and poor in another, for example, attested but modern in phrasing, or archaic in surface but skewed in vocabulary, the layers would disagree, and their disagreement would be the finding.

The five measuring layers largely agree, as is shown in Table 6.

Figure 3 draws the finding as a map: the DOEC at the centre, one contour line per layer, and on every contour the generated text and the original sitting together.

Figure 4 gives the measured distances behind the map. The agreement has two faces. On the source axis, every layer places the generated text next to the original: divergence 0.0081 in the frequency bins, cosine 0.982 at character level, cosine 0.9989 in embedding space, divergence 0.0081 in the syntactic triples. On the language axis, every layer places the generated text at a distance from the original in the DOEC: 0.086 against 0.081 at character level, 0.978 against 0.978 in embedding space, 0.164 against 0.160 in the triples. The second face is the stronger evidence. Closeness to the source could in principle be bought by copying; standing where the source stands, relative to a corpus the generator never saw as a reference, cannot. And the independence layer rules the copying out directly: 0.24% of tokens inside shared spans, none of them a full line.

The convergence has exceptions worth commenting on. The type count is low (8463 against 11,658), the rare bins are thin (3.5% of mass against 7.3%), and the qualitative reading noticed the safer synonym chosen too often. These three layers underline one characteristic of the generated text: the vocabulary is narrower than the original’s.

Because the layers converge, they can be combined without undesired results. When measures disagree, a composite hides the disagreement, and the profile must stay disaggregated. When they agree, a composite is a fair summary, and it answers a practical need: model selection, regression testing across generator versions, and comparison between systems all want one number with a confidence interval.

The proposal advanced in Section 4 is G, the geometric mean of seven bounded components. For the present text the components are A = 0.99997 (attestation), W = 0.99889 (word geometry), F = 0.99191 (frequency), S = 0.99191 (structure), C = 0.98189 (character), I = 0.99761 (independence) and L = 0.97550 (length), giving G = 0.9911, 95% CI [0.9906, 0.9916].

As is presented in Table 7 and Figure 5, the weakest components are length (the text runs 2.5% long) and the character surface, where the small JSD of 0.025 becomes a cosine of 0.982. No component falls below 0.975.

Three properties recommend this architecture. First, it is interpretable component by component. Second, it is strict: the geometric mean makes the score depend on its worst component, so that syntactic polish cannot compensate, for instance, for invented vocabulary. Third, it is falsifiable in the aggregate: G carries a bootstrap interval, so two generator versions can be compared with a significance statement rather than a bare ranking.

This architecture could be read in a wrong way, which should be discarded here. G is a compact summary of the seven diagnostics defined in Section 4, not a measure of grammatical correctness, historical authenticity, or human-level quality: a score of 0.9911 does not mean that the text is ‘99.11% correct’; it means that the text scores consistently high under these seven questions and no others. The components do not share units; that is to say, a character cosine, a frequency divergence and an attestation rate measure different properties on scales that merely happen to fit inside [0, 1], so the mean does not make them commensurable and the disaggregated components keep the primary linguistic interpretation at all times. Three components, moreover, partly restate the procedure: attestation, independence and length were enforced by the validator presented in Section 3, so that their high values measure successful compliance as much as linguistic quality; and the information in G is in the layers that no constraint could see: frequency, character, word structure, syntactic construction.

The objection can be put in quantitative form, and it deserves to be answered that way. If the three compliance components are removed and the overall score is recomputed as the geometric mean of the four free measurements alone, the result is G_free = (W·F·S·C)^1/4 = 0.9911, against G = 0.9911 for the full seven. The two differ by seven parts in a hundred thousand, well inside the bootstrap interval of either. The reason lies in the arithmetic of the geometric mean: A, I and L stand at 0.99997, 0.99761 and 0.97550, close enough to the ceiling and to each other that they neither raise nor lower the composite appreciably. The compliance components are therefore worth reporting, because they certify that the validator did its work and that the released corpus is the corpus described, but they do not produce the score. What moves G is the character surface at 0.98189 and, behind it, the frequency profile and the head–relation–dependent triples, none of which any constraint could see and none of which the generator was asked to optimise.

This architecture begs another question: Does a generator working under hard lexical constraints produce not Old English but a new, artificial variety assembled from attested pieces? As for vocabulary, the text contains no invented words in the strict sense: every surface form belongs to the pre-compiled lexicon of attested forms, and the residue of the independent attestation check is two types (0.0026% of tokens), both of them headwords in reference works outside the index. The proportion of neologisms is therefore zero by construction and near zero by independent verification. As for the deeper concern, a text could combine attested forms into combinations no scribe would produce, but this is exactly what the unconstrained layers test. The validator saw only surface forms, line lengths and copying; it could not see frequency balance, character texture, embedding geometry or clause structure. That the text nevertheless matches the original on those four layers, and stands at the distance of the original from the DOEC on each of them, is evidence that the constraints guided the generation toward the attested language rather than toward an artificial recombination of it. Had the output been a private idiom of attested pieces, the frequency bins, the n-gram profile and the head–relation–dependent triples would have indicated the departure.

As a direct answer to the plausibility question, the generated text is attested Old English throughout, to the limit of the reference resources: two word types in 114,451 tokens survive every linking layer, and both are headwords elsewhere. Its spelling, its balance of common and rare vocabulary, its word structure and its clause architecture all stand at the distance of the original from the language at large. It copies nothing beyond the lists and formulas that any faithful imitator would share. Its distinctive profile is mild: slightly heavier clauses, the finite verb at the end of the clause a little more often, and a vocabulary that draws on fewer types than Wærferth’s.

For philological purposes, adequacy rests on several procedures and criteria, in such a way that the assessment can be read against them one by one. Textual plausibility is guaranteed by construction and confirmed by measurement. Structural compositionality is measured and near the source. Discourse-register continuity is carried by the line-by-line design and visible in the qualitative reading. Lexical richness is the one construct on which the text falls short of its model, and the three layers that detect it agree on its size. Representational robustness is protected by the verified embedding mapping and by reporting three independent representation tracks. Inferential stability is carried by the bootstrap. The comparison with the DOEC deserves one more comment, because it is the methodological heart of the test. A generated text should not be asked to be at zero distance from the language at large; the original does not. It should be asked to be at the same distance as its source is. That reference behaviour turns every ‘how close is close enough?’ question into a measurement with a baseline.

Some limits remain. The structural comparison uses a bounded DOEC subset. This is a concession to tractability that must be flagged. The parser underperforms its published baseline by about three LAS points, and although all corpora are parsed with the same model, so that systematic error does not take place in the comparisons, relation-level conclusions inherit the weaknesses of the parser on clausal complements. The curated lexical probes of the wider pipeline have not yet been run on this text. Fine lexical relations are therefore measured only indirectly, through the embedding track. Semantic fidelity is measured only globally, for the reason given in Section 5: the centroid identifies the region, not the line.

Meaning preservation at the level of the individual sentence is not measured here, and it is worth setting out precisely what would measure it, since the corpus was built so that the measurement can be made. Each of the 4217 rows carries four aligned fields: the original Old English, a Present-Day English rendering of it, the generated Old English, and a literal Present-Day English rendering of the generated line. Three procedures follow from that design. The first computes a similarity per pair rather than per corpus, over the two Present-Day English renderings, which turns the single centroid cosine into a distribution of 4217 values whose low tail can be isolated and read. The second applies bidirectional lexical alignment between the two Old English lines, showing which content words of the source have a counterpart in the generation and which have been dropped or added. The third submits a stratified sample to philological adjudication, in which a reader marks each pair as preserving, narrowing, broadening or altering the sense of its source. Only the third settles the question; the first two identify the sentences on which it is worth spending. One caveat governs the first procedure and must be stated with any result it produces: the gloss of the generated line was written by the generator itself, so agreement between the two renderings is evidence about the internal consistency of the model rather than an independent judgement of fidelity.

A natural question is how the indexes would deteriorate if the constraints were removed one at a time. This ablation cannot be run on the present corpus, because the four constraints were enforced during generation rather than applied as filters afterwards. This means that disabling one changes the generation process itself, so each ablated condition requires a full regeneration of the 4217 lines, which constitutes a new sample from the model with intervals of its own (Section 3). What the present design does establish is the direction of the dependence. Three compliance components would degrade mechanically: without the lexicon, out-of-lexicon tokens would reappear, as pilot runs without the closed list showed; without the length band, L would fall with the token ratio; without the Jaccard ceiling, I would admit copies. The informative question is whether the four free layers (frequency, character, word geometry, structure) would deteriorate with them, and by how much: this factorial regeneration, one constraint disabled at a time under the recorded settings and scored by the same pipeline, is a complement to the adversarial tests proposed below. Nor can the ablation be reconstructed from the archive after the fact. The constraints operated inside a repair loop in which every flagged line was rewritten until it passed, so the deposited batch outputs are the repaired text and no unconstrained draft of any line survives. We therefore state the expectation that the experiment would test, so that it can be held against us. Because the four free layers were never visible to the validator, removing the lexical constraint should degrade attestation sharply while leaving character texture, frequency balance and syntactic triples substantially intact. Were the free layers instead to fall with it, the reading offered here, that the constraints guided generation toward the attested language rather than manufacturing the result, would be wrong, and the ablation is the experiment that would show it.

The blind detection experiments delimit what the G-index is and is not. On the one hand, the index is not redundant with expert judgement. On the other hand, the divergence it measures, mainly the depletion of rare and hapax vocabulary, 3.5% of token mass against 7.3% in the original (p < 10⁻⁶), is robust at corpus scale but inaccessible to experts working sentence by sentence.

The 4217 aligned pairs stand ready for the local analysis (one similarity per pair, the distribution examined, the low tail read philologically, with bidirectional lexical alignment where the reading needs support). This analysis remains a pending task for future work. Replication across genres, such as homiletic, legal, annalistic, and scientific prose, is needed before wider claims. Future research may also take further steps on the adversarial and the anchoring sides. On the adversarial side, G should be tested against texts built to fool it: a text that copies half the source and paraphrases the rest, a text in perfect Old English on the wrong topics, or a modern text in Old English spelling and, on the same side, generations with one constraint disabled at a time, to measure the contribution of each constraint to each layer. Each should fail on the component designed to catch it, and the composite should fall with it. On the anchoring side, the scale needs calibration from above: human rewritings of the same genre, scored by the same pipeline and judged independently by philologists, would show where good human work proceeds and what slight qualitative differences are worth.

7. Conclusions

The question addressed in this article was whether a multi-layer pipeline can give valid, interpretable and reproducible evidence about machine-made Old English when no parallel reference exists. The Gregory case answers it in the affirmative and adds something the earlier framing did not have: a demonstration that independent layers can converge and that, when they converge, their convergence evinces a single overall score.

The evidence can be considered with respect to the sources, the plausibility of the generated language, and the distance from the original. From the perspective of the sources, the generated text matches its original in attestation, frequency balance, spelling, lexical geometry, and clause structure, with divergences in the third decimal. Regarding the plausibility of the generated language, on every layer, the generated text keeps the distance of the original from the DOEC. As for the distance between the original and the generated text, their closeness is not achieved by copying, since no line repeats the source and shared spans cover a quarter of one percent of the text.

The proposed metric G folds the seven partial results into one number, 0.9911 with interval [0.9906, 0.9916], built so that the worst layer governs and every component stays open to inspection. The number is not the finding. The finding is the method behind it: a variation that is attested throughout, structurally faithful, orthographically plain, narrower in vocabulary than its model and different from the original. The broader implication for historical language generation is that progress depends less on finding one best metric and more on building a transparent set of measures, each with a clear function, explicit uncertainty and known relations to the others.

Blind detection experiments with expert judges confirmed that the divergence G measures is real but inaccessible to sentence-by-sentence reading—two-thirds of generated sentences passed as authentic—and fixed the index’s scope: it quantifies distributional fidelity at the level of the corpus, and it is at that level that it should be applied and cited.

8. Reproducibility and Data Availability

All data and materials supporting this study are openly available on Hugging Face at Nerthus-Project/Generated_OE_Gregory_Dialogues_Text_and_Evaluation (https://huggingface.co/datasets/Nerthus-Project/Generated_OE_Gregory_Dialogues_Text_and_Evaluation) (accessed on 13 September 2026). Two levels of reproducibility should be distinguished, as set out at the end of Section 3. The evaluation is reproducible from the deposit, which holds the generated corpus, the dependency parses, the embedding space, the evaluation scripts with the bootstrap code, the generation instructions and the validator, together with the numerical output behind every figure reported here. The two diagnostics added in revision, the rare-and-hapax contrast and the character-track cut-off, are deposited with the derived aggregate counts they read, so both recompute without access to any licenced resource. The Dictionary of Old English, its index and the corpus itself are licenced and cannot be redistributed, so the layers that consult them recompute only where those resources are held; the lexicon of 227,899 admissible forms is likewise not deposited, though the script that builds it from the dictionaries is. The generation, by contrast, is not exactly reproducible and cannot be made so by any measure under the control of the authors, since it ran through a commercial access point whose server-side configuration lies outside that control. What can be done, and has been done, is to fix the object of study rather than the process that produced it. The generated corpus is deposited as a single file, data/gregory_dialogues_generated_oe.tsv, whose SHA-256 digest is a11c1fb3c9f7717e0c55dea37b664b343a77c15a5fe3ce87f9438f8fdf3fe1ee, so that any later run can be compared against this one line by line even where it cannot be made to repeat it.