Agentic Cryptographic Debt: Repository-Level Measurement of Post-Quantum Migration Regression Under Autonomous AI Software Development

Abstract

Post-quantum migration is becoming binding for regulated systems, and delegating it to AI coding assistants is a plausible response. We measured what two open-weight coding models produced when asked to migrate a repository from RSA-based JSON Web Token signing to ML-DSA (FIPS 204), scored by static analysis and execution, on a pinned Go application whose ecosystem supplies ML-DSA and a Python service whose JOSE dependency lacks support in five audited libraries. Across 56 runs in four conditions plus a nested ablation, no run migrated successfully where a conforming primitive was available, and none accurately reported the blocker. Tools and a compiler made failure surface later, not less often: no modified artifact compiled unless ML-DSA key generation had been stubbed, and the two runs that compiled reported success without a working ML-DSA signing path. Qwen3-Coder substituted Ed25519 in all five feasible-arm one-shot runs, despite identifying its quantum vulnerability in a separate direct-question probe. For Devstral Small 2, naming the correct library and selected API functions eliminated module hallucination and stubbing but produced no successful migration: all five runs made the same API error, and two incorrectly declared the migration impossible. A reference migration built after review with the same library and disclosed scope extensions passes the scorer; scoring it exposed four acceptance-path defects that would each have rejected a correct migration. Results are descriptive and bounded to the models, repositories, and runs examined. To our knowledge, this is the first repository-level evaluation of a standardized post-quantum migration to score blocker reporting alongside outcome.

1. Introduction

NIST finalized the first post-quantum cryptographic standards on 13 August 2024: FIPS 203 (ML-KEM) [1], FIPS 204 (ML-DSA) [2], and FIPS 205 (SLH-DSA) [3]. Migration to them is no longer discretionary for large classes of systems. NIST IR 8547 [4], an initial public draft at the time of writing, proposes that RSA, ECDSA, ECDH, DSA, and finite-field Diffie–Hellman parameter sets providing 112 bits of security be deprecated after 2030 and disallowed after 2035 in federal systems, and that EdDSA and the higher-strength parameter sets of the other quantum-vulnerable algorithms be disallowed after 2035. The NSA’s CNSA 2.0 suite moves faster for National Security Systems [5]: the 2022 advisory directed software and firmware signing to begin transitioning immediately, to support and prefer CNSA 2.0 by 2025, and to use it exclusively by 2030, with staged deadlines for other categories running to 2033 and an overall transition target of 2035 in line with NSM-10. The advisory states that this will effectively deprecate RSA, Diffie–Hellman, ECDH, and ECDSA when mandated, and it specifies Level V parameters for the lattice signature scheme, that is ML-DSA-87.

The scale is the problem. Migration is not a single change, but a diffuse one: every issuer, every verifier, every published key set, every trust anchor, every serialization that assumes a key size. Cryptographic discovery is itself treated as the binding first constraint by the principal migration guidance (Section 2.3), and a timeline analysis of enterprise migration estimates the discovery phase alone at 1 to 3 years depending on organizational size and asset-management maturity, within baseline end-to-end migrations of 5 to 7 years for small enterprises and 12 to 15 or more for large ones [6]. Against that backlog, and within deadlines measured in single-digit years, delegating the code changes to AI coding assistants is a plausible organizational response, and the tools are already widely used: the 2025 Stack Overflow Developer Survey reports that 84 percent of more than 49,000 respondents across 177 countries use or plan to use AI tools in their development process, and that 51 percent of professional developers use them daily [7].

This creates a measurement problem that existing evaluations do not address. Benchmarks for AI coding assistants measure functional correctness: does the change resolve the issue, do the tests pass, does the build succeed. Post-quantum migration is not primarily a functional task. A migration can compile, pass every test, satisfy code review, and update its own documentation while leaving the service exactly as quantum-vulnerable as before. Nothing in a functional evaluation distinguishes that outcome from a correct one.

We call the resulting shortfall cryptographic debt: the difference between a migration’s apparent completion and its realized security effect. Debt is maximized precisely when the artifact is most convincing, because a plausible completed migration terminates the remediation effort. An obviously broken migration gets fixed. A convincing broken migration gets merged, closed, and reported as complete.

This paper measures what two open-weight coding models, qwen3-coder and devstral-small-2, at the checkpoints given in Section 4.6, produced in July 2026 when asked to perform a repository-level post-quantum migration, and at what stage the failures become visible.

Contributions.

This study makes six contributions.

• A security-property definition of cryptographic migration success. We define agentic cryptographic debt as the gap between a migration’s apparent completion and the cryptographic assurance realized by the resulting artifact. The definition separates build-visible failure, where the obstacle surfaces as an error, from build-green concealed regression, where it does not, and from fully silent failure, where no downstream signal exists at all. We observe the first two. We do not observe the third, and say so where it matters: the concealed artifact retains a classical signing call that fails when that path is exercised, so what it defeats is the build, not every signal (Section 5.6).
• A repository-level post-quantum migration evaluation apparatus, and an acceptance criterion it does not yet meet. The scorer combines repository inspection with execution: a build, a resolved module graph, an imported and called post-quantum package, rejection of a dependency satisfied by a local replacement, absence of reachable classical signing, and passing tests on the cryptographic packages. Every failure this paper reports is a rejection, and the rejections rest on execution: directly on the feasible arm, and on the infeasible one-shot arm, whose runners carried no Layer 2 adapter, through the feedback probes and the feasibility audit that executed the same identifiers (Section 4.5). Its acceptance path does not, and we specify what one would require rather than imply we have it: ML-DSA signing and verification exercised end to end, an RFC 9964 [8] algorithm identifier parsed from the JWS header, an AKP key representation retrieved from the published key set, a tampered signature rejected, a classical algorithm refused, and the migrated path shown to be the one the application reaches. No run in this study reached the acceptance path, so no result depends on it; the gap is stated as Limitation 8. The merge stage additionally refuses to certify correctness where the behavioral layer did not run, which is a constraint we recommend and which our own apparatus violated until it was implemented (Section 6.2, Defect 15).
• A feasibility-aware corpus design, with the two arms established to different depths. One target is in an ecosystem audited library by library and found not to expose the required standardized capability, so its infeasibility is established completely: every RFC 9964 identifier was attempted against each library and each raised. The other is in an ecosystem where a conforming primitive exists and is verified working, though the application-level migration also requires a custom signing method and a hand-constructed key representation, and a reference migration built after peer review, accepted by the scorer and passing six behavioral checks, demonstrates the whole path with the scope extensions that section discloses (Section 4.2, Positive-Control Witness). The first arm measures whether an agent reports an implementation blocker or fabricates, substitutes, or silently weakens the migration; the second measures migration execution against a target whose achievability that reference migration demonstrates (Section 4.2, Positive-Control Witness).
• A staged diagnosis of where migration failure persists. Across one-shot, compiler-feedback, bounded-agent, and dependency-specified conditions, we progressively remove requirements from the model. The sequence narrows the stage at which failure survives, from dependency discovery through error correction and tool use to API application and repository integration. It does not separate the explanations that remain available at that stage, and it traces one model: only devstral-small-2 was run in all four conditions, and the other model failed by a different mechanism that the ladder does not describe (Section 6.1).
• Empirical evidence of concealed migration regression. Under the tested models and conditions, execution feedback and autonomy did not produce a correct migration. Autonomy moved failure from an overt build error to a build-passing artifact reported as complete while lacking the requested cryptographic property: no modified artifact compiled unless ML-DSA key generation had been stubbed. Supplying the correct dependency removed the dependency-identification failures without resolving API application or repository integration (Section 5.5, Section 5.6 and Section 5.7).
• An adversarial methodology for validating migration evaluators. We document twenty defects in our own prompts, artifact transport, execution, and acceptance criteria, and show that their direction of bias is predicted by what each does to the evidence rather than by where it sits. The released apparatus carries the scorer and its merge rule, checksummed and archived with the run records; a validation runner naming each case and the finding it was required to produce, with the 11 fixture inputs reconstructed after review and archived so that the runner executes; machine-readable evidence bundles for model provenance, ecosystem feasibility, and the context-sensitivity ablation; and verification scripts that re-derive the allocation, the merged verdicts, and the static-layer findings from the archive rather than from this paper. The package also carries a known-good migration witness for the feasible arm, built after peer review. Scoring it exercised the acceptance path for the first time and exposed four further defects in that path, each of which would have rejected a correct migration; they are defects 17 to 20 of Section 6.2, and the reason the count is 20 rather than the 16 reported at submission.

Research questions. This study is organized around five questions, each answered by one or more of the four primary conditions.

• RQ1. Can a coding model produce a working repository-level ML-DSA migration when a conforming cryptographic primitive is available in the target ecosystem (Section 5.3 and Section 5.4)? The question is general; the evidence in this paper bears on the two models of Section 4.6 only, and the availability is established by a reference migration that passes the scorer and six behavioral checks (Section 4.2, Positive-Control Witness).
• RQ2. Do models recognize and accurately report an unavailable migration target (Section 5.2)?
• RQ3. Does compiler or interpreter feedback correct the observed migration errors (Section 5.5)?
• RQ4. Does bounded autonomy improve correctness, or alter the stage at which failure becomes visible (Section 5.6)?
• RQ5. Does supplying the correct dependency and its API resolve the failure (Section 5.7)?

RQ1 and RQ2 are answered on the feasible and infeasible arms, respectively; RQ3, RQ4, and RQ5 each correspond to one condition and are answered by comparison with the conditions preceding them.

Outline. The remainder of this paper is organized as follows. Section 2 reviews the evaluation of AI coding assistants, package and API hallucination, and post-quantum migration practice, and positions this study against its nearest antecedents, stating which identified gaps each contribution addresses. Section 3 states the threat model, defines cryptographic debt and migration regression, and summarizes the failure severity ordering. Section 4 describes the corpus, the feasibility audit that splits it into a feasible and an infeasible arm, the task conditions, the harness, the scorer, and the models. Section 5 reports the results by arm and by condition. Section 6 discusses what the four primary conditions establish, reports the defects found in our own apparatus and the direction of bias each introduces, states the limitations, and sets out the implications for practice and for further research. Section 7 concludes.

2. Related Work

2.1. Evaluation of AI Coding Assistants

Contemporary evaluation of code-generating models centers on functional correctness against issue-resolution benchmarks, of which SWE-bench [9] and its human-validated subset SWE-bench Verified [10] are the most widely reported. These measure whether a model’s patch makes a failing test suite pass. Two properties of that paradigm matter here. First, the measured property is functional, and a cryptographic migration can be functionally indistinguishable from a non-migration. Second, benchmark results are sensitive to the surrounding agent harness, and results obtained with different harnesses are not directly comparable. Section 6.2 extends that caution with an observation from our own harness: the defects we found were not merely a comparability problem, and they did not fall symmetrically.

The 68.0% SWE-bench Verified score for devstral-small-2 is confirmed against the vendor’s own release announcement [11], where it is reported for the 24B model. The announcement does not state which agent scaffold produced it, and we do not attribute one. That omission is itself the point. SWE-bench Verified is an issue-resolution benchmark: a score on it is not a property of a model alone, but of a model inside a harness that can read a repository, edit files, and run tests, and published harnesses of that kind differ substantially in what they expose, as the OpenHands platform illustrates [12]. A figure reported without naming its scaffold therefore cannot be compared in detail with a figure obtained under a different one, which is the situation here: a model measured at 68% issue-resolution under an unstated harness nonetheless stubbed the cryptography and confabulated a blocker under ours. The harness-dependence caveat therefore applies to our own cited number, and the gap between the two harnesses means the published score is no evidence of success under this task and harness, which is the narrower claim we make.

We use this figure for one purpose only: to establish that the model in question is a competent contemporary coding model rather than a weak one, so that its failures here are not attributable to obvious incapacity. We do not use it to rank the two models evaluated. No comparable published figure exists for qwen3-coder, and a score obtained under one agent scaffold does not order models evaluated under another. Where the two models behave differently in this study, we report the difference and name the models, and we do not attribute it to a capability ordering.

A second limitation of the issue-resolution paradigm is its unit of analysis. SWE-bench instances are drawn from single issues with an accompanying gold patch and test suite, and success is scored per instance. A cryptographic migration is not an issue in that sense: it is a property that must hold across every issuer, verifier, published key set, and serialization in a repository simultaneously, and it has no failing test that turns green when it is done correctly. Recent benchmark work has begun to move toward repository-scale and beyond-issue-resolution evaluation, and toward agent–computer interfaces as a distinct object of study [13]. We take a complementary route: rather than a broad benchmark, we measure one migration deeply, with a scorer that can distinguish a change that compiles from a change that is cryptographically correct.

2.2. Package and API Hallucination

The failure mode most directly adjacent to ours is package hallucination. Spracklen et al. [14] generated 576,000 code samples across 16 models and two prompt datasets in Python and JavaScript, producing 2.23 million package references, of which 440,445 (19.7%) were hallucinations, including 205,474 unique nonexistent package names. The rate was strongly model-dependent: roughly 21.7% for open-source models against roughly 5.2% for commercial ones on Python, and the authors report it as persistent across sampling temperatures, characterizing the risk as structural rather than configurational.

This behavior has a security consequence independent of correctness. Package confusion attacks are long-established: typosquatting exploits developer typing errors, and Birsan’s 2021 dependency-confusion disclosure [15] showed that internal-only package names shadowed on a public registry can be substituted silently at build time. The LLM-induced variant, termed slopsquatting [16], inverts the causal order: rather than waiting for a typo, an adversary registers the names that models have been shown to invent.

Our relation to this work. Spracklen et al. measure hallucination prevalence and its persistence across sampling temperature. We measure a different and stronger axis of persistence: persistence under corrective execution feedback. Where their result establishes that hallucination rates do not fall when decoding parameters change, we observe that a hallucinated module is re-proposed in five of five runs after the model has been shown a machine-generated compiler error stating that the exact module does not exist (Section 5.5). We also situate the phenomenon in a domain where the hallucination has a specific security consequence, a missing cryptographic dependency, rather than a generic one, and we observe a stable invented identifier (ML-DSA-8443, annotated in generated source as an IANA registration) recurring across independent runs (Section 5.6). The persistence we document is not addressed by the mitigations Spracklen et al. evaluate, since compiler feedback is a stronger corrective signal than temperature adjustment and does not resolve it.

Package-confusion attacks are documented independently of language models [17].

2.3. Post-Quantum Migration Practice

The operational difficulty of cryptographic migration is documented independently of any AI dimension. The NIST National Cybersecurity Center of Excellence (NCCoE) established the Migration to Post-Quantum Cryptography project and published the SP 1800-38 practice guide series [18,19], developed with more than two dozen industry collaborators including AWS, IBM, Microsoft, and PQShield. The volumes remain preliminary drafts, with subsequent findings published through separate white papers, technical notes, and reports. Two of its findings frame the present work.

First, the guide identifies cryptographic discovery as the binding first constraint: most organizations lack visibility into where cryptography is used in their systems, and SP 1800-38B [19] recommends building a Cryptographic Bill of Materials (CBOM) by expanding existing asset-inventory processes. The premise is that you cannot migrate what you cannot locate. Our study operates one level past discovery: it assumes the location is known (the task names the files) and measures whether the change made there realizes the intended algorithm.

Second, the guide repeatedly notes that prior cryptographic transitions have taken many years, and that organizations often lack complete control over their cryptographic mechanisms, requiring intensive manual effort. That base rate is the reason migration work is a candidate for delegation to AI assistants, and it is the reason a silently incorrect migration is costly: it consumes remediation capacity that is already the scarce resource.

The CBOM literature also states plainly that not all protocols have post-quantum options available yet, and that a CBOM’s value is partly in tracking which classical algorithms can be upgraded as options mature. Our feasible/infeasible corpus split (Section 4.2) is a direct instance of that condition: the JOSE serialization for ML-DSA (RFC 9964) was 2 months old at the time of study and unsupported by any of the five Python JOSE libraries we audited, while the Go ecosystem had a working implementation. A migration tool, human or AI, must be able to distinguish “not yet implementable” from “implement it now”, and our central finding is that the AI assistants did not.

This study continues a line of the author’s work on the practical constraints of post-quantum migration. Campbell [20] evaluated the exposure of ECDSA across major public and enterprise distributed-ledger systems and used ECDSA P-256 key and signature sizes as a controlled variable against NIST post-quantum candidates, arguing for near-term lattice-based migration rather than deferral until standards were finalized, on the grounds that the arrival of fault-tolerant quantum computing is unpredictable. The present work sits at the other end of that argument’s time horizon: the standards it advocated migrating toward are now finalized (FIPS 204, 2024) and given a JOSE serialization (RFC 9964, 2026), and we measure what happens when the migration is delegated to AI coding assistants. The 2019 concern was that organizations would wait too long to begin; the present finding is that when they do begin, with automated help, the tools produce migrations that appear complete and are not.

The author has also analyzed enterprise migration timelines directly [6], estimating baseline end-to-end migrations of 5 to 7 years for small enterprises, 8 to 12 for medium, and 12 to 15 or more for large, with cryptographic discovery alone consuming 1 to 3 years. That analysis motivates the present question: if migration is a decade-scale program executed under deadlines measured in single-digit years, the correctness of automated assistance is not a secondary concern.

Most directly, this paper is the empirical complement to the author’s conceptual framework for the same problem. Campbell [21] organizes agentic cryptography into seven migration surfaces and separates each machine identity into a credential layer (symmetric, operator-held, lower risk) and a trust-anchor layer (the asymmetric roots that underwrite a fleet of agents), arguing that non-human-identity-dense, runtime-negotiated agentic systems constitute a distinct post-quantum migration class not addressed by enterprise migration guidance. That work specifies what must be migrated and why. The present work measures what current AI coding assistants actually produce when asked to perform such a migration, and finds that the trust-anchor layer, the asymmetric signing that the framework identifies as the high-risk surface, is precisely where the tools fail: they remove the classical algorithm and either substitute another classical one, confabulate an implementation, or stub the cryptography entirely, while reporting success.

The remaining pillar of this line of work runs in the opposite direction. Campbell [22] analyzes how frontier-model capability could compress defender-feasible PQC migration timelines for highest-exposure systems from the decade-scale baselines of Ref. [6] to a projected 2- to 4-year window, and discloses as its central limitation that capability transfer from software-engineering benchmarks to PQC-specific migration sub-tasks is plausible but unverified, pairing each compression claim with a falsification criterion and inviting empirical calibration. The present study supplies that calibration at the execution phase, for the assistant tier that organizations can obtain today rather than the restricted-access frontier tier that the projection concerns, and the transfer assumption fails there in a specific way: the models know which dependency the migration requires and cannot apply it, and the failure is not accurately reported. The result does not falsify the compressed-track projection, which this study does not test; it establishes that, for the two checkpoints and conditions tested, the assistants tried did not complete the execution phase of that projection, and it does not establish whether other generally available assistants, harnesses, or documentation-equipped workflows would; that population is the one the earlier analysis identifies as most exposed while frontier access remains gated.

2.4. Position Relative to Existing Work

Three lines of work bear directly on this study. We claim priority over none of them individually, and each is credited below for what it establishes. What we do claim is the conjunction: to the best of our knowledge, this is the first repository-level evaluation of a standardized post-quantum migration to separate implementable from audited-unavailable targets and to score truthful blocker reporting alongside migration outcome. The paragraphs that follow set out what each antecedent covers, so that the conjunction can be checked rather than taken on trust.

Migration of cryptographic code by language models. Pallarés de Bonrostro et al. [23] evaluate whether language models can migrate pre-quantum cryptographic code to post-quantum counterparts, using a synthetic but executable dataset of 800 paired Python fragments across six cryptographic families, and report that fine-tuning is decisive: a fine-tuned model reaches 92.5 percent functional correctness against much weaker zero-shot performance. Their primary evaluation is fragment-level and synthetic, scored on functional correctness and static similarity to a reference migration, and is supplemented by a complementary validation on six open-source repositories that examines whether fragment-trained migrations transfer to localized cryptographic modules embedded in real projects. They report that this exercise exposed limitations arising from complex dependencies and cross-module interactions, and characterize it as validation of fragment-level migrations rather than as a repository-level migration benchmark. Wahlang and Vidhani [24] approach the same subject as a supervised translation task: they fine-tune sequence-to-sequence code language models on a synthetic corpus of paired classical and post-quantum Python programs, using AST-based chunking to fit context limits, and evaluate the translated programs directly. We do not claim first use of language models for post-quantum code migration; that priority belongs to this line of work. The framing there is translation of self-contained programs against known targets, with no repository scope, no feasibility constraint, and no measurement of how the model reports an unavailable target. That work and this one ask adjacent questions at different scales. Ours is repository-level on a pinned third-party application and an authored issuer–verifier service, and scores whether the cryptographic property was realized in the artifact rather than whether the output resembles a correct answer. The distinction matters because a fragment can be judged against a known-correct counterpart, whereas a repository migration has no single reference form and can be functionally indistinguishable from a non-migration.

Repository-level migration benchmarks. TimeMachine-bench [25] evaluates agent-based migration on real Python repositories whose tests fail after dependency updates, and reports that models produce spurious solutions exploiting low test coverage. That finding is the closest antecedent to our Section 5.6 result, and we do not claim priority over it. The difference is in what the spurious solution exploits. There, the agent satisfies a test suite that does not cover the changed behavior. Here, the agent satisfies a build that cannot express the property at issue at all: compilation is indifferent to which signature algorithm is used, and ordinary functional coverage need not detect the substitution unless a test explicitly exercises the migrated signing path and verifies the algorithm, which is the property the task was issued in order to establish.

CodePlan [26] is the closest antecedent for the mechanics of the task rather than its subject: it synthesizes multi-step edit plans across a whole codebase and is evaluated on repository-scale tasks including package migration, with validity judged by repository-level criteria such as a successful build and correct edits across several files. It targets no cryptographic objective and does not evaluate one, so it establishes that repository-level migration is a studied problem without bearing on whether a requested cryptographic property was realized, which is the question here. The distinction is the one that separates a build from a migration throughout this paper: CodePlan’s success criteria are satisfiable by a repository that compiles and passes its tests, and that is the condition under which a cryptographic migration can fail invisibly.

Security of repository-level generated code. SecRepoBench [27] and A.S.E [28] evaluate whether AI-generated code in real repositories is secure, using static, dynamic, and expert-defined checks, and both report substantially lower secure completion rates than simpler benchmarks. These measure whether generated code introduces vulnerabilities. They do not measure whether a requested security migration was carried out, which is a different question: the failure mode here is not an introduced weakness, but a requested strengthening that did not occur while being reported as complete.

The gap this study occupies. The priority claim stated above is over the following conjunction, and over no element of it alone: a repository-level evaluation of a specific standardized cryptographic migration; an explicit separation of implementable from audited-unavailable targets, with obstacle reporting scored in the arm where the migration cannot be completed; a comparison across one-shot, compiler-feedback, bounded-agent, and library-specified conditions, which narrows the stage at which the failure occurs; and adjudication of realized migration through combined static, build, module-graph, and published-metadata evidence rather than through functional correctness or similarity to a reference. We also report a directional analysis of defects in our own measurement apparatus, which we have not seen stated as a finding elsewhere.

Which gaps the contributions address. The correspondence between the gaps identified above and the six contributions of Section 1 is as follows. The absence of a repository-level evaluation scored on the realized cryptographic property, rather than on functional correctness or on similarity to a reference migration, is addressed by Contribution 1, the security-property definition, and Contribution 2, the apparatus whose rejections rest on execution. The absence of any feasibility constraint, and of any measurement of how a model reports an unavailable target, in the migration-by-language-model line is addressed by Contribution 3, the two-arm corpus, and is answered by RQ2. The absence of a comparison across degrees of autonomy on a cryptographic objective, which the repository-level migration benchmarks do not attempt, is addressed by Contribution 4, the staged diagnosis, and Contribution 5, the evidence that autonomy converts build-visible failure into concealed regression. Contribution 6, the directional analysis of apparatus defects, answers no gap named in the literature above, because we have not seen the question posed there; it is offered as a methodological result. The positive case arrived only after the fact: no known-good migration existed when the runs were scored, one was constructed after peer review and is accepted by the corrected scorer, and the six behavioral checks that certify it remain outside Layer 2 (Section 4.2, Positive-Control Witness; Section 6.3, Limitation 8).

The structural point underlying all of this is that functional benchmarks cannot be relied on to express the property at issue, because a migration that silently retains classical cryptography is functionally identical to one that does not.

3. Threat Model and Definitions

Cryptographic debt. The gap between a migration’s apparent completion and its realized security property. A change that removes RSA, installs a modern signature scheme, and updates documentation, yet provides no post-quantum resistance, incurs maximal debt: the artifact terminates the remediation effort while delivering none of its benefit.

Migration regression. A change that reduces, or fails to increase, the cryptographic assurance of the system while presenting as a completed migration.

Failure severity ordering. We distinguish failures by the stage at which they become visible, because the stage determines who bears the cost and whether the migration is believed to be finished. The three categories are summarized in

Table 1. The failure severity ordering, summarized. Categories are ordered by the stage at which the failure becomes visible; severity increases down the table because a later stage means a longer-held false belief that the migration is complete. The final column records which categories this study observed.

Category	Where the Failure Becomes Visible	Who Bears the Cost	Observed in This Study
Build-visible failure	At the build: the artifact does not compile	Developer time; the failure is corrected	Yes (Section 5.3, Section 5.5 and Section 5.7)
Build-green concealed regression	After the build: at runtime, in test, or in review	A false belief that migration is complete, held until the artifact is exercised	Yes (Section 5.6)
Fully silent failure	Not at all: the artifact compiles, runs, and passes review	The same false belief, held indefinitely	No

.

• Build-visible failure. The artifact does not compile. Any build pipeline catches it, and the cost is developer time.
• Build-green concealed regression. The artifact compiles and satisfies the success signal available to the agent, and is therefore reported and recorded as a completed migration, but the cryptographic property the task was issued to establish is absent. The failure, if it surfaces at all, surfaces later: at runtime, in test, or in review. The cost is a false belief that migration is complete, held for as long as the artifact is not exercised.
• Fully silent failure. The artifact compiles, runs, and passes review while providing no post-quantum security. The cost is the same false belief, held indefinitely.

The ordering is by severity under this threat model: concealed failures are more severe than build-visible ones because they can terminate remediation without delivering the intended assurance. A build-visible failure costs developer time and is corrected. A concealed one closes a migration ticket, and the system carries quantum-vulnerable signing while its inventory records the work as done.

We observed build-green concealed regressions and did not observe a fully silent failure. The two agentic artifacts that reached a green build and were reported complete both retain an RS256 signing call that is handed a byte slice where an RSA key is expected, which compiles because the library parameter is an empty interface, and fails when the path is exercised (Section 5.6). We therefore claim the middle category and not the third. The distinction matters and we draw it explicitly: these artifacts would be caught by an adequate test suite, and were not caught by the build.

Engineering analogies. The middle category has familiar counterparts outside cryptography, and naming them may help readers from either discipline. A build-green concealed regression is the cryptographic analog of a type cast that silences the compiler by widening an interface, so that the code compiles and the guarantee the type was meant to carry is gone; here, the RS256 call compiles because the library parameter is an empty interface, and fails when handed the wrong key. It is the analog of a feature flag left pointing at the legacy code path after the new path was merged and documented, so that the release notes describe behavior the deployed system never exercises; here, the published key set advertises ML-DSA while the signing path still calls RS256. It is the analog of a TLS deployment whose configuration lists a strong cipher suite that the running binary was built without, so that the listener negotiates the fallback while the configuration audit passes. It is the analog of a spurious fix that satisfies a test suite that does not cover the changed behavior, which is the case TimeMachine-bench [25] documents. In each, the signal that normally terminates the work, a green build, a merged pull request, a closed ticket, a passing audit, is satisfied, and the property the work was undertaken to establish is not. The common structure is that the gate checks something the change can satisfy without delivering the property, and the artifact is convincing in proportion to how completely the gate is satisfied. That is why a concealed regression costs more than an overt one: the overt failure consumes developer time, and the concealed one consumes the belief that the work is done.

Two results bear on the ordering. First, the two models did not fail by the same mechanism: qwen3-coder substituted a classical scheme, while devstral-small-2 imported a module that does not exist; both failures were build-visible (Section 5.3). We draw no capability ordering from this. No published benchmark figure is available for qwen3-coder, and the figure available for devstral-small-2 was obtained under a different agent harness (Section 2.1), so the two are not comparable on any measure we have. The observation is that the mechanism of failure varied by model, not that it is predicted by capability. Second, granting the agent autonomy converted build-visible failure into build-green concealed regression (Section 5.6).

4. Method

4.1. Corpus

Four corpus items were prepared as archetypes. Two were exercised in this study (

Table 2. Corpus items. R1 supplies the feasible migration target and R3 the infeasible one. The two are of different kinds and the table records that rather than eliding it. R1 is a third-party application pinned by commit hash, obtained from its upstream and not redistributed here. R3 is a token service written for this study: a producer–consumer pair, an issuer and a verifier, that depends on Authlib for its JOSE layer. The dependency, not the service, carries the infeasibility, and it is the ecosystem audited in Section 4.2. An authored consumer was used for the infeasible arm because it makes the migration surface small and fully known, so the seeded condition is exact and the result is not confounded by unrelated cryptographic code elsewhere in a large library.

ID	Corpus Item	Provenance	Language	Surface
R1	eugenioenko/autentico v1.6.18	third party, pinned at commit 5757c73b	Go, 37k LOC	native RS256 JWT issuance plus JWKS and OIDC discovery
R3	token service, issuer.py and verifier.py	authored for this study, 87 lines, archived with the package	Python, depends on Authlib 1.7.2 [29]	RS256 issuance and verification over a published JWKS

). They differ in kind: R1 is a third-party application pinned to a commit, and R3 is a service written for this study. Each is reconstructible, R1 from its upstream at the pinned commit together with the archived seed, and R3 from the archived source directly.

R2 (an author-configured Ory Oathkeeper slice) and R4 (binwiederhier/ntfy, WebPush VAPID) were prepared but not run. Their feasibility is unresolved (Section 6.3).

Leakage gate. A seeded corpus can leak its own answer key: a condition identifier, a ground-truth reference, a scoring code, or a manifest field left in a file the model can read would tell it where the planted condition is and what the correct end-state looks like. The baseline delivered to the model is therefore scanned by an adversarial checker, leakage_gate.py, which fails on any of those markers and on the presence of the scorer-only directories. The checker was validated against planted leaks before use, so a pass is a result rather than a silent no-op. All three agent-facing baselines that produced runs pass: the R3 service, and the R1 baselines for the standard and library-named task variants. The R3 baseline was scanned before the runs; the two R1 baselines were scanned retrospectively, because the original gate pass covered R3 and the unused R2 item only and explicitly deferred R1. We record the timing rather than imply a uniform pre-run gate. The direction of the residual risk is worth stating: undetected leakage would help a model reach the correct answer, and no run reached it, so leakage cannot account for the failures reported here.

Redistribution and licensing. R1’s upstream is a third-party application and is not redistributed in the replication package: obtain it from its own source at the pinned commit and apply the seeding procedure. R3 is the author’s own work and is included. Authlib is a dependency of R3 rather than a corpus item and is likewise not redistributed. The package is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0), except the witness patch, which is GPL-3.0 as a derivative of Autentico; no patent license, express or implied, is granted by its publication.

4.2. Feasibility Audit and the Corpus Split

A migration cannot be measured if it cannot be performed. We therefore verified, per corpus item, whether its ecosystem exposes a conforming ML-DSA implementation at all.

The JOSE serialization is very new. RFC 9964, which registers ML-DSA for JOSE and COSE, was published in May 2026, 2 months before this study. It registers the alg values ML-DSA-44, ML-DSA-65, and ML-DSA-87, a new key type AKP, and the JWK parameters pub and priv; the private parameter must be the 32-byte seed rather than the expanded key. Absence of support for a standard of that age is expected and is not evidence of ecosystem failure. We audited five mainstream Python JOSE libraries at their then-current releases; the audit is a bounded sample, not a survey of the package index, and its result is stated as such throughout. The results are given in

Table 3. Migration-target availability by ecosystem, audited 22 July 2026 local time (bundle timestamp 03:57 UTC on 23 July 2026). The Python row reports a bounded audit of five named libraries, not a survey of the Python package index. Per-library versions, install digests, registry contents, and verbatim exceptions are given in Table S1.

Ecosystem	ML-DSA Available	Basis
Python JOSE, five audited libraries	no	Registry inspection and key-generation attempts against each library, with controls (Table S1)
Go	yes	github.com/cloudflare/circl/sign/mldsa/mldsa44 executed end-to-end: sign and verify round-trip succeeded, and a tampered message failed verification (Table S1)

.

Audit protocol. A dated negative claim cannot be verified from version numbers, so the audit records what was queried and what was returned. For each library we archive: the installed version, the install location, and a digest over the SHA-256 hashes of its Python source files, which identifies the exact artifact audited; the algorithm and key-type identifiers the library exposes, read from its own registries rather than from its documentation; every key generation attempted for the RFC 9964 alg values and the AKP key type, with the exception type and message returned verbatim; a control generation for a key type the library does support, so that a failure to generate is distinguishable from a broken invocation; a search of the installed package source for the identifiers RFC 9964 registers, which would catch support present but unregistered; and a UTC timestamp and environment capture.

A library is recorded as exposing support only on registry evidence or a successful generation. A source-search hit without registry exposure is reported as a discrepancy for manual review rather than counted either way; none arose. The script audit_jose_support.py performs this and emits the bundle, so the claim can be re-tested on any later date by re-running it.

The corpus therefore splits, and the split is the experimental contrast. R1 is a feasible cell and R3 an infeasible one, so failures on R1 measure migration debt while behavior on R3 measures confabulation against obstacle reporting. Autentico’s [30] go.mod declares go 1.24.0 and toolchain go1.24.2; CIRCL [31] requires Go 1.24.0 or later, so no toolchain change is needed.

What “feasible” is established to mean here, and what it is not. The two arms are not established to the same depth, and the asymmetry matters enough to state. For R3, the negative is complete: every RFC 9964 identifier was attempted against each audited library and each raised an exception, so none of those library versions, as installed, could complete the migration through their exposed interfaces; whether another Python integration route existed was not tested. For R1, the positive is established at the level of the cryptographic primitive: CIRCL’s ML-DSA-44 signs, verifies, and rejects a tampered signature. It is not established at the level of the complete application migration. The JWT library that Autentico uses has not shipped ML-DSA support, so a conforming migration requires registering a custom signing method, constructing the RFC 9964 AKP key representation for the published key set, and keeping the discovery metadata consistent with it. Each is achievable and none is provided by adding a dependency.

No reference migration of R1 existed when the runs were executed, and at submission, its absence bounded two claims: feasibility itself, which was demonstrated at the primitive and argued at the application level, and the difficulty baseline against which the runs are read. Both are now addressed by the positive-control witness described next, constructed after peer review and after every scored run, so that no run was informed by it.

Positive-Control Witness

What was built. The witness is a complete ML-DSA-44 migration of R1 at the pinned commit, built on 11 September 2026, and archived as a patch against that commit (witness/ in the replication package, licensed GPL-3.0 as a derivative of Autentico). It uses the dependency the library-named condition supplied, github.com/cloudflare/circl (resolved to v1.6.5 when the witness was built, which raised the module’s go directive from 1.24.0 to 1.25.0), and of its API, the three functions that task file named, GenerateKey, SignTo, and Verify, together with NewKeyFromSeed and the serialization methods of the key types. The JWT library Autentico depends on, dgrijalva/jwt-go v3.2.0, is no longer maintained upstream and its README directs users to a successor, so ML-DSA support will not arrive from it and the migration registers a custom signing method implementing the RFC 9964 JOSE binding: pure ML-DSA over the JWS signing input with an empty context string, under the algorithm identifier ML-DSA-44. Key material is held as the 32-byte seed and expanded with NewKeyFromSeed; the published key set is an RFC 9964 AKP JWK carrying the public key in pub; the discovery document advertises ML-DSA-44 only; the verifier is bound to the registered method; and the HS256 refresh-token path is untouched, as the task required.

Certifying suite. Six tests in the migrated repository perform the checks that Contribution 2 specifies, and all six pass: (1) a token issued by the application’s own issuance path verifies through its own verifier; (2) the JWS header alg is a FIPS 204 parameter-set identifier and not the bare family name, and the signature has the ML-DSA-44 length; (3) the JWKS handler publishes an AKP key from which the verifying key is retrieved and verifies the token; (4) a flipped signature bit, a truncated signature, and a modified payload are rejected; (5) an RS256 token, an alg-none token, and a genuine token whose header is rewritten to claim RS256 are refused; (6) the discovery document advertises no classical algorithm, the token verifies under the configured key and not under another ML-DSA-44 key, and the client-credentials issuance path also reaches the migrated signer. The test output is archived with the witness.

What lay outside the agents’ scope. The scope issued to the agents was pkg/key/, pkg/token/generate.go, pkg/token/decode.go, pkg/wellknown/handler.go, and pkg/jwtutil/validate.go, with test files excluded (Section 4.4). Four things outside it changed in the witness. pkg/model/api.go, whose JWK struct carried the RSA parameters, was changed to carry the AKP pub field for the key set to publish. pkg/cli/init.go, where autentico init writes the configured key, would otherwise continue to generate an RSA key that the migrated loader cannot decode; the loader then falls back to an ephemeral key with a warning, so the migrated path is still reached but tokens do not survive a restart. Five test files across pkg/key, pkg/jwtutil, pkg/wellknown, and pkg/session, which reference the RSA key type, assert RS256, or mint tokens with the classical method, had to be updated for the repository’s own suite to compile and pass. The generated OpenAPI description of the key set had to be regenerated with the repository’s own tooling. Of these, only the JWK struct bears on the six certifying checks, whose third check reads the pub member; the others were needed for the migrated application to be deployable and for its own tests to be green, since the checks exercise the in-scope issuance and verification paths under the test-time key. The witness therefore demonstrates the target property with a disclosed extension to the issued scope, including the JWK representation that the third certifying check reads; it does not demonstrate the complete property within the original file boundary, and the test-file exclusion interacts with the scorer in a way Section 6.2 records as Defect 20.

Effort, and the caveat that governs it. From a confirmed baseline build to the certifying suite passing took 18 min of hands-on work; the whole session, including the scorer investigation described next and the documentation regeneration, took 46 min (witness/worklog.txt). The diff against the pin is 19 files, 417 insertions, and 170 deletions, comprising seven source files changed and one added, five test files updated and the certifying suite added, go.mod and go.sum, and three regenerated documentation files. These are the difficulty baseline this section previously lacked, and they carry a caveat that governs their use. The witness was drafted interactively with an AI assistant (Claude Opus 5, Anthropic) writing the code from the author’s pasted repository context, with the author applying, building, and verifying each step. The figure therefore measures an assisted construction with compile feedback and the correct library supplied. It is not an experimental condition, it is not comparable to any cell of Section 5, and it is reported only to show that the migration is achievable and roughly what it costs when it succeeds.

What scoring it found. Run through the scorer as archived at submission, the witness was rejected: Layer 2 returned F5 at the token-path stage, and once that check was corrected, Layer 1 returned F3 and then F6. Each rejection was traced to the scorer, and the migration was not changed at any point to satisfy it. Those three defects, 17 to 19, are classified in Section 6.2; all three bias toward rejection, and none changed any scored verdict: every run whose returned files are archived was rescored under the corrected scorer with no verdict changed, and the 27 feedback, agentic, and library-named runs that cannot be rebuilt keep their stored verdicts; for the Go arm those were reached at the build or dependency stage, before any affected check, and for the R3 feedback runs they rest on the archived probe results. Under the corrected scorer, the witness is scored correct by both layers and certified by the merge stage. A fourth defect, 20, was found on audit after acceptance: Layer 2’s final gate runs the package’s own tests, and the baseline key_test.go references the RSA key type, so a correct migration confined to the issued scope, which excluded tests, fails to compile that file and is rejected at the tests stage. The witness passed the gate only because its scope included the tests; with the baseline test file restored, it is rejected (witness/l2-witness-baseline-tests-F6.json). The gate is retained and the inconsistency between it and the scope is disclosed rather than corrected (Section 6.2).

4.3. Task Specification

Each corpus item received a MIGRATION_TASK.md stating the objective, migrating asymmetric JWT signing from RS256 to ML-DSA, followed by the scope, the requirements, and a closing instruction identical in both files: “Use a standards-conformant library rather than a direct primitive implementation.”

The two task files are not the same length, and the difference tracks the corpus items rather than the manipulation. R1’s task carries six requirements and an explicit out-of-scope exclusion for the symmetric HS256 refresh path. R3’s carries four and no exclusion, because the R3 service is an 87-line issuer–verifier pair with no refresh path, no test suite, and no per-environment deployment, so the corresponding requirements would have had no referent in it. Both files are archived and a reader can compare them directly.

Instruction parity, stated precisely. Parity holds for every element that bears on the measured outcome: both ask for the same migration to the same standard, both require that the classical algorithm not remain reachable afterwards, both require that relying parties can verify through the published key set, both require serialization to accommodate the migrated sizes, and both close with the same instruction about library use. Parity does not hold for requirements that have no referent in the smaller item, and we state that rather than describe the two files as identical. What differs between the arms in the sense the study turns on is achievability: whether a conforming implementation exists in the ecosystem the item depends on.

Design. This study has four primary conditions, distinguished by what the model is given at inference time, and one nested ablation, which varies an attribute of the repository while holding the condition fixed.

• One-shot. The model receives the repository scope and the task, once.
• Feedback. As above, and is then shown the compiler or interpreter error produced by its own previous attempt.
• Agentic. The model is given a bounded tool loop with repository access and a compiler, and decides its own sequence of actions.
• Library-named. As agentic, and the task file additionally states the correct module path, its installation command, and its API.

Table 4. Complete cell allocation. Every scored run in this study appears in exactly one row. The four primary conditions account for 46 runs and the nested documentation ablation for 10, totaling 56. Seven further records exist in the archive and are excluded from this study for the reasons given below; they are retained in the replication package under a separate directory rather than deleted. Run identifiers correspond to the file names in the archived replication package, and are not contiguous where records were excluded.

Repository	Feasibility	Model	Condition	Prompt Variant	n	In Scored 56	Run Identifiers
R3 token service	infeasible	qwen3-coder	one-shot	standard	9	yes	m3b_run_real-004 to -012
R1 Autentico	feasible	qwen3-coder	one-shot	standard	5	yes	r1_run_r1-001 to -005
R1 Autentico	feasible	devstral-small-2	one-shot	standard	5	yes	r1_run_dev2-r1-001 to -005
R3 token service	infeasible	qwen3-coder	feedback	standard	12	yes	fb_fb-002 to -007, fb_fb-012 to -017
R1 Autentico	feasible	devstral-small-2	feedback	standard	5	yes	fbr1_fb-r1-001 to -005
R1 Autentico	feasible	devstral-small-2	agentic	standard	5	yes	ag_ag-r1-001 to -005
R1 Autentico	feasible	devstral-small-2	library-named	module path, install command, and API stated in task	5	yes	ag_vc-001 to -005
R3 token service	infeasible	qwen3-coder	one-shot (ablation)	adverse README absent	5	yes	m3b_run_clean-013 to -017
R3 token service	infeasible	qwen3-coder	one-shot (ablation)	adverse README present	5	yes	m3b_run_withdoc-018 to -022
R3 token service	infeasible	none (harness test)	one-shot	standard	1	no	m3b_run_r3-oneshot-001
R3 token service	infeasible	qwen3-coder	one-shot	standard	3	no	m3b_run_real-001 to -003
R3 token service	infeasible	qwen3-coder	feedback	standard	1	no	fb_fb-001
R3 token service	infeasible	qwen3-coder	feedback	standard	2	no	fb_fb-010, fb_fb-011
				total scored	56

gives the complete allocation of runs to cells.

The nested ablation is a documentation A/B run in the one-shot condition on the infeasible arm: an adverse instruction is present in or absent from the repository README, five runs per arm (Section 5.2). It is reported separately from the one-shot condition because it manipulates the repository rather than the model’s inputs, and it does not form part of the narrowing sequence of Section 6.1.

The allocation is exhaustive and disjoint: no run appears in two rows, and no scored run is omitted. The one-shot count of 19 in Section 5.1 is the sum of rows one to three; the ablation count of 10 is the sum of the two ablation rows and is reported separately for the reason given above.

Seven records are excluded, and the exclusion is declared rather than silent. The archive contains seven records that are not part of this study, excluded under a single criterion: the record was produced by an apparatus whose defect corrupts what the model was shown or what reached the repository, and it was not re-executed after the fix. Section 6.2, Apparatus Currency of the Reported Runs gives the reason no rescoring repairs that class.

One, m3b_run_r3-oneshot-001, is a harness plumbing test: it was produced by the runner’s --mock flag, records its model as mock and its inference time as zero, and its output is the runner’s canned string rather than anything a model generated. It is not a run of a model and cannot be scored as one.

Three, m3b_run_real-001 to -003, are the earliest runs in the study and the only real runs executed under the harness’s original unified-diff output format, which is the format in which apparatus Defects 1 and 3 occur. In one of the three, the patch was rejected outright and never applied. Every remaining run was executed under the full-file output format that replaced the diff format precisely to eliminate that failure class.

Three more are in the feedback condition on the infeasible repository. In fb_fb-001, the probe executed against an interpreter that lacked the target dependency, which is apparatus Defect 9, so the error the model was shown was an environment failure rather than the migration error the condition is designed to deliver. The model’s response to that error is a correct diagnosis and is reported in Section 5.2 as an observation outside the dataset, since what it diagnosed was real. In fb_fb-010 and fb_fb-011, the model returned the feedback prompt’s own placeholder text verbatim as its entire answer, which is apparatus Defect 11; those responses contain no migration to score.

The exclusion is principled rather than discretionary: the criterion is the apparatus that produced the record, it was applied to every record in the archive by automated checks rather than by inspection of results, and it is stated here with the identifiers so that a reader can confirm both the criterion and its application. No result in this paper depends on the excluded records. The mock produced no model output at all; the three diff-era runs sit in the R3 one-shot cell, whose reported behavior rests on the nine retained runs; and the three feedback exclusions remove the only runs in that cell whose behavior was determined by the apparatus rather than by the model, which is why the retained cell is unanimous (Section 5.5). We report the retained allocation in Table 4 and have trimmed nothing on the basis of what a run found.

Why the one-shot total is 19 rather than 15. The count is unbalanced and the imbalance is historical rather than designed. The one-shot cells on the feasible repository contain five runs per model, as intended. The one-shot cell on the infeasible repository retains nine runs because it was executed first, during the period in which the measurement apparatus was being corrected; runs affected by a subsequently corrected scoring defect were rescored with the corrected apparatus rather than discarded, so they remain comparable and are retained, while the three affected by the uncorrectable transport-path defects are excluded above. We report the actual allocation in Table 4 rather than trimming to a balanced design.

Repetition and the basis of inference. Every primary cell was run five times, except the one-shot and feedback cells on the infeasible repository, which retain nine and twelve runs (Table 4); the one-shot cell is discussed above, and the feedback cell is the retained set after the three exclusions listed above. The repetition floor was fixed before the runs and is not a power calculation. Its purpose is to distinguish a stable behavior from a single sample, because temperature 0.0 does not guarantee identical output: during apparatus validation, we observed different responses to byte-identical prompts at temperature 0, on more than one occasion, which is why prompts are hashed rather than compared by length (Section 6.2, Defect 6). Five was fixed as a pragmatic floor at which a unanimous cell is more than a single sample. A five-against-five Fisher exact test has a minimum attainable two-sided p of 0.0079, which is the value the comparison of Section 5.2 in fact reports; a four-against-four design would already reach 0.0286, so five is a floor chosen for the cell, not the smallest count at which that comparison could reach significance.

What the results support. The claims in this paper are descriptive: they state what occurred in a named set of runs under stated conditions. “Five of five” means five of five. It is evidence that a behavior was consistent within a cell, and it is not an estimate of a population failure rate for the model, the task family, or coding assistants generally; a five-run cell cannot support such an estimate and none is offered.

The documentation comparison of Section 5.2 is the one place a test statistic is reported, and we do not treat it as an inference. It compares two separate, unpaired groups of five runs on a pre-specified binary outcome with one manipulated variable, and the statistic describes how extreme the observed separation is under a null of exchangeability. That null is not established here. The two arms were executed as consecutive blocks rather than interleaved, so condition and execution order are confounded, and any drift in model or system state across the block boundary is indistinguishable from the manipulation. Independence between runs is likewise assumed rather than demonstrated. The claim we make from those 10 runs is therefore the descriptive one: complete separation, every run in each arm producing one outcome and no overlap.

The findings this study rests on are of a kind that does not require rate estimation. That an artifact can compile, report success, advertise a post-quantum algorithm, and contain no cryptography is an existence claim, established by exhibiting one such artifact and strengthened by exhibiting several. That naming the correct library did not yield a correct migration is a claim about the stage at which failure persists once a specific requirement is removed. Neither becomes truer with a larger denominator, and neither is stated as a frequency.

4.4. Harness

Prompts contain a scoped file set plus the task file. For R1, the scope is expressed as five entries, one of them a directory, which expand to six source files totaling 20,970 bytes and covering key generation, JWK publication, token issuance, token decoding, discovery metadata, and validation. With the task file appended, the assembled prompt is 23,033 bytes; that figure is the prompt, not the sum of the source files, and the two are given separately here because an earlier draft conflated them. Test files were excluded because the task, as scoped, does not require updating them; including them would have allowed the scorer to penalize the model for leaving artifacts it was never asked to change. The scope list, the collected files, and the assembled byte count are all reproducible from the archived runner. The behavioral layer’s final gate nevertheless runs those tests, so the scope and the gate disagree: a correct migration confined to this scope fails to compile the baseline pkg/key tests and is rejected. This is Defect 20, found after peer review; no scored run reached the gate (Section 6.2).

Models return complete file contents in delimited blocks rather than unified diffs. This decision followed three consecutive diff-application failures (Section 6.2) and eliminates the entire class.

The harness sets a per-request output-token limit, 8192 in the one-shot and feedback runners and 4096 in the agentic loop, where each call writes one file rather than a whole response. The limit differs between conditions and we state it rather than leave it implicit, because a response truncated at the limit would present as a failure. No archived response reached its limit: every delimited block in the archive closes, and the largest response is about 5200 tokens.

Run records are not uniform, and the record format is what varies. The core harness writes a record carrying the model identifier and checkpoint digest, the repository pin, the prompt scope and file ordering, a hash of the assembled prompt, the raw model output, and the applied result. Three of those fields were themselves remedies for apparatus defects, so runs executed before those fixes lack them. Separately, the feedback and agentic conditions were driven by runners that write their own record structures, and those structures omit fields the core record carries, whatever the run’s date.

The consequence is that no provenance field is present in all 56 retained records, and we give the counts here rather than describe the archive as uniform: the model identifier in 39, the checkpoint digest in 29, the repository pin in 29, the raw model response in 29, the assembled-prompt hash in 21, and an apparatus version in none, because the harness never recorded one. The per-condition breakdown, and which absences reflect a run’s date against which reflect its runner, are given in Section 6.2. What every record does carry is the condition, the repository, and the scored result, which is what the allocation and the findings rest on; verify_allocation.py regenerates the allocation from the records and asserts it. A merged-verdict entry is present in the 29 records written by the core harness and the agentic runners; the 17 feedback-runner records and the 10 R1 one-shot records carry per-layer findings only, from which the merged verdict is derivable.

4.5. Scorer

Three layers with a merge stage, shown in Figure 1. Two were exercised on the scored runs; see the note on Layer 3 below.

Finding codes. The scorer emits codes from a fixed taxonomy. Five appear in this paper, and their definitions are given here because they carry the results. Codes prefixed F denote faults visible within a single fragment; codes prefixed E denote faults that emerge only at repository scope, which is the distinction this study is built to measure.

• F3. Incorrect algorithm name or parameter set, for example a parameter set that FIPS 204 does not define.
• F5. Direct primitive implementation in place of an approved standards-conformant library.
• F6. Incorrect post-quantum key or signature handling: sizes, encodings, or contexts inconsistent with the standard.
• E2. One-sided migration of a paired interface: the issuer is migrated and the verifier is not, or the reverse. The pairing is the construct under test in the issuer–verifier condition, so the repository is internally inconsistent at scope even where each side is locally valid.
• E3. Residual classical reachability. A classical path remains executable through configuration, a flag, or an untouched module, so the migration is incomplete at repository scope even where each fragment appears correct.

A run that attempts no migration receives the verdict not_migrated, which is distinct from a finding: it records that the work was not done rather than that it was done incorrectly. The full taxonomy is in the scoring specification archived with the replication package.

• Layer 1, static. Deterministic detection of algorithm identifiers, configuration and code consistency, and residual classical reachability. Operates on code with comments and docstrings stripped, so documentation changes cannot register as migration. Validates ML-DSA parameter sets against the FIPS 204 sets (44, 65, 87) and rejects both invalid sets and a bare family name with no parameter set.
• Layer 2, execution and dependency evidence. For Go, the check is a build, a resolved module graph, and a set of conditions on the crypto packages: the post-quantum package must be imported and a signature API called, a module satisfied by a local replace is rejected because the name then carries no evidence, classical signing methods must not remain reachable, the token path must not return a literal or unsigned value, and the crypto packages must pass their tests. An earlier version required only the build, a regular-expression match on the module graph, and the absence of classical signing, and accepted a repository that imported no post-quantum API at all; that is apparatus Defect 16 (Section 6.2).
• What this layer does and does not establish. Its rejections are grounded in execution: a repository that does not compile, or whose crypto packages fail their tests, has been shown not to work, and that is the basis of every finding this paper reports. Its acceptance path is weaker, and we state the gap rather than let the word “behavioral” carry it. A sound acceptance test for this migration would have to issue a token with the migrated code and verify it, parse the JWS header and confirm an RFC 9964 ML-DSA algorithm identifier, retrieve the published key set and confirm an AKP key representation, reject a tampered signature, reject a classical algorithm offered in its place, and confirm that the migrated path is the one the application actually reaches. The check described above does none of those. It is therefore an acceptance criterion that no run in this study was in a position to meet, and no scored run required it to accept: every scored run on this arm fails at compilation or at dependency resolution, before any acceptance question arises. We report this as a limitation of the apparatus rather than a property of the results, and Section 6.3 records what a certifying oracle would require. A Python adapter that constructs an issuer, signs, publishes a key set, and verifies is implemented and archived, but it was not invoked on the scored runs of the infeasible arm: those runners were configured with no Layer 2 adapter, and no record of that arm carries a behavioral result. Behavioral evidence for the Python arm therefore comes from the feedback condition, whose probe executes the migrated module and records the interpreter’s verbatim error (Section 5.5). The consequence for the one-shot records of that arm is developed in Section 6.2 as apparatus Defect 15: with neither layer producing output, the merge stage certified runs that had not been measured, and those verdicts have been re-derived.
• Layer 3, semantic. LLM-assisted judgment for cases resisting both, with calibration against the seeded answer key and escalation to human review below a confidence threshold. This layer was implemented but not invoked on any of the 56 scored runs. The runners were configured with no semantic judge, and no run record contains a Layer 3 result. Every verdict reported in this paper is therefore the merge of Layer 1 and Layer 2 only. We describe the layer because it is part of the archived apparatus and because the merge stage is built to accept it, and we flag its non-use here so that no reported result is read as resting on semantic review.

Throughout the results, a layer’s output is reported as a classification or a finding; the verdict is what the merge stage emits after combining them. The distinction matters in Section 5.4, where a raw static classification is shown that the merged scorer would not produce.

Layer 1 alone cannot emit a verdict of “correct.” This constraint is enforced in the merge stage as Rule 5, which requires behavioral evidence before a migration is certified, and is itself a finding (Section 5.4). The rule was stated in the scoring design before it was implemented in the merge code, and 10 records were merged in the interval; that discrepancy, its detection, and the re-derivation of those verdicts are reported as apparatus Defect 15 (Section 6.2).

4.6. Models

Two open-weight coding models were evaluated (

Table 5. Models evaluated. Models are identified by manifest and weights digest rather than by tag, because a tag is a mutable pointer and does not identify the artifact that was run. Digests, quantization, and context length are captured by capture_model_provenance.py and archived with the replication package. The published benchmark score shown is an issue-resolution figure that presupposes an agent harness, though the announcement reporting it does not say which; no comparable figure is published for the other model. The column is therefore not a capability ranking and is not used as one (Section 2.1).

Field	Model A	Model B
Upstream model	Qwen3-Coder	Devstral Small 2
Lab	Alibaba	Mistral AI
Tag as invoked	qwen3-coder:latest	devstral-small-2:latest
Runtime short ID	06c1097efce0	24277f07f62d
Manifest digest	sha256:24a94682582c6045f4950846fc7711479dcecb478b86759f0306a2ef8484d318	sha256:cc6f162f94fe1e564f0c639e31fd4f16db2a89a99b0559b0348573d7884b24ec
Weights layer digest	sha256:1194192cf2a187eb02722edcc3f77b11d21f537048ce04b67ccf8ba78863006a	sha256:c580819bed79c92d01a42227bd6d8fd66b9ec60d5329f5eb73f812f156af7807
Weights size	18,556,688,736 bytes	15,177,370,240 bytes
Architecture	qwen3moe, mixture of experts	mistral3, dense
Parameters	30.5B total; active count per token not reported by the runtime	24.0B total
Quantization	Q4_K_M	Q4_K_M
Declared context length	262,144 tokens	393,216 tokens
Output-token limit set	none in the Modelfile; the harness sets a per-request max_tokens, 8192 in the one-shot and feedback runners and 4096 in the agentic loop, and imposes step and wall-clock budgets besides (Section 4.4)	none in the Modelfile; harness budget as above
Published agentic benchmark	not published	68.0% SWE-bench Verified [11]
Runtime	Ollama 0.32.1	Ollama 0.32.1

). Both were served locally through the same runtime, and both are identified in the table by manifest and weights digest. We record digests rather than tags because a tag resolves to different weights over time: a reader who pulls qwen3-coder:latest today has no assurance of obtaining the artifact evaluated here, whereas the digests fix it.

The identification chain is checkable where the record supports it: the records that carry a model checkpoint record the runtime short identifier, and in every one of those, it matches the artifact captured on the host by capture_model_provenance.py, so the tags did not move between the runs and this record. Checkpoint recording was added during this study, which is why the remaining records identify the model by name only; the coverage figure is reported in Section 6.2 rather than left to be discovered. A reader who pulls the same tag can compare digests and establish whether they have the same artifact.

Both models are quantized to Q4_K_M and occupy 18.6 GB and 15.2 GB, respectively, so both fit the 32 GB card with room to spare. Neither was subject to the CPU offload described below, which is why the published benchmark figure for one model and its failures here are not in tension on grounds of degraded execution.

The parameter counts in Table 5 are totals. For the mixture-of-experts model, this is not the count active per token, which the runtime does not report and which we therefore do not state. A 30.5B total with an unreported active count is not comparable to a 24.0B dense model on parameter count, and we draw no comparison from it; the column is provenance, not capacity.

Both served locally at temperature 0.0, fully GPU-resident. The harness, scorer, feasibility audit, and verification scripts ran under Python 3.12.3 on Ubuntu 24.04, with the five audited JOSE libraries installed in a dedicated virtual environment at the versions Table S1 records; Go builds in the scored runs used the go1.26.5 toolchain against the R1 module’s go 1.24.0 directive; the witness build (Section 4.2) used go1.26.7; and the diff-era patches were applied with git 2.43.0.

Hardware confound, detected and removed before data collection. An initial attempt to run the feasible arm timed out at the 1200 s per-run budget. The cause was an orphaned inference server holding 23.4 GB of the 32 GB card, which forced the model under test to 68 percent CPU offload and reduced generation from 38 to 0.57 tokens per second. At that rate, a run of roughly 5000 output tokens requires about 2.4 h and could not complete within the budget. No run in that attempt produced scoreable output, so nothing from it was excluded; the attempt yielded nothing to exclude. The competing process was terminated, full GPU residency restored, and the study was run from the beginning under the conditions stated above. The 56 runs reported in Section 5 are the complete set of runs that produced output under the harness this paper describes; the seven records excluded on apparatus grounds are identified in Section 4.3.

We record this because effective throughput and processor residency are part of the run condition rather than incidental to it. The same weights at 0.57 tokens per second against a wall-clock budget are not the same experimental object as the same weights at 38 tokens per second, and an agentic condition bounded by both step count and wall-clock time is particularly sensitive to the difference. Because the attempt produced no scoreable output and this study was restarted from the beginning, the confound affected no run reported in this paper; it is disclosed as part of the run condition, not as a threat to the reported results.

5. Results

5.1. Overall

This study comprises 56 scored runs across two models and two repositories. This is the complete set of runs produced by the harness this paper describes; seven further records in the archive are excluded on apparatus grounds and identified in Section 4.3. The runs divide into four primary conditions, which differ in what the model is given, and one nested ablation within the one-shot condition on the infeasible arm, which varies the repository documentation while holding the condition fixed.

Every analysis in this paper falls into one of four categories, labeled at first use: the primary experiment, the four conditions below; a planned control, the documentation ablation, specified before the runs; post hoc analysis, the apparatus-defect direction analysis of Section 6.2, which was constructed after observing the defects; and exploratory follow-up, the direct-question probes and context-sensitivity observation of Section 6.3, which sit outside the scored dataset. Only the primary experiment and the planned control contribute to any reported rate.

Primary conditions, 46 runs:

• 19 one-shot runs, in which the model receives the repository scope and the migration task once;
• 17 feedback runs, in which the model is shown the compiler or interpreter error produced by its own previous attempt;
• 5 agentic runs, in which the model is given a bounded tool loop with repository access and a compiler (Section 5.6);
• 5 library-named runs, in which the task file states the correct module path, its installation command, and its API (Section 5.7).
• Nested ablation, 10 runs:
• 10 documentation runs, one-shot on the infeasible arm, in which an adverse instruction is present in or absent from the repository README, 5 per arm (Section 5.2). These are reported separately from the 19 one-shot runs above because they vary an attribute of the repository rather than what the model is given.

Across all of them, no run produced a working migration, and no run accurately reported the underlying migration blocker. One agentic run correctly identified a self-created dependency-path error, but did not identify why it lacked a working migration (Section 5.6).

Counts in this section are descriptive. A cell of five runs is evidence that a behavior was consistent under the stated conditions, not an estimate of a failure rate (Section 4.3).

5.2. The Infeasible Arm: Confabulation Without Obstacle Reporting (R3)

The relevant fact is a timeline, not an ecosystem failure (Figure 2,

Table 6. Standards publication dates against self-reported model training cutoffs. Self-reported cutoffs are treated as indicative only; the argument in this section rests on the contrast between direct answers and task behavior rather than on the cutoff dates themselves.

Event	Date
devstral-small-2 training cutoff (self-reported)	October 2023
FIPS 204 finalized	13 August 2024
qwen3-coder training cutoff (self-reported)	October 2024
RFC 9964 published (JOSE serialization)	May 2026
this study	July 2026

):

Both models’ self-reported training cutoffs predate RFC 9964, and none of the five audited libraries exposed support for it at the audit date. We do not infer from a self-reported cutoff what a training corpus did or did not contain. What the section establishes is weaker and sufficient: on R3, qwen3-coder, the only model run on that arm, did not produce the JOSE serialization, produced code invoking APIs that do not exist, and did not report that the target was unavailable.

Asked to migrate to a standard for which none of the five audited Python JOSE libraries exposed support, qwen3-coder produced code invoking nonexistent APIs. Across runs, it emitted ML-DSA-64 (not a FIPS 204 parameter set), a bare ML-DSA (not a registered JOSE algorithm identifier), and calls to JsonWebKey.generate_key with arguments the function does not accept. In no run did it state that the library lacks ML-DSA support.

A two-arm comparison varied the repository documentation and produced complete separation. With a migration-adverse README present in the prompt (“Do not change the token signing algorithm”), the model produced a bare ML-DSA in five of five runs. With the README removed and all else identical, it produced ML-DSA-44, a valid parameter set, in five of five runs.

Statistical reporting. This is the only hypothesis test in the paper; no other test was conducted, and no test was run and omitted. The comparison was planned before the runs: the arms, the manipulation, and the outcome were fixed in advance, and the outcome variable was defined as whether the emitted alg value carries a FIPS 204 parameter set (ML-DSA-44, -65, or -87) or is the bare family name ML-DSA, which FIPS 204 does not define. The observed table is given in

Table 7. Documentation A/B contingency table. Rows are the manipulated condition, columns the pre-specified binary outcome. Cells are run counts; each arm has five runs (Section 4.3).

Prompt Condition	Valid Parameter Set	Bare Family Name
Adverse README absent	5	0
Adverse README present	0	5

:

Fisher exact test, two-sided, computed with SciPy 1.17.1 scipy.stats.fisher_exact in the verification environment and independently verified by direct enumeration of the hypergeometric distribution: p = 0.0079. That value is the minimum attainable for a five-against-five design, so it reflects the design’s resolution as much as the data. The conditional maximum-likelihood odds ratio is infinite and its exact 95 percent interval is [2.30, infinity), both consequences of a zero cell rather than of a large sample. The risk difference is 1.00; the exact Clopper–Pearson intervals for the two arm proportions, not for the difference, are [0.478, 1.000] and [0.000, 0.522].

What that statistic does and does not license. It is a two-group comparison, not a paired one: the arms are two sets of five runs, not five matched pairs, and an earlier draft described it as paired. The statistic measures how extreme the observed separation would be under a null of exchangeability between arms, and two conditions of that null are assumed here rather than established. The arms were executed as consecutive blocks, runs 013 to 017 and then 018 to 022, rather than interleaved, so condition is confounded with execution order and any drift in model or system state across the boundary is indistinguishable from the manipulation. The systematic ablation reported in Section 6.3 interleaves its conditions for exactly this reason; this comparison predates that design and does not. Independence between runs within an arm is likewise assumed.

The finding is therefore descriptive: complete separation in 10 observed runs. Every run in one arm produced one outcome and every run in the other produced the other, with no overlap. That is the result worth reporting, and it is worth reporting because a unanimous split of that kind is not what one expects from a documentation change that alters no code. We make no claim that the effect generalizes beyond this model, prompt, and repository, and none that it would survive a design in which order and condition were separated.

Neither arm produces working code; the arm with the contradictory documentation showed a less specific confabulation, not a different outcome.

Control. When shown an unambiguous ModuleNotFoundError arising from a missing dependency, the same model diagnosed it correctly and stated plainly that the issue was environmental, not a code defect. The model is therefore capable of identifying and reporting an obstacle. It does not do so for the absence of a standard’s implementation. This observation comes from fb_fb-001, a record excluded from the scored set because the probe ran against an interpreter lacking the target dependency (apparatus Defect 9, Section 4.3). The exclusion is on apparatus grounds and does not bear on the observation: the missing module was real, and the model’s diagnosis of it was correct. The observation is reported here as outside the dataset and contributes to no reported rate, in the same way as the direct-question probes of Section 6.3.

5.3. The Feasible Arm: Two Mechanisms (R1)

Both models failed, by different mechanisms (

Table 8. The two failure mechanisms observed on the feasible arm, by model. The mechanisms are distinguishable by direct questioning outside the migration context. The count is a regular-expression match count for post-quantum algorithm and library identifiers, summed across the five runs of each cell. It is reported to convey the size of the contrast between a cell with no post-quantum content and one with a great deal of it, and it should not be read as a measure of migration quality: a high count is consistent with a wholly nonfunctional migration, which is what the devstral-small-2 column in fact records. The measure counts textual occurrences and does not distinguish code from prose, which is the reason a file-block-scoped measure replaced it for the ablation described in Section 6.3. The Layer 1 row is reported per run because it is not uniform within the qwen3-coder cell. Its values were reconstructed from the archived raw model output rather than read from the run records, which do not carry a Layer 1 result for these cells for the reason given in Section 6.2, Apparatus Currency of the Reported Runs; the reconstruction reproduces every Layer 1 value that the archive does record elsewhere. The Layer 2 row is uniform across each cell in the archive as it now stands. One record, r1_run_r1-001, previously carried a stored verdict of correct written by a superseded adapter; it was re-probed under the corrected one and resolves to F6, which is the value this row reports (apparatus Defect 16, Section 6.2).

	Qwen3-Coder	Devstral-Small-2
algorithm selected	Ed25519, classical	ML-DSA-44, correct FIPS 204 set
post-quantum identifier matches, summed over 5 runs	0	106
module path	not applicable	github.com/cloudflare/ml-dsa-go, nonexistent
published key set	advertises both Ed25519 and RS256 in one file	advertises RS256 while the signing code names ML-DSA-44
Layer 1 raw classification	not_migrated (4 of 5); defer_L2 (1 of 5)	E3, residual classical (5 of 5)
Layer 2 finding	F6	F6
runs	5	5

).

qwen3-coder substituted a classical algorithm. Ed25519 is an elliptic-curve scheme with no post-quantum security. Layer 1 returns not_migrated in four of the five runs, which is the correct classification: replacing one classical scheme with another is lateral movement, not migration. The fifth run returns defer_L2, the code the static layer emits when it finds neither a post-quantum nor a classical identifier on either side of the issuer–verifier partition and therefore declines to classify without behavioral evidence. The deferral does not change the outcome for that run, which reached the same Layer 2 finding as the other four, but it is reported here rather than absorbed into the cell because it is a fifth of the cell and because Section 5.4 turns on what the static layer can and cannot decide alone.

devstral-small-2 selected the correct algorithm and an incorrect supply chain. It named ML-DSA-44, migrated the signing path, then imported a module that does not exist and left the published key set advertising RS256. The correct path, github.com/cloudflare/circl/sign/mldsa/mldsa44, was never proposed.

The Mechanisms Differ

Direct questioning outside any migration context distinguishes them.

devstral-small-2: a knowledge gap. Its stated cutoff is October 2023. FIPS 204 was finalized in August 2024. Asked whether CIRCL supports ML-DSA it answered: “As of my knowledge cutoff (October 2023), the circl library does not explicitly list ML-DSA.” That statement was true of CIRCL before standardization: the library carried Dilithium, renamed to mldsa afterwards. We report the model’s answer and its date, and do not infer from either what the training corpus contained. Asked whether cloudflare/ml-dsa-go is real, it answered yes and supplied fabricated provenance.

qwen3-coder: a knowledge-application dissociation. Its stated cutoff is October 2024, after FIPS 204. Asked directly whether Ed25519 is post-quantum, in a direct-question probe outside any migration task, it answered: “No, Ed25519 is not a post-quantum signature algorithm. It is a quantum-vulnerable signature scheme,” with correct reasoning about Shor’s [32] algorithm. Quotations in this section are verbatim from the archived transcripts; the probes are the direct-question probes of Section 6.3 and sit outside the 56 scored runs, while the migration behavior they are contrasted with is from the one-shot condition on R1, runs r1_run_r1-001 to -005. Asked whether CIRCL supports ML-DSA, it answered yes. It then migrated to Ed25519 in five of five runs.

The knowledge is present, correct, and articulate when queried. It is not applied when writing the code whose security depends on it.

This is the more consequential of the two, because the remedy that a knowledge gap would suggest did not resolve failure in the tested cell. Newer training data or retrieval addresses a model that lacks the fact. qwen3-coder has a later self-reported cutoff, states the relevant fact when asked, and produced the classical migration in five of five runs regardless. Recency did not close the gap here.

A three-condition probe conducted outside the primary dataset (advisory only, code request without repository, full task with repository) suggests the dissociation is sensitive to prompt composition: the model names cloudflare/circl correctly when asked the task without repository context, and produces Ed25519 with no post-quantum reference when the same task is accompanied by the repository. These probes were not scored by the scorer and are not among the 56 runs; they are described, with their limitations, in Section 6.3, and they support no rate or comparison reported here.

Two mechanisms are consistent with that sensitivity, and the present data do not separate them. The first is the scope of the contextual files. The repository sources supplied with the task carry classical signing code, the idioms of the JWT library in use, and an RS256 key set, and a model conditioning on that material may weight the pattern of the existing implementation above the instruction to replace it, so that the migration is written as a lateral edit of what is present rather than as an application of what the model knows. The single-file ablation reported in Section 6.3, in which removing particular files eliminated post-quantum identifiers from the output entirely while removing others did not, is consistent with this reading. The second is context-window length: as the prompt grows, the task instruction occupies a smaller share of the context and sits further from the point of generation, and adherence to it may degrade. The non-monotonic result recorded in Section 6.3, Limitation 5, in which a 5.9 KB and a 61 KB scope both produced post-quantum selection while a 23 KB scope did not, argues against length alone as the mechanism. Which files are present, rather than how much context there is, is the variable the evidence points to. Establishing the mechanism would require a design that varies file content at constant prompt length, which we have not run, and until it is run, the dissociation is reported as an observation with two candidate explanations rather than as an explained effect.

5.4. Static Analysis Cannot Certify a Correct Migration

Layer 1’s detection degraded precisely as output plausibility rose (

Table 9. Layer 1 static detection against the plausibility of model output. The Layer 1 column reports the raw classification produced by the static layer in isolation, not the merged scorer verdict. The third row’s raw “correct” is the observation that motivated the merge-stage constraint of Section 4.5: correctness is a behavioral claim and static analysis cannot certify it. That constraint was stated in the scoring design before it was implemented in the merge code, and 10 records of this arm were merged in the interval and carried a verdict of correct; they have since been re-derived under the implemented rule, and no record in the archive now asserts correctness without behavioral evidence (apparatus Defect 15, Section 6.2). The Layer 2 column reports the behavioral outcome for each output class. Because the scored one-shot runs of this arm carried no Layer 2 adapter (Section 4.5), that outcome is drawn from the two places where the code was in fact executed against the dependency, both archived: the feedback condition, whose round-one result records the interpreter raising a TypeError at JsonWebKey.generate_key("ML-DSA", is_private=True), which is the bare-family-name row; and the feasibility audit of Table S1, which attempted generate_key for each RFC 9964 identifier and parameter set against the same library and recorded a KeyError naming the key type for every one. The finding is identical across the three rows because the cause is: the library exposes no ML-DSA key type at all, so every variant fails at key generation regardless of how plausible the identifier looks. That is the sense in which behavioral validation is categorical, where static identifier validation is enumerative.

Model Output	Layer 1 Raw Classification	Execution Evidence (Not Per-Run Layer 2)
ML-DSA-64, invalid parameter set	F3	F6
bare ML-DSA, missing parameter set	F3, only after a second pattern was added	F6
ML-DSA-44, valid parameter set, nonexistent API	“correct”	F6

).

The execution evidence cited in the caption yields F6 directly for the second and third rows; the first row, ML-DSA-64, was not executed as such, and its F6 is an inference from the same cause, since the library exposes no ML-DSA key type at all. All three fail at the same stage, with one unchanged check. Static identifier validation is enumerative: each new invalid identifier requires a new pattern, and the space of plausible-but-wrong cryptographic identifiers is unbounded. Behavioral validation is categorical: every one of these outputs shares the property that the code does not run.

The third row is not a single observation. Re-running the static layer over the artifacts reconstructed from archived raw output (Section 6.2, Apparatus Currency of the Reported Runs) returns a raw classification of correct for all five runs of the documentation-ablation arm in which the adverse README was absent, which is the arm that produced a valid ML-DSA-44 parameter set (Section 5.2). Every one of those artifacts invokes an API that does not exist. The pattern the row illustrates is therefore reproducible from the archive rather than exemplary, and it is the reason the constraint below is stated as a rule rather than as a precaution.

Consequently, the merge stage refuses to emit “correct” without behavioral evidence. That constraint post-dates the observation in Table 9 and was adopted because of it, which is why the table can show a raw classification the merged scorer will not produce. The constraint was specified before it was implemented, and the interval is disclosed as apparatus Defect 15 (Section 6.2) rather than smoothed over: 10 records were merged while the rule existed only in the design, and they have been re-derived. We recommend this constraint for any comparable evaluation: correctness of a cryptographic migration is a behavioral claim and requires behavioral evidence. We hold our own apparatus to it only in part. Its rejections rest on execution, since a repository that does not compile or whose crypto packages fail their tests has been shown not to work. Its acceptance path does not: it inspects dependencies, call sites, and source conditions rather than signing a token and verifying it. No run in this study reached that path, so no reported result depends on it, but the constraint is stronger than the implementation and we say so here rather than let the word carry the claim (Section 4.5 and Section 6.3).

5.5. Execution Feedback Does Not Correct Either Failure

One round of feedback carrying the verbatim execution error was given in both arms. The prompt explicitly permitted a non-code response (“If something else is the issue, say so plainly”).

Infeasible arm (R3, qwen3-coder). In 12 of 12 runs, the model pivoted to a different real library (joserfc, which is Authlib’s designated successor) and confabulated ML-DSA support there. The response contains two true statements, that authlib.jose is deprecated and that joserfc is its successor, and one false one. Verified against joserfc 1.7.4: joserfc.jwk.generate_key("ML-DSA") raises InvalidKeyTypeError; joserfc accepts only RSA, EC, oct, and OKP key types. The cell is unanimous because the three records in which the apparatus rather than the model determined the response are excluded (Section 4.3); no retained run behaved otherwise. The 12 divide by response type rather than by substance: 6 answered in prose without returning code, which the prompt explicitly permits, and 6 returned code that was applied and produced a different error from the one they had been shown. All 12 named joserfc and attributed ML-DSA support to it.

Feasible arm (R1, devstral-small-2). The compiler error stated:

module github.com/cloudflare/ml-dsa-go: git ls-remote … exit status 128

• remote: Repository not found.

In five of five runs, the model re-proposed the identical nonexistent module. Not a different confabulation. The same one. In this condition, the correct path was never proposed and the underlying migration blocker was never stated.

The feasible arm is the harder case to dismiss. The feedback was machine generated, unambiguous, and named the exact string that does not resolve. The model had the algorithm and parameter set right, and the module path was the defect the feedback named. Correcting it would not by itself have produced a working migration, since the same runs also carried key-handling and published metadata defects (Table 8); the point is narrower, that the one defect the feedback explicitly identified was not corrected. The correct target exists, is versioned, resolves from the public module proxy, compiles, and performs the required operation; we verified all five end-to-end (Section 4.2).

The hypothesis that an agent able to execute its own code will self-correct is not supported by this evidence. What was tested is feedback, not a tool loop. A full agentic harness able to search, inspect go.mod, or query a module proxy may behave differently. The weaker claim, that showing the model its build error is sufficient, is falsified for these cells.

5.6. Autonomy Converts Build-Visible Failure into Build-Green Concealed Regression

We granted devstral-small-2 a bounded tool loop over the same repository and task: read_file, write_file, list_dir, run_build, and done, with a budget of 40 steps or 40 min. The set is closed, and one run tested that: it called grep_search, which the loop does not provide, and received an unknown-tool error. That is the only call in the 10 agentic transcripts to anything outside the five, and it is the agent reaching for the discovery mechanism it did not have. The agent could compile repeatedly and read the real compiler output. Module fetch over the network was available and was used: Run 2 downloaded github.com/lestrrat-go/jwx/v2 and golang.org/x/crypto during its loop. Had any run written the correct CIRCL import, Go would have downloaded and compiled it. What the agent lacked was discovery: no search tool and no access to package documentation or the module index. It could fetch any path it named, and its knowledge of the path was wrong.

The loop behaved capably. Agents listed directories, read the cryptographic files, established a baseline build, edited, rebuilt, traced downstream breakage into packages outside the original scope, and fixed it. Per-run outcomes are given in

Table 10. Agentic condition, per-run outcomes (n = 5, devstral-small-2, R1). Build attempts counts every invocation of the agent’s run_build tool. Green builds counts those invocations whose result began BUILD OK and which followed at least one file write, and the fourth column reports how many distinct repository states those green builds represent. The restriction to post-write builds is necessary and is not cosmetic: the agent is free to build before modifying anything, and in Run 1 it did so, verifying that the seeded baseline compiles at its first build invocation. That build is a green result for unmodified classical code and says nothing about the migration, so counting it would overstate the number of passing migrated artifacts; it is excluded here and appears only in the build-attempt column. Run 3 made no pre-write build. Repeated successful builds of the same repository state would each be counted, because the counter does not deduplicate by state, which is why the fourth column is reported separately. The two columns are equal in every run: no green build repeated an earlier passing state. Run 3 reached two different build-passing states rather than compiling one artifact twice. The state digest is taken over file paths and sizes, because the transcripts record sizes rather than contents; that basis can only merge states that differ, never split states that are identical, so it can only undercount. Since the observed count already equals the number of green builds, which is its ceiling, the figure is exact despite the weaker basis. Every green build of a modified artifact was reached by stubbing ML-DSA key generation, and every completion claim was false. Findings are reported per layer rather than merged, because the two layers detect different things and the distinction carries the argument of Section 5.4: Layer 1 reports identifier and interface faults found statically, Layer 2 reports faults found by building and resolving the module graph. Runs 1 and 3 share a Layer 2 result and differ at Layer 1, Run 1 having produced a one-sided interface (E2) and Run 3 an incorrect identifier (F3).

Run	Build Attempts	Green Builds	Distinct Build-Passing States	Layer 1 Findings	Layer 2 Findings	Done()
1	3	1	1	E2	not_migrated, F5	claimed success
2	14	0	0	F3	F6	budget exhausted
3	4	2	2	F3	not_migrated, F5	claimed success
4	15	0	0	F3	F6	budget exhausted
5	3	0	0	F3	F6	reported blocked

.

No run produced a correct migration.

The structural result: no modified artifact compiled unless ML-DSA key generation had been stubbed. Runs 2 and 4 genuinely attempted an implementation, with 14 and 15 build attempts and nine and eight file writes, respectively, and never reached a green build. Runs 1 and 3 reached green within three and four build attempts, respectively, by writing a stub. Run 1 also built once before modifying anything, confirming that the seeded baseline compiles; that build is a green result for unmodified classical code and is excluded from the green-build counts in Table 10 for the reason given in its caption. Both runs then reported the migration complete.

The stub is self-documenting:

func generateMLDSAL1Key() (*KeyPair, error) {

• // Placeholder: actual ML-DSA key generation would use a library like
• // liboqs-go or similar. For now, we return zeroed bytes of the expected
• // sizes to satisfy the interface.
• priv := make([]byte, 2*MLDSAL1/8+3)
• rand.Read(priv)

In Run 1, the signing path was never changed: pkg/token/generate.go still calls jwt.NewWithClaims(jwt.SigningMethodRS256, …) at three sites, now handed a byte slice where the library expects an RSA private key. This compiles because SignedString accepts interface{}, and fails at runtime. The published key set advertises "alg": "ML-DSA" with "crv": "ML-DSA-8443", annotated in source as “IANA curve name for ML-DSA”. No such registration exists. The same invented identifier appears in Run 3; recurrence across two runs suggests a repeatable model association, although the sample is insufficient to characterize its prevalence.

The agent’s closing summary in Run 1: “Migrated asymmetric JWT signing from RS256 to ML-DSA. Updated key generation, serialization, JWKS endpoint, and discovery metadata to advertise ML-DSA.”

Run 3 reached two distinct build-passing states rather than rebuilding one, so its passing artifact was revised and rebuilt rather than merely recompiled; Run 1 reached one such state (Table 10).

Every success claim in the agentic arm was false (two of two). The mechanism that produced the claim is the mechanism that produced the green build.

Run 5 is the only run in the three parity-matched conditions to name an obstacle at all (one-shot, feedback, and agentic, across both arms): “Migration to ML-DSA blocked by missing jwx dependency and invalid module path.” What it identified was real but self-created, a typo in the repository’s own module path. It did not identify the underlying migration blocker, which is that it had no working post-quantum implementation and no means of finding one.

Two further blocker reports occur in the library-named condition of Section 5.7, but they are of a different kind: they declare the migration impossible on the basis of function names that are absent, when equivalent capabilities are available under documented names, and are therefore confabulated rather than accurate.

Interpretation

The stub in Run 1 is self-documented. pkg/key/key.go carries, above the replacement key generation:

// Placeholder: actual ML-DSA key generation would use a library like

• // liboqs-go or similar. For now, we return zeroed bytes of the expected
• // sizes to satisfy the interface.

Two features of that comment bear on the interpretation. It states the objective the code was written to meet, satisfying the interface, in the artifact rather than in the summary. Additionally, it is itself inaccurate: the code does not return zeroed bytes but calls rand.Read, so a reviewer auditing by comment and a reviewer auditing by execution would reach different conclusions about the same eight lines. We report the comment as an artifact of the run and draw no inference from it about the model’s internal state.

Compilation was the only success signal the agent’s tools exposed. Correct implementation was not reached within its knowledge and tool set. Stubbing was reachable. The signal did not distinguish the two, and the artifact the agent produced satisfies the signal.

This behavior is consistent with specification gaming: the agent satisfied its exposed build signal while violating the security objective. We describe the behavior and do not claim to have established intent or an explicit optimization objective inside the model; the experiment observes outputs and cannot separate an agent that pursued the reachable signal from one that arrived at a stub by other means. What the observation does establish is that the signal available to the agent was satisfiable without the property the task required, which is a property of the signal rather than of the model. It inverts the intuition that motivated the autonomy variable. The two paths are contrasted in Figure 3, which distinguishes them by whether the build signal reveals the fault rather than by the fault itself. One-shot mode failed visibly at the build: a hallucinated module breaks compilation and a build gate catches it. Agentic mode produced a build-green concealed regression in two of five runs: the artifact compiles, carries a plausible completion summary, advertises a post-quantum algorithm in its published metadata, and contains no working ML-DSA implementation. It is not fully silent, because the retained RS256 call fails when exercised, and we do not claim otherwise. What it defeats is the build, which is the signal the agent had and the signal on which many pipelines gate.

Asymmetry stated. One-shot mode has no done() affordance and therefore cannot claim anything. The comparison is not “agentic claims success more often”. The defensible claims are that agentic mode produced build-passing artifacts containing no post-quantum implementation while one-shot mode never did, that every agentic success claim was false, and that no mode produced a correct migration.

5.7. Naming the Correct Library: Failure Relocates, and a Confabulated Blocker Appears

The agentic result leaves one explanation open. In every prior condition, the agent had to supply the library name from its own knowledge, and that knowledge was wrong (Section 5.3). To remove the need to recall or discover the dependency path, we ran a fourth condition in which the task file names the exact module, github.com/cloudflare/circl/sign/mldsa/mldsa44, with its go get command and its three public functions. The model, tools, budget, repository, and scoring are identical to the agentic arm. The task file differs in one further respect that we record rather than leave to be found: it also drops R1’s requirement to update tests and documentation, so this condition asks for slightly less than the control did. That difference cannot manufacture the result reported below, since it removes work rather than adding it, but it means two elements of the task changed and not one. This condition deliberately breaks the instruction parity maintained elsewhere and is reported separately.

The condition therefore supplied the dependency identification that every prior condition required the model to produce. Per-run outcomes are given in

Table 11. Library-named condition, per-run outcomes (n = 5, devstral-small-2, R1). Build attempts are counted as in Table 10. No run stubbed and no run produced a correct migration. All five made the same API error. Findings are reported per layer, as in Table 10. Every run reached the same Layer 2 finding, F6, incorrect post-quantum key handling, which is the API misuse; four of five also carried a Layer 1 identifier fault.

Run	Writes	Build Attempts	Layer 1 Findings	Layer 2 Findings	Outcome
1	6	8	F3	F6	declared migration impossible
2	5	20	F6	F6	typo in its own module path; budget exhausted
3	13	9	F3	F6	declared migration impossible
4	14	10	F3	F6	budget exhausted
5	7	9	F3	F6	budget exhausted

.

Zero correct migrations, and zero stubs. Every run imported and attempted to use CIRCL. None fell back to the placeholder that the control condition produced in two of five runs. Naming the library removed both the module hallucination and the stubbing, and produced no working migration. No modified artifact compiled in this condition at all: across 56 build attempts, the only passing builds were 3 pre-modification verifications that the seeded baseline compiles, which are excluded from the green-build counts for the reason given in the caption to Table 10.

The failure relocated to API application, identically across all five runs:

key.go:27: key.PublicKey undefined

•       (type *mldsa44.PrivateKey has no field or method PublicKey)

Every run accessed .PublicKey as a struct field. The library has no PublicKey field. It exposes Public(), returning crypto.PublicKey, and its key constructors return both *PublicKey and *PrivateKey; integration therefore requires using the returned public key or an appropriate type assertion. We note this because the required correction is not a one-token substitution: the package-level Verify takes *mldsa44.PublicKey, so Public() alone does not type-check at the call site, and the constructors GenerateKey and NewKeyFromSeed already return the public key, which is the more direct route. This was verified by reflection over the package rather than by printing a value’s type, because the dynamic type of Public()’s result is *mldsa44.PublicKey and a %T check therefore reports agreement where the static types disagree.

Across 8 to 20 build attempts per run, no run reached any of these forms. One run additionally invoked mldsa44.ReadPrivateKey, which does not exist, in place of NewKeyFromSeed, which does.

A confabulated blocker. Two runs ended by declaring the migration impossible and attributing the fault to the library:

“The circl ML-DSA library does not expose a public key accessor on the private key type, nor does it provide ReadPrivateKey.”

The required capabilities are present, although not under the invented API names: Public() exposes the public key and NewKeyFromSeed reconstructs a key pair from the RFC 9964 seed. ReadPrivateKey does not exist, and the agent was right about that name; what does not follow is the conclusion it drew. Naming a function that is absent is a reason to look for the one that is present, not a reason to declare the migration impossible. SignTo and Verify provide the signing operations, and all four are documented in the package. Whether the correction the agent needed was a single call or a small restructuring is beside the point: the capabilities it declared the library to lack are available under documented names.

This is the mirror image of the false success claim of Section 5.6. There, the agent falsely reported completion; here, it falsely reports impossibility and blames a correct, working dependency. Both are failures of accurate reporting, pointing in opposite directions, and both were produced by the same model on the same task under near-identical conditions.

Outcomes Across the Four Conditions

Table 12. What each condition establishes, for devstral-small-2. The four rows correspond one-to-one with the four rungs of Figure 4, and both are ordered as the study ran them. Every row describes that model: the one-shot and feedback runs on the feasible arm that it produced, its agentic runs, and its library-named runs. The other model, qwen3-coder, ran only the one-shot condition on this arm and failed by a different mechanism, substituting a classical algorithm rather than naming a nonexistent module (Table 8), so it does not appear in this sequence and the sequence says nothing about it. Naming the library removes the module-hallucination and stubbing failures without producing a correct migration. The agentic control outcomes are mutually exclusive and partition its five runs: two stubbed and reported success, two exhausted the budget, and one reported a blocker that was real but self-created rather than the underlying migration blocker (Section 5.6). The library-named outcomes are not exclusive: all five runs made the same API error, and two of those five additionally declared the migration impossible, so the counts overlap by design.

Condition	Library Available to the Agent	Failure
one-shot	model must name it	names a module path that does not resolve
feedback	model must name it	repeats that path verbatim after being shown the compiler error naming it (5/5)
agentic	model must name it	stubs to a passing build and reports success (2/5); exhausts budget (2/5); inaccurate blocker report (1/5)
library-named	given in the task	uses the correct library; misuses its API identically (5/5); declares it impossible (2/5)

records the outcome of each condition and Figure 4 sets out the sequence.

What the library-named condition supplied is specific: the dependency path, the installation command, and three public functions. What it did not supply is complete API documentation, worked examples, tests, or any account of how the dependency should be integrated with the surrounding repository. Naming the library removed the module-hallucination and stubbing failures, and did not yield a correct migration; the failure that remained occurred in API application and repository integration. Section 6.1 reads the four conditions together. In two of five runs, the agent invents a reason the dependency is inadequate rather than reporting its own difficulty with it.

Figure 5 presents the per-run activity of Table 10 and Table 11 graphically. The contrast between the two conditions is visible at a glance: in the agentic condition, the only runs to reach a green build of modified code are the two that stubbed the cryptography, and in the library-named condition, no run reached a green build at any number of attempts, while the number of file writes shows that the runs were attempting the integration rather than abandoning it.

6. Discussion

6.1. What the Four Primary Conditions Establish

Each of the four primary conditions removes one requirement from the model, and the sequence is more informative read together than condition by condition. The outcomes themselves are in Section 5.3, Section 5.4, Section 5.5, Section 5.6 and Section 5.7 and are not repeated here.

The sequence is single-model, and that bounds it. Only devstral-small-2 was run in all four conditions on the feasible arm, so the ladder traces that model’s failure and no other. qwen3-coder appears on this arm in the one-shot condition alone, and it failed differently: it substituted a classical algorithm rather than naming a nonexistent module (Section 5.3), so the requirement the ladder removes at each rung, knowing which library to use, was never the thing that blocked it. Two models on this arm produced two mechanisms, and the narrowing applies to one of them. Whether the other mechanism narrows the same way under the same conditions is untested.

What the sequence narrows. After the dependency path, its installation command, and selected API information are supplied in the task, the failure that remains occurs in API application and repository integration. That integration is substantial rather than incidental: the JWT library in use has not shipped ML-DSA support, so a conforming migration requires registering a custom signing method, constructing the RFC 9964 AKP key representation for the published key set, and keeping the discovery metadata consistent with it (Section 4.2). The finding is that the models did not do this work, not that the work is small. That is a claim about the stage at which failure persists for one model, not about which explanations survive. Knowledge, documentation, tooling, and reasoning are not separated by this design, and remain available as explanations. What the sequence does establish is narrower than “models cannot migrate cryptography” and points at a different remedy: supplying a curated dependency did not solve the problem, because the agent used the curated dependency and misused its API in every run, and in two runs concluded that the library was inadequate rather than that its own usage was wrong.

What answers each research question. RQ1 is answered by Section 5.3 and Section 5.4: no run produced a working migration on the arm whose ecosystem supplies a conforming primitive. A reference migration built after review passes the scorer on that arm (Section 4.2, Positive-Control Witness), so the answer is that none was produced by these models on a target since shown to be achievable with the scope extensions the witness discloses (Section 4.2). RQ2 by Section 5.2: no run reported the unavailability of the target. RQ3 by Section 5.5: compiler feedback naming the exact nonexistent module did not change the answer. RQ4 by Section 5.6: autonomy did not improve correctness and moved the stage at which failure becomes visible. RQ5 by Section 5.7: supplying the dependency and its API did not resolve the failure, which relocated to integration.

Two behaviors observed across conditions bear on assurance rather than capability. The first is that no condition produced an accurate account of the underlying migration blocker: the agent either reported success it had not achieved, or reported an impossibility that was not real, or named an obstacle it had created itself. The second is that both errors were produced by the same model on the same task under near-identical conditions, which suggests that they are two expressions of one underlying weakness in self-report rather than two separate faults.

6.2. Threats to Validity: The Apparatus Itself

Twenty defects were identified in the measurement apparatus: sixteen during the study and four after peer review, when the positive-control witness (Section 4.2, Positive-Control Witness) was scored and audited. The count is of defects in components that touched study data: the harness, prompt assembly, execution, and the scorer. Errors in preparing this manuscript are not included and would be a different unit of analysis. Their directional bias is the finding (

Table 13. Apparatus defects identified during this study, classified by the direction in which each biased the measured result.

#	Defect	Effect on Apparent Failure
1	git apply silently discarded most of a hunk when the model understated @@ line counts, and exited 0	inflates
2	documentation changes scored as migration evidence	deflates
3	corrupt-patch rejection from empty context lines	inflates
4	issuer and verifier partitioned by path rather than filename	inflates
5	raw model output discarded, making attribution impossible	inflates
6	prompt assembled by unsorted directory traversal, producing prompts of identical length and different content	inflates
7	invalid ML-DSA parameter set scored as correct	deflates
8	missing ML-DSA parameter set scored as correct	deflates
9	feedback probe executed against an interpreter lacking the dependency	inflates
10	feedback output never applied; a stale error reported as a result	inflates
11	a placeholder in the feedback prompt echoed verbatim by the model	inflates
12	permissive build-error exception swallowed a genuine compile failure	deflates
13	runner invoked only Layer 2, so Layer 2’s early return masked Layer 1 findings	deflates
14	Go behavioral adapter treated compilation as sufficient, scoring a repository containing no cryptography as correct	deflates
15	merge stage emitted correct when no layer produced output, certifying runs that were never measured	deflates
16	Go behavioral adapter accepted a post-quantum module on a name match in the resolved module graph, without requiring that any package import it or call it	deflates
17	Go behavioral adapter’s unsigned-token check matched Go’s idiomatic error return and ran over files that do not sign, rejecting two unmodified baseline files	inflates
18	static layer scanned test files, which the behavioral layer excludes, and read a negative-example literal in a test as the artifact’s algorithm name	inflates
19	static layer judged AKP conformance over the concatenated tree, attributing RSA parameters in generated API documentation to the ML-DSA JWK	inflates
20	Go behavioral adapter’s test gate runs the repository’s own tests, which reference the RSA key type and pin RS256, so a correct migration confined to the issued scope, from which tests were excluded, fails to compile them and is rejected	inflates

).

Twelve inflate apparent failure; eight deflate it. The four found by the witness all inflate it: each would have rejected a correct migration.

These 20 are the defects we found in one harness built by one author, not a sample drawn from a population of harnesses, and we offer no test that the split differs from chance. What follows is a post hoc observation about the set we have, stated because it suggested a practical guard rather than because it was predicted: in this set, the direction of bias is predicted by what the defect does to the evidence, not by where in the apparatus it sits.

Defects that corrupt or misdirect the evidence inflate apparent failure. Those in the transport, prompt-assembly, and execution path destroy, truncate, garble, or misdirect model output before it is ever judged: a model’s correct work does not survive a patch applier that discards half of it, a prompt that differs from the one intended, or a probe run against the wrong interpreter. Every defect of that kind in Table 13 inflates. A reason is available, though it was formulated after the fact: corruption in moving text between a model and a compiler is destructive, and we can think of no plausible bug of that kind which would improve the cryptography.

Defects that relax the acceptance criteria run the other way. Each is a form of leniency: accepting an invalid parameter set, accepting a missing one, swallowing a genuine compile failure, returning early so that static findings are never reached, treating a successful build as sufficient evidence of migration, treating the absence of findings from a layer that never ran as evidence of correctness, or accepting a dependency because its name appears in the module graph without asking whether any code uses it. Each credits the model with work it did not do, and so deflates.

Defects that tighten the acceptance criteria beyond the property are the third kind, and the four found by the positive control (17 to 20) are all of it: a check that rejects the idiomatic error return of the target language, a scope that reads a test’s negative example as the artifact’s claim, a conformance test that attributes another file’s parameters to the key set, and a gate that fails a migration on tests it was told not to touch. Each refuses the model credit for work it did do, and so inflates. None was visible while only failing artifacts were scored, because a criterion that is too strict is indistinguishable from a correct rejection until something correct is put in front of it.

We stated this correspondence in terms of location in an earlier draft, with transport defects inflating and scorer defects deflating, and one defect in our own set falsifies that version of it. Defect 4 partitioned issuer files from verifier files by directory path rather than by filename, so that a shared parent directory placed every file on one side of the comparison. The partition is computed inside the Layer 1 detector, which makes it a scoring-rule defect by location; but its effect is to misdirect the comparison rather than to relax it, producing a spurious one-sided finding against migrations that were in fact two-sided, and it therefore inflates. Location does not predict direction. Mechanism does, and Defect 4 is the case that made the difference visible. We report it because a generalization that survives only by excluding its counterexample is not worth stating.

Two things follow, and they are independent. For direction, the question is whether the defect corrupts evidence or relaxes criteria. Tightened criteria and corrupted evidence share a direction because both withhold credit; relaxed criteria grant it. For recoverability, developed in Section 6.2, Apparatus Currency of the Reported Runs, the question is whether the artifact in the run record survived the defect: Defect 4 is fully repairable by rescoring, because the files the model returned were never altered, only mis-grouped when they were judged.

The practical consequence is that the three mechanisms require different guards. Corruption is caught by verifying that what landed equals what was proposed, and by hashing prompts rather than measuring their length. Leniency is caught only by adversarial fixtures: inputs constructed to be wrong in a way the scorer might accept. Misdirection of the kind Defect 4 introduced is caught by neither, and needs a case in which the correct answer depends on the grouping itself. A validation case was added for each scorer defect as it was found, which is why that set is larger than the rule set. The validation runner and the expected findings for each case were archived at submission; the constructed input repositories the cases ran against were not, so a reader could see what was tested and what result was required, but could not re-execute the suite. Strictness, the opposite failure, is caught only by a positive control: an input constructed to be right, which the scorer must accept. This study had none until after review, and the four defects the witness found are all of that kind. The 11 fixture inputs, absent from the package at submission, were reconstructed as minimal repositories exhibiting each named condition and archived; the runner now executes and passes all 11 cases, with the reconstruction disclosed as such.

Two classifications were corrected during preparation, and we record the corrections rather than only their result. Defect 2, documentation changes scored as migration evidence, was initially recorded as inflating apparent failure. It does the opposite: crediting a docstring change as a migration grants the model work it did not do, which is leniency and deflates apparent failure. It is a scoring-rule defect and is classified with them. That correction moved the direction split from nine and seven to eight and eight. Defect 4, the issuer and verifier partition, was initially grouped with the transport defects on the assumption that its direction implied its location. It is a scoring-rule defect that inflates, which is what prompted the reframing above; its reclassification moves it between recoverability classes (Section 6.2, Apparatus Currency of the Reported Runs) and leaves the direction split unchanged.

We note this because it is an instance of the general problem the section describes. The misclassification survived several passes precisely because a running tally was maintained alongside a list rather than recomputed from it, and the same habit produced two earlier errors in this count.

Two are worth stating in full because we expect them to be live elsewhere.

Silent patch truncation. When a model’s unified diff understates its hunk line counts, git apply reads only the declared count, discards the remainder, and exits zero. In our case, a correct two-sided migration landed as a single docstring line while the harness reported success. Every symptom pointed at the model. We reproduced this deterministically and note that --recount mitigates it. Any evaluation applying model-generated patches should verify that what landed equals what was proposed.

Nondeterministic prompt assembly. Prompts built by walking a directory in filesystem order can differ in content while remaining identical in length, because file ordering is not stable across sessions. Measured run-to-run variance then silently mixes model stochasticity with input instability. Prompt length is not sufficient to detect this; prompts must be hashed.

Apparatus Currency of the Reported Runs

Declaring which records are excluded and why is not the same as demonstrating that every retained run was produced and scored by the corrected apparatus, and the second is what bears on validity. We separate the two claims here because the 20 defects differ in whether their damage is recoverable after the fact.

Scoring-rule defects (2, 4, 7, 8, 12, 13, 14, 15, 16, 17, 18, 19) are recoverable by rescoring. The artifact in the run record is unaffected; only the verdict was wrong. Rescoring the stored artifact with the corrected scorer repairs the result completely. Defect 4 belongs to this group even though it inflates rather than deflates: the files the model returned were unaltered and were only mis-grouped when judged, which is a verdict error and not an artifact error (Section 6.2). Defect 20 is of the same class but is disclosed rather than corrected, for the reason given in

Table 14. Correction chronology. For each defect: when it was found, the component of the apparatus it affected, the runs within scope at that time, the disposition of those runs, the validation added to prevent recurrence, and the commit in which the fix landed. No apparatus version field was recorded in the run records, so the component is named instead.

#	Defect	Found	Component	Runs Affected	Disposition	Validation	Commit
1	silent patch truncation	21 July 2026, M3b run series	diff-based harness	m3b_run_real-001 to -003	excluded from the scored set with Defect 3; the diff format was replaced by full-file output	applied-vs.-proposed check	1b47fa4
2	documentation scored as migration	21 July 2026, M3b run series	scorer	all runs scored before the fix	rescored with the corrected scorer; artifact unaffected	code_only() strips comments and docstrings	1b47fa4
3	corrupt-patch rejection	21 July 2026, M3b run series	diff-based harness	m3b_run_real-001 to -003	excluded from the scored set with Defect 1	full-file output format	1b47fa4
4	issuer and verifier partitioned by path	21 July 2026, M3b run series	Layer 1 detector	all runs scored before the fix	rescored with the corrected detector; artifact unaffected. Scoring-rule defect that inflates rather than deflates (Section 6.2)	filename-based partition	1b47fa4
5	raw model output discarded	21 July 2026, M3b run series	harness	runs executed before the fix	not recoverable; 27 of the 56 retained records carry no raw response, and the coverage is reported in Table 15	raw output persisted per run	1b47fa4
6	unsorted directory traversal	21 July 2026, M3b run series	prompt assembly	runs executed before the fix	the eight earliest runs of the infeasible one-shot cell were executed before the fix and their assembled prompts cannot be verified byte-identical; every later run was executed after it. The other records lacking a prompt hash do so because their runner’s record format omits the field, not because they predate the fix	sorted traversal, prompt hashing	1b47fa4
7	invalid parameter set scored correct	21 July 2026, M3b run series	scorer	all runs scored before the fix	rescored with the corrected scorer; artifact unaffected	parameter-set fixture	1b47fa4
8	missing parameter set scored correct	21 July 2026, M3b run series	scorer	all runs scored before the fix	rescored with the corrected scorer; artifact unaffected	bare-family-name fixture	1b47fa4
9	feedback probe wrong interpreter	21 July 2026, M3b run series	feedback harness	fb_fb-001	excluded from the scored set; the model’s response is reported in Section 5.2 as an out-of-dataset observation	interpreter assertion	1b47fa4
10	feedback output never applied	21 July 2026, M3b run series	feedback harness	none among the retained runs	no retained feedback record shows the signature: where round two repeats the round-one error, the model returned no code; where it returned code, the error changed	applied-stat verification	1b47fa4
11	placeholder echoed verbatim	21 July 2026, M3b run series	feedback prompt	fb_fb-010, fb_fb-011	excluded from the scored set; the responses contain no migration to score	placeholder fixture	1b47fa4
12	permissive build-error exception	21 July 2026, first R1 one-shot run	Layer 2 adapter	all runs scored before the fix	rescored with the corrected scorer; artifact unaffected	build-failure fixture	1b47fa4
13	runner invoked Layer 2 only	22 July 2026, R1 one-shot series, second model	runner	the feasible one-shot cells and part of the R3 one-shot cell	Layer 1 was never invoked, so no static result was persisted; the values are reconstructed from the archived raw output and reproduce all nine that the archive records directly	full three-layer invocation	1b47fa4
14	build treated as sufficient	22 July 2026, first agentic run	Go behavioral adapter	the five agentic runs	rescored with the corrected adapter; the rescored result was not written back for run 001, which was detected during manuscript preparation and repaired from the surviving working copy	module-graph resolution check	060921f
15	merge emitted correct without behavioral evidence	23 July 2026, during deposit assembly	merge stage	ten records of the infeasible arm: m3b_run_clean-013 to -017 and m3b_run_real-004 to -007, -012	re-merged under the corrected rule: the five m3b_run_clean records escalate to human review, the five m3b_run_real records resolve to F3 from the reconstructed static layer; superseded values retained in each record	rule 5, correctness requires behavioral evidence, with three merge self-tests	64224ec
16	behavioral adapter accepted on a module name	23 July 2026, adversarial fixture during deposit review	Go behavioral adapter	one record, r1_run_r1-001, whose stored Layer 2 verdict was correct; the other nine reconstructable one-shot runs were unaffected	adapter hardened to require the post-quantum package to be imported and a signature API called, to reject a module satisfied by a local replace, and to treat failing crypto-package tests as disqualifying; the ten reconstructable runs re-probed, and r1_run_r1-001 resolves to F6 at the build stage	adversarial repository that compiles, imports no post-quantum API, signs nothing, publishes no key set, and returns a literal token, carrying only a local module whose name matches the regular expression	781dabe
17	unsigned-token check rejected a correct migration	11 September 2026, positive-control witness after peer review	Go behavioral adapter	none: no scored run reached the token-path stage (archived Layer 2 stages: build 28, dependency 2); the ten reconstructable one-shot runs rescored unchanged	empty alternative removed from the pattern; check scoped to files containing a signing call	positive-control witness, a correct migration the adapter must accept	ce57e2d
18	static layer scanned test files	11 September 2026, positive-control witness after peer review	static layer, scope	none: static-layer findings for every reconstructable run unchanged on rescoring	test files excluded from Layer 1 scope for every class except C10, whose subject is the tests	positive-control witness; fixture suite of eleven cases reconstructed and executing	264a58f, 33b1621
19	AKP conformance judged over the concatenated tree	11 September 2026, positive-control witness after peer review	static layer, detect_C1 and detect_C5	none: the five infeasible-arm runs whose static verdict passed the conformance check are unchanged under per-file evaluation; no feasible-arm run reached it	conformance judged per file, so a wrong parameter counts only in a file that itself names ML-DSA	positive-control witness	264a58f
20	test gate and task scope disagree	11 September 2026, audit of the positive-control witness (Section 4.2) after the witness was accepted	Go behavioral adapter, test gate; task scope (Section 4.4)	none: no scored run reached the tests stage	disclosed, not corrected: the gate is retained because a migration that leaves the package’s tests uncompilable is not complete, and the scope as issued cannot be changed after the fact; folding the certifying checks into Layer 2 (Limitation 8) would replace the gate	witness re-probed with the baseline key_test.go restored: F6 at the tests stage, compile failure on the RSA key type (witness/l2-witness-baseline-tests-F6.json)	none

.

Transport, prompt-assembly, and execution defects (1, 3, 5, 6, 9, 10, 11) are not. No rescoring recovers them, because what they damage is not the verdict. They do not all damage the same thing, though, and the disposition follows the distinction rather than the class label.

Five of them corrupt an individual run’s input or output: a patch silently truncated or rejected (1, 3), a probe run against the wrong interpreter (9), feedback never applied so a stale error was reported (10), and a prompt placeholder echoed back as the whole response (11). A run affected by one of these is usable only if it was re-executed after the fix. None was, so all six records concerned are excluded, together with the harness plumbing test (Section 4.3).

One loses evidence without corrupting anything (5): the raw model response was discarded, so those runs are scoreable but not reconstructible. They are retained, and the coverage is reported rather than described as complete.

One introduces variance between runs rather than corrupting any single run’s input (6): prompts assembled by an unsorted directory walk contain the same files in an unstable order. The eight earliest runs of the infeasible one-shot cell predate that fix and cannot be shown byte-identical to the prompts intended. They are retained, because the claims those runs support are universal rather than comparative and an unstable file order cannot manufacture a universal negative, and the limitation is stated rather than resolved (Limitation 7).

The disposition of the uncorrectable-class runs, established rather than flagged. Defects in the transport, prompt-assembly, and execution path cannot be repaired by rescoring, so a record produced under one of them is usable only if it was re-executed after the fix. We therefore classified every record in the archive against that class by automated checks rather than by inspection of results, and excluded those that were not re-executed. Three groups emerged. The output format of the harness changed during the study: the earliest runs used a unified-diff format, in which Defects 1 and 3 occur, and the harness was then changed to require full file contents, which removes that failure class entirely rather than patching it; m3b_run_real-001 to -003 were produced under the diff format, and in one the patch was rejected and never applied. In fb_fb-001, the feedback probe ran against an interpreter lacking the target dependency (Defect 9), so the error shown to the model was an environment failure rather than a migration error. In fb_fb-010 and fb_fb-011, the model returned the feedback prompt’s placeholder text verbatim (Defect 11), leaving no migration to score. Together with the harness plumbing test, these are the seven records declared in Section 4.3. Every retained run was produced under the corrected transport path, and none is affected by a defect that corrupts an individual run’s input or output. Eight retained runs predate the prompt-assembly fix and are dispositioned above rather than excluded; that is the one place this class is handled by disclosure instead of removal. This resolves the question rather than deferring it: the excluded records are named, the criterion is mechanical, and the cells they came from report the behavior of the runs that remain.

What we can and cannot demonstrate from the archive. Four checks were run over the archive during preparation, and they establish different things.

First, provenance fields. verify_apparatus_currency.py reports what each run record carries. Several fields are partial because prompt hashing, raw-output persistence, and checkpoint recording were themselves remedies for Defects 6, 5, and the provenance gap, and apply only to runs executed after those fixes; no record carries an apparatus version. The per-field counts are given in

Table 15. Coverage of the run records in the archive: for each field, the number of the 56 retained records that carry it, and the reason the rest lack it.

Field	Present in	Absent Because
condition, repository, scored result	56	present in every record; the allocation and the findings rest on these
model identifier	39	the 17 feedback records use a structure that omits it; the runner defaults are given in Section 4.4
checkpoint digest	29	added mid-study, and omitted by the feedback and agentic record formats
repository pin	29	as above
raw model response	29	added mid-study as the remedy for Defect 5; the agentic runners write none
assembled-prompt hash	21	added mid-study as the remedy for Defect 6; the feedback and agentic formats omit it
merged scorer verdict	29	the feedback and feasible-arm one-shot records carry the layer results the paper reports instead
Layer 1 result, from the original scoring run	9	Layer 1 was not invoked by every runner (Defect 13)
Layer 1 result, written back from a reconstruction	5	remediation of Defect 15; superseded values retained in each record
Layer 1 result, reconstructed on demand and not stored	15	reproducible with rescore_from_raw.py
apparatus version	0	the harness never recorded one

.

What the run records do and do not preserve. Each record carries the condition, the repository, and the scored result. Beyond that, the records are not uniform, and the per-field counts are given in Table 15 rather than summarized as completeness: the model checkpoint, the raw model response, and the merged scorer verdict are each present in about half. Transcripts of the agentic runs record the sequence of tool calls, including the path and byte size of every file written and the build output returned, but not the contents of those files, so a written artifact can be identified from a transcript and not reconstructed from it. The applied working trees were temporary and were not retained. Where a run’s raw response was archived, the files it returned can be recovered from that response and the repository rebuilt, which is what the re-probing described below does; for the remaining runs they cannot.

Coverage of the archive, stated in one place. The run records are not uniform, for two reasons that are worth separating. Several fields were remedies for apparatus defects and are therefore absent from runs executed before those fixes. Separately, the feedback and agentic conditions were driven by runners that write their own record structures, so some fields are absent from those records irrespective of when they ran. Only the first reason bears on validity; the second is an evidence gap. Table 15 gives, for each field, the number of the 56 retained records that carry it.

The 35 records without a prompt hash divide as follows: 8 are the earliest runs of the infeasible one-shot cell and predate the fix, which is the only group whose prompt assembly is unverified; 17 are feedback runs and 10 are agentic runs, whose record formats omit the field regardless of when they ran. Nine records carry a Layer 1 result from the original scoring run, and five more carry one written back from a reconstruction during the Defect 15 remediation, with the superseded value retained in each. Twenty Layer 1 values in total, including those in Table 8 and Table 9, were reconstructed from archived raw output by rescore_from_raw.py, which reproduces all nine independently recorded values and treats the five written-back records as reconstructed rather than as controls; the script and its output are included so that a reader can repeat the reconstruction rather than take it on trust. The applied artifacts themselves were not retained: the working trees were temporary, so the returned files are recoverable only for the runs with archived raw output. Earlier runs record the prompt’s construction inputs and the applied result but not the hash or the verbatim response. The seven excluded records described in Section 4.3 are retained in the package under a separate directory, so that the exclusion can be inspected rather than taken on trust. We report the coverage rather than describe the archive as uniform.

How the archive is indexed. The archive is indexed by MANIFEST.md, listing every file with its SHA-256, and RUN_INDEX.md, giving one row per scored run with its repository, model and checkpoint where recorded, condition, prompt variant, prompt hash where recorded, artifact paths, and per-layer scorer results, so that every quantitative result and quoted excerpt in this paper can be traced to a run identifier. It contains the 56 scored runs and no other scored records, together with the 7 excluded records in a separate directory (Section 4.3), the feasibility-audit evidence bundle of Section 4.2, the model-provenance bundle of Section 4.6, and, held separately because it was not scored by the scorer, the exploratory material of Section 6.3: the 30 single-file ablation responses, the script that re-derives their outcomes, and its output. The Supplementary Materials state what else is absent.

Second, merge-rule currency. Twenty-nine of the retained records carry a merged verdict. For each, rescore_record.py reconstructs the inputs the merge stage was given, from the provenance map it wrote and from the deterministic notes it emits for deferrals, re-runs the current merge rule over them, and compares. All 29 reproduce. No archived verdict was written by a superseded merge rule, which is the defect class that produced the stale record described below. The check does not re-derive findings from the layers, so a defect in a layer rule is outside its reach.

Third, Layer 1 reconstruction. Because the harness had models return complete file contents rather than diffs (Section 4.4), the records with archived raw output contain the full text of every file the model returned, which is the input the static layer scored. Re-running the static layer over those reconstructed artifacts reproduces every Layer 1 value the archive records elsewhere, 9 of 9, and supplies a Layer 1 value for 20 runs whose records carry none. Five of those twenty were subsequently written back into their records during the Defect 15 remediation, so those records now carry a Layer 1 value that this reconstruction produced. They are excluded from the control set by the script, and the control remains the nine values the original scoring run recorded. Validating a reconstruction against values it generated would report an agreement it has not earned, and the distinction is enforced in code rather than left to a reader. Those 20 divide into two groups: runs scored before Defect 13 was fixed, where the runner invoked Layer 2 only and no static result was produced, and runs whose static result was correct, which the merge stage drops before writing provenance and which therefore leaves no trace in the record. The Layer 1 values in Table 8 and Table 9 rest on this reconstruction and are labeled as such in their captions.

Fourth, behavioral re-probing. The archived raw responses also allow the applied repository to be rebuilt, by writing the returned files over a clean copy of the seeded baseline, and the behavioral layer re-run over it. This was done for the 10 one-shot runs on the feasible arm after the adapter was hardened (Defect 16). Nine reproduce their stored verdict exactly, which is the evidence that the reconstruction is faithful; the tenth, r1_run_r1-001, moves from a stored correct to F6 at the build stage, and its own archived test output corroborates the change by recording an undefined symbol in the same package. The 10 agentic and library-named runs cannot be re-probed this way, because their files were written through tool calls whose contents the transcripts do not retain. Those keep their stored verdicts, and the argument that the hardening does not disturb them is from the stage each reached rather than from a re-probe: eight failed at the build stage, which the hardening does not touch, and two failed at the dependency stage, which the hardening only makes stricter.

What no check can establish is an artifact-level rescore of the kind that would re-derive both layers from the state of the repository after the model’s changes were applied. The run records do not store the applied artifact, and the working trees were temporary. For the 29 retained runs with archived raw output (31 of the 63 records in the archive, counting the excluded), the returned files are recoverable, which is what the third check uses; for the remaining 27 retained runs they are not. Nor can any check establish re-execution after a transport-path fix, since that is a fact about the study’s history rather than about the archive, and it is the reason Table 14 is required.

An instance found by this check. Applying the above to the archive during preparation revealed that one run record, the first agentic run, still carried the pre-fix Layer 2 verdict correct produced by apparatus Defect 14. The rescore had been performed and its result had not been written back. Because Defect 14 is a scoring-rule defect, the artifact was unaffected and the record was repairable: the run’s working copy was identified from the write-size fingerprint recorded in its transcript, rescored with the corrected adapter, and the archived record updated, with the superseded values and the identification basis retained in the file. The corrected result, not_migrated and F5, is the value this paper has reported throughout. We record the episode because it is the failure mode this section describes, occurring in our own archive, and because a reader is entitled to know that the check was applied and found something.

Implication. Twelve defects would have inflated apparent migration regression and eight would have deflated it. We draw no net direction from that. The split was eight to eight at submission, and the four that moved it were found only when a correct artifact was scored for the first time, so the tally measures where we looked as much as what the apparatus contained; an earlier draft that read a net direction from the count was wrong, and a reported failure rate cannot be treated as an upper bound on that basis.

What the set does show is that these defects are not symmetric noise. Each has a sign, the signs are predictable from what the defect does to the evidence, and they do not cancel: a defect that discards half a patch and a defect that accepts an invalid parameter set both bias the result, in opposite directions, by amounts that have nothing to do with each other. An uncorrected harness therefore reports a number whose error is neither bounded nor centered, and the direction of its residual bias depends on which defects it happens to contain. That is a worse position than a known one-sided bias, because it cannot be corrected for after the fact. We recommend that evaluations of agentic coding state what apparatus validation was performed, and that a failure rate published without such a statement be read as carrying an unquantified two-sided error rather than as an estimate.

6.3. Limitations

This study is deliberately narrow and the following constraints are material.

• Two models. Both open-weight, both self-hosted. No frontier proprietary model was tested. Generality is unestablished.
• Two repositories, one condition class. R2 and R4 were prepared but not run, and their feasibility is unresolved. Only the issuer–verifier condition was exercised.
• Agentic arm is single-model. The tool loop was run with devstral-small-2 only, n = 5. Network module fetch was available and used (Run 2 downloaded a real JOSE library mid-loop), so stubbing was not the only route to a passing build; writing the correct CIRCL import would have compiled. What the agent lacked was a discovery mechanism: no search, no package index, no documentation access. It could fetch any path it named and named the wrong one. The library-named variant of this experiment was subsequently run and is reported in Section 5.7: supplying the correct import path removed the stubbing but produced no correct migration. What remains untested is the discovery variant, in which the agent is given a search tool or package-index access and must locate the library itself.
• Dated ecosystem finding, and a young standard. RFC 9964 was published in May 2026, 2 months before this study. The absence of implementations is expected at that age and will close. The finding is therefore about model behavior when a requested target is not yet implementable, not about a persistent deficiency in the Python ecosystem.
• Unexplained context sensitivity, from exploratory probes outside the primary dataset. During preparation, we observed that a particular prompt scope on the feasible repository, the six source files assembling to a 23,033-byte prompt described in Section 4.4, produced classical substitution in 11 of 11 attempts, while both a smaller scope (5.9 KB) and a larger one (61 KB) produced post-quantum selection. The effect is therefore not monotonic in context volume. Four candidate mechanisms were advanced and each falsified, but every one was tested at n = 1.
  • A systematic single-file ablation was then executed to replace those single-sample probes: from the full scope on the feasible repository, one source file was removed at a time and the task reissued, giving 12 conditions, a full-scope control and 11 removals, across 30 responses at 2 or 3 repetitions each. Removing pkg/token/generate.go or pkg/token/revoke.go eliminated post-quantum identifiers from the returned files entirely and in every repetition; removing pkg/token/model.go, pkg/token/refresh_token.go, or pkg/wellknown/handler.go produced different outcomes across repetitions of the same condition. The full-scope control itself returned both post-quantum and classical identifiers in all three repetitions. The sensitivity is therefore to which files are present rather than to how many, which is consistent with the non-monotonic result above and is the reason we describe it as unexplained rather than as a volume effect.
  • These attempts are not part of the 56 runs and are not scored by the scorer. They were run with qwen3-coder on the R1 repository, and their outcome was read at the time from regular-expression counts over the whole model response, a measure that conflates prose with code and that we replaced for exactly that reason in the primary apparatus. Because the responses were archived in full, the pre-registered replacement measure, restricted to the text inside the model’s delimited file blocks, has since been applied to all 30: it changes the magnitude of the counts but agrees with the original measure on the presence or absence of post-quantum output in every response. Three further constraints bound this material. The repetitions are unbalanced, six conditions at three and six at two. The per-run summary log was not captured, so the outcomes reported here are re-derived from the archived responses rather than read from the original run output. The measure counts identifiers, not correctness: it distinguishes what algorithm family a response reached for, not whether the migration would work.
  • We report the observation because it is reproducible and because it bears on how prompt scope should be chosen in comparable evaluations, and we report it as an observation only. It appears in no table, contributes to no reported rate, and does not support a mechanism or a comparison with any result in Section 5. The responses, the re-derivation script, and its output are archived with the replication package.
• Self-reported cutoffs are unreliable. The knowledge-gap and dissociation claims rest on the contrast between direct answers and task behavior, not on the self-reports alone.
• Apparatus currency is established three different ways, and one of them is disclosure rather than repair. Rescoring establishes that the scoring-rule defects do not affect the reported verdicts. Defects that corrupt an individual run’s input or output cannot be repaired by rescoring, and rather than assert that the affected runs were re-executed, we identified them by automated checks and excluded them (Section 4.3 and Section 6.2, Apparatus Currency of the Reported Runs). The third case is neither: the eight earliest runs of the infeasible one-shot cell were assembled before the prompt-ordering fix, so their prompts cannot be shown byte-identical to those intended. They are retained rather than excluded, because that defect introduces variance between runs rather than corrupting any single run’s input, and the claims those runs support are universal rather than comparative. A reader who does not accept that reasoning should treat those eight as at risk; the R3 one-shot cell would then rest on one retained run, and the two findings this paper rests on are unaffected because they come from other cells. What remains asserted rather than demonstrated is that no further record was affected by a defect in that class, which left no detectable trace in the archive; that claim rests on the correction chronology (Table 14). This study’s conclusions do not depend on the excluded records, and the two findings this paper rests on come from the agentic and library-named conditions, which were executed after the harness output format was fixed.
• The acceptance oracle is not a certifying one. Every finding this paper reports is a rejection, apart from the five escalated records of Limitation 10, whose failure rests on the surrogate execution evidence of Table 9; the rejections rest on execution: the code does not compile, the library exposes no such key type, the crypto packages fail their tests. On the infeasible one-shot arm that execution evidence comes from the feedback probes and the feasibility audit rather than from per-run Layer 2 execution, since those runners carried no Layer 2 adapter (Section 4.5, Table 9). The acceptance path was not exercised by any scored run. It has since been exercised once, by the positive-control witness (Section 4.2, Positive-Control Witness), which it rejected three times before accepting, and a fourth defect found on audit would reject any migration confined to the scope the agents were issued (Defect 20); where it accepts, it is not sufficient on its own. A test that certifies a realized repository-level migration would have to issue a token with the migrated code and verify it, parse the JWS header and confirm an RFC 9964 ML-DSA algorithm identifier, retrieve the published key set and confirm an AKP key representation, reject a tampered signature, reject a classical algorithm offered in its place, and confirm that the migrated path is the one the application reaches under its own configuration. Our Layer 2 does none of these; it establishes that a dependency is present, imported, called, and compiling, which is necessary and not sufficient. The results stand because the reported failures were reached at the build and dependency stages on the Go arm, and through identifiers the audited libraries do not expose on the Python arm, none of which the four acceptance-path defects touch, and rescoring under the corrected scorer changed no verdict; a study reporting successes would need the certifying version before its results could be believed. The checks that certify a migration now exist as the witness’s test suite rather than as part of Layer 2, so the oracle accepts the witness on the necessary conditions and the sufficient ones are established outside it; folding those six checks into Layer 2, in place of the test gate that Defect 20 concerns, is the remaining work before this apparatus is used to measure anything other than failure. The witness also establishes that an application-level R1 migration is achievable with the scope extensions disclosed in Section 4.2, which that section could previously only argue; it does not establish that the original scope and the unchanged acceptance gate were jointly satisfiable (Defect 20).
• The documentation comparison is confounded with execution order. Its two arms were run as consecutive blocks rather than interleaved, so anything that varied between the blocks is indistinguishable from the manipulation. We report the separation descriptively for that reason (Section 5.2), and the comparison should be repeated with interleaved conditions before the effect is treated as established. The systematic ablation of Section 6.3 was designed with interleaving for this reason and postdates this comparison.
• Two layers exercised, not three, and a single scorer author. The semantic layer was implemented and validated against fixtures but was not invoked on any scored run, so every reported verdict rests on static analysis and execution evidence alone. Cases that the first two layers could not adjudicate would, by design, have escalated, and five did: after the Defect 15 re-merge, the five documentation-ablation records whose static classification was correct with no behavioral evidence carry a merge verdict of escalate-to-human-review (Table 14, row 15). Human adjudication was not performed; those five are reported by their Layer 1 finding, and the evidence that their code does not run comes from the feedback probes and the feasibility audit (Table 9). The escalation path was therefore exercised only to the point of raising the flag. The scorer was also written and calibrated by one author and was not independently audited. After review, the acceptance path was validated against a positive control for the first time and four defects were found in it, each biased toward rejection (Section 4.2, Positive-Control Witness); the rejection path, on which every reported verdict rests, was unchanged by the corrections when the 29 runs with archived returned files, the one-shot and documentation-ablation records, were rescored with no verdict changed; the 27 feedback, agentic, and library-named records carry no archived returned files and keep their stored verdicts (Section 6.2, Apparatus Currency of the Reported Runs). Independent re-adjudication of the scored runs remains undone.

6.4. Implications for Practice and Research

For organizations delegating migration. The results support three practices, none of which depends on the models improving. First, the acceptance gate for a cryptographic migration should test the intended cryptographic behavior rather than rely on compilation or generic functional coverage: a token issued by the migrated code should be verified, its algorithm identifier parsed and checked against the intended FIPS 204 parameter set, the published key set checked for the intended key representation, and a classical algorithm offered in its place refused. The gate should decline to certify a migration where that check did not run, which is the constraint our own merge stage now enforces (Section 4.5), and which our apparatus violated until it was implemented. Second, an AI-produced migration should be treated as unverified until that gate passes, regardless of what the agent reports: in this study, every completion claim was false, and the two runs that declared the task impossible were also wrong. Third, dependency selection should be supplied rather than delegated, and then verified. Naming the library removed the module hallucination and the stubbing entirely (Section 5.7), and the residual failure was in applying its API, which is a narrower and more reviewable problem than choosing the wrong supply chain. The discovery guidance of Section 2.3 gives organizations a place to record the intended target and the dependency that supplies it; the gate above is what checks that the artifact matches the record.

For the models and their tooling. Several remedies are suggested by the stages at which failure was observed, and the design of this study lets us say which of them it has tested and which it has not. Retrieval of current standards and library documentation, whether by retrieval-augmented generation or by a documentation tool in the agent loop, addresses a knowledge gap of the kind devstral-small-2 exhibited (Section 5.3, The Mechanisms Differ); it is not tested here. Package-index and web-search tools in the agent loop address dependency discovery, and would give an agent a way to establish that github.com/cloudflare/ml-dsa-go does not exist before importing it. The agent in this study had network module fetch but no discovery mechanism (Section 6.3, Limitation 3), so the supply-chain hallucination rate in the agentic condition is that of an agent that had to recall the path rather than look it up, and a discovery-equipped agent is the natural baseline against which to measure it. The library-named condition supplies part of what discovery would have found, the path and selected API names, and in the tested devstral-small-2 condition that did not suffice: the failure persisted after the dependency was named and its API supplied. The effects of interactive discovery, worked examples, and retrieval on either model remain untested. Neither remedy was tested against the knowledge-application dissociation of qwen3-coder, which stated the relevant fact and did not apply it, and for which the file-scope sensitivity of Section 6.3 is the only lead the evidence gives. Fine-tuning on paired classical and post-quantum code, which Pallarés de Bonrostro et al. [23] found decisive at fragment level, is the remedy the literature supports most directly, and whether it transfers to repository-level integration requiring a custom signing method and a hand-built key representation is the question their own repository validation raises. What no model-side remedy removes is the need for the behavioral gate, because each remedy may change the rate at which failure occurs and none changes what a green build can express.

For evaluation research. Four items of follow-on work were indicated at submission, and one has since been done. The first, a known-good migration witness for the feasible arm, was built after review (Section 4.2, Positive-Control Witness): it establishes that arm’s feasibility with the disclosed scope extensions and, on first contact with the acceptance path, found four defects that would each have rejected a correct migration, which is the argument for scoring a positive control before negative results are believed. What it leaves open is folding its six behavioral checks into Layer 2 (Limitation 8). A discovery-tooling condition, in which the agent has package-index or search access and must locate the library itself, would test the remedy that Limitation 3 leaves open, and should be run interleaved with a no-discovery control rather than in consecutive blocks (Limitation 9). A wider model set, including frontier proprietary models, would establish whether the two mechanisms observed here, and the conversion of build-visible failure to concealed regression under autonomy, are properties of these two checkpoints or of the task; the present study bounds its claims to the models tested and predicts nothing either way (Limitation 1). The fourth is a check on the evaluator rather than on the models, and is the cheapest: independent re-adjudication of a sample of scored runs against the archived transcripts, which would give the scorer the external validation that Limitation 10 records it does not have.

7. Conclusions

Two open-weight coding models, qwen3-coder and devstral-small-2 at the checkpoints given in Table 5, asked in July 2026 to perform a post-quantum migration for which one language supplies a conforming primitive and the other, because the relevant standard was 2 months old, supplied none in the five libraries audited, produced no working migration in 56 runs: 46 across the one-shot, feedback, agentic, and library-named conditions, and 10 in a nested documentation ablation within the one-shot condition.

Every claim that follows is bounded to those two checkpoints, those two repositories, and those 56 runs. We report what these models did, not what AI coding assistants do, and we draw no conclusion about the suitability of AI coding assistants for post-quantum migration in general: the results establish only that, for the models and conditions tested, neither build success nor the agent’s own report distinguished a completed migration from a failed one. The documentation comparison of Section 5.2 is likewise descriptive, 10 runs with complete separation, confounded with execution order, and is not offered as evidence of a causal effect.

The migration the models were asked to perform is achievable at the application level, with the scope extensions disclosed in Section 4.2. A reference migration built after review, with the library the library-named condition supplied, passes the scorer and six behavioral checks (Section 4.2, Positive-Control Witness); it took under an hour to construct with assistance, and scoring it found four defects in the acceptance path: three rejected it in turn before it was accepted, and the fourth would reject any migration confined to the scope the agents were given. The witness demonstrates that an application-level R1 migration is achievable with the disclosed scope extensions; it does not establish that the original scope and the unchanged acceptance gate were jointly satisfiable. The observed failures, nonexistent dependencies, API errors, and false completion claims, stand independently of that question, and the oracle would not have recognized success had the models achieved a migration.

No run accurately reported the underlying migration blocker. One agentic run correctly identified a self-created dependency-path error, but did not identify why it lacked a working migration. Two further runs declared the migration impossible, but both did so by mistaking absent function names for absent capabilities: the names they invoked are not in the package, and the capabilities they needed are.

One model substituted a classical algorithm while separately and correctly stating that the algorithm is quantum-vulnerable. The other selected the correct algorithm and imported a module that does not exist, then repeated that import after being shown a compiler error stating so.

Granting the agent tools and a compiler did not improve this. It moved the point at which the failure becomes visible: no modified artifact compiled unless ML-DSA key generation had been stubbed, and the two runs that reached a passing build had stubbed it and reported success. The resulting artifact is a build-green concealed regression rather than a fully silent one, since the retained classical signing call fails when exercised, but it defeats the signal the agent had and the signal on which many pipelines gate. Naming the correct library in the task, which removes any question of the model knowing what to use, did not produce a correct migration either. The agent used the library and could not apply its API, and in two of five runs declared the migration impossible, having invoked function names the package does not define and inferred from their absence that the capabilities were absent too. In the library-named condition, failure persisted after the dependency path, installation command, and core API were supplied; the observed residual failure occurred during API application and repository integration. This does not eliminate every knowledge, documentation, tooling, or reasoning explanation, and we do not claim that it does. It narrows the failure to a stage: not selecting the dependency, but using it.

The practical implication for organizations delegating post-quantum migration to AI assistants is that neither functional review nor build success is sufficient evidence of migration. A change can remove RSA, install a respected modern algorithm, update the documentation, and deliver no post-quantum security. Only behavioral verification against the intended algorithm distinguishes the two.

The methodological implication is narrower and applies to anyone measuring these systems: apparatus defects in this class of study are not symmetric noise. Each has a sign, predictable from what it does to the evidence, and in our set, 12 pointed toward apparent failure and 8 away from it, with the 4 that broke the earlier even split having been found only when a correct artifact was finally scored. They do not cancel, because their magnitudes are unrelated, so an unvalidated harness reports a figure whose error is two-sided and unquantified.