Fixing, Breaking, or Faking It? An Execution-Calibrated Evaluation of LLM Vulnerability Patching in JavaScript, Python, Go, and Java, and the Limits of LLM-as-Judge

Abstract

Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness. Neither reveals whether working code was broken. We evaluate eight commercial and open LLMs on 922 JavaScript vulnerability patches, scoring neutralisation and functional preservation. Lacking tests, we score at scale with a reference-based LLM judge, calibrated against execution on a 144-patch benchmark and 254 Java-CVE patches, plus a cross-family judge. The best model fixes 23% of vulnerabilities (judge-based), and cost-efficiency inverts the accuracy ranking. Our central finding concerns the instrument: both judges flag more over-fixes than execution confirms (precision 5–10%), yet on the functional axis agree far more with each other ($\kappa =0.75$) than with execution ($\kappa \le 0.26$), so judge–judge agreement measures reliability, not validity. On real Java code the over-reporting persists, while the judges’ correctness estimates diverge, leaving no single judge trustworthy. Over-fixing is genuine but, under execution, uncommon: a few percent of vulnerability-removing patches, or under 2%, excluding one artefact-prone scenario, both lower bounds. Only adequately tested execution measures the functional-preservation rate, so security-patch evaluation must run the code, use a judge only to rank models, and weigh costs. We release the harness and executable benchmark.

1. Introduction

Software vulnerabilities are disclosed faster than they are remediated. Large language models (LLMs) that read code and propose edits are an obvious tool for closing this gap, and a growing literature reports that models can, in some fraction of cases, produce a fix that resembles the one a developer committed [1,2,3]. This has produced benchmarks and leaderboards that rank models by how often their output matches, or is textually similar to, a reference fix.

Two questions hide inside a single “fix rate.” The first is whether a patch removes the vulnerability; the second is whether it preserves the program’s behaviour. A patch can do the first without the second: it can shut down the vulnerable path and, in doing so, break legitimate functionality. This is the security-repair instance of the classical overfitting problem in automated program repair (APR), where a patch passes the tests used to guide repair but breaks untested-but-desired behaviour [4,5]. We call this security form over-fixing. It is invisible to similarity metrics and to any check that asks only “was the CWE removed?” From an operational standpoint, an over-fixing patch is not a fix: it trades a security bug for a functional regression.

But there is a subtler trap, and it is the one this paper is really about. Because most real vulnerability corpora provide no tests, researchers increasingly measure correctness with a second LLM acting as a judge. If that judge is itself unreliable on the very axis that distinguishes a fix from an over-fix, namely functional preservation, then a study can report a dramatic “over-fixing” or “fix-rate” number that is an artefact of its measuring instrument rather than a property of the patches. A recent audit found that every one of the 72 LLM-for-security papers it examined exhibited at least one such methodological pitfall [6]. We therefore treat the measurement instrument as an object of study, not a given.

We evaluate eight commercial and open LLMs on 116 real vulnerable JavaScript functions, scoring each patch on the two axes above. At scale, we use a reference-based LLM judge; we then calibrate that judge against an execution oracle on a purpose-built benchmark of runnable vulnerabilities, and we report their agreement. This calibration is what lets us separate real findings from instrument artefacts. Finally, we push the calibration onto real repository-scale code, running 254 patches across 33 real Java CVEs under full Maven execution.

Our contributions are as follows: (1) An execution-grounded characterisation of LLM JavaScript vulnerability patching that reports per-model correctness with confidence intervals and finds that even the best model fixes only a minority of the vulnerabilities. (2) The demonstration that an LLM judge cannot reliably measure functional preservation. On a runnable benchmark it reports over-fixing several times as often as execution finds it (precision 5–10%, recall 12–38%), and even its more reliable verdict, whether a patch is correct, erodes on hard real code. Over-fixing itself is a genuine but, when measured by execution, uncommon failure mode: among vulnerability-removing patches it occurs in 7.1% on the benchmark, 8.9% across three languages, and 4.2% on Java (1 of the 24 neutralising patches, or 1 of 254 overall), and each of these figures is a lower bound. Its apparent concentration in one weakness class is an artefact of a single constructed test. (3) Confirmation that textual similarity is uninformative about correctness ($\kappa =0.15$). (4) A cost analysis showing that cost-efficiency inverts the accuracy ranking. (5) A repository-scale replication on 254 patches across 33 real Java CVEs under full Maven execution. There, the commercial-over-open gap holds under a model-level test, though it is confounded by model scale and memorisation of pre-cutoff fixes, and both judges inflate the over-fix rate (28 and 43 labels to execution’s one). Their correctness estimates diverge in opposite directions (one over-credits at $\kappa =0.09$, the other under-credits at $\kappa =0.57$), so inter-judge agreement itself breaks down on hard code. In practice, security-patch evaluation must run the code in order to measure any such rate. We release the evaluation harness, the judge prompts, and the executable benchmark.

We organise the study around four questions, plus a repository-scale replication. RQ1: How often do current LLMs correctly fix real vulnerabilities, and how do commercial and open models compare? RQ2: How common is over-fixing, and can an LLM judge measure its incidence? RQ3: How reliable are the two scalable correctness proxies the field relies on (textual similarity and an LLM judge) compared to execution? RQ4: How do accuracy and cost trade off across models? We then ask whether the answers carry over to real, repository-scale code (Section 4.5).

2. Related Work

LLMs for vulnerability repair. Pearce et al. [1] first studied zero-shot vulnerability repair with LLMs and warned that a passing regression test is a weak proxy for a correct fix. Fine-tuning approaches (VulRepair [2], VulMaster [7]) report exact-match or CodeBLEU against the developer fix; Wu et al. [3] benchmark models on reproducible Java vulnerabilities and find that even the best model repairs only about 20% of them. Agentic pipelines [8] reach a similar ceiling.

Over-fixing is APR overfitting, specialised to security. The phenomenon we study is not new to program repair. Smith et al. [4] showed that using the same tests to both guide and evaluate repair fails to distinguish correct patches from ones that “overfit the available tests and break untested but desired functionality”. Qi et al. [5] then distinguished plausible patches, which pass the suite, from correct ones, and found that most generate-and-validate patches are plausible-but-incorrect, often equivalent to deleting functionality. The two forms differ mechanically: classical APR overfitting arises when a search games a weak test suite, whereas our patches are single-shot generations with no test in the loop, so the security form is better described as over-restriction (a guard that rejects legitimate inputs) than as suite-gaming. Our contribution is not to discover over-fixing but to (a) quantify its security form per weakness class on real CVEs and (b) show that the LLM judges now used to measure it cannot see it reliably.

Benchmarks and joint verification. Commit-derived corpora (CVEfixes [9], BigVul [10]) are noisy, since 40–75% of a fixing commit is unrelated refactoring [11], which motivated cleaned derivatives (CleanVul [12], used here) and re-labelled sets (PrimeVul [13]). Execution-based benchmarks are the current best practice, from repository-scale issue resolution (SWE-bench [14]) to security-specific suites. PatchEval [15] runs both security PoC tests and functionality tests over 1000 CVEs and explicitly observes patches that pass security tests but fail functionality tests (that is, over-fixing), arguing against similarity-only scoring. SecRepoBench [16] sweeps 19 models with execution and memorisation control, and CWEval [17] jointly scores functionality and security for code generation. Secure-code-generation benchmarks (CyberSecEval [18], SecurityEval [19]) measure a model’s propensity to emit insecure code with static detectors. We build on this execution-based tradition. Our distinct focus is not another fix-rate leaderboard but the reliability of the correctness signal itself. Concretely, we ask whether an LLM judge, the scalable alternative to execution, can measure functional preservation, and we add a per-CWE over-fixing cut and a cost-per-correct analysis.

Measuring patch correctness. The plausible-vs.-correct distinction and its overfitting threat are surveyed in [20], extending a line of automated patch-correctness assessment [21,22] grounded in execution-based APR benchmarks such as Defects4J [23]. Similarity metrics (BLEU, CodeBLEU [24], CrystalBLEU [25]) are cheap but track functional correctness weakly. LLM-as-judge for patch correctness is scalable but prompt-sensitive and biased [26]. Code-specific judge methods such as ICE-Score [27] and CodeJudge [28] instruct an LLM to score code directly, and LLM judges more broadly exhibit position, verbosity, and self-enhancement biases [29,30], including a documented preference for their own outputs [31]. Our results add direct evidence of a specific, consequential failure: LLM judges inflate the over-fix rate, and inter-judge agreement does not certify validity. The nearest multi-model, per-CWE studies (LLM4CVE [32], VulnLLMEval [33]) score with similarity and do not isolate over-fixing or calibrate their signal against execution.

3. Materials and Methods

Table 1. The four corpora, their scoring oracles, and the role each plays. Tier-1 is judged at scale; the Tier-2 benchmark and the real-Java Vul4J corpus are execution-scored and calibrate the judge.

Corpus	n	Oracle	Language(s) and Role
Tier-1 (CleanVul)	922	LLM judge (Haiku, GPT-5.2)	JavaScript; RQ1 correctness, RQ4 cost, commercial-over-open gap (judged)
Tier-2 benchmark	144	Execution + 2 judges	JS and Python(v3.11); RQ2/RQ3 judge calibration (over-fix precision, $\kappa$)
Tier-2 benchmark (+Go)	128	Execution + 2 judges	JS, Python(v3.11), Go; language robustness of the over-reporting
Vul4J (real Java)	254	Maven execution + 2 judges	Java; repository-scale replication, strong-oracle over-fix rate

summarises the study at a glance. One large corpus is scored by an LLM judge at scale; a purpose-built execution benchmark and a real-Java corpus are scored by execution and are what calibrate and stress-test that judge.

3.1. Corpus

We draw vulnerable–fixed function pairs from CleanVul [12], an LLM-cleaned, function-level derivative of open-source security-fix commits, using its highest-cleanliness split (label correctness ≈ 97%). Restricting to entries with both a CVE and a CWE label, within a 3–400-line size window, and excluding test files, yields a usable set that is effectively all JavaScript, drawn from the npm advisory ecosystem. Our run covers 116 vulnerable functions (66 CVEs). We treat two facts as explicit threats (Section 6): the corpus is web-centric and single-language, and the ground-truth chain is LLM-cleaned corpus → LLM judge (Section 3.4), which the execution oracle is designed to break.

3.2. Models and Generation

We evaluate eight models spanning the commercial/open and large/small axes (

Table 2. Per-model correctness (Wilson 95% CI), cost, and latency. Correct rate is judge-based (a calibration-dependent estimate; Section 4.3); n is the completed per-model attempt count. Dollar figures are recomputed from the released token counts and July-2026 provider list prices (the price constants ship with the harness); the databases’ stored per-call cost is a superseded run-time value. The token counts are the primary artefact and let the costs be recomputed at current prices; their per-model distribution is reported in Section 4.4. For 28 of 115 Qwen attempts the provider returned no usage block, so that model’s cost is averaged over the 87 attempts that did report usage (mean imputation); the affected attempts returned patches and are scored normally, so no correctness figure depends on this. No other model is affected.

Model	Kind	Correct [95% CI]	$/Call	$/Correct	Lat. (s)
Claude Opus 4.8	comm.	0.235 [0.17,0.32]	0.0235	0.100	9.0
GPT-5.2	comm.	0.198 [0.14,0.28]	0.0036	0.018	5.6
GPT-5-mini	comm.	0.165 [0.11,0.24]	0.0029	0.018	16.3
DeepSeek-V3.2	open	0.148 [0.09,0.22]	0.0003	0.0019	15.7
Claude Haiku 4.5	comm.	0.140 [0.09,0.22]	0.0035	0.025	3.9
Llama-3.3-70B	open	0.090 [0.05,0.16]	0.0002	0.0021	17.1
Codestral-2508	open	0.088 [0.05,0.15]	0.0005	0.0062	4.2
Qwen2.5-Coder-32B	open	0.043 [0.02,0.10]	0.0005	0.0124	8.9

): four commercial (Claude Opus 4.8, Claude Haiku 4.5, GPT-5.2, GPT-5-mini) and four open-weight via OpenRouter (DeepSeek-V3.2, Qwen2.5-Coder-32B, Llama-3.3-70B, Codestral-2508). We accessed each model through its provider API, and we give the exact identifiers so that the runs can be reproduced: claude-opus-4-8 and claude-haiku-4-5-20251001 (Anthropic); gpt-5.2 and gpt-5-mini (OpenAI); and deepseek/deepseek-v3.2, qwen/qwen-2.5-coder-32b-instruct, meta-llama/llama-3.3-70b-instruct, and mistralai/codestral-2508 (OpenRouter). All eight identifiers were confirmed live on 12 August 2026. Each model receives an identical prompt (the vulnerable function, the CWE, and an instruction to return a minimal, behaviour-preserving fix) at temperature 0 where permitted. Three commercial reasoning models (Opus 4.8, GPT-5.2, and GPT-5-mini) reject a custom temperature and run at their non-zero default, so sampling temperature covaries with the commercial/open split; we therefore treat it as a confound of the class comparison (Section 6) rather than a mild cross-model quirk. We record token usage, latency, and cost per call. The evaluation run was terminated for compute reasons after 922 of a planned 1280 attempts (a target of 160 functions across the 8 models); the 116 functions with completed coverage give the per-model denominators of 111–116 reported explicitly. We verified that the completed subset preserves the intended CWE distribution. Because the denominators are not identical across models, we also repeat the Tier-1 comparison on the 108 functions scored for every one of the eight models (864 attempts). The ranking is unchanged, at Spearman $\rho =0.99$ against the full-set ordering. The top five positions and the bottom one are identical, and the single swap is between two models that the common subset leaves exactly tied at 10 of 108. The commercial-over-open difference is 0.093 on the full set and on the common subset alike (Section 4.1). Unequal coverage therefore does not drive any Tier-1 result.

3.3. Two Axes and the Outcome Taxonomy

Each patch is scored on neutralised (vulnerability removed) and functional (behaviour and interface preserved), giving four outcomes: CORRECT (neutralised ∧ functional), OVER-FIX (neutralised ∧¬functional), INEFFECTIVE (¬neutralised), and ERROR (a missing or malformed patch, or a judge verdict that could not be parsed). ERROR is rare: the 7 excluded Tier-1 attempts are judge-parse failures on otherwise substantive patches, and the 1 excluded Java attempt returned no patch, leaving the 254 scored; both are folded into the unparsed or excluded counts rather than shown as their own bucket. A fix rate that reports only “neutralised” hides the OVER-FIX class. These four outcomes are the title of this paper in miniature: a CORRECT patch fixes the vulnerability, an OVER-FIX breaks working code while removing the flaw, and a patch that a judge certifies but execution rejects is a faked fix.

3.4. Scoring and the Execution Calibration

Because CleanVul provides no tests, we score at scale with a reference-based LLM judge (Claude Haiku 4.5) that compares each candidate against the developer fix on both axes and returns a structured verdict. An unvalidated judge is exactly the shortcut the field’s audit warns against [6], so we calibrate it against execution. We built a benchmark of 18 runnable vulnerabilities across two languages, 10 in JavaScript and 8 in Python (v3.11), spanning 11 weakness classes: XSS, prototype pollution, path traversal, code injection, SSRF, ReDoS, CSRF, CRLF/header injection, SQL injection, open redirect, and broken authentication. Each case carries a security test, a proof-of-concept that fails if and only if the vulnerability is present, and a functional test, a small check of one to three legitimate-input behaviours. We return to the strength of this oracle in Section 6. Running all eight models yields 144 patches whose outcome is decided by execution (under Node.js and CPython), the reference criterion against which we measure judge agreement (Cohen’s $\kappa$) and per-outcome precision. We call this criterion the execution oracle rather than ground truth throughout: it is a strong but imperfect criterion whose own limits we quantify in Section 6. The benchmark additionally includes Go cases (executed in a container); a four-model run over all three languages (Section 4.3) checks robustness across languages. To separate a general LLM-judge property from a single-model quirk, and to address self-bias (the primary judge is itself one of the evaluated models), we score every benchmark patch with a second, cross-family judge (GPT-5.2, outside the Claude family) and report both judges’ agreement with execution and with each other. GPT-5.2 is itself one of the evaluated models, so this design bounds cross-family self-preference rather than eliminating each judge’s preference for its own outputs [31]; no judge sits fully outside the evaluated set. The judge prompt is blind to authorship: it presents the vulnerable function, the CWE, the developer fix, and the candidate patch, and never discloses which model produced the candidate. Any self-preference must therefore act through stylistic recognition of a model’s own output rather than through an explicit identity cue, which is the mechanism Panickssery et al. report. We did not add an instruction to suppress self-recognition, and we report the residual signal we can measure (Section 4.3). Similarity metrics (exact match, token Jaccard, BLEU-4) are computed but are secondary and never determine correctness.

3.5. Statistics and Implementation

We refer throughout to two evaluation tiers: Tier-1, the 922-attempt CleanVul corpus scored by the LLM judge, and Tier-2, the 144-patch benchmark scored by execution. Rates are reported with Wilson 95% confidence intervals, and the commercial-over-open difference uses a bootstrap clustered over CVEs. The Tier-2 patches are not independent either: the 144 come from 18 underlying vulnerabilities, each patched by all eight models, so patches sharing a vulnerability share its difficulty and its oracle. Every $\kappa$ confidence interval we report is therefore a vulnerability-clustered percentile bootstrap (8000 replicates resampling the 18 vulnerabilities with replacement and taking all patches of a drawn vulnerability), which is wider than, and more honest than, treating the 144 patches as independent. Because four models per class make the model the natural unit, we also test that difference at the model level with a Welch t and a rank-based Mann–Whitney U over the four-per-class means, for which the largest possible U is 16. Because attempts on the same function are correlated and per-CWE Tier-1 cells rest on few distinct CVEs (CWE-79 on 25, but CWE-918 and CWE-352 on only 2 each), we ground per-CWE claims in the Tier-2 execution benchmark and report distinct-CVE counts where we emphasise a per-CWE cell. The harness is written in Python and uses a unified client across the three providers, with concurrent execution, per-call cost metering, and a SQLite store; the execution oracle materialises each case in an isolated directory and runs the tests under the runtime the case requires, namely Node.js, CPython, or a Go container. All code, the judge prompt, the executable benchmark, and raw per-attempt records are released, so every number in Section 4 is reproducible.

4. Results

We address four questions: correctness (RQ1), over-fixing and how reliably it can be measured (RQ2), evaluation-signal reliability (RQ3), and cost (RQ4), and we then extend the execution grounding to real repository-scale Java code (Section 4.5). Contamination, that is, whether models recall public fixes rather than reason about the code, cannot be assessed on this pre-cutoff corpus and is treated as a limitation. Tier-1 covers 922 attempts (915 judged; 7 unparsed, 0.8%), and Tier-2 is the 144-patch execution benchmark.

4.1. RQ1: Correctness Is Low, and Rankings Overlap

Figure 1 and Table 2 give per-model correctness. By the LLM judge, the best model (Claude Opus) correctly fixes 23.5% (95% CI [17,32]); the worst (Qwen) 4.3% [2,10]. These are judge-based estimates, not executed rates (we calibrate the judge in Section 4.3), and because the corpus predates the models’ training cutoffs, some of the signal may be recall of public fixes rather than repair (Section 6). Adjacent models overlap: Opus [0.17,0.32] and GPT-5.2 [0.14,0.28] are not distinguishable, so we report the ordering as indicative, not strict. Restricting to the 108 functions attempted by all eight models leaves this picture intact: the ranking is preserved (Spearman $\rho =0.99$), Opus stays first at 23.2% and Qwen last at 4.6%, so the unequal per-model denominators have no material effect here. The robust comparison is by class: the four commercial models reach a correct rate of 0.185 against 0.092 for the four open ones, a difference of 0.092 with a CVE-clustered bootstrap 95% CI of [0.04,0.15] that excludes zero (0.093, [0.04,0.15] on the common subset). We stress that these eight models are a purposive, not a random, sample of commercial and open systems, and that the two groups also differ in scale, provider, sampling temperature, and likely memorisation (Section 6). The contrast is therefore a statement about these eight models, not an effect of openness as such. We describe it throughout as a commercial-over-open gap among the models we selected. It should not be read as evidence that open-weight models are inherently weaker at vulnerability repair. A model-level test over the four-per-class means points the same way, though the rank-based Mann–Whitney ($U=15/16$) is significant one-sided only (two-sided $p\approx 0.057$), so the Tier-1 gap rests mainly on the Welch $t=3.12$ and the clustered bootstrap. These judged rates are estimates whose bias direction depends on difficulty. On the easy calibration benchmark the judge is high-precision on CORRECT, so rates there would be lower bounds. On hard real code it over-credits, and does so more for commercial patches than for open ones, which inflates the judged size of this gap (Section 4.5). We therefore claim no strict bound on CleanVul and read the gap’s magnitude cautiously, even though its direction survives on execution. Either way, the absolute rates are low, and even doubling them leaves a majority of vulnerabilities unfixed by every model, consistent with prior Java results [3].

4.2. RQ2: Over-Fixing Is Uncommon, and the Judge Over-Reports It

At Tier-1 scale, on the harder CleanVul corpus, the LLM judge reports that 53–80% of vulnerability-removing patches also break functionality. Taken at face value this would be the paper’s headline number; the calibration below shows it cannot be read as a measured over-fix rate. CleanVul provides no tests, so we cannot execute it directly. Instead, we audit the judge on runnable code and read the Tier-1 figure in that light (Figure 2). Because the judge’s bias is difficulty-dependent (Section 4.5), this audit does not carry over as a numeric correction to the harder corpus, so we report no corrected Tier-1 over-fix rate, only the demonstrated unreliability of the judge that produced the 53–80%. On the 144 executed patches (18 vulnerabilities across JavaScript and Python × 8 models), execution finds 104 CORRECT, 32 INEFFECTIVE, and only 8 OVER-FIX: over-fixing is 8/112 = 7.1% of vulnerability-removing patches (Wilson 95% CI [3.7%,13.5%]), and this is a lower bound under a single-test oracle (Section 6). On these identical 144 patches, the primary judge (Haiku) labels 20 patches OVER-FIX, and the second, cross-family judge (GPT-5.2), labels 30: the judges flag over-fixing 2.5–3.75 times as often as execution confirms it. Nor is this one judge’s blind spot: of the 20 Haiku labels, execution confirms one (precision 5%, catching just one of the eight real over-fixes, recall 12%), and of the 30 GPT-5.2 labels, three (precision 10%, recall 38%). Each scorer applies its own notion of which patches remove the vulnerability, so the three over-fix rates rest on different denominators (execution 112, Haiku 77, GPT-5.2 81) and are not directly comparable. Two denominator-free readings make the comparison exact. First, on the common set of all 144 patches the raw label counts are 8, 20, and 30. Second, restricted to the 112 patches that execution finds vulnerability-removing, execution scores 8 over-fixes (7.1%) where Haiku scores 17 (15.2%) and GPT-5.2 27 (24.1%). Figure 2 shows both views, and

Table 3. Judge label versus execution outcome on all 144 benchmark patches, for both judges. Rows are the execution oracle, columns the judge. The diagonal is agreement. Neither judge ever calls an execution-OVER-FIX patch CORRECT, and every judge-CORRECT label is execution-CORRECT (57 of 57 and 51 of 51), so a judge’s CORRECT verdict is trustworthy on this corpus even though its OVER-FIX verdict is not: of 20 and 30 OVER-FIX labels, execution confirms 1 and 3. The excess labels sit overwhelmingly on patches execution finds CORRECT (16 and 24), a different failure mode from the one seen on Java (Section 4.5).

Judge	Execution	CORRECT	OVER-FIX	INEFFECTIVE	Total
Haiku	CORRECT	57	16	31	104
	OVER-FIX	0	1	7	8
	INEFFECTIVE	0	3	29	32
	Total	57	20	67	144
GPT-5.2	CORRECT	51	24	29	104
	OVER-FIX	0	3	5	8
	INEFFECTIVE	0	3	29	32
	Total	51	30	63	144

gives the full cross-tabulation from which every one of these numbers can be recovered. On this benchmark, though, the label counts alone do not certify that the judge is wrong. Reading the rationales, many of the “false” over-fix labels flag genuine interface or error-contract differences that the single happy-path functional test never exercises, such as throwing rather than returning, a changed error type, or a path-traversal fix that switches to basename and drops sub-directory structure. Others are plainly spurious, as when a judge marks an XSS patch OVER-FIX for rendering the apostrophe as ' rather than ', the same HTML entity. The weak oracle therefore cannot sign the disagreement here, and the 5% precision is a lower bound on the judge’s true precision (Section 6). This interaction deserves to be stated precisely, because oracle weakness and judge error push the same statistic in opposite directions. Precision is measured against the set of over-fixes execution detects. A functional test of one to three legitimate-input behaviours has limited recall for behavioural change, so some genuine regressions go unobserved, the true over-fix set is larger than eight, and the denominator of “real” over-fixes is under-counted. Every judge precision we report is therefore a floor rather than a point estimate, and the measured 5–10% would rise under a stronger oracle. By the same token the execution over-fix rate is a floor. What the benchmark alone cannot establish is how much of the gap is judge error and how much is oracle blindness. That is exactly why we do not rest the argument here: on Java, where a full Maven regression suite replaces a handful of assertions, the excess labels fall on patches that do not even compile, which no oracle-recall argument can excuse (Section 4.5). The over-labelling is shared across two cross-family judges rather than being a single-model quirk, and it persists under a structurally different judge prompt (Section 6). The 53–80% Tier-1 figure is thus a judge estimate on harder, unexecuted code that the same judges’ demonstrated over-labelling tells us to distrust, not a measured rate.

Over-fixing is nonetheless a genuine failure mode, and where execution finds it on the benchmark it is concentrated in one place, though that concentration must be read cautiously (Figure 3,

Table 4. Execution-measured outcomes by CWE on the 144-patch Tier-2 benchmark (JS + Python; n = patches). Over-fixing concentrates in CSRF; six of the eight over-fixes are CSRF, most likely an oracle artefact (Section 6). The CRLF/header-injection case is labelled CWE-93 here, with CWE-113 (HTTP response splitting) its most specific applicable class; the released corpus records it under the parent class CWE-74. A few labels are likewise parent classes (for example, CWE-94 for the eval-based code injection, whose specific child is CWE-95).

CWE	n	Neutralised	Correct	Over-Fix
CWE-352 (CSRF)	8	0.75	0.00	0.75
CWE-918 (SSRF)	16	0.56	0.50	0.06
CWE-94 (Code Injection)	16	0.88	0.81	0.06
CWE-79 (XSS)	24	0.92	0.92	0.00
CWE-1321 (Prototype Poll.)	16	0.44	0.44	0.00
CWE-22 (Path Traversal)	16	0.94	0.94	0.00
CWE-89 (SQL Injection)	8	0.88	0.88	0.00
CWE-601 (Open Redirect)	8	0.50	0.50	0.00
CWE-93 (CRLF Injection)	8	0.88	0.88	0.00
CWE-287 (Broken Auth)	8	1.00	1.00	0.00
CWE-1333 (ReDoS)	16	0.81	0.81	0.00

). Six of the eight benchmark over-fixes fall in a single cross-site-request-forgery scenario (CWE-352), so the benchmark’s 7.1% over-fix rate rests heavily on one fragile cell. That concentration is most likely an artefact of the constructed oracle rather than a property of the patches, and the cell is unreliable on both axes. Its functional test exercises one hard-coded token-passing convention (it invokes the handler with a matching csrfToken/session.token pair), so a correct guard that reads the token from a different but equally valid interface throws on that single call and is scored OVER-FIX. Several of the six patches also add a POST-method contract rather than a real token check, so their neutralised verdict is itself an artefact of the single proof-of-concept. The corrected baseline matters enough to state on its own: excising the whole CSRF cell leaves only two non-CSRF over-fixes among the benchmark’s vulnerability-removing patches, an over-fix rate below 2%. The headline 7.1% and this corrected figure should be quoted together, because the first is inflated by a single fragile scenario and the second is the rate the benchmark supports once that scenario is set aside. One of those two remaining cases, an SSRF fix, is itself an oracle artefact: the model wrote a stronger guard (DNS resolution plus private-IP rejection) that fails only because the test’s legitimate host does not resolve in the offline sandbox. The genuine benchmark over-fix count is therefore one or two, and every point estimate of the over-fix rate here is a lower bound. Both readings agree with the strong-oracle evidence: the full-regression Java corpus finds over-fixing in 1 of 254 patches overall, which is 1 of the 24 that neutralise the vulnerability (4.2% on the same among-neutralising denominator; Section 4.5). The finding is therefore not “most LLM fixes break the code”, nor is it a robust per-class law. It is something more precise: over-fixing is an uncommon failure mode that similarity and LLM judges systematically over-report, and that only execution measures. Server-side request forgery (CWE-918) makes the same point from the other direction. The Tier-1 judge scored it 0% correct and 71% neutralised, yet execution finds it 50% correct with near-zero over-fix, a further sign that dramatic Tier-1 judge numbers need not survive execution.

4.3. RQ3: Neither Similarity nor an LLM Judge Measures Functional Preservation

Table 5. Two cross-family LLM judges vs. the execution oracle on 144 runnable patches (Cohen’s $\kappa$, with vulnerability-clustered 95% confidence intervals: 8000 bootstrap replicates resampling the 18 underlying vulnerabilities with replacement, so patches of the same vulnerability move together). Both judges are weakest on the functional axis; both inflate the over-fix rate (Haiku precision 1/20, GPT-5.2 3/30). The judges agree with each other far more than with execution, and on the functional axis the judge–judge interval does not overlap either judge–execution interval.

Axis	Haiku vs. Exec	GPT-5.2 vs. Exec	Haiku vs. GPT-5.2
Correct	0.40 [0.24,0.60]	0.35 [0.19,0.55]	0.91 [0.80,0.99]
Neutralised	0.41 [0.24,0.58]	0.45 [0.27,0.63]	0.83 [0.72,0.93]
Functional (over-fix axis)	0.26 [0.14,0.42]	0.19 [0.08,0.35]	0.75 [0.57,0.89]

reports both judges against execution ($n=144$). Agreement is moderate on correctness (Haiku $\kappa =0.40$, GPT-5.2 $\kappa =0.35$) and neutralisation (Haiku $\kappa =0.41$, GPT-5.2 $\kappa =0.45$) but weakest on the functional axis (Haiku $\kappa =0.26$, vulnerability-clustered 95% CI [0.14,0.42]; GPT-5.2 $\kappa =0.19$ [0.08,0.35]), the axis that separates a fix from an over-fix. All $\kappa$ intervals here are clustered on the 18 underlying vulnerabilities rather than on the 144 patches (Section 3.4); clustering widens them by roughly half, and we report the wider intervals throughout. This weakness is not an artefact of the skewed over-fix base rate: the prevalence- and bias-adjusted kappa is also low (PABAK $=0.28$ and $0.08$; Byrt et al. [34]), and the prevalence-robust precision figures below tell the same story. We therefore treat $\kappa$ as one of several views of agreement and lead with precision, which does not depend on the class balance. Because the execution criterion is itself imperfect (Section 6), these judge–execution $\kappa$ values are attenuated: part of any judge–execution disagreement is oracle noise, so the true judge validity could be somewhat higher, and we read execution as a strong but imperfect convergent criterion rather than an infallible one. The inter-judge agreement below, by contrast, involves no oracle and is not attenuated, so the reliability-versus-validity gap (judge–judge far above judge–execution) cannot itself be an artefact of oracle noise. On the benchmark, a judge is high-precision in one direction (of the patches it calls CORRECT, execution overwhelmingly agrees), but this property is difficulty-dependent: on the real Java corpus one judge keeps it by under-crediting while the other loses it by over-crediting (Section 4.5). Even where a judge cannot measure a rate, it still tracks the model ranking: judge and execution agree on the per-model correctness order at Spearman $\rho =0.95$ on the benchmark (tie-corrected, over the $n=8$ models), falling to $0.79$ on Java. Still, neither judge can reliably tell whether a single neutralising patch preserved behaviour.

A key methodological result concerns judge–judge agreement. The two judges agree strongly with each other ($\kappa =0.91$ on correctness and $\kappa =0.75$ on functional preservation) while both agree far less with execution on the functional axis ($\kappa =0.26$ and $0.19$). The gap is not an artefact of treating correlated patches as independent: under the vulnerability-clustered bootstrap the judge–judge functional interval is [0.57,0.89], which lies entirely above the judge–execution intervals [0.14,0.42] and [0.08,0.35]. The intervals do not overlap, so the separation survives the more conservative estimator. Inter-rater agreement between LLM judges, a common reliability proxy, would thus give false confidence: two judges concur precisely where both are wrong. This is the textbook distinction between reliability (agreement among raters) and validity (agreement with a criterion): a judge validated only by inter-judge consensus can be reliable yet invalid, and here the criterion is execution [35,36]. This diagnosis is specific to the easy benchmark, where the judges are reliable but invalid. On the hard Java corpus they lose even mutual reliability, with inter-judge agreement falling to $\kappa =0.25$ (Section 4.5), so inter-judge consensus is false comfort on easy code and simply absent on hard code. To test self-bias directly at scale, we re-scored all 922 Tier-1 patches with the cross-family, non-Claude judge (GPT-5.2). The two judges produce an identical model ranking (Spearman $\rho =1.0$); the second judge returns uniformly lower correct rates and is, if anything, harsher on Claude Haiku (rating it 0.06 versus the Haiku judge’s 0.14), which rules out the possibility that the primary judge inflates the scores of the Claude family. The signal on self-preference is more equivocal: Claude Haiku, the primary judge’s own model, shows the largest cross-judge drop of any model (a retention of 0.43 against Opus’s 0.88), consistent with a model-level own-output preference even though Opus’s stability rules out a family-level one. The commercial-over-open gap replicates under both judges (GPT-5.2: 0.136 vs. 0.052).

The pattern reproduces on a third language and a fresh model subset. A run adding Go (128 patches across JavaScript, Python, and Go, four models drawn from both classes) gives an execution over-fix rate of 8.9%, with both judges again over-estimating it (Haiku 27 labels, 3 real; GPT-5.2 38 labels, 3 real) and inter-judge agreement ($\kappa =0.92$ on correctness) again far exceeding judge–execution agreement (functional $\kappa =0.25$ and $0.18$). The over-reporting therefore persists across three language implementations and two judge families. Because the Python and Go suites re-implement the shared scenario set (Section 6), this demonstrates robustness to how a scenario is implemented rather than independent sampling, and we rest the real-corpus external validity on Vul4J.

Textual similarity is even worse. Exact match to the developer fix occurs 0–5% of the time and is essentially unrelated to correctness: $\kappa =0.15$. BLEU-4 clusters between 0.57 and 0.73 for every model, best and worst alike, so ranking models by similarity is misleading. The combined message of RQ3 is decisive: the two scalable proxies the field relies on, similarity and LLM judges, both fail on functional preservation, so security-patch correctness must be established by execution.

4.4. RQ4: Cost-Efficiency Inverts the Accuracy Ranking

Table 2 and Figure 4 report cost. Per-call cost spans roughly 120×, from $0.0002 (Llama) to $0.0235 (Opus); the measured input and output token counts are released so that the figures can be recomputed at current prices. Because providers price input and output tokens differently, often by an order of magnitude, and revise those prices frequently,

Table 6. Per-model input and output token counts per Tier-1 attempt (mean and median). Input and output tokens are priced separately and unequally, so releasing them separately lets any reader recompute Table 2 under a different price schedule. n counts the attempts for which the provider returned a usage block; it is the full per-model attempt count except for Qwen, where 28 of 115 calls returned none (Section 6). GPT-5-mini is the outlier on output tokens, which is a reasoning-token effect rather than a longer patch.

Model	n	In (Mean)	In (Median)	Out (Mean)	Out (Median)
Claude Opus 4.8	116	806	408	779	392
GPT-5.2	116	494	256	453	211
GPT-5-mini	115	497	258	1403	1189
DeepSeek-V3.2	115	534	269	471	204
Claude Haiku 4.5	116	618	306	577	246
Llama-3.3-70B	114	518	267	430	174
Codestral-2508	115	509	259	434	182
Qwen2.5-Coder-32B	87	385	235	290	146

reports the two distributions separately so that every dollar figure here can be recomputed under any future price schedule. The distributions also explain an apparent anomaly in Table 2. GPT-5-mini costs almost as much per call as GPT-5.2 despite a much lower list price, because it emits roughly three times as many output tokens (mean 1403 versus 453). This is a reasoning-token effect that a price comparison alone would hide. Normalising by correctness inverts the ranking: DeepSeek-V3.2 achieves the lowest cost per correct patch, while Claude Opus, the most accurate model, is the most expensive per correct patch, at roughly 52× the cost of DeepSeek. Because the correct-count denominator is a judge estimate of uncertain bias, cost per correct patch should be read as indicative. Recomputing it under the cross-family GPT-5.2 judge leaves the ordering intact, with DeepSeek still the cost-per-correct leader and Opus the costliest (a ratio of roughly 35× rather than 52×), so the inversion is not an artefact of one judge. Because the extreme comparison is cross-provider, we treat the ratio as directional and treat the provider-independent token counts as primary. Latency varies by a factor of four (from 3.9 s for Haiku to 17 s for Llama), independently of cost.

4.5. Real Java CVEs: Robust Over-Reporting, but Judges Diverge on Repository-Scale Code

To test whether our findings extend beyond constructed benchmarks to real repository-scale code, we scaled the evaluation to 254 patches across 33 real Java CVEs from Vul4J [37]. Each patch is scored by full-project Maven execution: the project’s proof-of-vulnerability test (vulnerability neutralised) plus its regression suite (functionality preserved), with the JDK that each CVE requires selected automatically. We restrict the corpus to CVEs whose developer fix rewrote a single production file, which is the task we set the models. We exclude one attempt that returned no parseable patch, and we flag nine attempts whose generation hit the output-token budget or whose build exceeded the test timeout (all execution-INEFFECTIVE by construction, and removing them leaves the correct count unchanged, at $23/245=9.4\%$).

Three of our results carry over, and the picture on the judge sharpens. First, correctness is low but not zero: $23/254=9.1\%$ (95% CI [6.1,13.2]) of patches are execution-CORRECT, well below the isolated-function rates, since real classes with cross-file behaviour are markedly harder. Second, the commercial-over-open gap replicates: commercial models fix $20/130=15.4\%$ versus open models’ $3/124=2.4\%$ (difference $0.13$, CVE-clustered bootstrap 95% CI [0.05,0.22]). A model-level contrast over the four-per-class means also separates the classes (Welch $t=5.6$, Mann–Whitney $U=16/16$). These Java class-comparison statistics are exploratory and uncorrected. The sweep was stopped once the commercial-over-open contrast reached significance, which is optional stopping, so the nominal Type-I error of this interval and of the model-level test alike is inflated and neither should be read as a confirmatory test (Section 6). We report them as a corroborating effect estimate on a pre-specified corpus of single-file Vul4J CVEs, and rest the confirmatory inference on the Tier-1 replication, whose stop was not outcome-contingent. The gap also stays entangled with model scale and with memorisation of pre-cutoff fixes (Section 6). Opus and GPT-5.2 lead at 18.2% each, while one open model fixes nothing. Third, fixability is sharply vulnerability-dependent: 25 of the 33 CVEs are fixed by no model, while a handful (including Apache Commons Compress path-traversal and infinite-loop issues) are fixed by most, so an aggregate “Java fix rate” hides a bimodal reality.

The evaluation-methodology findings are what this exercise was designed to test, and they split by axis. On the over-fix axis the benchmark result carries over cleanly and across judge families: full-regression execution finds over-fixing in a single patch of 254 (a genuine functional regression in Apache HttpClient), yet the primary judge labels 28 patches as functionality-breaking and the cross-family judge 43, of which execution confirms one and zero, respectively. The over-labelling is real in count, but its mechanism differs from the benchmark. It is worth separating the two failure modes explicitly, because they have different causes and different remedies. The first mode, functional-regression over-estimation, is the judge calling a patch behaviour-breaking when execution shows that the code works; it dominates on the constructed benchmark, where 16 of Haiku’s 20 and 24 of GPT-5.2’s 30 over-fix labels land on patches that execution finds CORRECT (Table 3). The second, neutralisation-blindness, is the judge issuing an over-fix label on a patch that never worked in the first place. This mode dominates on Java: 26 of the 28 Haiku labels and 40 of the 43 GPT-5.2 labels fall on patches that execution finds INEFFECTIVE, patches that do not compile or do not neutralise the flaw. On repository-scale code, the judge’s error is therefore less a matter of inventing a regression in working code than a failure to notice that a patch is dead on arrival, a neutralisation-blindness that a reference-based judge without the build or the test suite cannot escape. The distinction matters for the remedy: phantom regressions might in principle be reduced by a better prompt or a stronger judge, whereas neutralisation-blindness needs the build and the tests, which is precisely what a judge does not have. Neither judge’s count matches the executed one, so the robust reading holds on real code: a judge’s over-fix headcount is not an executed rate. Over-fixing itself is genuinely rare once a real regression suite, rather than one hand-written check, defines functional preservation: 1 of the 254 patches overall, and 1 of the 24 that actually neutralise the vulnerability (4.2%, the same among-neutralising denominator as the benchmark’s 7.1%). Each figure is a lower bound under the project’s own test coverage. On the correctness axis, by contrast, the two judges do not merely diverge from execution; they diverge from each other, in opposite directions. The primary judge (Haiku) over-credits: it labels 63 patches CORRECT, of which execution confirms 9 (precision $0.14$; Cohen’s $\kappa =0.09$, essentially chance), reporting many “fixes” that do not compile or do not fix anything. The cross-family judge (GPT-5.2) instead under-credits: it labels only 17 CORRECT, 12 of them real (precision $0.71$, $\kappa =0.57$), buying apparent accuracy by missing most genuine fixes. Inter-judge agreement, $\kappa =0.91$ on the easy benchmark, itself falls to $\kappa =0.25$ here: on hard real code the judges are no longer even reliable with each other, let alone valid, and swapping in a “better” judge only trades over-crediting for under-crediting. Both judges were reference-based (each saw the developer fix), so this is not an artefact of withholding the reference; what they lacked was the repository context and the test execution the oracle had, the same context limitation we raise for the generators (Section 6). One property does survive, if imperfectly: a judge still tracks the per-model correctness ranking (Spearman $\rho =0.79$ against execution here, versus $0.95$ on the benchmark). The primary judge nonetheless ranks its own family model first on Java, where execution places it fourth, so the rank is usable for coarse selection but not for crowning a winner. Only execution reliably tells a fix from a fake.

5. Discussion

The results tell an evaluation-methodology story with a practical edge. LLMs correctly fix only a minority of isolated-function JavaScript vulnerabilities, commercial models roughly double the rate of open ones, and the accuracy ranking inverts once cost is considered. The more important finding, however, concerns measurement. Both of the scalable proxies the field leans on fail on the axis that matters: textual similarity is uninformative about correctness ($\kappa =0.15$), and an LLM judge cannot reliably tell whether a neutralising patch preserved functionality (functional $\kappa \le 0.26$ for both judges; over-fix precision 5–10% and recall 12–38%). Had we trusted the judge, we would have reported that a majority of LLM security patches break the code; execution puts that figure far lower. On a common denominator, over-fixing among vulnerability-removing patches is 7.1% on the constructed benchmark, 8.9% across the three-language run, and 4.2% on the full-regression Java corpus (1 of 254 patches overall). Every oracle places it in single digits, and each figure is a lower bound under that oracle’s own test coverage, so over-fixing is a genuine but uncommon failure mode rather than the norm. Its rarity is a secondary result; the central finding is that neither similarity nor an LLM judge measures the functional-preservation rate reliably, and that only adequately tested execution does.

One qualification softens the prescription. As Section 4.3 shows, a judge that cannot measure a rate can still rank models: judge and execution agree on the per-model correctness order at $\rho =0.95$ on the benchmark, and the two judges agree with each other at $\rho =1.0$ on Tier-1. A practitioner choosing between models may therefore reasonably use a judge, whereas measuring an over-fix or correctness rate requires execution. The ranking property is itself difficulty-dependent. It weakens to $\rho =0.79$ on the hard Java corpus, where the judges also stop agreeing with each other, so it should be relied on for coarse model selection rather than fine distinctions.

This bears directly on deployed auto-patching assistants (e.g., pull-request bots that generate security fixes). Such systems should gate on a functional oracle, not on CWE-removal alone, and evaluations of them should not substitute an LLM judge for execution on the functional axis. Where projects lack adequate tests (the common case, and the reason our corpus had none), human review and property-based or differential testing are the practical fallbacks. Note that the calibration that makes the judge usable here does not transfer to an untested target codebase, so a team that cannot execute also cannot know its own local over-report rate. The failure modes split across stakeholders: a security team most fears an INEFFECTIVE-but-plausible patch (the vulnerability silently remains); a maintainer most fears an OVER-FIX (a silently broken feature in an auto-generated PR). Both evade the similarity and CWE-only checks the field commonly uses.

A reporting-integrity corollary follows from treating the judge as gameable. Because a judge can be satisfied without the code being fixed, and because LLM evaluators favour their own outputs [31], a judge-based fix rate is not merely imprecise but optimisable. A vendor can tune a patcher to please a judge, and a vendor that uses its own model as both patcher and judge inflates its numbers through self-preference. We therefore recommend that comparative fix-rate claims be execution-backed, that any unavoidable judge come from a provider family different from that of the patcher, and that self-judged fix rates be treated as unreliable by reviewers and procurement.

Our findings describe a single-shot, offline evaluation, and it is worth being explicit about how they change under multi-turn agentic or execution-guided repair. In a test-driven loop the model proposes a patch, the suite runs, and failures are fed back until the tests pass, as in agentic issue-resolution and CVE-repair settings [8,14]. Two consequences follow. First, such a loop presupposes exactly the executable oracle whose absence motivates an LLM judge in the first place, so it is not a substitute for our prescription but an instance of it: where tests exist, run them. Second, and more interesting, the loop changes what the judge is. An offline judge emits a terminal verdict that nothing downstream can contradict, which is the setting where our results say it should not be trusted to measure a rate. Inside a feedback loop, the same model acts as an interactive verifier whose proposals are continuously falsified by the suite, so its errors become recoverable rather than final: a false over-fix label costs an extra iteration instead of a wrong number in a results table. That reframing does not rehabilitate judge-reported rates, because the loop’s authority still comes from execution, but it does suggest the judge’s productive role is generative (proposing what to try next and ranking candidates) rather than evidentiary. It also raises a question our design cannot answer: whether a judge that has seen failing test output becomes well calibrated on the functional axis, which would require a corpus with both adequate tests and repository context. We flag it as the natural next experiment.

Two caveats bound even the surviving claims. First, our correctness figures are single-shot; an iterative, test-in-the-loop pipeline would plausibly fix more and over-fix less, so 23% should be read as a single-shot operating point rather than a deployable ceiling. Second, the models saw one function plus a CWE, whereas the developer had the whole repository; some of the low correctness therefore reflects a lack of context rather than a pure model limitation, which is itself an argument for repository-level evaluation.

6. Limitations and Threats to Validity

Language coverage. The large-scale Tier-1 corpus (CleanVul) covers only JavaScript and web code; the execution benchmark spans JavaScript, Python, and Go, and we additionally run 254 patches across 33 real Java CVEs (Vul4J) under full Maven execution (Section 4.5). That real-Java corpus is itself restricted to single-production-file fixes and, within Vul4J, to CVEs whose projects build cleanly in the pinned container. Several classes fall outside our coverage entirely: multi-file fixes, the remaining Vul4J CVEs, memory-safety classes in C/C++, and input-independent classes such as weak cryptography, insufficient entropy, and information exposure. The calibrated claims are therefore confined to the guard and input-validation weaknesses the benchmark exercises, and the over-fixing mechanism we identify, fixing by adding a guard or check, is plausibly specific to those classes. The benchmark cases are synthetic minimal reproductions. The released harness defines a superset of cases; the 18 behind the 144-patch calibration and the 32 behind the three-language run are the subsets recorded in the result databases. The Python and Go suites largely re-implement the same core scenario set as the JavaScript suite, so the cross-language runs test the judge’s robustness to a scenario’s implementation rather than sampling independent vulnerabilities. We accordingly reserve the external-validity weight for the real Vul4J corpus. We note that PatchEval’s public release provides function pairs but not its executable sandboxes, so large-scale real-CVE execution across additional languages remains dependent on such infrastructure.

Judge-based scale, execution-calibrated. Tier-1 correctness is judged, not executed; we calibrate two cross-family judges ($\kappa =0.40$/$0.35$ correct, $0.26$/$0.19$ functional) and we report correct rates as calibration-dependent estimates and never report a judge’s over-fix figure as established fact. The calibration benchmark is constructed ($n=144$, 18 vulnerabilities across two languages), and its cases are easier than the real corpus, so $\kappa$ is likely optimistic for real functions; a still-larger execution set drawn from the corpus would further tighten the estimate. The two central claims differ in kind. That both judges inflate the over-fix rate is the robust, cross-family claim (benchmark precision 5–10%; on Java 28 and 43 over-fix labels to execution’s one). That no single judge measures correctness reliably on hard code is supported differently: on the 254-patch Java corpus the primary judge over-credits ($\kappa =0.09$), while the cross-family judge under-credits ($\kappa =0.57$), and the two agree with each other only at $\kappa =0.25$, so the “high-precision-on-CORRECT” property is judge-specific, not a general guarantee.

The execution oracle is parameterised by test strength, on both axes. Functional preservation is decided by a single functional test on the constructed benchmark and by the full regression suite on Vul4J, and neutralisation is decided by a single proof-of-concept on the benchmark and by the project’s proof-of-vulnerability test on Vul4J. A single functional test can miss a genuine regression, so the benchmark’s 7% over-fix rate is a lower bound under a weak oracle, and CORRECT on the benchmark is correspondingly an optimistic estimate. The security axis is symmetric: a single proof-of-concept establishes only “not vulnerable to this payload”, so NEUTRALISED (and hence CORRECT) may be over-credited there too. A consequence we state plainly: where a weak-recall benchmark oracle disagrees with a judge, we cannot always sign the error, because the judge may catch a regression the one test does not exercise. The single-PoC neutralisation criterion is weakest for classes with many attack variants, such as server-side request forgery (bypass via DNS rebinding or IP encodings), prototype pollution (several pollution vectors), and ReDoS (a length cap that defeats one input). For these classes, the NEUTRALISED, and hence CORRECT, rates reported in Table 4 are correspondingly optimistic, so the single-PoC weakness affects the RQ1 correctness figures and not only the over-fix rate. The SSRF case in particular models the fetch target as an in-memory allow-list lookup rather than a live outbound request, so its verdict reflects the guard logic and not network-level defences. The CSRF over-fix cell is the clearest functional-side artefact: its functional test hard-codes one token-passing interface, so a correct guard that reads the token differently throws on that single call and is mislabelled as an over-fix, which is an interface artefact of the constructed oracle rather than a regression. We therefore treat “only execution measures functional preservation” as “only adequately tested execution does”. The generality of the judge-over-reporting finding rests on the convergence of the two-judge, three-language, and full-regression-suite Java evidence, where the two judges label 28 and 43 over-fixes to execution’s one, rather than on any single benchmark cell.

Prompt robustness. Beyond varying the model, we re-scored the 144-patch benchmark with a structurally different judge prompt that drops the token-level reference comparison and explicitly instructs the judge that a check rejecting only malicious or invalid inputs is functionality-preserving. The over-estimation persists: the variant still over-labels over-fixing (precision 3/20 = 0.15) with a comparably low functional-axis $\kappa =0.30$, so the effect is not an artefact of one prompt. (This prompt-variant re-scoring was run separately and its per-label judgements are not persisted in the released databases; we flag it as an auxiliary check rather than a result reproducible from the released data.) Varying the model likewise rules out an explanation based on a weak or small judge, since the frontier GPT-5.2 judge is, if anything, worse on the functional axis. A still-broader prompt sweep would strengthen this further.

The commercial-over-open gap is confounded, and stopping was not pre-registered. These eight models are a purposive sample chosen to span the commercial/open and large/small axes, not a random draw from any population of models, so the contrast is a comparison among the models we selected and supports no general claim about open-weight systems. The class contrast is entangled with model scale, provider family, differential memorisation of public fixes (the corpus predates model cutoffs), and sampling temperature (the three commercial reasoning models run at their non-zero default while the open models run at 0). The classes also overlap: individually, Haiku (commercial) sits below DeepSeek (open). With four models per class the effective unit is the model, not the patch. Because the model is the natural unit, we also ran a model-level test on the four-per-class means. The gap is clear on Java (Welch $t=5.6$, Mann–Whitney $U=16/16$) but weaker at Tier-1, where the Welch $t=3.12$ is significant, while the rank-based $U=15/16$ reaches only one-sided significance (two-sided $p\approx 0.057$); the Tier-1 evidence therefore rests mainly on the clustered bootstrap and the parametric test. A further confound is specific to the judged Tier-1 rates. The primary judge over-credits commercial patches more than open ones: on Java its judged correct rate is 0.40 for commercial versus 0.09 for open, a judged gap of 0.31 against execution’s 0.13. The magnitude of the Tier-1 commercial-over-open difference is therefore partly a judge plausibility bias, even though the direction is confirmed by execution on Java. With that replication the aggregate claim stands, but it should not be read as an effect of openness per se, and memorisation in particular means some of the gap may reflect recall of public fixes rather than repair. The Tier-1 corpus is also concentrated in one weakness class, with CWE-79 (XSS) making up 41% of attempts (25 CVEs), so the RQ1 rates are XSS-weighted. The commercial-over-open gap holds both within XSS and within the remaining classes. The Tier-1 run was terminated at 922 of 1280 planned attempts for reasons unrelated to the results (a compute limit), leaving balanced per-model denominators (111–116); that stop was not outcome-contingent. The Java sweep of 33 single-file Vul4J CVEs, by contrast, was stopped once the commercial-over-open contrast reached significance. That is optional stopping, and it inflates the Type-I error of every test computed on that Java snapshot, the bootstrap interval and the model-level contrast alike. We therefore rest the commercial-over-open inference on the Tier-1 replication, whose stop was not outcome-contingent, and read the Java gap as corroborating rather than independently confirming. Neither run followed a pre-registered rule.

Pseudoreplication. Per-CWE Tier-1 cells rest on few distinct CVEs (e.g., CWE-918 and CWE-352 on two CVEs each); we therefore ground per-CWE claims in the Tier-2 execution benchmark and report distinct-CVE counts. Absolute per-CWE Tier-1 rates should be read with this in mind.

LLM-to-LLM ground truth. The reference fixes come from an LLM-cleaned corpus and are compared by an LLM judge, which is exactly the chain our thesis warns against. The execution oracle breaks that chain, but it is small; wider human verification of reference fixes is future work.

Contamination, cost, single-shot. The corpus predates model cutoffs, so memorisation cannot be separated from reasoning [38]. Dollar costs depend on listed prices; we release token counts. Generation is single-shot: each model produces one patch per function, at temperature 0 where the provider permits it and at the provider’s non-zero default for the three commercial reasoning models that reject a temperature setting (Section 3.2). We considered multi-sample estimation, either pass@k or averaging over repeated draws, and did not adopt it, because the budget bought either repeated sampling on a small corpus or single-shot coverage of a large one, and coverage was worth more for the judge-calibration question this study asks. That choice has a consequence for the intervals we report, which we want to state plainly. The Wilson 95% intervals in Table 2 describe binomial sampling over functions. They answer one question: how much would this rate move on a different sample of vulnerabilities? They do not include generation stochasticity. For the three models running at a non-zero default, they therefore understate total uncertainty by an unmeasured amount, and a re-run over the same corpus would not land on exactly the same point estimate. We did not measure run-to-run stability of the outcome labels, so we cannot bound that component. It is an unquantified source of label noise, it affects the commercial models asymmetrically, and it is a further reason to read adjacent-model orderings as indicative rather than strict. It does not bear on the central judge-calibration result, which compares scorers on one fixed set of patches: judge and oracle see identical generations, so any generation noise is common to both and cannot manufacture the reliability-versus-validity gap. (Self-bias, a first-round concern because the primary judge is an evaluated model, is addressed by the cross-family non-Claude judge, which returns a slightly lower correct rate).

7. Conclusions

Large language models correctly fix only a minority of isolated-function JavaScript vulnerabilities, and the choice of model trades accuracy against cost. This low fix rate persists, with the same commercial-over-open gap, across 254 patches on 33 real Java CVEs scored by full Maven execution. The paper’s central lesson, however, is about evaluation: an LLM judge cannot reliably measure functional preservation. Both a Claude and a cross-family GPT judge report over-fixing far more often than execution confirms, at 28 and 43 Java labels to execution’s one. On Java most of those labels fall on patches that never compile, so the judge is failing to see dead-on-arrival patches as much as it is inventing regressions in working code. On hard real code their correctness estimates also diverge in opposite directions, one over-crediting and the other under-crediting, so their agreement with each other breaks down and strong agreement on easy code gives false confidence. Over-fixing itself, that is, removing a vulnerability by breaking functionality, proves uncommon rather than the norm once execution decides the question: single-digit percentages of vulnerability-removing patches across oracles, 1 of 254 at repository scale, and a lower bound throughout. Neither textual similarity nor an LLM judge can measure its rate reliably. Only execution, with adequate functional tests, measures it; a judge may still be used to rank models, not to quantify a rate. We therefore argue that security-patch evaluation should run the code to measure a rate, report uncertainty rather than point estimates, and weigh cost per correct patch. We release the evaluation harness, the judge prompt, the executable benchmark, and the resumable real-Java (Vul4J) execution runner so that such evaluations become routine.