VeriCrypt-Agent: Evidence-Grounded Multi-Agent Verification for Cryptographic Dependency Updates

Abstract

Cryptographic dependency updates can install cleanly and pass visible regression tests while changing project-facing behavior or leaving an application inside a vulnerable version range. This study presents VeriCrypt-Agent, an evidence-grounded workflow for bounded-autonomy cryptographic maintenance. The system combines a frozen snapshot of release/advisory evidence, a public catalog of project-level executable probes, cross-version execution under old and candidate dependency versions, role-specialized LLM reasoning, and a deterministic decision gate. Automatic merging is allowed only when mandatory probes pass, policy constraints are satisfied, and the audit record is complete. On CryptoUpdate-Mini-30, ten unique package-version transitions are evaluated through 30 controlled executions. Visible tests accept all five non-mergeable transitions, while advisory-only screening accepts two of five. The deterministic All-Probes Gate performs best on this benchmark, with perfect transition-level decisions and no review cases, when all relevant probes are already known and inexpensive. The full workflow observes no unsafe auto-merges (0/5; exact 95% CI: 0.000–0.522) and routes 9/30 executions to review (0.30; exact 95% CI: 0.147–0.494). Removing grounding accepts three of five non-mergeable transitions, and a fixed probe budget accepts two of five. On 48 blinded static crypto-API audit files, including 32 unsafe cases, two-pass review detects 20/32 unsafe cases (0.625; exact 95% CI: 0.437–0.789) and reaches macro-F1 $=0.556$. These point estimates characterize evidence-backed release decisions only under the evaluated conditions; they do not establish general superiority over exhaustive deterministic testing or a deployable static vulnerability detector.

1. Introduction

Large language models (LLMs) are used across software-engineering activities such as code generation, testing, review, migration, documentation, and release analysis [1,2,3]. These uses create a recurring control problem: an LLM may help propose or interpret a software change, but the operational decision to merge or deploy that change should depend on evidence that can be checked independently of the model’s explanation. Dependency maintenance makes this distinction concrete because a seemingly routine version change can alter both application behavior and security exposure.

Applications inherit compatibility constraints, vulnerabilities, and transitive behavior from third-party packages. Supply-chain research therefore emphasizes application-specific evidence, provenance, and update governance rather than treating a dependency version as an isolated metadata field [4,5,6]. For a dependency update, the practical question is not only whether the package installs, but whether the candidate version preserves the downstream contract and satisfies release policy for the particular application.

Cryptographic dependencies sharpen this problem because small API or default changes can affect security-sensitive behavior. A package upgrade can preserve a visible encode–decode path while changing a project-facing return type, validation default, key format, accepted algorithm, serialization representation, or malformed-input rejection behavior. Conversely, visible functionality can remain unchanged while the candidate version still lies inside a known affected range. Thus, visible regression success, version metadata, and natural-language model confidence provide different kinds of evidence and cannot safely substitute for one another.

Prior work provides several mature ingredients for this setting: dependency repair and migration systems use repository and environment context; testing systems use generated or differential execution; supply-chain methods use vulnerability and provenance information; and agentic workflows separate planning, generation, checking, or critique. Section 2 reviews these mechanisms in detail. The unresolved point addressed here is narrower: these ingredients are rarely combined as an explicit release-control contract in which a model may plan and interpret evidence but cannot authorize a cryptographic dependency update when the required executable or policy evidence is missing or blocking. The study therefore focuses on the boundary between model reasoning and release authority rather than on unconstrained update recommendation.

This study presents VeriCrypt-Agent, an evidence-grounded workflow for cryptographic dependency maintenance. The workflow uses three evidence sources: a frozen release/advisory context, meaning an immutable pre-inference snapshot of relevant release notes, affected-version ranges, and advisory text; a public catalog of project-specific executable probes; and old-versus-candidate execution results. Role-specialized LLM components plan and interpret this evidence, while a deterministic gate controls the final release action. HOLD denotes blocking evidence against automatic merge, whereas REVIEW denotes missing, inconsistent, or incomplete evidence that requires human inspection. The Relation Critic is included as a planning-check role, but the present benchmark does not independently demonstrate that it improves safety once mandatory fallback coverage is active.

Three research questions organize the evaluation. RQ1: How do visible tests, advisory-only screening, exhaustive executable checks, and bounded agentic workflows differ in unsafe automatic merges and decision quality? RQ2: How do grounding, probe coverage, critic review, deterministic gating, and evidence budget affect safety and review burden? RQ3: Does a second structured review pass change source-grounded detection in the auxiliary static cryptographic API audit? These questions separate evidence availability from agent orchestration.

Four contributions follow from this design. First, the paper formulates cryptographic dependency maintenance as an evidence-gated release decision that separates model reasoning from merge authority. Second, it specifies a bounded agentic architecture in which planning and interpretation are constrained by executable evidence, affected-version policy, and explicit escalation semantics. Third, it evaluates the architecture against deterministic, metadata-only, test-only, single-agent, planner–verifier, and ablated conditions using safety and operational metrics, including a deterministic all-probes control that exposes how much of the benchmark is solved by the supplied evidence itself. Fourth, it reports the limitations of this controlled protocol explicitly, including the small number of unique transitions, single LLM backbone, authored probe catalog, and absence of a historical ecosystem-scale comparison.

2. Related Work and Research Positioning

2.1. Agentic Software Engineering Across the Development Lifecycle

Agentic software engineering in recent work decomposes complex development work into specialized roles. AdaCoder uses adaptive planning and multiple agents for function-level code generation, illustrating how role decomposition can enhance task handling while remaining sensitive to the underlying foundation model [7]. AutoReview separates issue detection, localization, and repair in security-oriented code review and combines retrieval, graph-based slicing, and iterative verification [8]. Human-in-the-loop code-review frameworks extend this pattern with critic roles and explicit checkpoints before high-stakes findings are finalized [9]. Execution-free enterprise program repair demonstrates another form of agentic decomposition, although the absence of runtime execution distinguishes repair reasoning from evidence-backed release validation [10]. A setup agent addresses end-to-end Java project configuration with a dual-model design, illustrating how environment preparation itself can become an autonomous software-engineering task [11].

Requirement-guided test-driven development integrates LLM generation with modular testing constraints, showing that tests can act as specifications during code synthesis [12]. Deterministic retrieval-augmented generation provides a complementary direction in which retrieval and reproducibility are emphasized over unconstrained sampling [13]. EvoGraphCoder represents code, tests, history, dependencies, and review feedback as a relational graph and performs repeated agentic refinement before merge-oriented validation [14]. DocSync applies critic-guided reflexion to documentation maintenance and grounds updates in structural code information and retrieval context [15]. These systems differ in task and implementation, but they share a broader design movement from one-shot generation toward structured, reviewable workflows.

AutoQALLMs connects LLM reasoning to Selenium for web-application testing [16]. A workflow-centric framework examines end-to-end code generation across coordinated development stages [3]. Agent-based evaluation has also been applied to commit-message generation [17], while comparative LLM evaluation has been used for code documentation and commit-message generation [18]. Multi-objective evaluation of LLM-based code modernization further illustrates the need to assess transformations through multiple quality dimensions rather than a single success label [19]. Together with a survey of LLM opportunities and threats in software security [20], these studies place testing, maintenance, code quality, and governance within a common software-engineering context.

2.2. Maintenance, Migration, and Dependency Evolution

Dependency maintenance differs from green-field generation because the target behavior is constrained by an existing project and by compatibility with an external ecosystem. Empirical work predating the current wave of agentic systems already examined how LLMs could help analyze software supply-chain security failures, establishing an early connection between model-assisted reasoning and dependency risk [21]. Early experiments on automatic library migration likewise showed that LLMs could support version-transition tasks before multi-agent migration pipelines became common [22]. DependaFix focuses on the repair of dependency-update pull requests after CI failures, using build logs and local builds to provide context for automated repair [23]. Constraint-driven dependency resolution addresses another part of the problem by combining formal satisfiability reasoning with selective LLM assistance when metadata are missing or ambiguous [24]. DI-BENCH uses repository executability as part of dependency-inference evaluation, so inferred dependencies are assessed in a concrete software environment [25].

Code migration research broadens the same principle beyond package resolution. CoMRA uses retrieval-augmented generation and coordinated agents for migration, including version and dependency evolution scenarios [26]. Environment-in-the-Loop argues that migration agents should observe execution outcomes from the target environment rather than infer correctness solely from source text [27]. AMRerank addresses the preceding choice problem by combining semantic multi-agent analysis with interpretable ranking for replacement libraries [28]. TestSyncer focuses on the downstream maintenance consequence: when production APIs evolve, change types, dependencies, and caller context are used to guide test-code repair [29]. These approaches motivate a maintenance workflow in which repository context, runtime evidence, and explicit update semantics are considered jointly.

Cryptographic dependency updates introduce an additional security constraint. A behaviorally compatible candidate can still be unacceptable when an advisory marks the version as affected. Conversely, a version outside an advisory range can still violate a project-level behavioral contract. VeriCrypt-Agent therefore treats policy evidence and executable evidence as complementary rather than substitutable signals.

2.3. Testing, Differential Execution, and Verification Feedback

Execution feedback is used as a validation signal in LLM-based software testing. Differential testing in release engineering uses LLMs to help automate comparisons across executions and versions, providing a direct conceptual precedent for old-versus-candidate evidence [30]. Large-scale unit-test experiments show that LLM-generated tests can remain vulnerable to compilation failures, hallucinated APIs, and fabricated dependencies under alternative prompting strategies [31]. These results provide a basis for validation architectures in which generated reasoning is checked against executable artifacts.

Multi-agent test generation provides several complementary mechanisms. MACO decomposes dependency construction, abstract test requirements, and engineering test generation, then uses coverage feedback to refine results [32]. UI-Most applies multi-agent collaboration to one-shot GUI testing, extending agentic testing to interactive interfaces [33]. HieraTest uses dependency-aware hierarchical reasoning and repair for LLM-based unit-test generation [34]. PALM combines program analysis with LLM generation to increase Rust test coverage, illustrating the value of hybrid symbolic and generative techniques [35]. A semi-structured review of LLM agents for autonomous system testing similarly identifies external tools, execution, and feedback as recurring ingredients in agentic testing systems [36].

Vulnerability-oriented testing also uses executable confirmation. VulFinder constructs call paths from downstream applications to vulnerable dependency APIs and uses specialized agents to generate and execute exploit tests for reachability validation [37]. SecVulEval shows that statement-level vulnerability localization remains difficult even for current LLM and multi-agent systems, despite richer context [38]. GenReachAI targets enterprise vulnerability reachability with agentic generative AI [39]. CQLLM generates CodeQL detection logic with LLM assistance [40], while code-semantic vulnerability models such as CosFormer address detection through learned structural representations rather than agentic orchestration [41]. These studies reinforce the distinction between proposing a security judgment and collecting evidence sufficient to authorize an operational action.

2.4. Supply-Chain Security, Provenance, and Bounded Autonomy

Software supply-chain security provides the closest conceptual setting for the present work. Application-specific security-test generation with ChatGPT shows that downstream developers benefit from executable demonstrations of how vulnerable libraries affect their applications [4]. A research agenda for the LLM supply chain expands the dependency concept to models, infrastructure, downstream applications, and associated security and privacy risks [5]. SBOM-to-AIBOM work extends software bills of materials with agentic orchestration and reproducibility information, describing provenance and traceability for model-based software ecosystems [42]. GuardNPM focuses on malicious dependency detection in the NPM ecosystem and illustrates the need for adaptive supply-chain defenses [43].

Governance mechanisms in recent work move from passive reporting toward runtime control. TraceCaps attaches verifiable provenance to agent steps and associates accumulated risk with policy thresholds that can allow, warn, or block tool actions [44]. Human-certified module repositories similarly emphasize curated components, provenance, security review, and explicit trust boundaries for automatically assembled systems [6]. Work on governing updates in the LLM supply chain treats pre-deployment testing as a deployer-side control against opaque upstream changes [45]. Research on coding agents in real repositories also documents failure and rejection patterns that caution against treating generated output as equivalent to an acceptable software change [46].

Secure code generation and provenance research addresses adjacent risks. A systematic taxonomy identifies technical and process challenges that affect secure LLM-generated code [47]. A systematic review of watermarking and attribution addresses provenance of model-generated code [48]. These concerns differ from cryptographic dependency compatibility, but they also distinguish evidence about artifacts and actions from model confidence or natural-language rationale.

2.5. Position of VeriCrypt-Agent

Table 1. Positioning of representative related approaches against the evidence and control dimensions used in this study. Evaluation scales use the unit reported by the cited study and are not directly comparable across tasks.

Approach	SE Task	Primary Evidence	Agent/LLM Role	Operational Control	Reported Evaluation Scale
GateLens [49]	Release analytics	Tabular data + formal IR	Reasoning and code generation	Executable analytical pipeline	Industrial real-world release datasets; unit is analytical queries, not update transitions
DependaFix [23]	Dependency-update repair	CI logs and local builds	Diagnose and repair	Validated repair PR	FSE tool-demonstration evaluation on Java dependency-update repair; not an accept/decline corpus
Differential testing [30]	Release engineering	Cross-execution comparison	Classify behavioral differences	Differential evidence for release work	Microsoft production workflow with hundreds to thousands of behavioral differences; different decision unit
MACO [32]	Unit-test generation	Dependency context + coverage	Specialized agents	Coverage-driven refinement	Methods selected from 6 open-source Java projects
VulFinder [37]	Vulnerability reachability	Static paths + executed exploits	Generate and validate	Execution-supported reachability	25 vulnerabilities; 50 reachable and 45 unreachable downstream applications
TraceCaps [44]	Agentic SE governance	Provenance + accumulated risk	Monitored agent actions	Allow/warn/block thresholds	NIER prototype; no transition-level release benchmark reported
Environment-in-the-Loop [27]	Code migration	Target-environment feedback	Generate/revise migration	Environment-constrained migration	Framework/position study; no acceptance benchmark reported
AMRerank [28]	Library migration choice	Semantic + heuristic evidence	Multi-agent analysis	Interpretable reranking	GT2014 benchmark with 188 source libraries
DocSync [15]	Documentation maintenance	AST + retrieved code context	Generate and critic-refine	Factual-consistency critic	CodeXGLUE proxy maintenance task; different unit from dependency transitions
GuardNPM [43]	Malicious dependency detection	Package features + benchmark data	Adaptive feature generation	Package detection/reporting	NPM-package detection benchmark; different unit from update transitions
VeriCrypt-Agent	Dependency acceptance	Advisory + paired execution	Plan and interpret evidence	Deterministic ACCEPT/HOLD/REVIEW	10 unique transitions/30 controlled executions; 48 auxiliary audit files

summarizes representative nearby approaches. No single dimension is unique by itself: agent specialization, executable tests, provenance, migration support, and policy gating all appear in prior work. The specific position of VeriCrypt-Agent is their combination for a narrowly defined release decision. Agent roles may choose and interpret evidence, but the final decision remains constrained by cross-version execution, affected-range policy, mandatory coverage, audit completeness, and explicit escalation semantics.

3. Materials and Methods

3.1. Problem Formulation and Design Requirements

VeriCrypt-Agent decides whether a candidate cryptographic dependency update can be merged automatically. Input consists of compact project code, old and candidate dependency versions, a frozen release/advisory context, and a public catalog of executable probes. Here, frozen means that the release notes, advisory text, and affected-version ranges used for a case are snapshotted before inference and remain identical for all methods that receive grounding; agents cannot query a changing external advisory feed during evaluation. LLM roles are permitted to plan evidence collection and interpret structured outcomes, but they are not permitted to approve an update unless a deterministic gate is satisfied. This separation follows the broader direction of trustworthy agentic software engineering, where provenance and policy can be enforced inline rather than reconstructed after an action [44]. It also reflects human-in-the-loop review designs that retain expert authority for consequential or ambiguous cases [9].

Four design requirements follow. Evidence must be project-relevant, because a generic package-level statement may not capture the contract used by a downstream application. Evidence must be differential when compatibility is at issue, so that the same probe is observed under both old and candidate versions. Release policy must remain external to the LLM, so an affected candidate cannot be accepted merely because generated reasoning appears persuasive. Finally, uncertainty must be represented explicitly: missing coverage and contradictory evidence should trigger review rather than be silently collapsed into either acceptance or rejection.

For candidate update c, the public input is

$$c=\left(p,v_{\mathrm{old}},v_{\mathrm{cand}},s,a,\mathcal{P}\right),$$

(1)

where p is the package, $v_{\mathrm{old}}$ and $v_{\mathrm{cand}}$ are the old and candidate versions, s is compact project source, a is frozen release or advisory context, and $\mathcal{P}$ is the public probe catalog. The system produces an evidence record containing selected probes, execution outcomes in both environments, candidate-side exceptions, role outputs, and the final decision.

3.2. Evidence-Grounded Agentic Workflow

Figure 1 summarizes the workflow. An Update Agent first proposes a risk-aware verification plan. An optional Relation Critic reviews whether the proposed plan covers project-facing contracts, cryptographic round trips, invalid-input behavior, and the available policy context. Its role is advisory: mandatory fallback coverage is enforced independently by the execution controller.

The control skeleton in Figure 1 is intentionally reusable, but its instantiation in this study is cryptography-specific at the evidence and policy layers. The public probes include cryptographic round-trip and malformed/tampered-input behavior; project-contract checks target security-sensitive API semantics such as token representations, key/algorithm handling, validation defaults, and rejection behavior; and the gate consumes affected-version information from security advisories in addition to execution outcomes. Thus, the general architectural pattern is “reason over evidence, then gate the action,” whereas the evidence contract evaluated here is specific to cryptographic dependency behavior and vulnerability policy. Extending the control skeleton to non-cryptographic updates would require a different mandatory-probe catalog and domain policy, not merely reuse of the current oracle.

This organization differs from agentic systems whose primary objective is to generate a software artifact. Multi-agent code-generation frameworks can distribute planning, coding, testing, and debugging across roles [7]. Security code-review systems can similarly distribute detection, localization, repair, and critique [8]. In VeriCrypt-Agent, role specialization is used only upstream of a bounded release decision. A correct-sounding explanation cannot compensate for missing executable evidence, and a failed mandatory probe cannot be overridden by the Audit Agent. The present ablation does not show an independent aggregate safety benefit from the Relation Critic once controller-enforced fallback coverage is active; accordingly, critic checking is treated as a workflow component rather than as a separately validated source of safety.

3.3. Evidence Planning over Public Probes

Update Agent receives the dependency transition, task description, compact source, frozen context, and probe catalog. It returns a structured risk assessment and a selected probe set. Four pre-authored checks are available. The visible probe captures the workflow exposed by the client project. The api_contract probe checks project-facing assumptions such as return types, serialization representations, and direct metadata access. The crypto_roundtrip probe checks whether cryptographic operations remain mutually usable under the project configuration. The invalid_input probe checks rejection behavior for malformed, tampered, or otherwise unsuitable inputs.

Subset selection models budgeted evidence collection. Validation targets can differ in runtime cost, environment setup, and diagnostic value, so a practical agent should not be assumed to execute every conceivable test. Prior test-generation work shows that dependency context and feedback can guide iterative test selection and refinement [32]. Program-analysis-guided test generation likewise demonstrates the value of combining learned reasoning with structured program evidence [35]. In this study, mandatory fallback coverage means that, after agent selection, the controller automatically adds any probe required by the case type but omitted by the planner/critic. This prevents learned subset selection from becoming the sole safety mechanism in the full condition.

Compatibility-oriented updates require visible behavior, the project-facing API contract, and cryptographic round-trip evidence. Advisory-defined security updates require all four probes. The Relation Critic can refine the selected set but cannot generate unrestricted test code, alter probe semantics, or access the private oracle. This restriction keeps the benchmark focused on evidence use rather than unconstrained test synthesis.

3.4. Cross-Version Execution

Each selected probe $q\in {\mathcal{P}}^{*}$ is executed under isolated environments containing $v_{\mathrm{old}}$ and $v_{\mathrm{cand}}$:

$$e_q=\left(\mathrm{run}(q,v_{\mathrm{old}}),\mathrm{run}(q,v_{\mathrm{cand}})\right).$$

(2)

Each execution record contains a success state and, when applicable, a compact exception summary. Because the same project code and probe logic are used in both environments, a candidate-only failure can be attributed to the dependency change rather than to a changed test harness. This construction is closely related to the use of differential testing in release engineering, where behavioral comparison across executions provides paired behavioral evidence rather than a candidate-only outcome [30].

Environment-grounded migration research reaches a similar conclusion from a different task: execution in the target environment reveals incompatibilities that text-only transformation may miss [27]. Vulnerability reachability work also combines static paths with executed exploit tests to reduce the gap between a nominally vulnerable dependency and an application-relevant exploit path [37]. VeriCrypt-Agent applies the same evidence principle to update acceptance: old-versus-candidate execution is not used to generate a patch but to constrain whether the candidate may be merged.

Audit Agent receives structured execution outcomes rather than unrestricted terminal output. It returns a provisional decision, a concise rationale, and an evidence_complete flag. Operationally, evidence_complete is true only when the audit output is parseable and refers to the selected evidence fields required by the gate. This flag can prevent automatic acceptance, but it cannot override a failed candidate probe or an affected-version policy violation.

3.5. Bounded Decision Gate

Final update status is determined by a deterministic gate:

$$\begin{array}{cc}\hfill \mathtt{ACCEPT}⟺& \underset{q\in {\mathcal{P}}^{*}}{\bigwedge }\left(e_q^{\mathrm{old}}=\mathrm{pass}\wedge e_q^{\mathrm{cand}}=\mathrm{pass}\right)\hfill \\ \hfill & \wedge \neg \mathrm{Affected}\left(v_{\mathrm{cand}}\right)\wedge \mathrm{Coverage}\left({\mathcal{P}}^{*}\right)\hfill \\ \hfill & \wedge {\mathrm{Complete}}_{\mathrm{audit}}\wedge d_{\mathrm{audit}}=\mathtt{ACCEPT}.\hfill \end{array}$$

(3)

A candidate-side probe failure produces HOLD. A failed old-environment probe, missing mandatory coverage, parsing failure, or incomplete audit evidence produces REVIEW. Accordingly, HOLD represents known blocking evidence against automatic merge, while REVIEW preserves uncertainty for human inspection. Runtime risk-enforcement systems such as TraceCaps similarly distinguish analysis from action by applying policy thresholds to agent operations [44]. Update-governance work in the LLM supply chain also treats pre-deployment testing as a control boundary rather than a descriptive quality signal [45].

3.6. Auxiliary Static Cryptographic API Audit

A complementary blinded audit task uses the same structured-output discipline but not the same execution mechanism. Input is a blinded Java source file, and output is safe, unsafe, or uncertain, accompanied by issue category, line-level evidence, explanation, and remediation. Java files are not compiled or executed. A structural validator checks only that cited line intervals are non-empty and fall within the source file; it does not establish that the explanation is semantically complete.

This auxiliary task reflects a broader security-analysis setting in which LLMs must localize or explain vulnerabilities. SecVulEval shows that fine-grained vulnerable-statement identification remains difficult even when contextual information is available [38]. CQLLM uses LLMs to synthesize CodeQL vulnerability-detection logic, representing a different route from direct natural-language classification [40]. Secure-code research also documents recurring limitations that arise when models generate or assess security-sensitive code [47]. Consequently, the audit track is reported separately from executable dependency-update verification and is not used to claim end-to-end update safety.

3.7. Experimental Protocol

Two evaluation tracks are used. The primary track is cryptographic dependency update acceptance, where only ACCEPT authorizes automatic merging. The auxiliary track is blinded static crypto-API audit.

Table 2. Overview of the two evaluation tracks. Public artifacts are available to the system during inference; private artifacts are used only for scoring.

Protocol Dimension	CryptoUpdate-Mini-30	CryptoAPI-Bench-Mini-48-Blinded
Evaluation unit	Thirty records generated from ten package-version transitions under three controlled client contexts	Forty-eight blinded Java source files with original identifiers and labels withheld
Agent-visible input	Pinned old and candidate requirements, compact client source, task description, public probe catalog, and frozen release or advisory context	Blinded Java source code without original class names or ground-truth labels
Private oracle	Expected mergeability, expected decision, cached old/candidate environments, and oracle traces	Expected safe/unsafe verdicts stored outside the public benchmark directory
Required output	ACCEPT, HOLD, or REVIEW, selected probes, and rationale	Verdict, issue type, line-level evidence, explanation, and remediation
Execution setting	Isolated Python environments with selected probes executed under both old and candidate versions	Static source-level review without Java compilation or execution
Primary target	Avoid unsafe automatic merges despite passing visible functional checks	Detect insecure cryptographic usage with structurally valid line references
Reported measures	Accuracy, macro-F1, UAMR, compatibility-failure recall, review rate, probe budget, latency, and coverage	Accuracy, macro-F1, unsafe recall, safe specificity, uncertainty, evidence validity, latency, and coverage

summarizes public inputs and private evaluation artifacts.

3.7.1. CryptoUpdate-Mini-30

CryptoUpdate-Mini-30 contains executable scenarios for PyJWT, bcrypt, python-jose, pycryptodome, and cryptography. Ten package-version transitions are each instantiated under three controlled client contexts that vary secrets and payload literals while preserving the underlying maintenance outcome. The 30 records therefore test decision consistency under small client-level changes; they are not treated as 30 independent historical ecosystem updates.

Compatibility and security cases are both represented. Compatibility cases preserve visible behavior while violating a project-level contract, such as the PyJWT change from byte-string to string token returns [50] or bcrypt metadata access affected by the removal of the __about__ attribute [51]. As a result, the visible-test baseline is expected to fail on these cases by construction because the benchmark targets failures that visible checks alone cannot detect. Security cases use frozen advisory snapshots derived from public vulnerability records [52], where a candidate inside the affected range is non-mergeable even when visible functionality succeeds.

Each public case provides old and candidate requirements, compact project code, a task description, frozen context, and four executable probes: visible, api_contract, crypto_roundtrip, and invalid_input. The probes, controlled client contexts, and frozen advisory snapshots are researcher-authored benchmark artifacts and were deliberately constructed to expose the targeted compatibility or security distinction. They are available at inference time and are sufficient for the deterministic All-Probes Gate on this benchmark. Evaluation therefore measures how methods use a supplied evidence set for release decisions; it does not measure open-world defect discovery, autonomous test invention, or robustness to incomplete/incorrect advisories. This construct choice limits the amount of causal credit that can be assigned to agent orchestration.

3.7.2. Methods, Baselines, and Ablations

All LLM conditions use the same local Qwen2.5-Coder-7B-Instruct backbone with structured JSON outputs, as summarized in

Table 3. Method matrix. All LLM-based conditions use the same Qwen2.5-Coder-7B-Instruct backbone; All-Probes Gate is deterministic.

Condition	Release/Advisory Context	Executable Evidence Policy	Critic	Final Decision Authority
Existing Tests	None	Candidate visible probe only	No	Candidate visible outcome
SemVer–Advisory	Frozen advisory range only	No executable probes	No	Affected-range rule
All-Probes Gate	Frozen context and affected range	All public probes cross-version	No	Deterministic evidence gate
Single-Agent Tool User	Available to one LLM workflow	LLM-selected probes	No	Single LLM decision
Planner–Verifier	Available to separate planning and verification prompts	Planner-selected probes	No	Verification-role decision
Full VeriCrypt-Agent	Available to all role prompts	Critic-checked selection with mandatory fallback coverage	Yes	Deterministic evidence gate
Ablations of the full system
No Grounding	Withheld from LLM prompts and release-policy gate	Full executable probe regime	Yes	Deterministic evidence gate
No MetaTest	Available	Visible regression probe only	No	Deterministic evidence gate
No Critic	Available	Planner-selected probes; fallback coverage remains mandatory	No	Deterministic evidence gate
No Audit Gate	Available	Critic-checked selection with mandatory fallback coverage	Yes	Audit-Agent decision directly
Fixed Budget	Available	Fixed visible-test and cryptographic-round-trip budget	Yes	Deterministic evidence gate under the same fixed budget

. This is a controlled within-backbone comparison: it isolates workflow and evidence-policy changes while holding the model fixed, but it cannot separate architecture effects from limitations specific to this 7B backbone or establish cross-model generality. Conditions include Existing Tests, SemVer–Advisory, a deterministic All-Probes Gate control, Single-Agent Tool User, Planner–Verifier, the full VeriCrypt-Agent, and five ablations. All-Probes Gate executes all public probes cross-version and applies the same affected-range rule and candidate-failure blocking policy without LLM planning or audit interpretation. This control measures how much of the benchmark is solvable by the public probe set alone.

Full pipeline execution uses three role prompts. The Update Agent proposes evidence, an optional Relation Critic reviews the proposed coverage, and the Audit Agent interprets structured outcomes. Mandatory fallback coverage is enforced independently of the critic. Ablations remove grounding, supplementary probes, critic review, or deterministic gate control. Fixed Budget replaces adaptive selection with visible and cryptographic-round-trip checks only. These conditions isolate which evidence sources and control mechanisms affect safety and review burden.

3.7.3. Blinded Cryptographic API Audit

CryptoAPI-Bench-Mini-48-Blinded contains 48 Java examples with blinded identifiers and private labels. The task is independent of update acceptance and evaluates static source review. The system predicts safe, unsafe, or uncertain, with issue type, line references, explanation, and remediation. Audit Rule detects selected recognizable patterns, including weak hashes, ECB mode, permissive hostname verification, predictable randomness, and obsolete TLS or SSL strings. Audit Single performs one structured LLM review, Audit Two-Pass adds a critic revision, and Audit Full further checks correspondence between cited code and verdict. Blinding removes original identifiers and labels from the inference input, but the study cannot establish whether semantically similar or identical code appeared in the pretraining corpus of Qwen2.5-Coder; the audit results should therefore not be interpreted as contamination-free capability estimates.

3.7.4. Execution Setting and Metrics

All LLM conditions run sequentially on one NVIDIA A100 GPU with 40 GB memory. Qwen is loaded in bfloat16 and decoded deterministically. Update and audit generations are limited to 360 and 420 tokens, respectively; executable probes use a 120-s timeout. These values were fixed before the reported runs as protocol/resource bounds; no grid, random, Bayesian search, or sensitivity sweep was performed. Consequently, the reported latency, parse behavior, and review routing should not be interpreted as optimized with respect to these settings. Runtime environments are cached by package-version pair. Completed cases are stored immediately in JSONL, allowing interrupted runs to resume. Decision-stage parsing failures default conservatively to REVIEW; critic failures mark evidence incomplete.

For updates, ACCEPT is the positive automatic-merge decision, while HOLD and REVIEW are non-acceptance outcomes. The principal safety measure is unsafe auto-merge rate,

$$\mathrm{UAMR}={\displaystyle \frac{\left|\left\{i\in \mathcal{U}:{\widehat{d}}_i=\mathtt{ACCEPT}\right\}\right|}{\left|\mathcal{U}\right|}},$$

(4)

where $\mathcal{U}$ denotes non-mergeable updates. Additional update measures are macro-F1, compatibility-failure recall, review rate, probe budget, latency, and coverage. For audit, unsafe cases form the positive class; reported measures include macro-F1, unsafe recall, safe specificity, uncertainty, evidence validity, latency, and coverage.

Because the evaluation units are small, binary rates are accompanied by two-sided 95% Clopper–Pearson exact confidence intervals rather than normal approximations. For UAMR comparisons against the full system, all methods are evaluated on the same five non-mergeable transitions, so two-sided exact McNemar tests are reported for the paired unsafe-auto-merge event. Macro-F1 is retained as a descriptive point estimate because a valid paired bootstrap requires the per-case prediction vector; no normal-theory standard error is inferred from the aggregate F1 alone. These analyses are intended to expose uncertainty, not to convert this compact benchmark into a high-powered significance study.

For binary outcome rates, we report two-sided 95% Clopper–Pearson intervals as exact binomial proportion intervals. This choice uses a Bernoulli working model in which each independent evaluation unit contributes one binary outcome (e.g., unsafe auto-merge versus no unsafe auto-merge). For UAMR, the evaluation units are the five unique non-mergeable package-version transitions; the three client-context executions per transition are not treated as independent observations. Likewise, unsafe recall in the auxiliary audit is computed over the 32 unsafe files, each contributing a detected/not-detected outcome. The binomial model is used only to quantify finite-sample uncertainty of these observed proportions. Because the benchmark cases are fixed and researcher-constructed rather than randomly sampled from a broader population, these intervals should not be interpreted as population-level confidence bounds or evidence of ecosystem-wide generalization.

4. Results

Update results are reported at two levels (

Table 4. Evaluation units and denominators used in the reported metrics.

Metric Family	Evaluation Unit	Denominator
Update UAMR	Non-mergeable unique transition	5 transitions
Update macro-F1	Unique package-version transition	10 transitions
Review rate/probes/latency	Controlled execution	30 executions
Audit unsafe recall	Unsafe blinded file	32 files
Audit safe specificity	Safe blinded file	16 files
Audit macro-F1/uncertainty	Blinded audit file	48 files

). UAMR and macro-F1 are aggregated over ten unique package-version transitions, while review rate, probe budget, and elapsed time are computed over 30 controlled executions. The auxiliary audit task is evaluated on 48 blinded Java cases containing 32 unsafe and 16 safe examples.

Figure 2 summarizes the update track. High macro-F1 does not necessarily imply safe automatic merging: SemVer–Advisory has the highest F1 among the simple baselines but still allows unsafe auto-merges, while the full system has an observed UAMR of 0/5 with a nonzero review burden. The exact interval reported below shows why this observation should not be read as proof of a zero population error rate.

4.1. CryptoUpdate-Mini-30 Results

Table 5. Update results. UAMR and F1 are transition-level metrics over 10 package transitions; review rate, probes, and time are computed over 30 controlled executions. P–V denotes Planner–Verifier. All-Probes Gate is deterministic; no LLM time is reported.

Method	UAMR ↓	F1 ↑	Review ↓	Probes	Time (s) ↓
Existing Tests	1.00	0.333	0.00	1.0	0.18
SemVer–Advisory	0.40	0.792	0.00	0.0	<0.01
All-Probes Gate	0.00	1.000	0.00	4.0	–
Single Agent	0.00	0.670	0.80	2.0	5.65
P–V	0.00	0.670	0.80	2.0	5.29
Full VeriCrypt-Agent	0.00	0.670	0.30	3.6	8.63
Ablations of the full system
No Grounding	0.60	0.670	0.00	3.5	7.19
No MetaTest	0.20	0.583	0.37	1.0	5.57
No Critic	0.00	0.670	0.30	2.0	5.64
No Gate	0.00	0.670	0.80	3.6	8.54
Fixed Budget	0.40	0.495	0.30	2.0	8.46

reports update results. Existing Tests accepts every non-mergeable transition (5/5), giving observed UAMR $=1.00$; this behavior follows from compatibility cases designed to pass visible checks while failing a project-facing contract. SemVer–Advisory accepts two of five non-mergeable transitions. All-Probes Gate reaches 10/10 correct transition-level decisions on this benchmark, confirming that the authored public probes are sufficient when all are executed.

Full VeriCrypt-Agent has observed UAMR $=0/5$ with macro-F1 $=0.670$ and a review count of 9/30. Single Agent and Planner–Verifier match its transition-level UAMR and macro-F1 but route 24/30 executions to review. However, All-Probes Gate is strictly stronger on the reported decision and automation metrics in this compact benchmark: it obtains macro-F1 $=1.000$, UAMR $=0/5$, and no review while executing all four public probes. Therefore, the current data do not justify preferring the agentic workflow when a complete, inexpensive, pre-authored probe set is already known. The narrower question studied by the full workflow is whether bounded planning and routing can operate safely under selective or budgeted evidence collection; demonstrating an advantage over exhaustive deterministic execution requires a larger setting in which probe cost or probe-space size makes exhaustive checking nontrivial.

The exact intervals are wide, and none of the paired UAMR contrasts against the full system reaches a two-sided 0.05 threshold. In particular, observing 0/5 unsafe auto-merges is compatible with a population unsafe-merge probability as high as approximately 0.522 under an exact 95% interval. The result is therefore evidence about these five benchmark transitions, not statistical proof of a generally zero unsafe-merge rate.

Ablations identify grounding and probe coverage as the largest observed factors in this benchmark, but the exact intervals above are too wide to support strong population-level effect claims. Removing release/advisory grounding yields 3/5 unsafe auto-merges; restricting the system to the visible regression probe yields 1/5; and a fixed two-probe budget yields 2/5. Removing the deterministic gate preserves 0/5 unsafe auto-merges in these executions but increases review routing from 9/30 to 24/30, indicating that the gate primarily changes how known blocking evidence is operationalized. Removing the Relation Critic leaves both aggregate UAMR and review count unchanged once mandatory fallback coverage remains active. Therefore, this experiment provides no measurable evidence that the critic is load-bearing for safety or automation under the current controller-enforced coverage regime.

4.2. CryptoAPI-Bench-Mini-48-Blinded Results

Figure 3 reports the static audit track. The rule baseline keeps high safe specificity but detects only 1/32 unsafe cases (0.031; exact 95% CI: 0.001–0.162). Single-pass review detects 9/32 unsafe cases and returns uncertain for most files. Two-pass review detects 20/32 unsafe cases (0.625; exact 95% CI: 0.437–0.789) and obtains the highest descriptive audit macro-F1 of $0.556$; its safe specificity is 8/16 (0.500; exact 95% CI: 0.247–0.753). Audit Full detects 19/32 unsafe cases (0.594; exact 95% CI: 0.406–0.763) with macro-F1 $=0.538$.

These absolute values in

Table 6. Results on CryptoAPI-Bench-Mini-48-Blinded. Unsafe examples are the positive class. Evidence validity checks whether predicted line references are non-empty and within the source file.

Method	Safe	Unsafe	Unc.	Macro-F1	U-Prec.	U-Rec.	Spec.	Evidence	s
Rule	46	2	0	0.271	0.500	0.031	0.938	0.000	<0.01
Single Pass	1	11	36	0.474	0.818	0.281	0.875	0.979	4.98
Two-Pass	1	28	19	0.556	0.714	0.625	0.500	0.979	9.16
Audit Full	1	27	20	0.538	0.704	0.594	0.500	0.979	9.33

, together with the large number of uncertain outputs, do not demonstrate a usable stand-alone vulnerability detector. The auxiliary track is retained only as an exploratory test of whether structured second-pass review changes source-grounded judgments.

4.3. Representative Cross-Version Failure Trace

Figure 4 shows CU-001, a PyJWT 1.7.1→2.0.0 update. The visible regression and cryptographic round-trip probes pass, so the visible-test baseline accepts the update. The project-facing API-contract probe fails in the candidate environment because the returned token is already a string and client code calls decode(). The deterministic gate therefore returns HOLD. Passing visible behavior can coexist with a project-facing compatibility break that appears only under a contract-oriented cross-version check.

5. Discussion

The results answer the research questions as follows. RQ1: exhaustive execution of all authored probes is the strongest condition on this benchmark (10/10 transition decisions, 0/5 unsafe auto-merges, no review), while the full bounded workflow trades decision quality for selective evidence/routing and therefore does not outperform exhaustive deterministic checking. RQ2: the largest observed degradations occur when grounding or probe coverage is removed, whereas the Relation Critic shows no independent aggregate effect under mandatory fallback coverage and the deterministic gate mainly reduces escalation by mapping known failures to HOLD. RQ3: a second structured audit pass increases observed unsafe detection from 9/32 to 20/32, but the wide interval, 19 uncertain outputs, and macro-F1 $=0.556$ mean that this track remains exploratory rather than a usable detection system.

5.1. Evidence Coverage and Agent Roles

Results indicate that executable evidence coverage and explicit policy constraints account for the largest observed changes in unsafe auto-merge behavior in the controlled update benchmark. All-Probes Gate reaches perfect transition-level decisions because the benchmark supplies targeted public probes. This distinction is central when interpreting the agentic contribution: multi-agent decomposition should not be credited for evidence already encoded in the catalog, and the present experiment does not show that agentic planning is preferable to simply running all four probes. Instead, VeriCrypt-Agent should be read as a bounded-control design evaluated under selective evidence conditions; its practical value over exhaustive checking remains an open question for larger and cost-heterogeneous probe spaces.

This interpretation is consistent with recent testing research. Large-scale unit-test generation shows that LLM outputs remain susceptible to hallucinated APIs and compilation failures, making automated validation necessary even when generation quality enhances [31]. MACO reports benefits from task decomposition but also relies on execution-based coverage feedback to optimize test quality [32]. Differential release testing directly treats execution comparison as a source of evidence rather than a narrative justification [30]. Accordingly, the present results should be read as evidence for a hybrid architecture in which learned reasoning and deterministic checks have different responsibilities.

Critic ablation provides an additional comparison. Once mandatory fallback coverage remains active, removing the Relation Critic does not change UAMR or review rate. Critic-style roles may still matter in larger search spaces, where many possible probes compete for limited budgets or where generated tests must be validated. Agentic code-review studies report reductions in hallucinated findings when critic roles check results against source code [9]. DocSync similarly uses critic-guided refinement to maintain consistency between evolving code and documentation [15]. Our compact benchmark, however, does not isolate such an effect for update acceptance, so the paper does not attribute a safety effect to the additional agent role in this setting.

5.2. Implications for Software Maintenance and Release Engineering

Dependency maintenance can fail at several different layers: resolution, build compatibility, project-facing behavior, security policy, and downstream vulnerability reachability. Existing systems in recent work specialize at these layers. Constraint-driven resolution addresses installability under incomplete metadata [24]. DependaFix addresses post-update build repair inside CI [23]. AMRerank addresses selection of replacement libraries when a dependency should be migrated [28]. VulFinder addresses whether a vulnerable dependency is reachable from a downstream application [37]. GuardNPM addresses malicious-package identification in the registry itself [43].

VeriCrypt-Agent occupies a different point in this chain: the candidate version and project are already given, and the task is to authorize or block automatic merge. This narrower formulation permits a more specific operational contract. Policy context can block known affected versions, while differential probes can expose project-specific incompatibilities that version metadata cannot express. Neither source subsumes the other. In the observed five non-mergeable transitions, removing grounding produces 3/5 unsafe auto-merges, restricting executable coverage produces 1/5, and a fixed budget produces 2/5; the exact intervals in

Table 7. Small-sample uncertainty for key binary outcomes. Confidence intervals are two-sided 95% Clopper–Pearson intervals. Exact McNemar p values compare unsafe-auto-merge events with Full VeriCrypt-Agent on the same five non-mergeable transitions.

Outcome/Method	Observed Count	Exact 95% CI	McNemar p vs. Full
UAMR, Existing Tests	5/5	[0.478, 1.000]	0.0625
UAMR, SemVer–Advisory	2/5	[0.053, 0.853]	0.500
UAMR, Full VeriCrypt-Agent	0/5	[0.000, 0.522]	–
UAMR, No Grounding	3/5	[0.147, 0.947]	0.250
UAMR, No MetaTest	1/5	[0.005, 0.716]	1.000
UAMR, Fixed Budget	2/5	[0.053, 0.853]	0.500
Review, Full VeriCrypt-Agent	9/30	[0.147, 0.494]	–
Review, Single Agent/P–V	24/30	[0.614, 0.923]	–

caution against treating these small-count differences as stable effect sizes.

Nearby systems in Table 1 solve different tasks—repairing broken dependency updates, classifying production behavior differences, generating tests, ranking replacement libraries, or validating vulnerability reachability—and are not executed here on a shared accept/decline corpus. Consequently, the present results establish only within-benchmark contrasts among the implemented baselines and ablations. A defensible claim of advancement over non-specialized update-decision systems requires a common historical corpus and adapters that expose the same candidate-update decision to competing methods; that experiment remains future work.

Release engineering also involves a trade-off between automation and escalation. GateLens demonstrates that LLM-based analytical support can reduce manual effort in software release analysis when reasoning is mediated through a structured representation [49]. In the present study, deterministic gating changes escalation routing: removing the gate increases review from 9/30 to 24/30 without increasing observed unsafe auto-merges. This occurs because known blocking evidence can be mapped directly to HOLD instead of being treated as unresolved uncertainty. However, review rate is only a count-based proxy for human burden. The study does not measure reviewer minutes, expertise, queue delay, false-review cost, or monetary cost, so wall-clock GPU latency plus review rate should not be interpreted as a complete operational-cost analysis.

5.3. Trust, Provenance, and Human Oversight

Bounded autonomy is also a governance choice. TraceCaps proposes runtime provenance and risk enforcement so that agent actions can be allowed, warned, or blocked according to policy rather than audited only after completion [44]. Human-certified module repositories emphasize provenance and human certification for reusable building blocks in model-assisted development [6]. AIBOM-oriented work similarly extends bills of materials to capture AI-specific orchestration and reproducibility information [42]. These systems differ from cryptographic update verification, but they share an architectural principle: operational trust should be attached to inspectable evidence and explicit control points.

At the release boundary, the HOLD/REVIEW distinction separates blocking evidence from incomplete evidence. HOLD is machine-actionable because blocking evidence is known: a candidate probe failed or a policy condition disallows the version. REVIEW is human-actionable because the evidence is incomplete or inconsistent. This routing preserves the reason for non-acceptance rather than reducing both cases to one confidence value. Human-in-the-loop code-review systems similarly retain an expert checkpoint for release-sensitive decisions [9]. Provenance research for model-generated code treats attribution and traceability as requirements separate from functional correctness [48].

5.4. Implications for Software Engineering Evaluation

Software-engineering evaluations of coding models use measures beyond generation accuracy. Coding-agent studies in real repositories show that generated pull requests can be rejected for functional and specification-level reasons even when agents produce code [46]. Multi-objective modernization research likewise evaluates transformations across multiple statistical and quality dimensions rather than a single pass/fail label [19]. Context-aware vulnerability benchmarks demonstrate that apparent model competence can decline sharply when evaluation requires fine-grained localization rather than coarse labels [38]. These studies use benchmark designs that expose operational failure modes directly.

For dependency maintenance, unsafe auto-merge rate directly measures unsafe automatic acceptance, whereas aggregate accuracy does not isolate this failure mode. SemVer–Advisory obtains macro-F1 $=0.792$ yet still permits unsafe automatic merges. Conversely, conservative LLM workflows can obtain zero UAMR while imposing a high review burden. Reporting UAMR together with macro-F1, review rate, probe budget, and latency therefore separates safety, decision quality, and operational cost. The same combination of safety, decision-quality, and operational measures can be used in other software-engineering tasks where model output can trigger an external action.

Research on secure LLM-generated code also treats automation controls as task-specific. A taxonomy of secure code-generation challenges shows that security quality is affected by model behavior, prompting, context, and process design [47]. Software-security surveys in Computers similarly document both opportunities and threats associated with LLM adoption [20]. Rather than proposing a general-purpose safety wrapper, VeriCrypt-Agent therefore defines a narrow contract around one maintenance action and exposes the exact evidence required for that action.

5.5. Threats to Validity and Limitations

Internal validity is constrained by the controlled benchmark construction. CryptoUpdate-Mini-30 contains only ten independent package-version transitions, of which five are non-mergeable; the 30 executions are repeated client contexts rather than 30 independent ecosystem events. Exact intervals are consequently wide: the observed 0/5 UAMR of several methods has a 95% exact upper bound of 0.522. All-Probes Gate provides a deterministic upper-bound control for this authored benchmark and should not be generalized to updates for which relevant checks are unknown, flaky, expensive, or absent.

Construct validity is strongly limited by researcher-authored evidence. The probe catalog, controlled client snippets, private mergeability oracle, and frozen advisory snapshots were designed together so that the relevant distinction is representable in the supplied evidence; this explains why All-Probes Gate can solve the update benchmark perfectly. The experiment therefore evaluates evidence selection, interpretation, and gating conditional on a sufficient catalog, not discovery of unknown failures in an open repository. UAMR captures unsafe automatic acceptance but not all consequences of conservative automation. Review rate is a workload proxy rather than measured reviewer time or accuracy, and probe count ignores heterogeneous setup/runtime costs. Evidence validity in the static audit track checks only that cited intervals exist and are in range; it does not establish semantic completeness.

Model and protocol validity are also limited. All agentic conditions use only Qwen2.5-Coder-7B-Instruct; therefore, high review rates or parsing behavior cannot be attributed uniquely to the orchestration rather than this backbone. No second model size/family was run, and no sensitivity analysis was performed for token limits, deterministic decoding, or the 120-s probe timeout. The study should consequently be interpreted as a controlled mechanism comparison under one fixed inference configuration, not as evidence that the same workflow ranking will hold across stronger, smaller, or differently trained models.

External validity is limited by package, language, and provenance scope. The update benchmark uses five Python cryptographic packages and controlled project fragments rather than a mined corpus of historical dependency pull requests or independently sampled CVE-driven upgrades. Results may differ for large monorepositories, transitive dependency graphs, native extensions, containerized services, polyglot systems, stateful integration tests, changing advisory feeds, or organization-specific release policy. Dependency-inference benchmarks such as DI-BENCH highlight the scale and repository diversity that broader evaluation can require [25]. Code-migration and vulnerability-reachability work likewise suggest that environment behavior and evidence availability vary substantially across ecosystems [26,37]. No claim of ecosystem-level performance is made from the current five-package sample.

Auxiliary Java audit is evaluated separately from update execution. Static source review does not compile or run the cases, the 48-file benchmark is too small to characterize general software-security capability, and blinding identifiers does not prove that code patterns were absent from the model’s pretraining data. The observed two-pass recall and macro-F1 are therefore exploratory. Future evaluation could combine statement-level vulnerability benchmarks, repository context, executable validation, and contamination-aware sourcing. SecVulEval provides one example of a larger context-aware benchmark design [38]; vulnerability reachability frameworks provide another direction in which static dependency paths are followed by generated exploit tests [37].

Future work should extend the update track in four directions. First, historical dependency pull requests and independently sampled CVE-driven upgrades should provide a common corpus on which deterministic package tooling, non-specialized update agents, and the proposed bounded workflow can be compared directly. Second, the public catalog should become a maintained CI artifact: candidate probes could be synthesized from changed APIs, retrieved from regression suites, deduplicated, validated against the old version, assigned estimated cost, and promoted to mandatory status only after reproducibility checks. Third, model robustness should be tested across at least one stronger and one differently sized/family backbone together with sensitivity sweeps for generation budgets and timeouts. Fourth, environment isolation should expand from cached Python environments to containers and polyglot microservices, where a probe may need to coordinate multiple services, persistent state, migrations, network policy, and stateful integration tests. Agentic migration and test-generation work offers candidate mechanisms for adaptive evidence synthesis and environment feedback [27,34], while runtime provenance could attach verifiable traces to CI release decisions [44].

The evidence-to-decision trace should retain the same diagrammatic structure used in Figure 4 even when the probe-generation stage becomes dynamic. The exact binomial intervals rely on a Bernoulli working model for the binary outcomes of the unique evaluation units; they quantify uncertainty within the controlled benchmark but do not imply that the benchmark transitions constitute an independent random sample from the broader dependency-update ecosystem.

6. Conclusions

VeriCrypt-Agent formulates cryptographic dependency maintenance as a bounded release-control problem. LLM roles plan evidence collection and interpret structured outcomes, while acceptance is authorized only when required cross-version probes pass, the candidate is outside any frozen affected range, coverage is complete, and the audit output is valid. Across ten unique transitions and 30 controlled executions, visible tests accept all five non-mergeable transitions and advisory-only screening accepts two of five. The full workflow observes 0/5 unsafe auto-merges (exact 95% CI: 0.000–0.522) and routes 9/30 executions to review (exact 95% CI: 0.147–0.494). Removing grounding yields 3/5 unsafe auto-merges, restricting supplementary probes yields 1/5, and a fixed probe budget yields 2/5. Removing the deterministic gate leaves the observed unsafe count at 0/5 while review routing increases from 9/30 to 24/30. On 48 blinded static crypto-API cases, two-pass review detects 20/32 unsafe cases (exact 95% CI: 0.437–0.789) and reaches descriptive macro-F1 $=0.556$, which is not sufficient to claim deployable static detection capability.

All-Probes Gate is the strongest condition on this benchmark: when all four relevant authored probes are known and inexpensive, exhaustive deterministic execution achieves perfect transition-level decisions without human review and is preferable to the evaluated agentic workflow. The contribution of the present study is therefore not a claim that multi-agent prompting outperforms deterministic testing, nor a state-of-the-art superiority claim. It is a controlled examination of a bounded release architecture in which model reasoning is prevented from overriding executable and policy evidence. Whether selective agentic planning becomes useful in larger, dynamically generated, cost-heterogeneous probe spaces remains to be established on historical corpora and across multiple model backbones.