Abstract
Large language models (LLMs) are increasingly being deployed as multi-agent systems (MASs), yet limited evidence exists on when such architectures are warranted and how reliably they perform in decision-critical workflows. This paper investigates these questions through veterinary insurance claims adjudication, a document-intensive, policy-driven task requiring structured reasoning, regulatory compliance, and human oversight. We first propose a four-condition framework for assessing MAS suitability and identify claims adjudication as the highest-priority veterinary application. We then develop a self-hosted MAS integrating specialised LLM agents with deterministic policy evaluation, immutable audit logging, and mandatory human review. To support reproducible evaluation, a synthetic benchmark is released, comprising 150 veterinary insurance claims with independently computed ground-truth adjudication outcomes. The proposed system achieves a 97.3% workflow completion rate and 90.0% adjudication accuracy, increasing to 92.5% for completed claims and approaching the corpus ceiling of 96.0%. The evaluation shows that reliable deployment depends primarily on system engineering rather than model capability alone. Explicit orchestration and verification eliminate common LLM tool-use failures, deterministic adjudication achieves perfect agreement with the reference oracle, and completing the inter-agent policy contract increases agreement from 91.8% to 100%. These results demonstrate that hybrid MASs combining probabilistic document understanding with deterministic policy execution provide a practical and auditable approach for decision-support applications in insurance and other policy-governed domains.
1. Introduction
The role of artificial intelligence has become an important factor in the healthcare sector [1,2,3]. Large language models (LLMs) [4,5] have rapidly evolved beyond conversational assistance into autonomous agents capable of reasoning, planning, using external tools, and interacting with complex software environments. Early work, such as ReAct, demonstrated that interleaving reasoning with tool execution substantially extended the capability of language models beyond text generation [6]. Subsequent studies, including those of Toolformer [7] and Gorilla [8], further showed that LLMs can learn to invoke external APIs and software tools, enabling their application to increasingly complex workflows. Building upon these advances, recent multi-agent frameworks, such as AutoGen [9], MetaGPT [10], CAMEL [11], ChatDev [12], and HuggingGPT [13], have proposed decomposing complex tasks into teams of specialised agents that communicate through shared memory, role assignments, or structured workflows. The appeal of multi-agent systems (MASs) is intuitive. Instead of relying on a single general-purpose model to perform every task, specialised agents can independently manage document processing, information extraction, policy interpretation, reasoning, or recommendation, while an orchestrator coordinates the overall workflow. Such decomposition promises improved modularity, parallel execution, easier maintenance, model tier optimisation, and enhanced explainability through explicit intermediate reasoning steps. These characteristics are particularly attractive for document-intensive decision-support applications in healthcare, insurance, finance, and public administration, where decisions must be transparent, auditable, and subject to human oversight [14,15,16,17].
Despite the growing popularity of LLM-based MASs, two fundamental research gaps remain. First, existing studies largely focus on proposing increasingly sophisticated agent frameworks rather than identifying when multi-agent decomposition is genuinely advantageous over a well-designed single-agent solution. Recent empirical evidence suggests that additional agents often introduce substantial coordination overhead without consistently improving task performance. Cemri et al. [18] systematically analyse popular MAS frameworks and identify fourteen recurring failure modes, concluding that naïve agent decomposition frequently increases system complexity while reducing reliability. Likewise, the τ-bench benchmark demonstrates that even state-of-the-art tool-using agents exhibit limited execution consistency, completing fewer than half of realistic tasks successfully and showing poor repeatability across identical executions [19]. These studies suggest that engineering reliability, rather than peak model capability, remains the principal obstacle to deploying autonomous LLM systems in production environments.
Second, relatively few studies investigate the operational behaviour of MASs in real-world deployments. Most published evaluations rely on synthetic benchmarks or laboratory tasks, while practical issues, such as tool compatibility, orchestration failures, deterministic computation, auditability, deployment cost, and human approval workflows, receive comparatively little attention. Consequently, important engineering questions remain unanswered. Should numerical calculations be delegated to LLMs or implemented deterministically? How should orchestration recover from tool-execution failures? When does workflow decomposition improve robustness, and when does it merely increase opportunities for coordination errors? Addressing these questions is essential for trustworthy deployment in high-stakes decision-support systems.
Veterinary insurance claim adjudication provides an appropriate case study for investigating these questions. The workflow combines heterogeneous document ingestion, policy interpretation, numerical benefit calculation, structured reasoning, and mandatory human verification. Unlike conversational tasks, adjudication requires reproducible decisions, transparent reasoning, complete audit trails, and strict adherence to contractual policy rules. Furthermore, veterinary insurance has experienced rapid commercial growth, with the North American market reaching approximately USD 5.2 billion in written premiums and over seven million insured pets in 2024. Although large veterinary insurance datasets have previously been used to study disease prevalence, epidemiology, and claim prediction [20,21,22], no publicly available benchmark provides adjudication outcomes, such as APPROVE, PARTIAL, DENY, or RETURNED, because these decisions remain proprietary to insurance providers. This lack of benchmark data has limited reproducible research on AI-assisted insurance adjudication.
Motivated by these gaps, we argue that multi-agent architectures should not be regarded as the default solution for LLM applications. Instead, MASs should be adopted only when the expected benefits justify the additional coordination complexity. We therefore propose a four-condition decision framework for assessing whether multi-agent decomposition is appropriate: (1) continuous monitoring of evolving external state; (2) high-volume or multi-channel information intake; (3) document-intensive workflows that naturally decompose into specialised processing stages; and (4) multi-tenant operation requiring isolated execution contexts. Tasks satisfying few or none of these conditions are better served by a single-agent architecture, whereas workflows meeting multiple conditions are more likely to benefit from coordinated specialist agents. This perspective complements recent studies on MAS reliability by providing practical design guidance before implementation.
Accordingly, this paper makes four contributions. First, we propose a practical decision framework for determining when LLM-based multi-agent systems are justified and apply it to veterinary applications, identifying insurance claims adjudication as the highest-ranking use case. Second, we design and deploy a nine-role adjudication pipeline combining specialised LLM agents, deterministic computation, immutable audit logging, and mandatory human approval on commodity hardware. Third, we construct and publicly release the first synthetic benchmark for veterinary insurance claim adjudication, comprising 150 claims with independently computed ground-truth adjudication labels to support reproducible evaluation. Finally, through extensive deployment experiments, we identify several previously underreported engineering failure modes—including tool protocol incompatibility, plan-only execution failures, and unreliable numerical reasoning—and demonstrate practical mitigation strategies. Collectively, these findings contribute empirical evidence toward the design of trustworthy, auditable, and economically viable LLM-based decision-support systems for regulated domains.
This research does not seek to establish that multi-agent architectures outperform single-agent alternatives. Rather, it investigates whether a multi-agent system can be engineered to operate reliably in a decision-critical, policy-governed workflow, and which system-level mechanisms are required to achieve such reliability. Accordingly, the evaluation focuses on workflow completion, adjudication accuracy, orchestration reliability, deterministic policy execution, auditability, and human oversight. A comparative evaluation against alternative architectural decompositions, including single-agent systems, constitutes a distinct research question and is left for future work.
2. Related Work
2.1. LLM-Based Multi-Agent Systems
LLM-based MASs have recently emerged as an important paradigm for solving complex tasks through collaboration among specialised agents. Unlike single-agent architectures, MASs decompose workflows into coordinated subtasks performed by multiple role-specific agents communicating through shared memory, messaging protocols, or orchestration frameworks. Recent survey papers have systematically categorised LLM-MASs according to agent profiling, communication mechanisms, memory management, planning strategies, and collaboration topologies, while identifying scalability, coordination, and reliability as major open research challenges [23,24,25].
Recent studies have also explored increasingly sophisticated collaboration mechanisms. AgentVerse provides a flexible platform for studying collaborative agent behaviours across multiple domains, while MacNet investigates collaborative scaling laws, demonstrating that carefully designed communication topologies can substantially improve multi-agent performance as the number of agents increases [24,26,27]. These studies have demonstrated that collaborative intelligence can outperform individual agents on challenging reasoning tasks, but they primarily evaluated synthetic benchmarks rather than operational decision-support workflows.
2.2. Reliability and Evaluation of Agentic AI
As LLM agents become increasingly autonomous, evaluating their reliability has become an active research topic. Recent benchmarks, such as AgentBench and TheAgentCompany, have assessed autonomous agents across interactive environments and realistic workplace tasks, respectively, revealing considerable variation in planning, tool execution, and long-horizon reasoning performance [16,28]. Despite rapid progress, recent surveys have consistently concluded that the robustness, communication efficiency, fault tolerance, and deployment reliability of LLM-based multi-agent systems remain insufficiently understood, particularly in production environments involving long-running interactions among multiple autonomous agents [25,29].
Current evaluation efforts largely emphasise task success rates or benchmark scores. Comparatively little attention is paid to deployment-oriented properties, such as orchestration failures, deterministic computation, auditability, operational cost, and engineering reproducibility. These practical considerations are particularly important for regulated decision-support systems, where consistent execution may be more critical than peak reasoning capability.
2.3. AI Decision Support in Insurance
Artificial intelligence has increasingly transformed insurance operations, including underwriting, fraud detection, customer service, and claims management. Reviews of digital transformation in insurance highlight the growing adoption of AI to improve operational efficiency, while emphasising the continuing need for explainability, governance, and human oversight in high-stakes decisions [22]. Although recent advances in generative AI have stimulated interest in LLM-assisted document understanding, existing insurance applications predominantly employ conventional machine learning or single-agent architectures rather than collaborative LLM-based systems.
Veterinary insurance claim adjudication represents a particularly challenging decision-support problem because it requires integrating heterogeneous clinical documents, policy interpretation, deterministic benefit calculation, and human verification within a single workflow. Unlike conventional document classification or information extraction tasks, adjudication must simultaneously determine policy eligibility, apply contractual exclusions and reimbursement rules, calculate payable benefits accurately, and provide an auditable rationale for every decision. These requirements demand both semantic reasoning and deterministic computation, making the task particularly suitable for hybrid LLM-based decision-support systems.
Large veterinary insurance databases have previously been leveraged for epidemiological research, disease surveillance, and predictive analytics. For example, Egenvall et al. employed nationwide insurance records to investigate the morbidity and mortality patterns in insured dog populations, demonstrating the value of insurance claims for population-level veterinary research [30]. More recently, Debes et al. developed machine learning models using 2.4 million insurance claims from 785,565 dogs collected over 17 years to predict 45 canine disease categories, illustrating the richness of insurance claims as a source of veterinary health data, but they focused exclusively on disease prediction rather than claims adjudication [31]. Similarly, Paynter et al. identified veterinary insurance databases as one of the most valuable sources of veterinary big data for epidemiology, precision medicine, and translational research, while highlighting the limited accessibility of operational insurance data for AI research [32].
However, these datasets were designed to support epidemiological studies or predictive modelling rather than operational insurance decision making. Publicly available veterinary insurance datasets do not include insurer adjudication outcomes, such as APPROVE, PARTIAL, DENY, or RETURNED, policy rule evaluations, reimbursement calculations, or human review decisions because these constitute commercially sensitive business information. Consequently, no publicly available benchmark currently supports the rigorous, reproducible evaluation of LLM-based veterinary insurance adjudication systems. This limitation motivates the development of our synthetic benchmark with independently computed ground-truth adjudication labels, enabling objective comparison of autonomous and human-in-the-loop decision-support systems.
2.4. Research Gap
The current literature reveals three important gaps. First, despite rapid advances in LLM-based multi-agent frameworks, there is limited guidance regarding when multi-agent decomposition is preferable to a simpler single-agent architecture. Second, most published studies evaluate MASs using research benchmarks rather than reporting deployment experiences in real operational workflows. Third, no public benchmark currently supports the reproducible evaluation of veterinary insurance claim adjudication with independently verified disposition labels.
Our work addresses these gaps by proposing a practical framework for deciding when MAS architectures are justified, deploying a complete multi-agent adjudication pipeline in a real insurance workflow, and releasing the first synthetic benchmark for veterinary insurance claim adjudication. Rather than focusing solely on task accuracy, we investigate deployment reliability, deterministic computation, orchestration robustness, and human-in-the-loop decision support, thereby complementing existing benchmark-oriented research.
2.5. Comparison with Recent Hybrid and Behaviour-Aware Architectures
The closest recent work to ClaimPilot in construction, though not in purpose, is the hybrid LLM and behaviour-aware framework of Polat et al. [33], which secures Model–Context–Protocol tool-calling systems against content- and time-based attacks. Both systems are deliberately hybrid: Polat et al. combine rule-based pre-filtering, LLM semantic analysis, and a Sentinel-Net CNN-LSTM anomaly scorer, whereas ClaimPilot combines LLM document understanding with a deterministic policy engine. Both also resort to synthetic data because their domains lack a public corpora. The reported accuracies are comparable (96.07% accuracy and 96.52% F1 in Polat et al.; 90.0% accuracy and 92.1% macro-F1 here), although the tasks differ and the figures are not directly commensurable. The methodological divergence is the point of contrast: their non-LLM layer is a learned classifier that scores the likelihood of an attack, while ClaimPilot’s non-LLM layer is an exact rule computation that reproduces policy arithmetic deterministically. Their deterministic component manages adversarial uncertainty; ours removes functional uncertainty. Crucially, Polat et al. characterise the very attack surface, prompt injection, tool abuse, and multi-step behavioural attacks that any production deployment of a tool-calling adjudication pipeline, such as ClaimPilot, would inherit; the security and privacy considerations in Section 7 adopt their threat framing.
Relative to the broader literature, the communication-centric survey of LLM-based multi-agent systems by Yan et al. [34] frames coordination, communication efficiency, security, and benchmarking as the field’s principal open challenges, and ClaimPilot’s file-based shared state, strict status chain, and completed inter-agent policy contract are concrete responses to the communication and reliability dimensions it identifies. The hybrid role-based reference architecture of Asici et al. [35] argues that LLM-enhanced multi-agent systems require engineering rigour beyond prompt-level role descriptions, with the roles treated as first-class runtime entities and a separation of deliberation from execution; ClaimPilot instantiates this principle by assigning judgement to LLM roles and all deterministic computation and state transitions to code.
As Table 1 shows, ClaimPilot is distinguished less by agent count than by the combination of a released domain benchmark with independent oracle labels, a mandatory human gate, and an exact deterministic adjudication component; the security-oriented hybrid of Polat et al. [33] shares the hybrid determinism principle but targets threat detection rather than task adjudication.
Table 1.
State-of-the-art comparison of representative LLM multi-agent and hybrid decision systems.
3. Selection of First Use Case in the Veterinary Sector
Veterinary medicine is an instructive proving ground for MASs because it inherits the documentation and decision burden of human medicine without the staffing and revenue cushions of large hospital systems, and because its insurance arm is large and growing. At industry scale, this implies millions of claims per year, each requiring document extraction, policy matching, and an adjudication decision—precisely the document-heavy, rules-bound, human-gated profile of condition.
To choose the first target, we scored four candidate veterinary domains against five weighted criteria: genuine MAS necessity (25%), research novelty and literature gap (30%), data and evaluability (20%), generalisability (15%), and regulatory/safety richness (10%). Table 2 reports the scores; veterinary claims adjudication ranks first (4.25/5), narrowly ahead of herd health monitoring (4.15), with diagnostic labs (3.28) and companion clinic operations (3.25) a tier below (Figure 1).
Table 2.
Use case selection matrix (scores 1–5; weighted total).
Figure 1.
Weighted MAS suitability scores across four veterinary domains.
As reported in Table 2, claims adjudication is selected first for three further reasons. Its harm model is financial rather than clinical, giving a cleaner safety story than diagnosis. It maps onto well-understood document AI primitives (extraction, rules, fraud signals) and reuses existing engineering. Moreover, its volume and homogeneity make per-claim automation economically meaningful at the market scale quoted above. Crucially, the system never pays or denies a claim: it produces a draft recommendation for a licensed human adjudicator, preserving human authority.
4. System Design
4.1. Architecture and the Status Chain
ClaimPilot is organised as a pipeline of nine roles operating over a file-based shared state on a single self-hosted gateway. A claim is one record in a JSON array; every role reads records at its input status, performs one responsibility, advances the status, and appends to an immutable JSON lines audit log. The status chain is strict received → extracted → policy_checked → risk_scored → adjudication_drafted → human_review; no role may skip a step, which makes the workflow auditable from end to end (Figure 2 and Figure 3). Eight roles are assigned to LLM agents, adjudication runs in code, and a human gate precedes any decision. Each LLM stage is wrapped in a verify-and-retry loop. The deterministic adjudication step consumes the structured policy evaluation. All roles share claims.json and an append-only audit log.
Figure 2.
Compact view of the nine-role pipeline and the status chain.
Figure 3.
Detailed agent architecture and data flow.
The roles are: an intake agent that registers inbound claim documents; a document extractor that parses invoices and records into structured fields; a policy matcher that applies coverage rules, exclusions, deductibles, and waiting period logic into a structured policy evaluation; a medical-necessity reviewer and a fraud/anomaly scorer that add decision-support annotations; a deterministic step that computes the payout and recommendation; a notifier that advances drafts to the human gate; an audit logger that reconciles the transition record; and a weekly pattern review role. Table 3 details each role, model, and transition.
Table 3.
Pipeline roles, model/engine, and status transition.
4.2. Model Tiering and the Human Gate
Rather than apply one model everywhere, ClaimPilot follows the per-stage tiering rationale common to capable agent frameworks: the mechanical steps run on a cheap, fast model and only genuine judgement warrants a stronger one. In the deployed configuration, every live-pipeline LLM role runs on a cheap-tier model; the weekly strategic review is the only role that runs on a frontier model. The human review gate is structural, not advisory: the status chain physically stops at human_review and no agent may emit an approved or denied status. This enforces in the code the principle that the system drafts and a licensed human decides.
5. Deployment: Model Selection and Reliability Engineering
We deployed the pipeline on a single commodity mini-PC (an embedded x86 board) running the gateway as a background service, with cheap-tier agents authenticated against a commercial LLM API. The system runs entirely on a cheap model tier and, with the three engineering decisions described here, completes claims dependably at negligible cost. We present these as design choices that make agentic pipelines reliable, each being grounded in a concrete deployment observation; the measured outcome of the configuration is reported in Section 6.
5.1. Choosing a Model That Works
Model selection in an agent system is governed by more than capability and price: the model must also accept the gateway tool schema. Of four candidates we evaluated, gpt-5-mini and a frontier model accepted the schema and invoked tools correctly, whereas the two others, a reasoning-tier and a mid-tier model, rejected every request with an HTTP 400 error on the tools field (Figure 4). We therefore standardised the live pipeline on gpt-5-mini, which is fast, inexpensive, tool compatible, and, as the worked example and evaluation show, sufficient for the extraction and policy reasoning the task requires, reserving the frontier model only for the weekly strategic review. The practical lesson is that tool protocol compatibility belongs, alongside accuracy and cost, as a first-class model selection criterion, and it is one that standard capability leaderboards do not report.
Figure 4.
Acceptance of the gateway tool schema by candidate model. The live pipeline is standardised on the tool-compatible cheap tier (gpt-5-mini).
5.2. Making Tool Execution Reliable
With a tool-compatible model chosen, the system must ensure that agents actually call their tools rather than describe the intended action in prose, a documented limitation of current agent loops. Two design choices secured this. First, prompt construction—explicit, numbered instructions ending in an unambiguous tool-forcing directive—raised first-attempt tool execution to approximately 100% across stages, whereas terse instructions were unreliable (Figure 5). Second, and decisively, an orchestrator-level verify-and-retry wrapper re-invoked each stage up to five times until the claim status demonstrably advanced. Together, these converted an individually probabilistic step into a dependable one, and they are what allowed the pipeline to complete the great majority of claims (146 of 150) in the evaluation in Section 6.
Figure 5.
Explicit, tool-forcing prompts raise first-attempt tool execution to near 100% across stages.
5.3. Using the Right Tool for Each Job
The third decision concerns the boundary between the agents and ordinary code. The adjudication step computes a payout, which is a closed-form function of structured fields and therefore not a judgement task. Implemented as an LLM agent it is unreliable; implemented in a few lines of code it is exact and free (Figure 6). We accordingly reserve the LLM agents for the genuinely judgemental stages, which require extraction from messy documents, policy interpretation, and necessity and fraud assessments, and place all deterministic computations in code. This division both improves reliability and makes the system’s error budget legible: as the evaluation shows, residual error originates in the judgemental LLM stages, never in the arithmetic.
Figure 6.
Adjudication implemented as an LLM agent versus in code: completion rises from 0/10 to 100%, with zero payout error.
5.4. Delivery Channels and Human-in-the-Loop Integration
Because ClaimPilot is built on a multi-channel gateway, the human gate need not live in a bespoke interface: drafts can be delivered, and approve, deny, or return decisions collected, through the messaging tools an organisation already uses. A single gateway can bind the same agents to Slack, Microsoft Teams, Discord, Telegram, or e-mail simultaneously, routing each draft to the appropriate reviewer or queue and accepting a one-tap reaction or a short reply as the human decision that advances a claim from human_review to its final disposition. This lowers the adoption cost, because reviewers act where they already work, while preserving the audit trail: every channel message is bound to the same claim record and immutable log. It also enables tiered routing of low-risk drafts to a shared review channel, high-risk or over-approval-prone claims (Section 6.3) to a senior adjudicator, and system alerts to an operations channel, a practical way to direct scarce expert attention to exactly the cases where the pipeline is least reliable.
5.5. From a Single Host to Headless Cloud Deployment
The prototype runs on a single self-hosted board, but the same gateway and agent definitions can be deployed unchanged to a headless cloud server. For a production or multi-reviewer setting, the gateway can run on a small virtual private server or container, exposing only an authenticated, loopback-bound control surface and reaching reviewers through the messaging channels above or a secure overlay network (for example a tailnet), so that authorised staff can review claims from anywhere without exposing the service to the public internet.
This topology separates the always-on adjudication service from the people who supervise it, supports horizontal scaling (additional gateway instances behind a work queue as claim volume grows), and keeps sensitive records within a controlled tenancy. We stress that any deployment touching real client and patient data requires the data protection posture noted in the limitations, access control, encryption in transit and at rest, a retention policy, and redundancy, none of which a single prototype host provides.
6. Synthetic Pilot Corpus and Component Evaluation
Rigorous evaluation requires labelled data, and here the field has a gap. Large veterinary claims datasets exist for research; for example, a corpus of data on 785,565 dogs with 17 years of insurance claims, used to predict disease categories [31] and general insurance claim cost datasets, are available on public platforms, but none provides ground-truth adjudication dispositions (APPROVE/PARTIAL/DENY/RETURNED), which are proprietary to insurers. Therefore, a synthetic corpus of 150 claims is constructed and released, spanning policy logic, with ground-truth labels computed by an independent reference oracle. The corpus contains 62 APPROVE, 22 PARTIAL, 48 DENY, and 18 RETURNED claims across accident (43), illness (36), wellness (21), orthopaedic (18), cosmetic (14), and missing/illegible (18) cases; the invoice totals average USD 1404 (median 1022; maximum 3745). Representative claims appear in Table 4; their full composition is shown in Figure 7 and Table 5.
Table 4.
Representative claims from the synthetic corpus (one per disposition).
Figure 7.
Composition of the 150-claim synthetic pilot corpus (independently labelled): (a) disposition mix, (b) peril mix.
Table 5.
Synthetic corpus summary statistics (N = 150).
6.1. Component Evaluation: The Deterministic Adjudication Engine
The deterministic adjudication engine achieved 100% agreement with the independent reference oracle when provided with complete structured policy information. The initial evaluation achieved only 91.8% agreement because the policy evaluation contract omitted an explicit wellness coverage flag. Once this information was included, agreement increased to 100% without modifying the adjudication logic. This demonstrates that in multi-agent systems, correctness depends not only on the individual components but also on the completeness of the information exchanged between them. Well-designed inter-agent contracts are therefore a prerequisite for reliable downstream decision making.
We distinguish the full architecture from the configuration used in the reported evaluation. The architecture defines nine roles, but the disposition and payout are determined by exactly two LLM stages, document extraction with the usability gate and peril classification, feeding into the deterministic adjudication engine; the remaining roles provide intake, risk annotation, delivery, audit, and periodic drift monitoring that do not alter the adjudication arithmetic. The evaluation therefore focuses on the two outcome-determining LLM stages plus the deterministic engine, so that the measured accuracy is attributable to the components that actually decide the claim, and is not over-credited to the nine-role decomposition.
6.2. Pilot Evaluation of the Full LLM Pipeline
To measure the LLM extraction and policy-matching stages end to end, we ran the released batch harness over the complete 150-claim corpus in a single uninterrupted run, with each claim processed in isolation so that no agent context exceeded a single record. The run took 5.71 h on the embedded testbed (137 s per claim). The throughput was dominated not by inference but by per-invocation gateway start-up: instrumenting a single claim took roughly 65 s of processing and session overhead against 3 to 8 s of model latency, so batching or connection reuse is the obvious lever for higher volume. The evaluated configuration kept the two LLM stages that can influence the disposition, document extraction with its usability gate and peril classification, while the intake, risk annotation, notification and audit ran in code; none of these fed into the adjudication arithmetic, so the measured disposition was unaffected. Three architectural changes distinguish this configuration from the earlier pilot: all policy arithmetic was moved from the language model into a deterministic engine, all state transitions became the responsibility of orchestrator code rather than of model output, and the agent session state was cleared between claims. The full reproducibility specification is given in Appendix A.
The results (Table 6; Figure 8, Figure 9 and Figure 10) show the end-to-end performance of the proposed pipeline. The system successfully completed 146 of 150 claims (97.3%), indicating that the orchestration and verification mechanisms were effective at preventing workflow failures. The overall disposition accuracy reached 90.0%, meaning that the pipeline produced the correct final adjudication outcome for 135 of the 150 claims. When the four stalled claims were excluded, the accuracy increased to 92.5%, showing that most residual errors are attributable to claim interpretation rather than execution failures. Furthermore, the measured corpus ceiling of 96.0% indicates that the remaining performance gap is partly explained by information unavailable in the submitted claim documents.
Table 6.
Pilot evaluation results.
Figure 8.
Confusion matrix of pipeline disposition versus ground truth (N = 150). Green diagonal = correct. The dashed OTHER column holds the four claims that never reached a terminal state.
Figure 9.
Per-class disposition recall and overall accuracy.
Figure 10.
True versus predicted disposition counts (N = 150). Predicted totals track true totals closely; residual gap is concentrated in DENY.
The classification performance was consistently strong across most disposition classes. APPROVE, PARTIAL, and RETURNED achieved recalls of 98.4%, 90.9%, and 100.0%, respectively, while the DENY recall was lower (75.0%), reflecting the system’s tendency to favour approval when uncertainty existed. This conservative behaviour resulted in an over-approval rate of 6.7%, while avoiding incorrect claim denials. Overall agreement with the independent reference oracle remained high, with a macro-averaged F1 score of 92.1% and a Cohen’s kappa of 0.856, indicating almost perfect agreement. Isolated evaluation of the policy-matching stage yielded 96.1% peril classification accuracy, confirming that most adjudication errors originated from upstream semantic interpretation rather than deterministic policy execution.
These figures should be read against a corpus ceiling. Running the deterministic engine over all 150 claims with oracle peril labels yields 96.0% accuracy (144/150); the residual six claims are denials whose grounds, a pre-existing condition, are never stated in the claim documents and are therefore not recoverable by any system reading only that document. The measured 90.0% is thus 6.0 points below a benchmark-specific ceiling under the current input conditions, not below a perfect score.
Table 6 reports both executions of the full corpus together with the 95% bootstrap confidence intervals, so that the precision of each estimate and its stability across runs can be read directly from the main results. The two runs differ by 6.0 percentage points in disposition accuracy (90.0% and 96.0%) and by 2.7 points in workflow completion (97.3% and 100%). This spread should be interpreted as a property of orchestration reliability rather than as measurement noise or model variability: the two runs assigned identical dispositions to 141 of the 150 claims, and the entire difference is accounted for by four claims that stalled at an inter-agent hand-off in the first run and never reached a terminal state. Where the pipeline completes a claim, its decision is stable; what varies between executions is whether every claim completes. The confidence interval for the first run (84.7 to 94.7) accordingly reflects the sampling uncertainty on the 150-claim corpus, whereas the run-to-run difference reflects an intermittent orchestration failure mode, and the two should not be conflated. Section 6.8 examines this behaviour in detail.
6.3. Error Analysis
Figure 11 shows that the errors are structured rather than random, and that their direction matters as much as their number. The DENY precision is 100%: every claim the pipeline denied was genuinely deniable, so the system never withheld payment in error. Conversely, all fifteen errors resolved in the direction of paying or of returning for more information. For a system whose output is a draft reviewed by a licensed adjudicator, this is the correct failure asymmetry; an over-approval is caught by a reviewer inspecting a payment, whereas a wrongful denial might never be appealed.
Figure 11.
Per-class precision, recall and F1 (N = 150). DENY precision is 100%: system never denies claim in error.
Attributing each of the fifteen errors to a cause (Figure 12) separates what can be engineered away from what cannot. Four claims (2.7%) never reached a terminal state and were infrastructure failures rather than judgement failures. Five (3.3%) followed from an incorrect peril classification by the language model. Five (3.3%) were the information-limited denials described above, where the document contained no signal for the ground-truth reason. One (0.7%) was a genuine gap in the deterministic engine: claim SYN-0025 was correctly classified as wellness but was denied in ground truth for falling within a waiting period, which the engine did not apply to the wellness peril. Ten of the fifteen errors are therefore addressable, and five are not.
Figure 12.
Attribution of all fifteen errors. Ten are addressable by engineering; five are information-limited and bound corpus at 96.0%.
The peril misclassifications are not scattered. Four of the five are the same confusion: a routine or preventive presentation that the ground truth labels as wellness is classified as illness (claims SYN-0010, SYN-0032, SYN-0121, and SYN-0125). Because the engine applies the wellness add-on requirement only to claims labelled wellness, this single confusion converts four denials into approvals and accounts for the whole gap between the 75.0% DENY recall and the other three classes. It is a prompt specification problem with a concentrated remedy, not evidence of a capability limit; the peril classification is otherwise 96.1% accurate.
The per-class metrics (Table 7) sharpen what the headline accuracy obscures. The macro-averaged F1 is 92.1% and Cohen’s kappa is 0.856, which on the Landis and Koch scale is an almost perfect agreement beyond chance. RETURNED, which the earlier pilot never once predicted, now has perfect recall and 94.7% precision, confirming that the missing status route rather than the model was responsible for that failure.
Table 7.
Per-class precision, recall, and F1 on the 150-claim pilot.
Combined with the fact that no claim was wrongly denied, this remains the clearest quantitative justification for retaining the human gate: the system is an assistant that drafts, and the reviewer’s residual task is now concentrated on a small, well-characterised class of over-approvals rather than spread across every disposition.
Because the estimates rest on 150 claims, we report the non-parametric confidence intervals. A bootstrap over 10,000 resamples gives 90.0% accuracy (95% CI, 84.7 to 94.7) and a 92.1% macro-F1 (95% CI, 87.3 to 95.9); the Wilson interval for accuracy (84.2 to 93.8) agrees. The lower bound of every interval exceeds the 80% threshold, indicating the result is not an artefact of a favourable sample.
The errors are also weighted by financial consequence rather than counted uniformly. Of the ten over-approvals, five computed a payout of USD 0 (wellness or waiting-period denials that carried no monetary exposure despite being incorrect dispositions) and five a disbursed value totalling USD 6216, with the single worst case of USD 2652. The financial risk is thus concentrated in five of 150 claims, which sharpens the reviewer’s task from inspecting ten errors to catching five high-value over-approvals, and confirms that the error direction and magnitude, not the error count, are what matter for an insurance system. The per-1000-claim linear projection of approximately USD 41,000 is reported only as a property of this corpus’s error profile, not as a real-world loss estimate.
6.4. Observed Failure Modes and the MAST Taxonomy
Our deployment errors are not idiosyncratic; each instantiates a category of the Multi-Agent System Failure Taxonomy (MAST) [18], which makes them comparable with the failures reported for other multi-agent systems (Table 8). Four of the five sit in the specification and verification categories rather than in the model’s raw capability, reinforcing that reliability here is an engineering problem of contracts and checks more than a question of the larger model.
Table 8.
Observed failure modes mapped to the Multi-Agent System Failure Taxonomy.
6.5. Effect of Each Design Decision
Table 9 consolidates the measured contribution of each engineering decision, reporting the same metric with and without the change. Together they convert an unreliable assembly of cheap agents into a pipeline that completes 97.3% of claims and adjudicates 90.0% of them correctly, within six points of the corpus ceiling. On the testbed, a single embedded x86 board, the system processed claims at approximately 137 s each end to end, dominated by the per-invocation gateway start-up rather than model latency; this suits batch or overnight review and would benefit substantially from connection reuse or claim batching at a higher volume.
Table 9.
Measured effect of each design decision.
The synthetic corpus is calibrated against published veterinary insurance descriptive statistics where these exist: the disposition mix reflects the industry pattern in which most submitted claims are approved in full or in part while a substantial minority are denied or returned for documentation, and the peril mix follows the broad categories used by pet insurers. This is calibration to aggregate statistics, not resampling of real claims: the corpus cannot capture the document heterogeneity, ambiguous phrasing, and adversarial or fraudulent patterns of operational data, and it encodes a single policy schedule. These are the reasons the corpus is positioned as a reproducible pilot benchmark rather than an epidemiological sample, and the reason operational data validation is deemed as required future work.
6.6. Synthetic Corpus: Construction, Labelling, and Validation
The benchmark contains 150 synthetic veterinary insurance claims generated from an explicit policy grammar rather than sampled from a model. Each claim is a structured record (species, breed, age, policy and claim dates, coverage add-ons, diagnosis, treatment, itemised line items, and invoice total) created by drawing a peril category and composing a clinically plausible diagnosis, treatment, and line-item set for that peril. The corpus is deliberately stratified to exercise every decision path: 62 APPROVE, 22 PARTIAL, 48 DENY, and 18 RETURNED by disposition; and 43 accident, 36 illness, 18 orthopaedic, 14 cosmetic, 21 wellness, and 18 with absent or illegible diagnosis by peril. The invoice totals span a realistic range (mean USD 1404, median USD 1022, maximum USD 3745). The claim complexity is varied along four axes: document completeness (18 claims carry missing or unreadable diagnoses to test the usability gate), excluded line items (dietary and supplement items carved out of otherwise-covered claims, producing PARTIAL outcomes), waiting-period edge cases (claim dates placed just inside or outside the peril-specific window), and add-on dependence (wellness claims covered only when a wellness rider is present). The reference labels were produced by an oracle that applied the modelled policy (annual limit, USD 10,000; USD 250 deductible; 80% reimbursement; peril-specific waiting periods; a fixed exclusion set; and pre-existing-condition rule) to each claim’s true underlying fields, independently of the LLM pipeline. The label validity was checked by re-deriving the disposition and payout for all 150 claims with the deterministic engine on the true fields: the agreement was exact (150/150; payout MAE USD 0.00).
6.7. Data Independence and Information Leakage
We address label system leakage directly because it bears on how the headline accuracy should be read. By design, the reference oracle and the deployed adjudication engine implement the same policy arithmetic; consequently, once the pipeline perceives a claim’s fields correctly, the engine reproduces the oracle’s disposition and payout exactly. The 100% engine–oracle agreement reported in Section 6.1 is therefore a property of shared, deterministic policy logic; it is expected by construction and is not evidence of the language model’s quality. The only place the pipeline can diverge from the oracle is the LLM perception stage, which conducts field extraction and peril classification from the claim document, and the 90.0% end-to-end accuracy reflects the errors introduced there, not in the arithmetic. No ground-truth field, the true peril, coverage flag, payout, or disposition, is exposed to any LLM prompt; the agents receive only the claim document and must recover the fields the oracle already knows. A residual limitation remains: because the surface document and the oracle label derive from the same peril, an unusually easy-to-classify document could flatter the perception stage, and future corpora should derive documents and labels through independent processes to remove this entirely.
6.8. Run-to-Run Reproducibility of the Pipeline
The reviewer feedback asked whether the results based on a single execution are stable. Because every LLM stage is non-deterministic, we repeated the full 150-claim evaluation under identical conditions—same corpus, same model snapshot, same retry policy and session handling—and report both runs rather than a single observation.
The disposition accuracy was 90.0% in the first run and 96.0% in the second; the workflow completion was 97.3% and 100%, respectively. The two runs assigned identical dispositions to 141 of 150 claims (94.0%). The entire difference was attributable to orchestration rather than to judgement: in the first run four claims (SYN-0020, SYN-0027, SYN-0076 and SYN-0090) stalled at an inter-agent hand-off and never reached a terminal state, whereas the second run completed every claim. Excluding those four claims, the two runs agreed on the disposition for every claim they both completed.
This is the central reliability finding of the study, and it is reported without qualification. A multi-agent pipeline of this design is capable of completing an entire corpus correctly, but it does not do so on every execution: the inter-agent hand-off is an intermittent failure mode that the verify-and-retry wrapper reduces without eliminating. The orchestration reliability, rather than model capability, therefore bounds the end-to-end performance, and a 6.0-point swing in accuracy between otherwise identical executions is the practical consequence. We regard this as the principal engineering lesson of the deployment: reliability in a decision-critical workflow must be demonstrated across repeated executions, not inferred from a single favourable run.
As with every result in this section, these reliability figures were obtained from a synthetic 150-claim corpus built from a single policy schedule and should be read as characterising the benchmark rather than operational deployment.
6.9. Component Ablation: Verify-and-Retry and Session Isolation
Two engineering components have been credited with improving reliability without having been isolated experimentally: a verify-and-retry wrapper, which re-invokes a stage up to five times until the claim status has demonstrably advanced, and per-claim session isolation, which clears the agent session state between claims. We therefore removed each in turn and re-ran the complete 150-claim corpus, holding the model, decision logic, prompts and scorer constant, so that each configuration is directly comparable with the two baseline executions reported in Section 6.8. Table 10 reports the outcome.
Table 10.
Component ablation on the full 150-claim corpus. Both baseline executions are shown so that the ablation effects can be judged against the run-to-run variation.
The result is not the one we anticipated, and we report it as measured. Neither ablation produced a configuration that fell outside the range spanned by the two unmodified baseline executions: the disposition accuracy across the baseline runs varied between 90.0% and 96.0%, and the two ablated configurations returned 95.3% and 94.0%, respectively, both inside that interval. Tested against the stronger baseline execution using McNemar’s exact test on paired claims, neither difference approached significance (verify-and-retry removed, p = 1.00; session isolation removed, p = 0.25). On this corpus, therefore, the individual contribution of either component cannot be resolved against the run-to-run variability in the pipeline itself.
A directional signal is nonetheless visible in the stall counts, which are the quantity these mechanisms are intended to affect: the unmodified pipeline stalled on zero claims in its better execution, whereas removing the verify-and-retry produced one stalled claim and removing the session isolation produced three. The ordering is consistent with both components acting as safety nets against intermittent hand-off failure, but a difference of one to three claims among 150 is not a basis for a quantitative claim, and we do not make one.
We interpret this in light of the development history documented in Section 6.8. Both mechanisms were introduced while the pipeline still contained latent defects: the language-model-owned state transitions that silently dropped claims, an auto-loading plugin that blocked on a dead connection, and unbounded growth of agent session state. Retrying and session clearing compensated for those defects. Once the defects themselves were removed, the compensating mechanisms had correspondingly less to recover, and their measurable contribution fell below the noise floor of the system. This distinction matters for anyone building a comparable pipeline: the design decisions that demonstrably determine correctness here are those that eliminate a defect outright—deterministic execution of the policy arithmetic, a complete inter-agent contract, and code-owned state transitions—whereas retry and isolation are mitigations whose value is contingent on the presence of the faults they mitigate.
Two limitations of this ablation should be stated plainly. First, absence of a measurable effect on a corpus of 150 claims is not evidence that these mechanisms are unnecessary; a safety net that is never called upon in a clean run may still be essential under provider degradation, rate limiting, or model drift, none of which occurred during these executions. Second, the baseline’s own 6.0-point spread sets a floor on the effect size this experiment can detect, so components contributing less than that cannot be distinguished here regardless of their true value. Establishing their contribution would require either a substantially larger corpus or deliberate fault injection, both of which we identify as future work.
As with every result in this section, these figures were obtained on a synthetic 150-claim corpus built from a single policy schedule and characterise the benchmark rather than operational deployment.
7. Discussion and Proposed Evaluation Protocol
The evaluation demonstrates that a hybrid architecture combining LLM-based information extraction with deterministic policy execution is a viable approach for veterinary insurance claim adjudication. The proposed system achieved 90.0% end-to-end disposition accuracy and 97.3% workflow completion, with its performance approaching the measured corpus ceiling of 96.0%, indicating that much of the remaining error is attributable to information absent from claim documents rather than limitations of the adjudication framework itself.
A key finding is that deterministic adjudication eliminates inconsistency once complete structured policy information is available. The adjudication engine achieved perfect agreement with the reference oracle after the policy evaluation contract was extended to include all required coverage information, highlighting that inter-agent communication and schema design are critical determinants of system correctness.
The remaining errors were concentrated in the LLM perception stage, particularly the confusion between wellness and illness claims, which accounted for most incorrect approvals. Importantly, the system never incorrectly denied a valid claim, producing an error profile that favours over-approval rather than wrongful denial. This behaviour is appropriate for a human-in-the-loop workflow, where reviewers can identify excessive payments before settlement while avoiding the higher risks associated with incorrect claim rejection.
The results also suggest that reliability depends more on system engineering than on language model capability alone. Most observed failures are attributable to orchestration, specification, or inter-agent communication issues rather than reasoning errors, consistent with their classification under the MAST. Engineering measures, such as deterministic rule execution, explicit tool invocation, verification-and-retry mechanisms, and code-managed state transitions, are therefore central to achieving stable operation.
The principal limitation of this study is the use of a synthetic evaluation corpus representing a single insurance policy. Although independently labelled, further validation using operational insurer data is required to assess generalisability. Future work should focus on improving peril classification, incorporating historical veterinary records, and evaluating the framework in real-world deployment settings. Overall, the findings indicate that dependable claim adjudication is best achieved through hybrid architectures that combine probabilistic document understanding with deterministic policy reasoning under human oversight.
7.1. Security and Privacy Considerations
ClaimPilot is evaluated here as a functional pipeline; a production deployment on real claimant and patient data introduces a security and privacy surface that we scope explicitly. As a tool-calling multi-agent system on a gateway, it inherits the Model–Context–Protocol threat model characterised by Polat et al. [33]: content-based attacks (prompt injection through claim documents, embedded instructions in invoice text, path traversal via crafted file names) and time- or behaviour-based attacks (multi-step tool-call chains that are individually benign but collectively malicious). Four properties of the current design bear on these risks. First, untrusted-input handling is declared in every agent contract, the document text is treated as data and never as instructions, a first-line but incomplete mitigation against prompt injection. Second, the deterministic policy engine is not LLM controllable, so no prompt content can alter the coverage, deductible, or payout; an injected instruction can only perturb the extracted fields feeding it. Third, the append-only audit log and strict status chain make anomalous transitions observable after the fact, providing the behavioural trace a monitor such as Sentinel-Net [33] would consume. Fourth, the mandatory human gate means no injected content can cause an autonomous payment. The residual risks identified as future work include sensitive data leakage through model providers (mitigable by on-premises or private-endpoint inference), cross-agent contamination through shared state (partly addressed by per-claim session isolation), unauthorised tool execution, and API abuse. A production deployment would pair ClaimPilot with a real-time behavioural monitor of the kind Polat et al. propose [33], together with document sanitisation, per-agent least-privilege tool scoping, encryption in transit and at rest, access control, and retention limits appropriate to veterinary and personal financial data.
7.2. Scope of the Generalisation Claim
Beyond the synthetic corpus limitation noted above, we delimit the generalisation claim itself. The engineering results—the deterministic execution of policy arithmetic, explicit tool-forcing, verify-and-retry, code-owned state transitions, and complete inter-agent contracts that convert unreliable cheap model agents into a dependable pipeline—are the properties we measured directly on this task. Their applicability to other policy-governed domains, such as other insurance lines, benefits adjudication, or regulatory compliance, is a reasoned expectation rather than an experimental result, and we frame it as a hypothesis for future work; establishing it would require evaluation on operational data and on at least one further policy domain.
7.3. Planned Human-in-the-Loop Evaluation
The workflow terminates at a human_review state, and the human gate is central to the safety argument; the reviewer’s operational performance, however, is not yet measured and is reported here as declared future work with a concrete protocol. That study will record, over a stratified sample of drafted claims, the reviewer decision time per claim, agreement between the reviewer and the system’s recommendation, the correction (override) rate, the proportion of drafts requiring modification, and the resulting workload reduction relative to unaided adjudication, with inter-rater agreement estimated across multiple reviewers. Until that study is run, the current evidence for the value of a human gate is indirect but concrete: the error direction analysis shows every residual error is an over-approval or a return rather than a wrongful denial, and the risk-weighted analysis localises the financial exposure to five of 150 claims, defining precisely the small, well-characterised set of drafts on which reviewer attention is most valuable.
We state explicitly that this evaluation is not carried out within the present study, and that the operational performance of the human-review stage therefore remains unmeasured. No reviewer decision time, agreement rate, override rate or workload reduction is reported here, and none should be inferred from the system-side results. This is a substantive limitation rather than a presentational one: the human gate is the mechanism by which the architecture converts a fallible draft into a safe decision, and its practical value depends on reviewer behaviour that we have not observed. We considered conducting a small pilot using the authors as reviewers, but rejected it on the grounds that a reviewer who has seen the reference labels cannot provide an unbiased estimate of agreement, and a single-site sample of that kind would risk overstating the evidence rather than strengthening it.
The indirect evidence available from the system side is reported for what it is worth, and no further. The error profile is asymmetric in the direction that human review can most readily correct: across the evaluated runs the pipeline never denied a claim that should have been paid, and the residual errors are over-approvals and returns. The risk-weighted analysis in Section 6.3 further localises the financial exposure to a small number of high-value claims. Together, these indicate where reviewer attention would be most productive, but they say nothing about how long that review would take, how often a reviewer would disagree with the draft, or how much work the system would save. Those quantities require the study described above.
8. Conclusions
This paper investigated the suitability and reliability of LLM-based multi-agent systems for veterinary insurance claim adjudication. We proposed a framework for identifying tasks appropriate for MAS deployment, developed a hybrid architecture combining LLM agents with deterministic policy execution, and released a synthetic benchmark to support reproducible evaluation. The proposed system achieved 97.3% workflow completion and 90.0% adjudication accuracy, demonstrating that practical performance is achievable using low-cost LLMs when supported by robust orchestration and deterministic decision logic.
The results show that reliable MAS deployment depends more on software engineering than on model capability alone. Explicit orchestration, verification, deterministic policy execution, and complete inter-agent contracts are essential for achieving dependable operation. Future work will evaluate the framework using operational insurance data, incorporate historical veterinary records, and investigate formal verification of inter-agent communication to further improve reliability in policy-governed decision-support systems.
These results should be read within the limits of the evaluation setting. The system was assessed on a synthetic corpus of 150 claims generated from a single insurance policy schedule, and although the reference labels were computed independently of the pipeline, the corpus cannot reproduce the document heterogeneity, ambiguous phrasing, or adversarial and fraudulent patterns of operational claim files. The reported accuracy, completion and reliability figures therefore characterise the behaviour of the system on this benchmark and under this policy, and are not offered as estimates of operational performance. The engineering findings concerning orchestration, deterministic policy execution and code-owned state transitions are properties we measured directly; their applicability to other policy-governed domains remains a hypothesis that requires validation on operational data and on at least one further policy schedule before it can be asserted.
Future work will include a systematic comparison between ClaimPilot’s multi-agent architecture and an equivalent single-agent baseline. This evaluation will examine the adjudication accuracy, workflow completion rate, end-to-end latency, computational cost, and failure modes under comparable model, tool, and prompting conditions. It will also investigate how architectural differences affect orchestration reliability, scalability, auditability, and human oversight requirements. This analysis will help determine when the additional coordination and complexity of a multi-agent architecture provide practical advantages over a single-agent approach.
Author Contributions
Conceptualisation, methodology, validation, writing—original draft preparation, and writing—review and editing, M.L.H.; visualisation, validation, software, data curation, and writing—original draft preparation, N.S. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Informed Consent Statement
Not applicable.
Data Availability Statement
Pipeline configuration, agent definitions, orchestrator, the 150-claim synthetic corpus with ground-truth labels, the batch-evaluation harness, the claim pilot results, and analysis scripts are released by the authors at https://github.com/TinyAnts/ClaimPilot (accessed on 25 August 2026).
Conflicts of Interest
The authors declare no conflicts of interest.
Appendix A
Reproducibility Specifications
To support exact reproduction, we specify the full configuration. Models: All live agents run on gpt-5-mini except the weekly pattern review agent (gpt-5.5), and the same model serves every stage in the evaluated fast path. Tool protocol: Agents use the gateway’s custom tool type, which gpt-5-mini and gpt-5.5 accept and o4-mini and gpt-5.4 reject with HTTP 400. Determinism controls: Generation uses the provider defaults for the released model snapshot (gpt-5-mini-2025-08-07); the evaluation sampler uses fixed seed 7 and the bootstrap analysis uses seed 42; the verify-and-retry wrapper is capped at five attempts per stage. State: Agent sessions are reset to a clean state between claims during evaluation and daily at 04:00 in the scheduled deployment. Orchestration: File-based shared state (claims.json) with an append-only audit log and a strict status chain (received, extracted, classified, risk_scored, adjudication_drafted, human_review) with a returned branch for unusable documents. The pipeline configuration, agent role prompts, orchestrator and evaluation scripts, the 150-claim corpus, the reference labels, and the scoring and figure-generation code are released at the project repository, and a seeded dataset generator that regenerates the corpus from the policy grammar will be added so the benchmark can be reproduced and extended.
References
- Hoang, M.L.; Matrella, G.; Ciampolini, P. Artificial Intelligence Implementation in Internet of Things Embedded System for Real-Time Person Presence in Bed Detection and Sleep Behaviour Monitor. Electronics 2024, 13, 2210. [Google Scholar] [CrossRef] [Scilit]
- Hoang, M.L. Stagewise Optimization Framework for Fall Direction Recognition from Wearable Sensor Data Based on Machine Learning. IEEE Sens. J. 2026, 26, 7755–7769. [Google Scholar] [CrossRef] [Scilit]
- Hoang, M.L.; Matrella, G.; Ciampolini, P. Metrological Evaluation of Contactless Sleep Position Recognition Using an Accelerometric Smart Bed and Machine Learning. Sens. Actuators A Phys. 2025, 385, 116309. [Google Scholar] [CrossRef] [Scilit]
- Sajjadi Mohammadabadi, S.M.; Kara, B.C.; Eyupoglu, C.; Uzay, C.; Tosun, M.S.; Karakuş, O. A Survey of Large Language Models: Evolution, Architectures, Adaptation, Benchmarking, Applications, Challenges, and Societal Implications. Electronics 2025, 14, 3580. [Google Scholar] [CrossRef] [Scilit]
- Peykani, P.; Ramezanlou, F.; Tanasescu, C.; Ghanidel, S. Large Language Models: A Structured Taxonomy and Review of Challenges, Limitations, Solutions, and Future Directions. Appl. Sci. 2025, 15, 8103. [Google Scholar] [CrossRef] [Scilit]
- Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; Cao, Y. ReAct: Synergizing Reasoning and Acting in Language Models. In Proceedings of the 11th International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, 1–5 May 2023. [Google Scholar]
- Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.; Cancedda, N.; Scialom, T. Toolformer: Language Models Can Teach Themselves to Use Tools. Adv. Neural Inf. Process. Syst. 2023, 36, 68539–68551. [Google Scholar] [CrossRef] [Scilit]
- Patil, S.G.; Zhang, T.; Wang, X.; Gonzalez, J.E. Gorilla: Large Language Model Connected with Massive APIs. arXiv 2023, arXiv:2305.15334. [Google Scholar]
- Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Li, B.; Zhu, E.; Jiang, L.; Zhang, X.; Zhang, S.; Liu, J.; et al. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv 2023, arXiv:2308.08155. [Google Scholar]
- Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Zhang, C.; Wang, J.; Wang, Z.; Yau, S.K.S.; Lin, Z.; et al. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. Int. Conf. Learn. Represent. 2024, 2024, 23247–23275. [Google Scholar]
- Li, G.; Hammoud, H.A.A.K.; Itani, H.; Khizbullin, D.; Ghanem, B. CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society. Adv. Neural Inf. Process. Syst. 2023, 36, 51991–52008. [Google Scholar] [CrossRef] [Scilit]
- Qian, C.; Liu, W.; Liu, H.; Chen, N.; Dang, Y.; Li, J.; Yang, C.; Chen, W.; Su, Y.; Cong, X.; et al. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, 11–16 August 2024. [Google Scholar]
- Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; Zhuang, Y. HuggingGPT: Solving AI Tasks with ChatGPT and Its Friends in Hugging Face. Adv. Neural Inf. Process. Syst. 2023, 36, 38154–38180. [Google Scholar] [CrossRef] [Scilit]
- Gunning, D.; Aha, D.W. DARPA’s Explainable Artificial Intelligence Program. AI Mag. 2019, 40, 44–58. [Google Scholar] [CrossRef] [Scilit]
- Amershi, S.; Weld, D.; Vorvoreanu, M.; Fourney, A.; Nushi, B.; Collisson, P.; Suh, J.; Iqbal, S.; Bennett, P.N.; Inkpen, K.; et al. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; ACM: New York, NY, USA, 2019; pp. 1–13. [Google Scholar]
- Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. AgentBench: Evaluating LLMs as Agents. In Proceedings of the International Conference on Learning Representations 2024, Vienna, Austria, 7–11 May 2024; pp. 52989–53046. [Google Scholar]
- Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. Holistic Evaluation of Language Models. Ann. N. Y. Acad. Sci. 2023, 1525, 140–146. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Cemri, M.; Pan, M.Z.; Yang, S.; Agrawal, L.A.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Klein, D.; Ramchandran, K.; et al. Why Do Multi-Agent LLM Systems Fail? In Proceedings of the Advances in Neural Information Processing Systems 38 Main Conference (NeurIPS 2025); Curran Associates, Inc.: Red Hook, NY, USA, 2025. [Google Scholar]
- Yao, S.; Shinn, N.; Razavi, P.; Narasimhan, K. τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv 2024, arXiv:2406.12045. [Google Scholar]
- Sutton, R.T.; Pincock, D.; Baumgart, D.C.; Sadowski, D.C.; Fedorak, R.N.; Kroeker, K.I. An Overview of Clinical Decision Support Systems: Benefits, Risks, and Strategies for Success. npj Digit. Med. 2020, 3, 17. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Topol, E.J. High-Performance Medicine: The Convergence of Human and Artificial Intelligence. Nat. Med. 2019, 25, 44–56. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Eling, M.; Lehmann, M. The Impact of Digitalization on the Insurance Value Chain and the Insurability of Risks. Geneva Pap. Risk Insur.-Issues Pract. 2018, 43, 359–396. [Google Scholar] [CrossRef] [Scilit]
- Luo, J.; Zhang, W.; Yuan, Y.; Zhao, Y.; Yang, J.; Gu, Y.; Wu, B.; Chen, B.; Qiao, Z.; Long, Q.; et al. Large Language Model Agent: A Survey on Methodology, Applications and Challenges. arXiv 2025, arXiv:2503.21460. [Google Scholar]
- Li, X.; Wang, S.; Zeng, S.; Wu, Y.; Yang, Y. A Survey on LLM-Based Multi-Agent Systems: Workflow, Infrastructure, and Challenges. Vicinagearth 2024, 1, 9. [Google Scholar] [CrossRef] [Scilit]
- Guo, T.; Chen, X.; Wang, Y.; Chang, R.; Pei, S.; Chawla, N.V.; Wiest, O.; Zhang, X. Large Language Model Based Multi-Agents: A Survey of Progress and Challenges. arXiv 2024, arXiv:2402.01680. [Google Scholar]
- Chen, W.; Su, Y.; Zuo, J.; Yang, C.; Yuan, C.; Chan, C.-M.; Yu, H.; Lu, Y.; Hung, Y.-H.; Qian, C.; et al. AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. arXiv 2023, arXiv:2308.10848. [Google Scholar]
- Qian, C.; Xie, Z.; Wang, Y.; Liu, W.; Zhu, K.; Xia, H.; Dang, Y.; Du, Z.; Chen, W.; Yang, C.; et al. Scaling Large Language Model-Based Multi-Agent Collaboration. Int. Conf. Learn. Represent. 2025, 2025, 41488–41505. [Google Scholar]
- Xu, F.F.; Song, Y.; Li, B.; Tang, Y.; Jain, K.; Bao, M.; Wang, Z.Z.; Zhou, X.; Guo, Z.; Cao, M.; et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. In Proceedings of the Advances in Neural Information Processing Systems 38 Main Conference (NeurIPS 2025); Curran Associates, Inc.: Red Hook, NY, USA, 2025. [Google Scholar]
- Xi, Z.; Chen, W.; Guo, X.; He, W.; Ding, Y.; Hong, B.; Zhang, M.; Wang, J.; Jin, S.; Zhou, E.; et al. The Rise and Potential of Large Language Model Based Agents: A Survey. Sci. China Inf. Sci. 2025, 68, 121101. [Google Scholar] [CrossRef] [Scilit]
- Egenvall, A.; Hedhammar, A.; Bonnett, B.N.; Olson, P. Gender, Age, Breed and Distribution of Morbidity and Mortality in Insured Dogs in Sweden during 1995 and 1996. Vet. Rec. 2000, 146, 519–525. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Debes, C.; Wowra, J.; Manzoor, S.; Ruple, A. Predicting Health Outcomes in Dogs Using Insurance Claims Data. Sci. Rep. 2023, 13, 9122. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Paynter, A.N.; Dunbar, M.D.; Creevy, K.E.; Ruple, A. Veterinary Big Data: When Data Goes to the Dogs. Animals 2021, 11, 1872. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Polat, O.; Doğan, F.; Türkoğlu, M.; Yolcu, F.; Baturay, M. MCP-Driven Real-Time Security Monitoring: A Hybrid LLM and Behavior-Aware Framework for Temporal Attack Detection and Severity Assessment. Comput. Netw. 2026, 287, 112540. [Google Scholar] [CrossRef] [Scilit]
- Yan, B.; Zhou, Z.; Zhang, L.; Zhang, L.; Zhou, Z.; Miao, D.; Li, Z.; Li, C.; Zhang, X. Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems. arXiv 2025, arXiv:2502.14321. [Google Scholar] [CrossRef] [Scilit]
- Asici, T.Z.; Gürcan, Ö.; Kardas, G. Towards Engineering LLM-Enhanced Multi-Agent Systems: A Critical Examination of Roles. In International Workshop on Engineering Multi-Agent Systems; Springer Nature: Cham, Switzerland, 2026; pp. 200–220. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.











