EKA—Enterprise Knowledge Assistant: Collaborative Multi-Agent AI for Large Claims Handling

Abstract

Large insurance claims handling is a complex, knowledge-intensive process that requires the analysis of heterogeneous information sources and the reuse of past experience distributed across multiple organizational data sources. Consequently, a significant portion of decision-making knowledge is embedded in historical claims records and internal documents, making systematic access and reuse challenging. This paper presents Enterprise Knowledge Assistant (EKA), a collaborative multi-agent AI system designed to act as a sparring partner for large claims handlers. EKA integrates claims structured and unstructured data with an archive of more than five thousand historical cases related to claims management, enabling retrieval, interpretation, and synthesis of relevant past cases and decision patterns. The system is organized as a set of specialized AI agents, each responsible for distinct tasks including claim context analysis, knowledge extraction, document synthesis, and interaction with human users. Through agent collaboration, EKA provides decision support by analyzing comparable historical cases, uncovering hidden correlations, and extracting insurance wisdom, while keeping the human expert firmly in control. The paper describes the system architecture and reports an industrial case study evaluating EKA in a real insurance environment. Results indicate improved knowledge reuse and reduced analysis effort in large claims handling.

1. Introduction

Large insurance claims handling involves complex decision-making processes that require the interpretation of heterogeneous information sources and the reuse of historical knowledge. While organizations accumulate extensive claims records and internal communications over time, systematically leveraging this information to support consistent and informed decision-making remains challenging.

This paper introduces EKA—Enterprise Knowledge Assistant, a collaborative multi-agent system (MAS) based on artificial intelligence (AI) and designed to support claims handlers in accessing and synthesizing relevant historical knowledge. Rather than automating decisions, EKA acts as a decision-support sparring partner, assisting human experts in exploring comparable past cases, identifying potential risks, and creating insurance wisdom. This work demonstrates the practical value of a human-centered AI-based MAS when deployed in a real industrial insurance environment. Through an applied case study conducted within Allianz Commercial, focusing on large claims handling activities, the paper shows how collaborative agent-based architectures can support knowledge reuse in large claims handling while preserving transparency, accountability, and human control. The novelty lies in automatic data-model generation, applied agent decomposition, controlled enterprise data access, and cross-source synthesis in a regulated setting.

The remainder of the paper is organized as follows. Section 1 proceeds with the explanation of the problem background, the relevant state of the art and the article’s contributions. Section 2 describes the industrial context, data sources, and methodological choices underlying the development of EKA. Section 3 presents the results of the industrial case study and discusses the observed outcomes. Section 4 provides a discussion of the findings, limitations, and implications of the proposed approach. Finally, Section 5 concludes the paper and outlines directions for future work.

1.1. Background and Motivation

In the insurance domain, a claim represents a formal request by a policyholder for compensation following a loss event covered by an insurance contract [1]. While many claims can be handled through standardized and partially automated processes, large corporate and specialty claims typically involve high financial exposure, multiple stakeholders, legal uncertainty, and evolving case facts and case dynamics. Examples include major property losses, industrial accidents, liability claims, and cross-border cases, where decisions often require negotiation strategies, expert assessments, and careful evaluation of precedents.

Due to their underlying complexity, claims handlers must evaluate diverse information sources, including structured claims records, expert assessments, and extensive written communications, to determine appropriate handling strategies, reserves and mitigation actions [2]. These decisions are often time-sensitive and require careful consideration of past experience and contextual factors. In practice, a substantial portion of relevant organizational knowledge is embedded in historical claims data and informal internal communications, such as email exchanges among claims handlers, legal experts, and external stakeholders. This knowledge often remains dormant, context-dependent, and difficult to retrieve systematically, leading to reliance on individual experience and manual information search [3]. Consequently, decision quality and efficiency may vary significantly depending on the availability of senior expertise.

Recent advances in AI offer new opportunities to support complex decision-making processes in regulated industrial domains [4]. Instead of fully automating decisions, there is growing interest in AI systems that augment human expertise by improving access to relevant information and enabling reflective analysis [5]. Collaborative and human-centered AI approaches are particularly well-suited to large claims handling, where accountability and expert judgment remain essential.

1.2. Related Work and State of the Art

The banking and insurance sectors have increasingly adopted AI applications to enhance operational efficiency, improve customer experience, and mitigate risk [6]. These applications span process automation [7], decision support [8], customer engagement [9], and fraud detection [10]. In the insurance domain specifically, AI is extensively applied in claims reserving, risk assessment, and underwriting processes [11,12]. A comprehensive review of AI applications in insurance can be found in [13]. Over the past decade, AI applications in insurance have focused on predictive analytics, fraud detection, automated underwriting, and claims severity estimation [14]. Companies are increasingly leveraging AI tools to process large volumes of structured and unstructured data, extract actionable insights, and support decision-making initiatives [15].

Recent work also demonstrates the application of natural language processing techniques to unstructured insurance documents, including claim descriptions, reports, and customer communications, enabling automated information extraction and text-based classification [16]. In parallel, research in enterprise knowledge management explores methods for capturing, organizing, and reusing organizational know-how distributed across heterogeneous systems [17]. However, traditional approaches often rely on centralized repositories and predefined taxonomies, which struggle to accommodate the dynamic and contextual nature of real-world decision-making processes. More recently, large language models (LLMs) have been investigated as flexible components for supporting enterprise knowledge work and decision-making tasks in the insurance domain [18]. Rather than being limited to document classification or extraction, LLM-based assistants are increasingly used to synthesize information, support exploratory analysis, and assist human experts in complex reasoning activities [19]. However, their application in enterprise settings raises challenges related to governance, explainability, privacy, and the integration of heterogeneous data sources within controlled operational workflows [20,21]. These limitations are particularly evident in large claims handling, where relevant knowledge emerges from interactions, negotiations, and evolving case contexts.

In response to these challenges, agent-based and tool-augmented approaches have gained attention as a means to structure and constrain the behavior of LLM-driven systems [22]. By decomposing complex tasks into specialized roles such as planning, information retrieval, extraction, and synthesis, agentic architectures enable multi-step reasoning, iterative interaction, and more transparent coordination between system components [23]. Recent frameworks for multi-agent LLM orchestration, such as AutoGen [24] and HuggingGPT [25], further demonstrate how complex enterprise tasks can be decomposed into coordinated sub-processes involving retrieval, tool invocation, and iterative refinement. Such approaches have been explored in enterprise assistant and workflow automation scenarios, highlighting the potential benefits of combining retrieval mechanisms with structured orchestration and human-in-the-loop interaction [26]. Furthermore, MASs provide a well-established paradigm for implementing these ideas, offering modularity, scalability, and flexibility through collaboration among specialized agents [27,28].

Nevertheless, much of the existing literature emphasizes theoretical coordination mechanisms or controlled experimental settings, and empirical evidence of collaborative AI-based MASs deployed in real-world insurance environments remains limited. Overall, while prior work has extensively explored AI applications in insurance, enterprise knowledge management, and MASs in isolation, there is a lack of documented approaches that combine these strands into a unified, human-centered decision-support system that is used and deployed in real-world large claims operations. In particular, scenarios that integrate a preprocessed, structured view and an ad hoc, unstructured view of the same data, while positioning AI systems as cognitive partners supporting human decision-makers rather than automated decision engines, are largely unexplored [29].

1.3. Research Gap and Contributions

Despite growing interest in applied AI and MASs, there is a limited body of documented industrial case studies demonstrating how collaborative agent-based architectures can support complex, knowledge-intensive decision-making processes in real insurance operations. In particular, existing research provides little insight into systems that leverage internal communications as a source of enterprise knowledge while maintaining human responsibility and control over final decisions. From a research perspective, this gap is not primarily algorithmic, but architectural and interactional. Existing work provides limited guidance on how LLMs, company data sources, and human expertise can be coherently orchestrated into a single decision-support system that operates within real organizational constraints. While recent agentic frameworks demonstrate how LLM-driven agents can coordinate tools and sub-tasks, these approaches primarily focus on general-purpose task automation or experimental multi-agent coordination settings. They do not explicitly address the integration of preprocessed structured enterprise knowledge with dynamically retrieved unstructured communications within regulated industrial workflows, nor do they systematically embed human accountability as a first-class architectural constraint. In contrast, this work focuses on a domain-constrained orchestration pattern tailored to large-claims decision support under real organizational governance requirements.

This paper addresses this gap by presenting an applied AI-based MAS, i.e., EKA, that is designed to support large insurance claims handling through structured enterprise know-how reuse. The main contributions of this work are as follows:

• The presentation of a technique to automatically derive a structured data model from multiple examples of a given data class and then to extract data from the examples into the structured data model;
• The design of a collaborative multi-agent AI architecture for industry knowledge support in the context of large insurance claims handling;
• A human-centered interaction model in which the AI system acts as a sparring partner, supporting expert reflection rather than automating decision-making;
• An industrial case study with expert-based evaluation assessing the proposed approach in a real-world insurance environment, highlighting both practical benefits and inherent limitations.

An overview of the positioning of this work within the existing literature, together with the identified research gap and the corresponding contributions, is presented in Figure 1.

Figure 1. Positioning of the proposed EKA system within existing research streams. The diagram illustrates (1) the three primary research streams relevant to this study, (2) the research gaps identified in the literature, and (3) how the proposed EKA system addresses these gaps with the related contributions.

Beyond the specific industrial context, the aforementioned contributions provide transferable design insights for the development of human-centered, agent-based AI systems in other knowledge-intensive and regulated domains. The principal scientific contribution of this work is the formalization and empirical validation of an architectural pattern for human-centered, multi-agent decision support in regulated environments, explicitly combining (i) structured schema induction from enterprise communications, (ii) dual-view knowledge representation integrating structured and unstructured data, and (iii) directive-ledger-based orchestration under human-in-the-loop governance.

2. Materials and Methods

This Section describes the industrial context, data sources, and methodological choices underlying the development of EKA. The focus is on system design, agent collaboration, and interaction with human users.

2.1. Use Case Description and Operational Context

Large insurance claims are characterized by high financial exposure and require careful evaluation of legal, technical, and negotiation-related factors. Typical large-claims scenarios considered in this study span complex industrial and liability contexts, such as heavy machinery breakdowns, large-scale industrial accidents, major fire events, environmental contamination incidents, aviation losses, multi-party personal injury claims, cross-border product liability disputes, and cyber incidents involving operational disruption or data breaches. Decision-making in this context is largely knowledge-driven and relies on the analysis of facts, insights from the adjustment process, and accumulated experience from past cases and historical documentation. EKA is designed to support large claims handlers by facilitating structured access to historical claims knowledge while preserving full human control over the final decision.

Rather than automating claims handling, the system acts as a decision-support tool that helps users explore relevant past cases and analyze previously unseen inter-dependencies, correlations, and commonalities across lines of business. As illustrated in Figure 2, EKA is embedded in the operational workflow of claims handlers and interacts with users through an iterative feedback loop. User questions to EKA trigger multi-step analysis, while user feedback allows the system to refine subsequent interactions, ensuring a human-centered and transparent decision support process. Indeed, users can refine queries and request additional evidence while the system re-plans accordingly.

Figure 2. High-Level EKA Workflow. The user question is first processed by the Planner agent, which decomposes the query into structured tasks and determines the required data sources. The DataSelector then retrieves relevant information from structured databases and vector-based representations of unstructured documents. The InfoExtractor analyzes the selected content to identify contextually relevant elements. Finally, the Summarizer agent synthesizes the extracted evidence into a structured response presented to the user. A feedback loop enables iterative refinement through human-in-the-loop interaction.

2.2. EKA Workflow and Operational Integration

EKA is implemented as a collaborative AI-based MAS designed to support modularity, extensibility, and transparent interaction with human users. In Figure 2, the high-level architecture of EKA is shown, integrating heterogeneous data sources with a sequence of specialized agents that collaboratively process user queries and generate decision-support insights.

The workflow begins with a user question submitted by a claims handler. The Planner agent interprets the input and translates it into a structured set of instructions that guide subsequent processing steps. Based on these instructions, the DataSelector agent identifies and retrieves relevant information from both the structured claims database and the unstructured email archive embedded in the vector-based database (more details on the data sources in Section 2.3). The selected data are then processed by the InfoExtractor agent, which extracts key facts, patterns, and contextual information required to address the user query. Finally, the Summarizer agent synthesizes the extracted information into a coherent and human-readable response. A feedback loop allows users to refine queries or provide additional input, enabling iterative exploration and reflection. The proposed workflow is event-agnostic and applies uniformly to different large-claims scenarios, including industrial accidents, major fire or explosion events, environmental losses, cyber incidents, aviation claims, and natural catastrophe-related cases, by processing the associated structured records and unstructured documentation within the same MAS orchestration framework.

Agents operate on a shared representation of the claim context, which is progressively enriched as intermediate results are produced and refined across the workflow; intermediate artifacts generated by individual agents, such as selected data subsets or extracted contextual elements, are indeed explicitly passed to subsequent agents to support coordinated multi-step analysis. The modular agent design also facilitates robustness, as individual components can be adapted or replaced without affecting the overall system behavior. This agent-based architecture supports multi-step reasoning, contextual knowledge synthesis, and human-in-the-loop interaction, distinguishing EKA from simpler retrieval-based approaches while maintaining transparency and user control.

2.3. Data Sources and Preprocessing

The system uses a single data source: an archive of more than 5000 highly condensed case extracts related to large insurance claims that are reported in email format. The corpus of the 5000 historical cases reflects several years of large-claims handling activity within the considered industrial context. Rather than representing a statistical sample, it constitutes an operational knowledge base covering a broad range of high-severity cases. The size and diversity of the corpus ensure meaningful coverage of recurring decision patterns and contextual variations typical of large-claims management. Although originating from a single archival source, the email corpus is transformed into multiple complementary representations that are treated as distinct data views during agent processing. The distilled claims case summaries, formatted as emails, are indeed processed along two distinct paths to yield structured and unstructured representations, which are combined at the agent level during online processing; the choice as to which data view or views to use to service an online query is made by the Planner agent at runtime. The preprocessing workflow follows two distinct but complementary paths: a structured data generation pipeline and an unstructured semantic retrieval pipeline. Both pipelines are executed prior to deployment and operate independently of the online reasoning and interaction loop.

To extract explicit and reusable claim-level information, a structured data model comprising approximately 30 fields (e.g., claim attributes, exposure values, and resolution outcomes) was generated through a manual, LLM-based preprocessing pipeline. This pipeline was designed to support the creation of a structured enterprise view of the email archive and does not perform learning or adaptation during runtime. The pipeline consists of two sequential LLM-driven agent invocations executed offline.

In the first step, batches of historical claims case summaries, formatted as emails, were analyzed by an Analyzer LLM-based agent. This agent was tasked with inspecting the email content and proposing candidate data fields together with semantic descriptions that capture recurring claim attributes and decision-relevant information. In the second step, the outputs produced by multiple Analyzer executions were consolidated by a Merger agent, also LLM-based, whose role was to reconcile overlapping field proposals, resolve inconsistencies, and incrementally refine the schema as additional email samples were processed. This consolidation process allowed information extracted from different subsets of emails to be progressively combined into a coherent and stable data model. The outcome of this two-agent preprocessing pipeline consists of a consolidated relational schema and the corresponding SQL DDL statements required to instantiate the table. In addition, the pipeline produces structured prompts that instruct downstream agents on how to populate individual rows of the table from new email instances. This structured representation is subsequently used by EKA as one of the available knowledge views during online query processing and can be readily extended to integrate additional structured enterprise data sources. To assess the reliability of the structured schema induction and extraction pipeline, a qualitative validation procedure was conducted on a randomly selected subset of historical cases. A sample of 50 structured records generated by the offline Analyzer–Merger pipeline was manually reviewed by domain experts and compared against the original email summaries. The audit focused on key decision-relevant attributes, including claim category, exposure range, involved entities, and resolution descriptors. The review indicated that approximately 92% of the evaluated structured fields were correctly populated and semantically aligned with the source content. Around 6% exhibited minor inconsistencies or partial omissions (e.g., abbreviated entity references), while approximately 2% of fields were missing or required clarification. Observed discrepancies were primarily attributable to ambiguous phrasing in the original email summaries, incomplete contextual descriptions, or heterogeneous terminology across cases. No systematic structural inconsistencies in the induced schema were identified. Correction effort was qualitatively assessed as low: in the majority the reviewed discrepancies, the issue could be resolved through direct inspection of the referenced historical case without additional data sources. This validation confirms that the structured preprocessing pipeline provides a sufficiently reliable enterprise data view to support downstream agent reasoning, while remaining subject to expert oversight within the human-in-the-loop framework.

In parallel, the same email corpus is processed to support semantic retrieval over unstructured content. Email-format cases are cleaned, normalized, and segmented into semantically coherent text chunks. Each chunk is transformed into a vector representation using embedding models, enabling similarity-based retrieval and context-aware access to relevant passages. The resulting embeddings are stored in a vector-based index, which serves as the primary access mechanism for agents when retrieving information from historical communications during online query processing.

All data are processed in compliance with internal data protection policies, including the removal or masking of personally identifiable information and sensitive references before downstream processing. This approach supports the retrieval of contextually meaningful information rather than simple keyword matching, which is essential for capturing implicit knowledge distributed across informal communications. All interactions rely on aggregated, curated, and access-controlled data representations. This preprocessing strategy enables EKA to integrate structured and unstructured enterprise knowledge within a unified decision-support workflow while preserving data confidentiality and governance requirements.

2.4. Technical Architecture

EKA is an AI-platform built around a web API framework, modular retrieval pipelines, a LLM service, and a relational document store. In this work, the term LLM service refers to a large language model (LLM) accessed via API, enabling generative text processing and contextual reasoning capabilities based on transformer architectures [30]. Such services provide scalable natural language understanding and synthesis without requiring on-premise model training. Implementation details, internal design choices, and infrastructure-specific configurations are intentionally omitted to comply with internal organizational policies, data protection requirements, and confidentiality constraints. However, the overall EKA architecture and system functionality are fully described in this Section.

The specific use-case settings are defined declaratively in a configuration database, specifying a directed graph of components—LLM agents, retrieval modules, document loaders, and Python snippets—validated at load time to prevent inconsistent routing or variable misuse. At the system core, there is a middleware server that instantiates and caches an asynchronous retrieval pipeline. Each user request is converted into a ledger object that carries all conversational state, external variables, control-flow metadata, token accounting, and document identifiers across components. Retrieval is handled by a modular sub-pipeline that performs embedding, vector-based search, and relevance ranking. LLM calls are managed through a configurable map of endpoints, with support for structured JSON outputs, tool calling and optional parallel execution for purely directive agents. Agent behavior is controlled through JSON schemas and a directive mechanism: the LLM returns structured directives that are interpreted to update ledger variables, modify document sets, and generate final answers. While the underlying LLM exhibits non-deterministic behavior by design, the use of directive-based orchestration, structured intermediate representations, and explicit agent role separation is intended to constrain variability and ensure reproducible analytical behavior at the system level. This architectural design is not intended as an end in itself, but directly supports the modular reasoning, low interaction overhead, and rapid time-to-first-insight observed in the evaluation results reported in Section 3. An overview of the technological architectural layers is represented in Figure 3.

Figure 3. High-level technological architecture of EKA. The agent orchestration layer acts as the central coordination component, interacting independently with the knowledge access layer and the foundation model layer. Unstructured email-format reports are accessed through vector-based representations, while the structured data that emerges from the preprocessing steps is retrieved from relational databases. The cloud infrastructure layer provides cross-cutting services such as security and identity management.

The structured view of the email-format reports is stored in a relational database and accessed through application-level APIs that enforce authorization and data governance policies. In parallel, the unstructured email content is transformed into semantically meaningful chunks and embedded into vector representations, which are stored in a vector layer within the relational database. This vector layer serves as the primary access point for agents when retrieving relevant information from historical communications, enabling semantic and context-aware retrieval rather than direct document access. EKA integrates with cloud-native services, including secure storage, secrets management, and containerized execution environments. Authentication and authorization are enforced through identity and access management services, supporting role-based access control and auditing of system interactions. Advanced natural language understanding and reasoning capabilities are provided through LLMs. It is important to note that agent decomposition, orchestration logic, and human-in-the-loop interaction are independent of the specific foundation model employed. This separation allows the proposed architecture to be adapted to alternative foundation models or future LLM generations without altering the overall interaction model or agent collaboration logic. The LLMs are accessed via managed services and operate within the aforementioned middleware that isolates enterprise data sources from direct exposure. By decoupling enterprise systems, vector-based knowledge access, and foundation model reasoning, EKA combines state-of-the-art AI capabilities with enterprise integration, supporting extensibility and compliance.

2.5. Agent Roles and Responsibilities

EKA relies on four specialized agents that implement distinct analytical and coordination responsibilities within the system. While large claims handling involves multiple conceptual aspects such as claim context understanding, communication analysis, risk evaluation, and exposure estimation, in addition, these responsibilities are operationalized through the aforementioned Planner, DataSelector, InfoExtractor, and Summarizer agents. A detailed overview of agents’ roles, inputs and outputs is shown in Table 1.

Table 1. Roles, inputs, outputs, for the EKA workflow specialized agents.

Agent	Role	Input	Output	Notes
Planner	From user queries to step-by-step instructions	User question	Set of instructions for downstream agents	Guides the workflow
DataSelector	Selects relevant data	Instructions from Planner, Claims DB, Email archive	Subset of relevant claims and emails	Filters structured and unstructured data
InfoExtractor	Extracts relevant information from selected data	Data from DataSelector; instructions from Planner	Key facts, numbers and insights	Elaborates and summarize relevant data
Summarizer	Constructs final response	Info from InfoExtractor; instructions from Planner	Complete answer	Produces human-readable output

Each agent operates within a clearly defined scope of responsibility, reducing functional overlap and supporting traceability of intermediate reasoning steps: the Planner decomposes user queries into structured instructions, the DataSelector identifies relevant data sources, the InfoExtractor extracts the necessary information, and the Summarizer produces the final response. At the same time, the decomposition of tasks across agents enables partial inspection of intermediate outputs, supporting transparency and facilitating user understanding of how final responses are constructed. This decomposition enables modular processing while supporting coherent knowledge synthesis across heterogeneous information sources.

2.6. Agent Collaboration and Interaction Model

Agents communicate through structured information exchange and shared representations of claim context, but those interactions are in every step subject to predefined constraints to ensure that generated outputs remain aligned with the available data and organizational policies. Intermediate results are progressively refined and combined across agents, enabling coordinated analysis rather than isolated outputs. Human users interact with EKA through an iterative query–response process, and at no stage do agents operate autonomously with respect to decision-making, as all system outputs are intended to support, rather than replace, human judgment. The system is indeed designed to act as a sparring partner by presenting comparable historical cases, novel insights, and potential risks, encouraging reflective decision-making rather than automated conclusions. Therefore, the outputs generated by EKA are advisory in nature and do not constitute final or binding decisions. The system is designed as a decision-support tool, with full responsibility and authority remaining with the human claims handler, who evaluates, validates, and ultimately determines the appropriate course of action. Through this structured division of responsibilities, EKA reflects the multifaceted nature of large claims handling while maintaining human oversight and accountability.

2.7. Illustrative Multi-Agent Interaction Example

To make the collaborative behavior of EKA more concrete, it is possible to show an illustrative example of a typical interaction between a claims handler and the system. The example is intentionally abstracted and does not refer to real claims, email content or sensitive data; its purpose is to clarify how the specialized agents collaborate during a decision-support task.

• Question for EKA: “I am working on this claim, with the following details: … Do we have similar large claims resulting in litigation, and what were the contributing or causative factors for them? Please advise on reoccurring commonalities, correlations, and interdependencies amongst those claims even across different lines of businesses!”

The system processes the request through the following sequence of agent interactions:

• The Planner interprets the user query and decomposes it into a structured set of analytical steps: (i) identification of comparable historical claims (ii) retrieval of related information (iii) extraction of escalation-related indicators, (iv) synthesis of risk-relevant insights, and (v) identification of reoccurring commonalities and correlations.
• Based on the Planner instructions, the DataSelector retrieves a subset of relevant structured claims records and related segments from the historical email archive, ensuring that access constraints and governance policies are respected.
• The InfoExtractor analyzes the selected data to extract key facts, contextual signals, and patterns associated with escalation events and negotiation outcomes observed in comparable cases. Also, it looks for inter-dependencies, correlations, and commonalities across the various cases.
• The Summarizer synthesizes the extracted information into a concise, human-readable response highlighting comparable cases, potential escalation risks, and alternative handling considerations, without producing binding recommendations.
• The claims handler may refine the query or request additional focus (as specific claim characteristics), triggering a new planning and analysis cycle while retaining full control over the final decision.

This example illustrates how task decomposition and agent collaboration enable multi-step reasoning and contextual synthesis across heterogeneous enterprise data sources, while positioning EKA as a decision-support sparring partner rather than an automated decision-making system. This case exemplifies how EKA enables the generation of insights that would not have been attainable through conventional analytical approaches, as reported by the large claims experts involved in the experimentation and tool development.

3. Results

This Section presents the results of the industrial evaluation of EKA. Given the decision-support and human-centered nature of the proposed system, the evaluation focuses on user-centered and operational metrics rather than task-level accuracy or predictive performance. Accordingly, the assessment focuses on usefulness, interaction process, and analysis efficiency, capturing how the system supports expert reasoning in complex claims handling scenarios. Finally, as EKA operates as an advisory system rather than an autonomous decision engine, the concept of a fixed error threshold for automated decision-making is not directly applicable. Instead, reliability is operationalized through human validation, with final responsibility and acceptance remaining entirely under expert control.

It is important to note that, for confidentiality and data protection reasons, certain implementation details are intentionally omitted, and selected results are reported in aggregated or range-based form rather than as exact numerical values; nevertheless, these results provide a meaningful indication of the effectiveness of the proposed solution.

3.1. Evaluation Setup

The evaluation of EKA was conducted through an industrial case study involving real large insurance claims drawn from an operational environment. The study was focused on 150 real large-claims cases that were not included in the knowledge base used to build EKA, i.e., representing newly handled claims and constituting out-of-sample evaluation instances. During the evaluation, EKA was used to support analysis of these cases by retrieving and synthesizing relevant information from the historical enterprise archive, without having prior exposure to the evaluated claims themselves. The selected cases reflect the diversity of scenarios typically encountered in large-claims operations, covering heterogeneous claim categories (e.g., industrial property damage, liability, environmental losses, aviation incidents, cyber-related events), multiple jurisdictions, exposure ranges, and negotiation contexts. The sampling strategy aimed to ensure broad coverage of realistic decision-making situations rather than focusing on narrowly defined or homogeneous cases.

Five senior claims professionals participated in the evaluation. These experts were selected based on their recognized domain expertise in large-claims handling and represent some of the most experienced practitioners within the considered organizational context. Importantly, none of the participating experts were involved in the technical development of EKA, thereby mitigating expectation and confirmation bias. The 150 case interactions were categorized into 30 Level-1 questions, which primarily require direct information retrieval and 120 Level-2 questions, which involve multi-step reasoning, contextual interpretation, and synthesis across multiple information sources [31]. An example of Level-1 question is the request to provide one or more detailed attributes of a certain claim, while a Level-2 question involves cross-case synthesis, identification of recurring commonalities and correlations among comparable claims, or contextual reasoning across multiple historical precedents, as exemplified in Section 2.7. These categories were defined before the evaluation to avoid post hoc classification bias.

To ensure methodological consistency, a structured task protocol was defined prior to the evaluation phase. For each evaluated case, experts were provided with a standardized interaction procedure. First, the case context was briefly introduced. Experts were instructed to use EKA as their analytical tool for the assigned objective. Upon reaching a satisfactory initial analytical understanding, they recorded their ratings (more details in Section 3.2). Evaluations were collected independently by each expert. While the limited number of experts reflects the specialized nature of large-claims roles, consistency across assessments was qualitatively reviewed, and no systematic divergence was observed. To reduce novelty and learning effects, experts underwent a short familiarization phase prior to formal evaluation sessions, and performance metrics were recorded only after this stabilization phase. In line with its intended role of sparring partner, EKA did not produce binding recommendations but aimed to surface relevant knowledge and novel insights while leaving final judgment to human experts. The following examples summarize typical tasks observed during the evaluation and contrast them with the corresponding effort commonly required without EKA:

• Identification of comparable historical claims: locating prior cases with similar exposure profiles and resolution patterns would normally require manual database queries or support from data specialists with access to structured claims repositories.
• Reconstruction of decision rationale from historical communications: understanding the reasoning behind past handling decisions typically involves manually searching and reviewing extensive internal documents and email distributed across different repositories.
• Synthesis of structured and unstructured information: combining claims data with insights embedded in communications to form a coherent case overview usually requires repeated tool switching, manual interpretation, and coordination among claims handlers, legal experts, and technical specialists.

Without EKA, these activities would require a combination of specialized competencies, including database access, data extraction skills, and familiarity with historical communication archives, and would typically involve substantial manual effort and coordination across multiple roles. EKA consolidates these tasks into a unified interaction process, enabling individual claims handlers to access and synthesize relevant enterprise knowledge while remaining fully responsible for final decision-making.

3.2. User-Centered Evaluation Results

The effectiveness of EKA as a decision-support system was assessed using three complementary user-centered and operational metrics to capture how the system supports expert reasoning and how efficiently it provides initial analytical value: (i) Perceived Usefulness, (ii) Time to First Insight, (iii) Interaction Rounds. These metrics were selected to reflect the intended role of EKA as a human-centered, exploratory decision-support tool rather than as an automated decision-making system. In the context of large claims handling, effectiveness is indeed better captured by how efficiently and meaningfully the system supports expert reasoning than by objective correctness metrics, which are difficult to define for context-dependent and knowledge-intensive decisions.

Perceived Usefulness captures whether claims handlers considered the support provided by EKA helpful in progressing their analysis of a claim, for example, by clarifying the case context, surfacing relevant historical precedents, or highlighting potential risks and alternative strategies. Perceived usefulness was measured using a structured 4-point Likert scale ranging from 1 (“Not useful”) to 4 (“Highly useful”) [32]. The scale was designed without a neutral midpoint in order to reduce central tendency bias and encourage decisive expert assessment. Given the ordinal nature of Likert-scale data, results are reported using the median and interquartile range (IQR), which represent robust measures of central tendency and dispersion appropriate for non-parametric distributions [33]. The interquartile range corresponds to the range between the 25th and 75th percentile and captures the central 50% of observed ratings. For Level-1 questions (n = 30), EKA achieved a median usefulness score of 4 with a narrow IQR range [4–4], indicating strong agreement among experts regarding its high practical value for direct retrieval tasks. For Level-2 questions (n = 120), the median usefulness score was 3 with an IQR range [3–4], reflecting consistently positive evaluations while acknowledging moderate variability in more complex, multi-step reasoning scenarios. Inter-rater consistency was qualitatively assessed through dispersion analysis, and no systematic divergence across evaluators was observed. Overall, EKA was rated at least “Useful” (score ≥ 3) in more than 90% of the evaluated interactions. Ratings were collected independently from each expert without group discussion, and dispersion analysis did not reveal systematic divergence across evaluators.

Time to First Insight captures the efficiency of the system in supporting early-stage analysis. This metric reflects the elapsed time between the initial user query and the point at which the claims handler reported having sufficient information to form a preliminary analytical framing, such as identifying comparable historical cases or relevant risk factors. For Level-1 tasks, the median time to first insight was consistently below 30 s, with minimal dispersion across experts. For Level-2 tasks, the median time remained below 2 min, reflecting the additional reasoning and synthesis required for cross-case analysis. These values indicate that EKA enables rapid access to relevant enterprise knowledge even in more complex multi-step analytical scenarios.

Interaction Rounds measures the number of user–system exchanges required before the system output was considered useful. This metric provides an indication of how effectively the multi-agent orchestration supports iterative refinement without introducing unnecessary interaction overhead. For Level-1 questions, usefulness was typically achieved within 1 interaction round, while Level-2 questions generally required 1 to 2 rounds to converge toward a satisfactory analytical synthesis. The limited interaction depth suggests that the collaborative agent orchestration supports refinement without generating excessive conversational overhead.

For each of the 150 evaluated cases, experts provided usefulness ratings immediately after reaching a satisfactory preliminary analytical understanding. Time to First Insight was measured based on system interaction logs, capturing the elapsed time between the initial user query and the moment at which the user reported having obtained sufficient information to form a preliminary understanding of the case. Interaction Rounds were computed as the number of user–system exchanges required before the output was considered useful. A summary of the three metrics is reported in Table 2. In accordance with the confidentiality and data protection considerations, results are presented in aggregated or range-based form rather than as exact numerical values. Nevertheless, the observed ranges provide a meaningful and representative indication of the effectiveness of EKA in supporting expert decision-making in large claims handling scenarios.

Table 2. Evaluation results of EKA by question type.

Question Type	Metric	Measurement	Result
Level-1 (n = 30)	Perceived Usefulness	Median (IQR)	4 [4–4]
	Time to First Insight	Elapsed time to first insight	<30 s
	Interaction Rounds	User–system exchanges	1 turn
Level-2 (n = 120)	Perceived Usefulness	Median (IQR)	3 [3–4]
	Time to First Insight	Elapsed time to first insight	<2 min
	Interaction Rounds	User–system exchanges	1–2 turns

3.3. Comparison with a Simple RAG-Based Baseline

To further contextualize the observed results, EKA was compared with a simple retrieval-augmented generation (RAG) baseline using both qualitative and quantitative perspectives. The objective of this comparison is not to establish a competitive performance benchmark, but to clarify the architectural, interactional, and operational differences between a conventional RAG pipeline and the proposed MAS, particularly in relation to the types of decision-support questions encountered in large insurance claims handling. This distinction is particularly relevant in considering the difference between Level-1 and Level-2 questions (see Section 3.1). To ensure interpretability and methodological transparency, the RAG baseline was implemented under controlled and comparable technical conditions. The RAG-based approach relies on single-step retrieval from a vector-indexed knowledge source containing the same 5000 historical large-claims cases summarized in email format used for EKA (Section 2.3). The pipeline does not include query decomposition, task specialization, iterative retrieval, or structured multi-step reasoning mechanisms. Documents were chunked into fixed-size segments and embedded using a transformer-based embedding model. Retrieval was performed using a top-k nearest-neighbor similarity search over the vector index, and the retrieved chunks were concatenated within a fixed context window before answer generation. No re-ranking or additional post-retrieval filtering mechanisms were applied. The same generation model family and user interface were used for both systems to ensure comparable interaction and latency conditions. A single-step prompt template was used, in which the retrieved chunks were concatenated and followed by an instruction to generate a concise analytical response grounded exclusively in the provided context. Table 3 summarizes the configuration of the RAG baseline.

Table 3. Technical configuration of the RAG baseline used for comparison.

Component	Configuration	Component	Configuration
Knowledge base	Same as EKA	Retrieval top-k	8
Chunking strategy	Fixed-size 512-token segments	Embedding model	intfloat/multilingual-e5-large
Re-ranking	No	Context window	Fixed 12,000 tokens
Generation model	Same as EKA	User interface	Same as EKA

While effective for addressing Level-1 questions primarily requiring direct retrieval, this baseline configuration does not provide explicit support for query decomposition, heterogeneous data integration, or coordinated multi-agent reasoning as implemented in EKA. Table 4 summarizes the main qualitative differences between the two approaches. While a simple RAG pipeline can effectively retrieve relevant information from a homogeneous knowledge source, it provides limited support for integrating heterogeneous enterprise data, reasoning across long and evolving documents, or supporting iterative human exploration, which are characteristic of Level-2 decision-support questions in large claims handling. In contrast, EKA adopts a collaborative MAS architecture in which specialized agents coordinate to decompose queries, select relevant data, extract contextual information, and synthesize results across structured and unstructured sources. Another key distinction lies in the interaction model. The RAG baseline primarily supports information retrieval, whereas EKA is designed to function as an interactive sparring partner that supports human-in-the-loop reasoning, allowing users to iteratively refine questions and explore alternative perspectives. This design choice reflects the requirements of large claims handling, where decision-making is context-dependent, knowledge-intensive, and cannot be reduced to single-step retrieval.

Table 4. Qualitative comparison between a simple RAG pipeline and the EKA MAS design. The comparison reflects typical characteristics of a simple RAG pipeline as implemented in the considered baseline, rather than an exhaustive representation of all possible RAG-based system variants.

Dimension	Simple RAG	EKA
Data sources	Single–Homogeneous data view	Preprocessed, structured view and ad hoc, unstructured data views
Reasoning process	Single-step retrieval and generation	Multi-step collaborative agents
Interaction model	Query–response	Interactive sparring partner with human-in-the-loop
Adaptability	Static pipeline	Modular and extensible MAS architecture
Decision support	Information retrieval	Reflective decision support and alternative strategies
Human-in-the-loop support	Limited	Central
Ability to handle Level-2 questions	No	Yes

Beyond the qualitative comparison, a quantitative contrast was also conducted using the same user-centered evaluation metrics introduced in Section 3.2. While not intended as a strict benchmark, these indicative measurements further contextualize the practical differences between the two approaches. When disaggregated by question type, the RAG-based baseline exhibited differentiated behavior across task complexity levels. For Level-1 questions, the baseline achieved a median perceived usefulness score of with an IQR range [3–4], indicating that direct retrieval queries could often be addressed in a satisfactory manner. However, this support required longer interaction cycles (typically between 3 and 5 turns) and a time to first insight exceeding 3.5 min in most cases, reflecting the absence of structured orchestration and task decomposition. In contrast, for Level-2 questions involving multi-step reasoning and cross-case synthesis, the RAG baseline consistently failed to provide meaningful analytical support. In these cases, the median usefulness rating was 1 with an interquartile range [1–1], indicating unanimous expert assessment of non-usefulness. As the baseline was unable to generate actionable or contextually coherent responses for such queries, time-to-first-insight and interaction-round metrics were not meaningfully computable.

Overall, the comparison highlights that EKA extends the capabilities of standard RAG-based systems by explicitly addressing Level-2 decision-support needs through modular reasoning, contextual synthesis, and human-centered interaction, rather than standalone information access. These characteristics are particularly relevant in complex industrial scenarios, where effective decision support depends not only on retrieving information but on enabling experts to interpret, combine, and reflect on dispersed organizational knowledge.

3.4. Failure Cases, Guardrails, and Evidence Transparency

In addition to the aforementioned positive evaluation metrics, a structured qualitative analysis was conducted to examine scenarios in which EKA provided limited added value and to document the architectural safeguards adopted to ensure responsible deployment in a regulated environment. Although EKA was rated as useful in the majority of the evaluated interactions, certain recurring patterns were identified in which the system’s contribution was more constrained. These situations include:

• Claims presenting unusual or rarely encountered configurations (e.g., emerging technical risks, atypical cross-border liability combinations, or novel regulatory contexts) for which no sufficiently comparable historical precedents were available in the enterprise corpus, resulting in contextual summaries rather than meaningful cross-case synthesis.
• User queries formulated in overly broad or under-specified terms, which initially produced high-level thematic responses and required subsequent clarification turns before converging toward a focused analytical objective.
• Requests implicitly expecting forward-looking legal interpretation, external market intelligence, or hypothetical scenario modeling not contained within the curated enterprise archive; in such cases, EKA remained strictly grounded in the available internal data and did not attempt speculative reasoning beyond the knowledge base.

In none of these cases did the system generate binding recommendations or autonomous operational actions, as EKA is explicitly designed as a decision-support instrument operating within a human-in-the-loop framework.

Given the probabilistic nature of LLM-based components, particular attention was devoted to hallucination risk mitigation. In operational mode, EKA applies strict retrieval grounding: analytical synthesis is performed exclusively on information retrieved from the data source (Section 2.3), and open-domain knowledge generation is not enabled. Responses are therefore constrained to the enterprise knowledge base. At the architectural level, the directive-ledger orchestration mechanism structures intermediate outputs and constrains multi-step reasoning through controlled variable passing, thereby reducing uncontrolled generative drift. In addition, agent role separation ensures that data selection, extraction, and synthesis remain modular and inspectable rather than executed as unconstrained end-to-end generation. Transparency is further reinforced through explicit evidence presentation mechanisms. EKA responses systematically reference identifiable historical claim instances, structured schema attributes, and retrieved communication excerpts forming the basis of the generated synthesis. Source elements are exposed within the user interface so that claims handlers can directly inspect the underlying evidence and independently verify contextual relevance. This design ensures that outputs are not opaque generative statements but grounded summaries whose provenance can be examined in real time. While these architectural safeguards do not eliminate the inherent non-determinism of LLMs, they constrain system behavior within enterprise-controlled knowledge boundaries and preserve full human validation. EKA therefore remains positioned as a transparent, evidence-grounded decision-support tool rather than an autonomous reasoning engine.

3.5. Summary of Results

The results presented in this Section provide a consolidated view of the effectiveness of EKA as a collaborative decision-support system for large insurance claims handling, while highlighting its role as an advanced knowledge exploration tool that facilitates structured access to dispersed organizational knowledge that would otherwise require substantial manual effort. Across real operational cases, the evaluation indicates that EKA consistently supports expert reasoning by facilitating access to relevant historical knowledge, synthesizing heterogeneous information sources, and enabling efficient interaction with complex enterprise data. User-centered evaluation metrics show that EKA effectively reduces the effort required to explore historical precedents and contextual information while maintaining a lightweight and focused interaction model. The comparison with a simple RAG-based baseline further contextualizes these findings. The results indicate that the EKA MAS architecture and collaborative workflow better align with the requirements of large claims handling, where decision-making is knowledge-intensive, context-dependent, and cannot be reduced to single-step retrieval.

In line with organizational confidentiality and data protection requirements, selected results are reported in aggregated or range-based form rather than as exact numerical values. Nevertheless, the observed ranges and qualitative findings provide a representative and meaningful indication of the proposed approach’s effectiveness in a real industrial setting.

4. Discussion

The results presented in Section 3 highlight the potential of collaborative AI-based MASs to support complex, knowledge-intensive decision-making processes in real insurance environments. While the experimental evaluation presented in this study is necessarily limited by industrial constraints related to data confidentiality, organizational processes, and system deployment, it reflects a deliberate methodological choice aligned with the human-centered and decision-support nature of the proposed system. Rather than focusing on algorithmic accuracy or predictive performance, the evaluation emphasizes operational metrics that reflect how effectively EKA supports expert reasoning in large claims handling.

4.1. Interpretation of User-Centered Evaluation Results

The high level of Perceived Usefulness observed during the evaluation indicates that EKA consistently provided value to claims handlers when analyzing complex cases. This finding suggests that the system successfully surfaces relevant historical knowledge, contextual information, and alternative perspectives that align with expert needs. Importantly, Perceived Usefulness reflects not only the relevance of retrieved information, but also the extent to which the system integrates smoothly into existing analytical workflows without disrupting human judgment.

The consistently low Time to First Insight further supports this interpretation. In large claims handling, early access to an effective understanding of a case is critical for framing subsequent analysis and decision-making. Achieving initial insights within a short time window indicates that EKA effectively reduces the overhead associated with manual data exploration and fragmented access to enterprise knowledge sources. In the absence of EKA, obtaining a comparable level of historical understanding of similar cases from scratch would typically require several hours of manual effort, involving the coordination of multiple skills and roles, such as database querying, document review, and the interpretation of distributed internal communications. This efficiency is particularly relevant in organizational contexts where claims handlers may not have direct access to all underlying systems or the technical skills required to extract information independently.

The limited number of Interaction Rounds required to reach useful outcomes provides additional evidence of effective system design. The low interaction number suggests that EKA balances responsiveness with flexibility: users can obtain meaningful support quickly while retaining the ability to refine queries when necessary. This behavior aligns with the intended role of EKA as a sparring partner that supports reflective reasoning rather than enforcing rigid interaction patterns.

Finally, the qualitative comparison with a simple RAG-based baseline comparison highlights that the added value of EKA does not stem solely from improved retrieval capabilities, but from its ability to orchestrate analysis, synthesize context, and support iterative human exploration. These characteristics are essential in domains where decisions cannot be reduced to single-shot question answering and where accountability remains with human experts.

4.2. Practical Implications for Insurance Operations

From an operational perspective, the results suggest that systems such as EKA can support a more systematic reuse of organizational knowledge in large claims handling. By consolidating access to structured claims data and unstructured historical communications within a single interaction framework, EKA reduces the fragmentation typically associated with complex claim analysis.

A particularly relevant implication concerns the horizontalization of analytical capabilities across claims handlers. In traditional workflows, effective access to historical data often depends on personal business knowledge and specialized technical skills, such as the ability to query databases, extract records, and navigate distributed document repositories. EKA abstracts these technical requirements by mediating data access and synthesis through agent-based orchestration, enabling claims handlers to focus on interpretation and judgment rather than on data manipulation. This shift has the potential to reduce reliance on individual seniority or support from technical specialists, while promoting more consistent analytical practices across teams. However, while EKA enables a more horizontal distribution of analytical capabilities, access to underlying data remains governed by existing authorization policies and audit mechanisms, ensuring that increased usability does not imply unrestricted data access.

Finally, EKA does not aim to replace expert decision-making; instead, it supports transparency and reflective reasoning by making relevant enterprise knowledge more accessible, while preserving full human accountability and control over final decisions. Such characteristics are particularly important in regulated insurance environments, where explainability, responsibility, and expert oversight remain essential.

4.3. Limitations and Future Work

While the presented results indicate the practical value of EKA, several limitations should be acknowledged to properly contextualize the findings and outline directions for future research.

First, the proposed system represents a specific architectural instantiation of a collaborative MAS approach. Alternative designs could employ different agent decompositions, orchestration strategies, or underlying LLMs, potentially leading to different interaction characteristics and outcomes. The results reported in this study should therefore be interpreted as evidence of the viability of the overall approach rather than as definitive validation of a single architectural configuration. Future work will also explore systematic evaluation across different foundation models to assess robustness and consistency of agent behavior. Furthermore, the evaluation relies on user-centered and operational metrics rather than traditional accuracy-based measures. This choice reflects the nature of the application domain, where decision support, contextual understanding, and expert reasoning are more relevant than objectively verifiable outputs. While appropriate for the intended decision-support role, such metrics inherently reflect subjective judgments and may not fully capture all dimensions of system performance.

An additional limitation concerns the inherently non-deterministic nature of LLMs employed as reasoning components within the system. Despite the use of controlled prompts and structured orchestration mechanisms, LLM-based outputs may still exhibit variability across interaction sessions due to stochastic generation behavior and ongoing model updates. During system usage, alternative formulations and emphases in generated responses were occasionally observed. To mitigate the practical impact of this non-determinism, EKA adopts several design-level measures, including explicit task decomposition across specialized agents, clear separation between data selection, information extraction, and synthesis phases, and the preservation of intermediate reasoning artifacts that constrain downstream processing. Moreover, the human-in-the-loop interaction model ensures that generated outputs are interpreted, validated, and contextualized by claims handlers, preserving expert oversight over final judgments. Nevertheless, these mitigation strategies do not fully eliminate non-deterministic effects, which remain an intrinsic characteristic of LLM-based systems. As a result, some degree of variability in system behavior may persist over time, requiring continuous monitoring, validation, and governance to ensure consistent and reliable operation in regulated insurance environments. This limitation highlights the importance of maintaining human accountability and reinforces the role of EKA as a decision-support and reasoning aid rather than an autonomous decision-making system.

Finally, the findings are derived from a single organizational context. Although the underlying design principles are transferable, further studies across different organizations and domains are required to assess generalization. Also, the evaluation focuses on short-term interactions within a defined set of cases and does not assess longer-term effects such as changes in user behavior, learning curves, or sustained adoption over time. Longitudinal studies would be required to better understand how systems like EKA influence expertise development, decision consistency, and organizational practices in the long run.

5. Conclusions

This paper presented EKA, a collaborative AI-based MAS designed to support large insurance claims handling through enterprise knowledge reuse. EKA integrates structured and unstructured views of historical communications data within a human-centered decision-support framework, positioning the system as a sparring partner that augments expert reasoning rather than automating decisions. The use case considered in this study is drawn from large claims handling operations within Allianz Commercial, providing a realistic industrial context for system design and evaluation.

The case study involved 150 real large claims, leading to an evaluation that suggested how EKA provides practical value to claims handlers. User-centered metrics indicate high perceived usefulness, rapid access to initial analytical insights, and efficient interaction patterns, highlighting the system’s ability to reduce the effort to explore complex and historically distributed information while preserving human control and accountability. Beyond the specific implementation, the results provide indications regarding the potential applicability of the collaborative multi-agent AI architectures in knowledge-intensive and regulated domains. By abstracting access to data sources and reducing reliance on specialized technical skills, EKA supports a more horizontal distribution of analytical capabilities across users, contributing to more consistent and transparent decision-making practices. Rather than aiming to provide algorithmic performance benchmarks, this work contributes practice-oriented evidence from a real industrial deployment, illustrating how collaborative multi-agent AI architectures can be effectively integrated into complex, regulated decision-making environments.

While the study is subject to limitations related to architectural choices, evaluation scope, and data availability, it provides empirical evidence within the studied context of how AI-based MASs can be effectively deployed in real industrial settings. More broadly, this work suggests that collaborative AI-based MASs can serve as effective cognitive partners in complex organizational decision-making, provided that architectural design, data integration, and human-centered interaction are treated as first-class concerns. Finally, although the presented findings provide empirical support within the evaluated industrial context, cross-organizational studies and controlled comparative evaluations are required to rigorously assess the transferability and generalization of the proposed architecture. Future work will therefore explore alternative agent configurations, additional sources of organizational expertise, and longitudinal evaluations to further assess the impact of such systems on expertise development and operational performance.