Developing an AI Assistant for Knowledge Management and Workforce Training in State DOTs

Abstract

Effective knowledge management is critical for preserving institutional expertise and improving the efficiency of workforce training in state transportation agencies. Traditional approaches, such as static documentation, classroom-based instruction, and informal mentorship, often lead to fragmented knowledge transfer, inefficiencies, and the gradual loss of expertise as senior engineers retire. Moreover, given the enormous volume of technical manuals, guidelines, and research reports maintained by these agencies, it is increasingly challenging for engineers to locate relevant information quickly and accurately when solving field problems or preparing for training tasks. These limitations hinder timely decision-making and create steep learning curves for new personnel in maintenance and construction operations. To address these challenges, this paper proposes a Retrieval-Augmented Generation (RAG) framework with a multi-agent architecture to support knowledge management and decision-making. The system integrates structured document retrieval with real-time, context-aware response generation powered by a large language model (LLM). Unlike conventional single-pass RAG systems, the proposed framework employs multiple specialized agents for retrieval, answer generation, evaluation, and query refinement, which enables iterative improvement and quality control. In addition, the system incorporates an open-weight vision-language model to convert technical figures into semantic textual representations, which allows figure-based knowledge to be indexed and retrieved alongside text. Retrieved text and figure-based context are then provided to an open-weight large language model, which generates the final responses grounded in the retrieved evidence. Moreover, a case study was conducted using over 500 technical and research documents from multiple State Departments of Transportation (DOTs) to assess system performance. The multi-agent RAG system was tested with 100 domain-specific queries covering pavement maintenance and management topics. The results demonstrated Recall@3 of 94.4%. These results demonstrate the effectiveness of the system in supporting document-based response generation for DOT knowledge management tasks.

1. Introduction

Knowledge management (KM) is a set of systematic organizational processes and tools for acquiring, organizing, sustaining, applying, sharing, and maintaining tacit and explicit knowledge so as to enhance organizational and personal performance [1]. In the transportation industry, effective KM is essential for maintaining operational efficiency, ensuring safety, and supporting strategic asset management. Transportation projects often span long timelines and involve intricate processes that depend on a well-organized body of knowledge, from technical specifications to regulatory requirements and maintenance records [2]. Knowledge management systems (KMS) provide engineers with quick access to relevant, up-to-date information, which enables them to maintain and repair critical infrastructure that ensures public safety and service continuity [3]. Furthermore, as infrastructure ages and transportation systems expand, the need to retain institutional knowledge becomes critical, particularly when key personnel retire or leave the organization. This also supports asset management by preserving data related to asset life cycles, maintenance schedules, and repair histories, which are instrumental in planning and resource allocation [4]. By utilizing KM, the transportation industry can improve knowledge retention, reduce reliance on individual expertise, and foster a more resilient and adaptable workforce.

Despite its importance, KM in the transportation industry faces several significant challenges. One of the primary obstacles is the reliance on manual processes for capturing and organizing knowledge, which often leads to inconsistencies and inefficiencies in data accessibility and usage [5]. The transportation sector typically generates vast amounts of data, including technical documents, maintenance logs, safety protocols, and regulatory updates, but much of this information remains stored in unstructured or semi-structured formats that are difficult to retrieve and analyze effectively [6]. Furthermore, the industry’s dependence on the expertise of seasoned professionals presents a risk when these experts retire, as critical institutional knowledge may be lost. Hence, the use of artificial intelligence (AI) technologies has been suggested to mitigate the limitations of traditional KM methods in transportation sector [2].

Advancements in AI models have significantly enhanced KM across various industries. For example, IBM Watson’s decision-support capabilities have been used in healthcare to deliver real-time information, assisting in complex decision-making scenarios where healthcare workers require fast access to critical data [7]. Google’s DeepMind has shown substantial potential in improving human decision-making in technical environments where immediate access to accurate information is essential [8]. Adaptive learning systems, such as Microsoft Azure’s AI-powered learning platforms, have advanced the personalization of training experiences by dynamically adjusting content based on individual performance metrics. Lim and Lee [9] highlight how these platforms allow workers to address skill gaps, which is particularly beneficial in fields where continuous upskilling is required.

Recently, large language models (LLMs) have been investigated in knowledge retrieval, particularly in reducing the time workers spend searching for information. Morandini et al. [10] found that chatbots built on these models significantly enhance on-the-job efficiency by quickly retrieving contextually relevant information. However, concerns regarding the accuracy of responses from AI models have been raised [11]. For example, Jaiswal et al. [12] emphasize ethical considerations, suggesting that while self-learning platforms can support human work, they should ideally complement rather than replace human expertise to avoid ethical and job-related issues.

While recent advancements in LLMs have improved information retrieval efficiency, their reliability remains a concern in high-stakes or knowledge-intensive domains. Consequently, the RAG paradigm has emerged as a promising solution by coupling LLMs with verifiable external knowledge sources [13]. RAG overcomes the limitations of LLMs, i.e., outdated information, hallucinations, and absence of domain knowledge. RAG does this by using the strengths of retrieval-based and generation-based approaches. The technique involves applying an information retrieval (IR) system to retrieve relevant information from a knowledge base and feeding it into an LLM to produce contextually correct responses. This combination of the LLM with actual-world information makes it more trustworthy and adaptable for different uses [14]. While previous RAG systems are based on pre-encoded content and embedding vectors, an agentic RAG model enriches data sources with descriptive metadata, allowing an agent to dynamically search through diverse data pools. Agentic RAG enhances RAG functionality by using an agent in the construction of proper retrieval actions, mimicking human logical processes for improved accuracy and use in output [15]. This adaptability and accuracy in data retrieval, particularly in complex and dynamic information environments, makes agentic RAG unique compared to normal RAG.

Despite recent advances, there remains a gap in deploying LLM-enabled knowledge tools in transportation settings, particularly for State DOT workforce training where domain-grounded accuracy is critical. Prior work often relies on single-pass RAG without explicit quality control, limited multi-step query refinement, or agent specialization. To address this gap, we develop a multi-agent RAG assistant for knowledge management and workforce training that (i) structures retrieval, generation, evaluation, and query refinement into specialized agents, (ii) grounds answers in agency documentation to reduce hallucinations, and (iii) reports transparent performance metrics under realistic workloads. In addition, (iv) this study extends conventional text-only RAG by treating figures as retrievable knowledge units rather than auxiliary illustrations, which enables visual evidence in State DOT technical documents to be indexed, retrieved, and directly influence question answering. Unlike prior RAG systems that primarily rely on textual descriptions or ignore figures altogether, the proposed approach allows performance trends, comparative analyses, and expert judgments embedded in figures to actively contribute to retrieval and response generation. The retrieved text and figure-based context are then provided to the Qwen3-4B-Instruct-2507 model, which generates the final responses grounded in the retrieved evidence.

2. Literature Review

2.1. KM in Infrastructure and Engineering

KM refers to the systematic processes and tools for acquiring, organizing, sharing, and maintaining both tacit and explicit knowledge within an organization [1]. In engineering-intensive sectors such as transportation and public infrastructure, effective KM is essential for maintaining operational efficiency, ensuring safety, and supporting strategic asset management. Transportation projects often generate vast bodies of knowledge, from technical specifications and standards to maintenance logs and regulatory requirements, which must be well-organized and readily accessible to engineers and decision-makers. By implementing KM systems, infrastructure agencies can provide personnel with quick access to up-to-date information needed to maintain and repair critical assets (e.g., roads, bridges) while upholding public safety and service continuity [2]. In the transportation domain, several State DOTs have also reported formal efforts to deploy knowledge management systems to support workforce continuity and operational decision-making. For example, National Cooperative Highway Research Program [16] documents knowledge management initiatives across multiple State DOTs, highlighting challenges related to document-heavy workflows, workforce turnover, and decentralized information repositories. These studies emphasize the need for scalable, searchable KM solutions that can operate over large collections of technical reports, manuals, and guidelines typical of DOT environments.

One driving concern for KM in high-complexity, public-sector organizations is the impending retirement of senior experts and the consequent risk of knowledge loss. As infrastructure networks age and seasoned engineers leave, capturing their tacit know-how and lessons learned becomes critical. KM initiatives help convert this tacit institutional knowledge into organizational assets, preserving data on asset life-cycles, maintenance history, and best practices for future generations of engineers. This not only reduces reliance on any single individual’s expertise but also fosters a more resilient and adaptable workforce capable of handling complex operations [4]. Early efforts in the transportation domain demonstrated the value of formal KM approaches. For example, Hammer et al. [5] explored distributed knowledge management architectures for transportation agencies. Cheung et al. [17] conducted a knowledge audit in a transit organization to identify critical knowledge assets and gaps. These studies highlighted longstanding challenges such as siloed information, fragmented documentation, and inconsistent knowledge capture practices.

Modern AI technologies are now transforming how organizations address these KM challenges. A key issue has been the heavy reliance on manual or ad-hoc processes for capturing and organizing knowledge, which leads to inefficiencies in data retrieval and use. Moreover, much of the knowledge in infrastructure agencies resides in unstructured formats (e.g., lengthy reports, PDFs, email communications), which makes it difficult for staff to retrieve relevant information in a timely manner. To tackle these problems, researchers have proposed integrating AI into KM systems [18]. For example, natural language processing and knowledge graph techniques can automatically index and interpret large volumes of engineering documents. Xu et al. [19] demonstrated an ontology-based KM system for highway construction inspection, which structured project knowledge in a machine-readable form to improve information exchange and support decision-making in the field.

Among emerging AI advances, LLMs and RAG techniques have gained significant attention for KM applications. RAG is an approach that combines an LLM’s natural language understanding with an external knowledge base or document repository [20]. In a RAG pipeline, when a user poses a query, the system first retrieves relevant documents or snippets from the organization’s knowledge base, and then the LLM generates a context-aware answer grounded in that retrieved information. By integrating the LLM’s output in trusted agency documents, RAG can overcome many limitations of standalone LLMs (such as hallucinations or outdated knowledge) and produce more accurate, domain-specific responses. Researchers are also examining advanced architectures, such as multi-agent or agentic RAG systems [21]. In these systems, specialized agents manage different subtasks. For example, one agent analyzes the query intent, another retrieves focused information from various data sources, and another validates the LLM’s answer. These steps aim to mimic human reasoning and enhance the reliability of results.

2.2. AI-Enhanced Workforce Training and Upskilling

Traditionally, agencies have relied on a mix of classroom-based instruction, apprenticeships or mentoring, and static documentation to train new engineers and technicians. These conventional methods, however, often result in fragmented knowledge transfer and inconsistent skill development. In large public agencies (e.g., state DOTs), new hires face steep learning curves due to the sheer complexity of operations and the decades of tacit knowledge held by senior staff. Compounding the challenge, as veteran employees retire, much of their experiential knowledge may leave with them, leading to a “brain drain” that training programs struggle to fill. This problem is especially acute in high-risk, safety-critical fields like infrastructure maintenance, where mistakes by an untrained workforce can have serious consequences. Indeed, recent surveys have found that employees often spend up to 20% of their work week searching for information or consulting colleagues for help on the job [22]. The need for more efficient, continuous learning solutions is evident.

AI-based tools are increasingly being explored to augment workforce training and bridge these knowledge gaps. One promising direction is the use of conversational agents and intelligent tutoring systems powered by large language models. Instead of attending formal classes, trainees can interact with an AI assistant that provides on-demand guidance, answers technical questions, and offers scenario-based problem solving. For example, researchers have developed prototype chatbots using state-of-the-art LLMs fine-tuned on an organization’s internal documents and manuals. Lee et al. [22] report a domain-specific conversational system for engineering document review: by fine-tuning GPT and LLaMA models on thousands of pages of in-house technical specifications, their system could automatically answer engineers’ questions about project requirements with a high degree of accuracy.

Beyond simple Q&A, AI-driven training platforms can personalize learning pathways and adapt to each employee’s progress. Adaptive learning systems use AI to dynamically adjust training content based on individual performance metrics. If a trainee is struggling with a particular procedure or topic, the system can recognize this and provide additional exercises or explanatory material targeting that weakness. Conversely, if an employee demonstrates mastery, the AI can recommend more advanced modules or even suggest practical on-the-job assignments to apply the skill. This level of personalization was traditionally impossible to achieve at scale in large organizations, but AI makes it feasible to deliver a tailored training experience for every worker. Gorowara et al. [23] describe an AI-powered training and reskilling framework that constructs individualized learning paths for employees, using machine learning to select optimal training content and pacing for the digital age workforce.

When integrating generative AI tools (like LLMs) into training environments, researchers stress the importance of maintaining accuracy, trust, and ethics. Ozmen Garibay et al. [11] cautions that heavy reliance on AI recommendations without proper checks can introduce socio-technical risks. In high-risk fields, even a small mistake in training (e.g., a misconstrued safety procedure) can have serious real-world repercussions. Therefore, successful implementations have human experts in the loop: AI-generated answers or training content are validated against authoritative sources or reviewed by instructors, especially in the initial deployment phases. Establishing clear oversight protocols helps build trainees’ confidence in the system. Moreover, a theme in recent studies is that AI should augment human mentors, not replace them [12]. While an AI coach can handle routine queries and provide 24/7 support, human trainers are still crucial for teaching nuanced hands-on skills, imparting organizational culture, and providing the empathetic encouragement that algorithms cannot.

Although prior studies on AI-enhanced knowledge management and training platforms have demonstrated the potential to capture, structure, and deliver organizational knowledge more efficiently, there remains limited empirical research on implementing such AI-driven systems within state transportation agencies. This disconnect from the operational realities of DOT environments, where technical documents are extensive, heterogeneous, and continuously updated, limits their practical value. This study addresses these gaps by introducing a multi-agent RAG framework that grounds responses in agency documentation, integrates query refinement and answer evaluation, and demonstrates its effectiveness using a large corpus of State DOT technical and research materials.

3. Methodology

This section presents the methodology used to develop the proposed multi-agent RAG framework for knowledge management and workforce training in State DOT environments. The methodology consists of four main components: preprocessing and indexing of heterogeneous document content, including both text and figures; a retrieval-augmented generation pipeline for grounding responses in agency documentation; an agentic workflow that enables structured retrieval, generation, evaluation, and query refinement; and the design of specialized agents that coordinate these tasks. The following subsections describe each component in detail.

3.1. Figure Preprocessing

RAG systems rely on text-based representations to perform embedding, similarity search, and retrieval. Consequently, visual elements such as figures, charts, and plots embedded within technical documents cannot be directly processed by standard RAG pipelines. Unlike plain text, figures are not natively represented as semantic tokens and therefore cannot be embedded or retrieved unless they are first converted into a textual form. Without explicit preprocessing, critical technical information conveyed visually such as performance trends, comparative evaluations, threshold values, and practitioner judgments would be excluded from retrieval and reasoning.

State DOT technical documents contain both textual and visual information, where figures often serve as the primary medium for communicating analytical results, model performance comparisons, and expert assessments. In many cases, these insights are only partially described, or not described at all, in the surrounding text. To capture this information in a machine-retrievable form, figures are first extracted from PDF documents and processed using a vision-language model.

The objective of figure preprocessing is to translate both the visual content and its associated contextual metadata into structured natural-language representations that preserve the semantic meaning of the original figure. Specifically, the vision-language model is prompted to extract (i) the figure caption which provides contextual grounding for the visual content, and (ii) a detailed figure description derived from the visual elements themselves. The generated descriptions include information about (a) the type of visualization (e.g., bar chart, line plot, pie chart), (b) the variables shown on each axis, including units and scales, (c) key quantitative relationships and trends such as increases, decreases, or relative differences, (d) categorical comparisons across methods, treatments, or conditions, and (e) qualitative interpretations conveyed through legends, annotations, or practitioner labels. Figure 1 illustrates the structured prompt used to guide this caption and description generation process. The resulting caption description pairs are treated as first-class knowledge units and indexed alongside textual document chunks in the vector database. By converting figures into semantically rich textual representations that combine contextual labels with visual evidence, the RAG pipeline can apply the same embedding, similarity search, and retrieval mechanisms to both figure-based and text-based information. This preprocessing step enables figure-derived knowledge to be retrieved and used directly during question answering, ensuring that important technical information encoded in figures is not overlooked.

Figure 1. Prompt used for vision-language-based figure description generation.

3.2. RAG

RAG is a hybrid framework that integrates external knowledge retrieval with generative modeling to enhance the accuracy and trustworthiness of large language models. The process begins with raw documents $D=\{d_1, d_2, \dots , d_N\}$, such as technical manuals, reports, or articles. Each document $d_i$ is split into smaller, coherent chunks $c_{ij}$, where i denotes the document index and j represents the chunk index. The resulting set of chunks $C=\left\{c_{ij}\right\}$ ensures that each text chunk is manageable and semantically meaningful.

Each text chunk $c_{ij}$ is transformed into a high-dimensional vector embedding $e_{ij}$ using an embedding model E:

$$e_{ij}=E\left(c_{ij}\right),e_{ij}\in {\mathbb{R}}^d$$

(1)

where,

• $E\left(c_{ij}\right)$ = Embedding function mapping the chunk $c_{ij}$ to its vector representation;
• $e_{ij}$ = Embedding of the chunk;
• d = Dimensionality of the embedding space.

The embeddings for all chunks in a document $d_i$ are grouped together as:

$$E_i=\{e_{i1}, e_{i2}, \dots , e_{im}\}$$

(2)

where $E_i$ represents the set of embeddings for the chunks of document $d_i$.

These embeddings for all documents are stored in the Knowledge Base V, which is defined as:

$$V=\{E_1, E_2, \dots , E_N\}$$

(3)

where,

• V = The knowledge base containing all chunk embeddings.

When a user submits a query q, it is embedded into a vector representation $e_q$ using the same embedding model:

$$e_q=E\left(q\right),e_q\in {\mathbb{R}}^d$$

(4)

The query embedding $e_q$ is compared with the document embeddings $e_{ij}$ stored in the knowledge base V to identify the most relevant chunks. Similarity metrics, such as cosine similarity, are used to measure relevance:

$$\mathrm{cosine\_similarity}(e_q,e_{ij})={\displaystyle \frac{e_q·e_{ij}}{\parallel e_q\parallel \parallel e_{ij}\parallel }}$$

(5)

Alternatively, Euclidean distance (L2) can be used to calculate the nearest vector:

$$\mathrm{distance}(e_q,e_{ij})=\parallel e_q-e_{ij}\parallel$$

(6)

The retrieved chunks are concatenated with the query q to form the context, which is the input for the language model. The context vector is represented as:

$$\mathrm{context}=[q, c_{\mathrm{top}\text{-}1}, c_{\mathrm{top}\text{-}2}, \dots , c_{\mathrm{top}\text{-}k}]$$

(7)

The language model G (e.g., Qwen3-4B-Instruct-2507) processes the input context to generate a response r:

$$r=G\left(\mathrm{context}\right)$$

(8)

The generative model predicts the response by modeling the probability distribution over the possible sequence of tokens:

$$P(r\mid \mathrm{context})=\prod _{t=1}^{T}P(r_t\mid r_{1:t-1},\mathrm{context})$$

(9)

where,

• r = The generated response;
• $r_t$ = The t-th token of the response;
• $P(r_t\mid r_{1:t-1},\mathrm{context})$ = The probability of generating token $r_t$ given the prior tokens and the context.

The output r is presented to the user as the final answer, completing the RAG process.

3.3. Agentic RAG

The traditional RAG approach performs well for most use cases but has limitations, such as the lack of iterative refinement and multi-step processing, which could enhance answer quality. To address this, we used an agentic RAG system where multiple agents collaborate to improve the response generation process. The system begins with a User Proxy Agent that captures user input, followed by a Retriever Agent that fetches relevant documents. A Generator Agent then formulates the initial response, which is assessed by an Evaluator Agent for accuracy and relevance. If necessary, a Query Refiner Agent further optimizes the query to refine the retrieval process.

Figure 2 illustrates three RAG approaches: Multi-Agent RAG, Standard LLM, and Agentic RAG, each varying in how queries are processed, information is retrieved, and responses are generated. In this paper, the proposed tool implemented in this work is based on the Multi-Agent RAG approach, where multiple specialized agents collaborate to refine the response generation process. It begins with an initial agent receiving the user query, which is then passed to a retrieval agent to fetch relevant context. Next, a generator agent formulates the response, followed by an evaluation agent that assesses its quality based on predefined criteria. If necessary, a refinement agent further improves the response. In contrast, the Standard LLM approach simply retrieves relevant context and feeds it, along with the query, to a large language model for response generation. The Agentic RAG approach extends standard RAG by introducing task-specific agents that enhance both retrieval and response generation.

Figure 2. Multi-Agent RAG, Standard LLM, and Agentic RAG architectures for retrieval and response generation.

The process begins by loading the documents, segmenting them into chunks, generating embeddings, and storing them in the database. The Retriever Agent then retrieves relevant context based on the given query, ensuring that only documents exceeding a predefined relevance threshold are selected. The retrieved context is passed to the Generator Agent, which extracts the answer along with any additional relevant details, such as examples or historical background, if available. The response is then formatted according to the specified structure before being sent to the Evaluator Agent, which assesses it based on predefined criteria, such as whether it is beginner-friendly. If the answer meets the required standards, it is presented as the final output, and the process ends. However, if the response is unsatisfactory, a refinement loop is triggered, where the Query Refiner Agent modifies the query based on feedback from the Evaluator Agent. To prevent infinite looping, query refinement occurs at most k times. If the system is still unable to generate a satisfactory response after k iterations, it displays a predefined message indicating that the answer could not be found. This automated workflow ensures a structured response, with each agent specializing in a specific task to enhance efficiency and accuracy.

3.4. Agent Design

In this research, we utilized the AutoGen framework to build the multi-agent system and facilitate communication between agents. AutoGen, developed by Microsoft Research, is an advanced framework designed for constructing multi-agent AI systems that interact, collaborate, and solve complex problems using LLMs. It provides a structured approach to creating AI agents capable of autonomous communication, task planning, and execution.

The query-answering process begins with a User Proxy Agent, followed by a set of Assistant Agents, including the Retriever Agent, which is specifically instructed to obtain context using the retrieve_contexts function. Next, the Generator Agent extracts the answer and formats it as required before passing it to the Evaluator Agent for assessment. If necessary, the Query Refiner Agent further refines the query based on feedback. In the following sections, we will provide a detailed discussion on the prompts used for each agent.

User Proxy Agent: The User Proxy Agent is the initial agent in the multi-agent RAG system. When user input is received, it is first processed by the User Proxy Agent before being passed to subsequent agents. Its primary role is to receive human input and relay it to downstream agents without modification or additional processing. By acting as a seamless intermediary, it ensures a smooth workflow throughout the system. Figure 3 illustrates the prompt for the User Proxy Agent. As shown, its functionality is clearly defined as to accept user queries and relay them to downstream agents without modification.

Figure 3. System prompt for the UserProxyAgent.

Assistant Agent: In the AutoGen framework, the assistant agent is a specialized component designed to handle specific tasks within a multi-agent system. It can be programmed with defined functions and behaviors, making it highly adaptable for various use cases. The assistant agent interacts with other agents, processes instructions, and executes designated tasks. It can be configured using prompts, function-calling capabilities, and other customization options. In this implementation, assistant agents were responsible for query processing, document retrieval, response generation, and evaluation. Within the Multi-Agent RAG approach, four distinct assistant agents were created and configured to optimize the system’s workflow. The retriever agent called the retrieval function to fetch relevant context from the document database. The generator agent extracted relevant information from the retrieved context, formulated a response, and structured it in a predefined format. The evaluator agent assessed the generated response to ensure quality and relevance. Finally, the query refiner agent refined the initial query based on feedback from the evaluator and, if necessary, retriggered the retrieval process to improve the response.

Retriever Agent: The Retriever Agent is responsible for fetching relevant context from the database based on the given query. Its primary function is to call the retrieve_contexts function, which performs a similarity-based search to identify and retrieve the most relevant context. Once retrieved, the context is passed to the subsequent agents in the pipeline, specifically the Generator Agent, ensuring a smooth and efficient workflow. Figure 4 illustrates the prompt for the Retriever Agent, which is explicitly instructed to call the retrieve_contexts function as its sole responsibility. After retrieving the context, it is passed to the downstream generator agent, which is responsible for generating the response.

Figure 4. System prompt for the RetrieverAgent.

Generator Agent: The Generator Agent is responsible for structuring responses in a clear and organized format. It follows specific instructions on formatting the context retrieved by the Retriever Agent and is explicitly guided to provide beginner-friendly responses. Figure 5 illustrates the prompt for the Generator Agent, which outlines its key responsibilities. It must extract the most relevant answer from the retrieved context, ensuring it is concise and directly addresses the user’s query. If the context includes relevant examples, trends, or comparisons, the agent summarizes them. Any background information related to the query topic is also included. Also, the agent formulates follow-up questions based on the query topic to enhance user engagement. It is responsible for citing the sources used in its response and is strictly instructed to generate answers solely from the provided context without incorporating external knowledge. The agent maintains a simple and clear tone, and if the retrieved context does not provide sufficient details, it explicitly states that the available information is insufficient to answer the query.

Figure 5. System prompt for the GeneratorAgent.

After extracting the response from the context and formatting it according to the given instructions, the response is then forwarded to the evaluator agent.

Evaluator Agent: The Evaluator Agent is responsible for assessing the response generated by the Generator Agent based on clarity, relevance, and completeness. It also ensures that the answer is beginner-friendly. If the response meets the defined evaluation criteria, the agent marks it as satisfactory. Otherwise, it deems the response unsatisfactory, provides feedback, and forwards it to the Query Refiner Agent for further refinement. Figure 6 illustrates the prompt for the Evaluator Agent, clearly outlining the evaluation criteria and specifying the next step based on whether the response is deemed satisfactory or unsatisfactory. Once the evaluator agent completes its assessment, it either determines the response to be satisfactory and returns the final answer or, if deemed unsatisfactory, triggers the query refinement loop by passing the evaluation and feedback to the query refiner agent.

Figure 6. System prompt for the EvaluatorAgent.

Query Refiner Agent: The sole responsibility of the Query Refiner Agent is to refine the query based on feedback from the Evaluator Agent. This agent is essential when the original query is ambiguous or not well-defined. By addressing these challenges, the Query Refiner modifies the query and passes it to the Retriever Agent, triggering the pipeline again. Figure 7 illustrates the prompt for the Query Refiner Agent, highlighting its primary function of modifying the query based on the provided feedback.

Figure 7. System prompt for the QueryRefinerAgent.

The retrieved text and figure-based context are then passed to the Qwen3-4B-Instruct-2507 model, which produces the final answers.

4. Case Study

This section presents a case study to evaluate the proposed system. It is divided into three parts: data description, experimental setup, and results discussion.

4.1. Data Description

In this case study, documents collected selected if they (1) were issued or sponsored by state DOTs, federal transportation agencies, or recognized research institutions; (2) focused on pavement maintenance, pavement management systems, materials, rehabilitation strategies, or related pavement decision-making topics; and (3) contained substantive technical content such as design guidance, experimental results, performance models, or implementation practices. Documents that were non-technical, administrative in nature, duplicated across repositories, or lacked sufficient technical depth were excluded. This process resulted in a curated corpus of 521 authoritative documents representative of practical and research-driven pavement management knowledge. Document lengths vary considerably, from technical manuals of just one or two pages to comprehensive reports extending up to 281 pages. Most documents fall within the range of 40 to 130 pages, with a notable portion exceeding 150 pages. The dataset draws on materials from a wide range of state DOTs, national repositories, and international sources. Texas is prominently represented due to the breadth of research and documentation available through TxDOT. In addition, the dataset incorporates data and policy documents from states including California, Kansas, Georgia, North Carolina, Washington, Illinois, South Carolina, Virginia, Oregon, South Dakota, Pennsylvania, Ohio, Missouri, Louisiana, Florida, Utah, Michigan, Maine, Minnesota, Maryland, Tennessee, and New Mexico.

The topics covered span critical aspects of infrastructure planning and management. Many documents focus on the development and evaluation of Pavement Management Information Systems (PMIS), including the calibration of performance prediction models for asphalt, continuously reinforced concrete, and jointed concrete pavements. Several reports examine revisions to utility curves, updates to decision-tree trigger criteria, and integration of non-destructive testing data into PMIS workflows. Others address comparative analyses between district-level rehabilitation and repair needs versus PMIS-recommended actions. Broader topics include asset management strategy, traffic data analysis, surface drainage modeling with LiDAR, skid resistance evaluation, four-year pavement planning processes, right-of-way acquisition, and the implementation of structural condition indices for network-level pavement evaluation. Some documents also detail specific treatments such as diamond grinding and explore material-related concerns.

Temporally, the dataset spans over three decades of transportation research and policy development. The oldest documents date back to the 1990s, while the most recent were published in 2024. The majority were released between 2000 and 2023, offering a comprehensive view of both historical evolution and contemporary practices. Many reports cite explicit publication years, such as 2006, 2009, 2010, 2012, 2016, 2017, 2018, 2020, and 2022, and several guidelines and manuals appear in updated editions, with the latest versions from 2023 and 2024.

Based on the collected documents, we extracted a set of recurring technical questions that reflect practitioners’ decision needs in pavement management, which focus on treatment service life, cost, application timing, and performance impacts. Table 1 summarizes representative questions used for subsequent analysis.

Table 1. Questions extracted from the curated pavement-management corpus.

No.	Key Investigative Questions
Surface Treatments: Service Life, Cost, and Application
1–7	What is the expected service life of: chip seal, micro-surfacing, scrub seals, fog seal, seal coat, thin overlays, and light texturing?
8–14	What is the typical cost associated with these surface treatments?
15–21	Under what specific weather, traffic, and pavement conditions should these treatments be applied?
Rehabilitation & Recycling: Strategic Planning
22–33	What is the expected service life for major interventions (e.g., HIR, CIR, FDR, HMA overlays, and various repair/patching methods)?
34–45	What are the unit costs for these rehabilitation and recycling strategies?
46–57	What are the critical triggers and conditions for applying these major treatments?
Performance Modeling & Effectiveness
58–64	What are the observed deterioration rates following surface and HMA overlays?
65–71	How is the long-term effectiveness of these treatments quantified?
Concrete Pavement & Technical Triggers
72–78	What are the texturing methods, use cases, and performance impacts for concrete? What are the specific triggers for Diamond Grinding?
System Integration: PMIS Modeling & Data Governance
79–90	How are performance models calibrated for different pavement types (Asphalt, CRCP, JCP), and how are utility curves or NDT data integrated into workflows?
91–100	Comparison of district-level needs vs. PMIS recommendations, multi-year planning, network-level metrics (skid resistance), and data quality governance.

4.2. Experiment Setup

Table 2 summarizes the key implementation choices and system parameters used in the case study.

Table 2. Implementation details and configuration settings of the proposed multi-agent RAG system.

Item	Specification
Embedding model	all-MiniLM-L6-v2
Embedding dimensionality	384
Evaluation metric	Precision@3, Recall@3
Chunk size	1200 characters
Chunk overlap	0 (no overlap)
Vector database	ChromaDB
Similarity metric	Cosine similarity
Relevance threshold	Not applied (top-k results always returned)
Vision-language model	Qwen3-VL-2B-Instruct
Response generation model	Qwen3-4B-Instruct-2507

4.2.1. Preprocessing Figures

Figure 8a shows pavement performance comparisons in terms of surface texture, skid resistance, roughness, and noise, while Figure 8b shows practitioner opinions summarizing model reliability. These figures were represented as caption-description pairs that capture both contextual labels and detailed quantitative and qualitative interpretations. The figure-derived representations were embedded using the same embedding model as standard text chunks and stored together in the vector database, which enables the RAG system to retrieve and reason over both text-based and figure-based information when answering user queries.

Figure 8. VLM Description of Figures. (a): Bar charts adapted from Buddhavarapu et al. [24]. Qwen3-VL-2B-Instruct generated description: This figure presents four bar charts illustrating the effectiveness of diamond grinding on different pavement surfaces. The top-left chart compares surface texture measured by MTD and MPD, showing that the ground surface has higher values than the unground surface for both metrics. The top-left chart evaluates skid resistance, indicating that the ground surface provides better performance than the unground surface. (b): Pie chart adopted adapted from Gharaibeh et al. [25]. Qwen3-VL-2B-Instruct generated description: This figure is a pie chart that displays the opinions of TxDOT’s practitioners regarding newly-developed pavement performance prediction models. The chart is divided into five segments, each representing a different level of prediction accuracy with corresponding percentages. The largest segment, colored green, accounts for 60% and represents “Reasonable” predictions.

4.2.2. Evaluation Metrics

The system was evaluated using 100 domain-specific questions (Table 1). For each question, an LLM (Qwen3-4B-Instruct-2507) was prompted to assess every retrieved document and determine whether it was relevant to the query. The LLM-assigned relevance labels were then manually reviewed and verified against ground truth. Retrieval quality was measured using Precision@3 and Recall@3, where Precision@3 measures how many of the top three retrieved documents are relevant, and Recall@3 measures how many of relevant documents are found within the top three results returned by the system.

$$\mathrm{Recall}@3={\displaystyle \frac{\mathrm{Number} \mathrm{of} \mathrm{relevant} \mathrm{documents} \mathrm{in} \mathrm{the} \mathrm{top} 3 \mathrm{retrieved}}{\mathrm{Total} \mathrm{number} \mathrm{of} \mathrm{relevant} \mathrm{documents}}}$$

(10)

$$\mathrm{Precision}@3={\displaystyle \frac{\mathrm{Number} \mathrm{of} \mathrm{relevant} \mathrm{documents} \mathrm{in} \mathrm{the} \mathrm{top} 3 \mathrm{retrieved}}{3}}$$

(11)

4.3. Results Discussion

To evaluate the performance of the proposed system, we tested it using 100 domain-specific queries (Table 1) and compared the system’s responses against manually verified ground truth. Retrieval quality was measured using Precision@3 and Recall@3. The system achieved a Precision@3 of close to 1.0 and a Recall@3 of 94.4%. A Precision@3 of 1.0 indicates that all documents retrieved within the top three results were relevant, while a Recall@3 of 94.4% indicates that nearly all relevant evidence was retrieved within the top three results. Together, these results show that the system consistently retrieves accurate and sufficient contextual information for downstream answer generation.

To assess result stability, retrieval metrics were computed across multiple queries and reported as macro-averages. Variability across queries was observed but remained consistent, which indicates stable retrieval behavior across different topics. After retrieval, the top-ranked text chunks and figure-derived descriptions were provided as context to the Qwen3-4B-Instruct-2507 model, which generated the final responses grounded in the retrieved evidence.

Single-Pass RAG vs. Multi-Agent RAG

To understand the contribution of the proposed multi-agent design, a study was conducted comparing a single-pass RAG baseline with the multi-agent RAG system. Both systems used the same document corpus, embedding model, and large language model, which ensures that the observed differences are due to the retrieval and reasoning strategy rather than model capacity. Retrieval performance was evaluated using top-k recall and precision. Using the same set of 100 domain-specific evaluation queries, the single-pass RAG system achieved a Recall@5 of 58%, which indicates that relevant documents were frequently missing even when up to five documents were retrieved. In contrast, the proposed multi-agent RAG system achieved a Recall@3 of 94%, meaning that relevant evidence was successfully retrieved within the top three results for nearly all queries. The improvement in recall is due to the multi-agent workflow, where retrieval is refined through evaluator feedback rather than executed only once. When an initial response lacks sufficient evidence, the Query Refiner Agent reformulates the query and triggers additional retrieval steps, allowing the system to recover relevant documents that are commonly missed in a single-pass setup. This comparison shows that the multi-agent RAG system finds important documents more reliably than a single-pass RAG by refining queries when key information is missing, which leads to better evidence retrieval for DOT-related questions.

5. Conclusions

This paper presented a multi-agent RAG system designed to support knowledge management and workforce training in State DOTs. By separating retrieval, response generation, evaluation, and query refinement into specialized agents, the system is able to produce structured and document-grounded answers rather than relying on single-pass language model outputs. Using a corpus of more than 500 technical and research documents from multiple State DOTs and a set of 100 domain-specific queries related to pavement maintenance and management, the system achieved a Precision@3 of 1.0 and a Recall@3 of 94.4%, which indicates that all documents retrieved within the top three results were relevant and that nearly all relevant evidence was successfully retrieved within this top-ranked set.

The primary limitation of the current study lies in the scope of the dataset. The proposed system is evaluated using pavement-related documents, which limits its applicability to other transportation domains. As a result, the findings primarily reflect the system’s effectiveness within the pavement engineering context and may not fully generalize to broader transportation infrastructure areas such as traffic operations, safety analysis, or bridge management.

This system is designed to operate under low-cost deployment constraints using standard computing infrastructure without specialized hardware or on-premise servers. Document ingestion is performed periodically, and figure-based information is currently incorporated through manual extraction and description generation rather than fully automated pipelines. While this approach reflects realistic State DOT workflows and avoids the complexity and cost of hard automation, it limits scalability and requires human involvement (e.g., manual screenshot capture of figures). These constraints highlight practical trade-offs between cost, automation, and scalability, and motivate future work on incremental automation of figure processing as deployment resources permit.

Future work will focus on expanding the dataset beyond pavement engineering to include a wider range of transportation domains. Incorporating reports related to traffic engineering, roadway safety, asset management, and multimodal transportation systems will enable a more comprehensive evaluation of the framework’s generalizability. This expansion will help assess the system’s adaptability across diverse transportation applications and support its potential use as a unified knowledge assistant for transportation agencies.