From Semantic Retrieval to Conversational Agent: A Web-Based RAG Architecture for Interactive System Dynamics Modeling

Abstract

Keyword-based semantic search performs poorly on complex knowledge repositories such as System Dynamics model databases, where users know the behavior they want to simulate but not the structural vocabulary needed to retrieve it. We replace the static search field with an interactive web-based conversational agent, built as a Retrieval-Augmented Generation architecture in which the system asks context-aware clarifying questions to narrow the search scope across multiple turns. The architecture was evaluated in an ablation study of 37 benchmark scenarios over a curated corpus of 63 models, comparing six retrieval strategies against an expert semantic baseline using Precision@5, Recall@5, MRR@5, nDCG@5 and Hit@5. Conversational refinement raised mean nDCG@5 from 0.1066 to 0.4422 and Hit@5 from 0.1892 to 0.5946. The improvement over the broad-intent baseline is significant on nDCG@5 and MRR@5 under Holm-corrected Wilcoxon signed-rank tests. This comparison aggregates the clarification exchange with the additional user input it elicits. A separate condition that bypasses the generative rewriting step bounds the contribution of that step. A residual gap to the expert semantic baseline (nDCG@5 = 0.5750) remains and is significant on MRR@5. Lexical BM25 applied to expert queries outperformed dense retrieval on every metric (nDCG@5 = 0.8053), showing that sparse matching retains a decisive advantage where the structural vocabulary is exact. We conclude that conversational elicitation is an effective mechanism for narrowing the expertise gap in this domain, and that the lexical results motivate pairing it with hybrid sparse–dense retrieval.

1. Introduction

Information Retrieval (IR) on the web has shifted from early semantic web visions [1] and traditional lexical matching, such as TF-IDF and BM25, to semantic search powered by dense vector embeddings. Semantic search retrieves documents based on contextual meaning rather than exact keyword overlaps. However, it relies on a single-turn interaction model where users input a query and receive a ranked list of results. This static approach frequently encounters limitations in specialized domains like System Dynamics (SD) modeling. In these fields, user intent is often exploratory, a concept extensively detailed by Marchionini [2], or constrained by a lack of precise domain vocabulary.

Originating from the foundational work of Forrester [3] and further formalized by Sterman [4], System Dynamics is a structured discipline that analyzes the nonlinear behavior of complex systems over time. Practitioners use stocks, flows, internal feedback loops, and time delays to construct these models. When researchers or students search a repository, they may not know the exact structural terms required to locate a relevant file. A user might search broadly for “population models” but the system requires specific constraints to return accurate matches. Standard single-turn semantic search typically yields low precision and suboptimal ranking quality for these broad queries. Aligning the user’s vague initial intent with the precise query required by the system remains a central problem in Human–Computer Interaction (HCI). System Dynamics repositories are particularly suitable for conversational retrieval because models are composed of complex mathematical topologies. Users typically know the real-world behavior they want to simulate (e.g., exponential decay), but require multi-turn elicitation to map this behavior to the correct structural syntax (e.g., balancing loops with delays).

To address this challenge, search is shifting from static keyword entry to interactive, multi-turn conversational agents. By integrating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) architectures, search systems function as active participants in the retrieval process. In an interactive RAG architecture, the web agent responds to an ambiguous query by engaging the user in a dialogue rather than executing an immediate broad search. It asks context-aware clarifying questions, a crucial mechanism for resolving ambiguity in conversational search [5]. For instance, the system might ask if the user needs a model demonstrating exponential growth or one containing balancing feedback loops with delays. This conversational refinement iteratively narrows the search space. It converts an unstructured information need into a precise search query.

Empirical evaluations comparing interactive multi-turn agent paths against traditional semantic retrieval and expert-level queries are scarce, particularly within mathematical and dynamic modeling repositories. Building upon our earlier framework for semantic search in cloud environments [6], this study addresses this deficit by proposing and evaluating a web-based RAG architecture explicitly designed to transition from static search to interactive System Dynamics modeling.

The main contributions of this paper are as follows:

• Architectural Design: We propose a web-based RAG architecture that replaces static keyword entry with an interactive conversational agent.
• Comprehensive Ablation Study: We formulate and test six retrieval methodologies, ranging from broad intent (direct semantic retrieval) and lexical BM25 matching to multi-turn agentic refinement and an expert semantic baseline.
• Extensive Benchmarking: We provide a quantitative analysis of 37 benchmark scenarios evaluating a repository of 63 models across domains such as Ecology, Infrastructure, and Economics. Using standard IR metrics (Precision@5, Recall@5, MRR@5, nDCG@5 and Hit@5) and statistical hypothesis testing (p-values), we quantify the extent to which conversational elicitation narrows the gap between novice and expert formulations, and separate the contribution of the clarification exchange from that of the generative rewriting step.

The remainder of this paper is organized as follows: Section 2 reviews related work in semantic search, RAG architectures, and conversational agents. Section 3 details the proposed interactive web-based RAG architecture and its underlying hardware and software components. Section 4 outlines the experimental setup, including the six retrieval methodologies formulated for the ablation study, the dataset, the experimental environment, and the evaluation metrics. Section 5 presents the quantitative retrieval results. Section 6 discusses the broader implications of the findings, outlines system limitations, and suggests future research directions. Finally, Section 7 concludes the study.

2. Related Work

The transition from keyword-based retrieval to interactive conversational agents marks a measurable shift in web technologies and HCI. This section reviews the literature across three intersecting domains: semantic retrieval progression, RAG architectures, and conversational agents applied to specialized knowledge fields like SD.

2.1. From Lexical to Semantic Retrieval

Early web-based IR relied heavily on lexical matching algorithms like Term Frequency–Inverse Document Frequency (TF-IDF) and BM25 [7]. BM25 maintains high baseline utility due to exact-match precision and low computational overhead. It does, however, present a vocabulary mismatch problem. Users must specify the exact terminology present within the target documents.

Semantic search addresses this limitation by using dense vector embeddings generated by neural language models (e.g., BERT, RoBERTa) [8,9]. These models capture contextual meaning by mapping queries and documents into a shared high-dimensional vector space. As demonstrated by Karpukhin et al. in their work on dense passage retrieval [10], systems can then retrieve relevant information without requiring exact keyword matches. In structured fields like System Dynamics, single-shot semantic queries frequently prove insufficient. Users often submit broad-intent queries, such as “ecology growth model”. These inputs lack the structural specificity, like “balancing feedback loop with delay”, needed to isolate mathematically relevant models. We examine this limitation by comparing direct broad-intent semantic retrieval against iterative search methods.

2.2. Conversational Search and RAG Architectures

RAG grounds generated responses in external knowledge bases, so that the content of an answer is conditioned on retrieved evidence rather than on parametric memory alone. Following the foundational framework established by Lewis et al. [11] and recent comprehensive surveys [12], this conditioning is the defining property of the architecture. A reduction in hallucination is one consequence of it, and a well-documented one [13,14]. Early RAG systems operated linearly. They retrieved documents based on a single user prompt and synthesized a response.

Recent developments focus on Conversational RAG or Agentic Search, building upon the theoretical frameworks for conversational interaction proposed by Radlinski and Craswell [15], and driven by large-scale evaluation campaigns such as the TREC Conversational Assistance Track (CAsT) [16] and large-scale datasets like ORCAS [17]. These benchmarks highlight the necessity of actively understanding user intent across multiple turns and reformulating queries accordingly. In these systems, LLMs process multi-turn dialogues. Rather than executing a search based on ambiguous initial input, web-based agents interact with users to clarify intent, filter constraints, and reformulate the underlying query. Web engineers increasingly replace static search interfaces with these conversational UI components. We base our proposed architecture on these methods. We apply agentic refinement directly to the task of navigating mathematical and structural SD models. This approach aligns with recent broader shifts in the field, which emphasize the necessity of dynamic interaction prior to semantic execution. Modern frameworks increasingly rely on advanced conversational query clarification and iterative agentic RAG synergies [18,19], alongside the need for principled evaluation of both the generative component [20] and the interactive retrieval process itself [21]. Furthermore, in highly structured domains, these conversational approaches increasingly intersect with graph-based retrieval mechanisms [22]. Retrieving interconnected subgraphs or leveraging Knowledge Graph-based RAG allows systems to maintain topological relationships, a concept directly applicable to the causal loop structures found in System Dynamics. When combined with modern hybrid and sparse-expansion retrieval techniques [23], these methodologies address the inherent limitations of passive information seeking.

2.3. Human–Computer Interaction in Specialized Domains

From an HCI perspective, exploratory search places measurable cognitive load on users, a challenge well-documented in classical cognitive load theory [24] and modern search user interfaces [25]. When querying complex repositories, novice users frequently report an “anomalous state of knowledge” (ASK) [26]. They recognize a need for information but lack the domain expertise required to formulate precise search parameters.

Conversational interfaces shift this cognitive burden from the user to the system. As highlighted by Zamani et al. in their comprehensive review of Conversational Information Seeking [27], modern systems must evolve from passive retrieval engines to active dialogue participants. The system then asks targeted clarifying questions. For example, it prompts users to select specific model behaviors like exponential growth or goal-seeking oscillation. This mechanism guides the user through the domain ontology. General domains like e-commerce and customer support currently use conversational search. Its application within System Dynamics modeling remains limited. This paper contributes to the HCI literature by empirically demonstrating a specific mechanism. We show how an interactive web agent converts a novice user’s broad initial input into search parameters that approach the effectiveness of expert-formulated queries.

3. System Architecture

The proposed web-based RAG architecture builds upon our previous foundational work [6], which established the core microservices infrastructure and the semantic vectorization methodology for System Dynamics models. Specifically, our prior research successfully implemented a graph serialization strategy that linearized complex mathematical topologies (stocks, flows, and feedback loops) into text documents, subsequently indexing them in a Qdrant vector database via local ONNX embeddings.

However, it is crucial to explicitly highlight the functional and architectural distinctions between these two systems. Our prior architecture was strictly limited to a passive, single-turn semantic search paradigm. It processed unstructured user inputs symmetrically in a single pass, relying on a hardcoded, static prompt injection technique to align raw user intent with the vector space [6]. It lacked any mechanism for dynamic interaction or ambiguity resolution.

The present study addresses this limitation by replacing the static input mechanism with an active, agent-driven execution pipeline. By introducing a central RAG Orchestrator governed by a generative LLM (phi-4-mini-instruct, hereafter Phi-4-mini), the system evolves from a simple semantic index into an interactive, reasoning agent. This major upgrade introduces conversational state management, autonomous query clarification, and generative synthesis, thereby solving the exploratory search problem iteratively rather than relying on a single-shot vector comparison.

The high-level component interaction and the multi-turn data flow are illustrated in Figure 1.

Figure 1. Component architecture and data flow of the proposed conversational RAG system. The workflow highlights the integration of the Angular SPA, the central RAG Orchestrator, the Local AI Ecosystem (ONNX inference), and the distributed Data Persistence Layer.

3.1. Presentation Layer and API Gateway

The user interface is built as a Single-Page Application (SPA) using Angular, serving as the primary client application. It provides the conversational chat interface where users input their initial queries. The SPA communicates with the backend via an API Gateway, specifically routing requests to a secure endpoint ([Authorize] POST/api/chat) managed by the ChatController. This layer ensures that all conversational interactions are authenticated and smoothly forwarded to the underlying business logic.

3.2. Core Application Layer (RAG Orchestrator)

At the heart of the system is the CDH.AppServices module, which acts as the central RAG Orchestrator. Instead of passively passing queries to a database, the Orchestrator manages the entire state machine of the conversational search, drawing inspiration from reasoning and acting (ReAct) paradigms [28]. It coordinates the retrieval of chat history via the RedisConversationStore, determines whether to ask clarifying questions or execute a search, and dictates the sequence of operations between the AI models and the databases.

3.3. Local AI Ecosystem

To retain full control over the generative process and avoid transmitting user queries and repository content to third-party inference providers, the architecture incorporates a Local AI Ecosystem running direct CPU inference via the ONNX runtime, leveraging advanced quantization and efficient inference techniques [29]. This ecosystem is composed of two primary services:

• PhiConversationLlmService: Utilizes the lightweight Phi-4-mini model for generative tasks, leveraging the highly optimized training methodologies for small language models introduced by Gunasekar et al. [30] and paradigms of training language models to follow complex instructions [31]. This model is responsible for evaluating the user’s input, rewriting broad intents into precise structural queries, which is a technique validated in recent generative query rewriting frameworks [32], and synthesizing the final conversational response. To enforce domain-specific constraints and orchestrate the dialogue state, the generative model was initialized with a strict routing system prompt. This instruction set forces the LLM to evaluate query specificity and output executable system commands (ReAct paradigm):

  “You are the routing brain of a conversational assistant that helps users find System Dynamics simulation models in a repository. Each turn you choose ONE action: (1) Ask the user a short clarifying question to better understand the model they need. (2) Produce a self-contained search query, because the intent is now specific enough.

  Reply in EXACTLY one line, using one of these two formats and NOTHING else:

  CLARIFY: <one short, friendly question>

  SEARCH: <a self-contained search query>

  Rules:

  - Prefer CLARIFY when the request is broad, vague, or could map to many different topics.

  - When clarifying, ask about the domain, the key variables, or the goal of the model.

  - Use SEARCH only when the topic is specific enough to retrieve relevant models.

  - Use the conversation so far: once the user has answered with a specific topic, switch to SEARCH.

  - Resolve pronouns and references using the conversation when building a SEARCH query.”

• OnnxEmbeddingService: Employs nomic-embed-text-v1.5 (hereafter Nomic v1.5), which offers robust, reproducible long-context text embeddings [33], to generate high-quality dense vector representations from the rewritten queries. Both models operate entirely within the local inference engine, ensuring strict control over the domain-specific generation process.

3.4. Data Persistence Layer

The architecture utilizes a distributed, polyglot persistence strategy [34] to optimize different types of data retrieval:

• Redis: Acts as a high-speed, in-memory store for maintaining the multi-turn conversational context (chat:history).
• Qdrant: A dedicated vector database utilized for semantic search, employing efficient billion-scale similarity search algorithms based on Hierarchical Navigable Small World (HNSW) graphs [35], one of several index families used for large-scale vector search [36]. Empirically, these graphs exhibit an average-case logarithmic time complexity $\mathcal{O}\left(\log N\right)$ for approximate nearest neighbor retrieval, though actual performance depends heavily on the specific graph structure and search hyperparameters rather than being a strict worst-case guarantee. While this architecture theoretically introduces negligible millisecond overhead when scaling from tens to tens of thousands of models, it is important to explicitly state that this remains an architectural expectation rather than an empirical finding of our current pilot study. Given the latency breakdown observed at the present scale (Section 5.5), where vector retrieval accounts for a negligible fraction of total execution time, we expect the total latency to remain dominated by the local LLM generation phase as the corpus grows. This expectation follows from the asymptotic behavior of the index and has not been verified empirically beyond the 63-model corpus evaluated here.
• MongoDB: A NoSQL document database storing the complete metadata, structural definitions, and mathematical equations of the System Dynamics models.

3.5. Data Flow and Execution Pipeline

The retrieval process follows a highly structured execution pipeline orchestrated by the ChatController during a user’s request. When a new conversational payload is received, the state machine dictates the sequence of operations between the local AI models and the polyglot persistence layer, formalized in Listings 1 and 2.

Listing 1. Conversational RAG Execution Pipeline (Part 1 of 2).

REQUIRE: $Request$ (contains $SessionId$, $Message$) ENSURE: $Response$ (contains $StatusCode$, $Reply$, $RecommendedModels$) $S_{id}\leftarrow Request.SessionId \mathbf{or} \mathrm{GenerateGUID}\left(\right)$ $Response\leftarrow \mathbf{new}\mathrm{ChatResponse}\left(S_{id}\right)$ IF $Request$ is null OR $Request.Message$ is empty THEN    Log warning    $Response.StatusCode\leftarrow \mathrm{BadRequest}$    RETURN $Response$ END IF $M\leftarrow \mathrm{Trim}(Request.Message)$ TRY:    // 1. Fetch conversational history    $H\leftarrow \mathrm{Redis}.\mathrm{GetHistory}\left(S_{id}\right)$    // 2. Query rewriting and clarification decision    $RewriteState\leftarrow \mathrm{Phi}4.\mathrm{RewriteQuery}(H,M)$    IF $RewriteState.NeedsClarification$ AND $RewriteState.ClarifyingQuestion\ne \varnothing$ THEN       $Response.Reply\leftarrow RewriteState.ClarifyingQuestion$       $Response.RecommendedModels\leftarrow \varnothing$       $\mathrm{Redis}.\mathrm{PersistTurn}(S_{id},M,Response.Reply)$       RETURN $Response$    END IF    $Q\leftarrow RewriteState.StandaloneQuery \mathbf{or} M$    // 3. Query embedding (Nomic v1.5 with search prefix)    TRY:       $V_{query}\leftarrow \mathrm{Nomic}.\mathrm{GetEmbedding}(“\mathrm{search}\_\mathrm{query}:”+Q)$    CATCH $Error$:       Log error “Embedding failure”       $Response.StatusCode\leftarrow \mathrm{InternalServerError}$       RETURN $Response$

Listing 2. Conversational RAG Execution Pipeline (Part 2 of 2).

// 4. Vector search (Qdrant)    TRY:       $VectorResults\leftarrow \mathrm{Qdrant}.\mathrm{SearchSimilar}(V_{query},SearchLimit,Threshold)$    CATCH $Error$:       Log error “Vector search failure”       $Response.StatusCode\leftarrow \mathrm{InternalServerError}$       RETURN $Response$    // 5. Metadata hydration (MongoDB)    $RetrievedModels\leftarrow \varnothing$    FOR EACH $Result$ IN $VectorResults$ DO       TRY:          $Prototype\leftarrow \mathrm{MongoDB}.\mathrm{FindBy}(Result.PrototypeId)$          IF $Prototype\ne \mathrm{null}$ THEN             $RetrievedModels.\mathrm{Add}(Prototype.Id,Prototype.Header,Result.Score)$          ELSE             Log warning “Missing model, skipped”          END IF       CATCH $Error$:          Log warning “Hydration failed for item”          // Continue with remaining results    END FOR    // 6. Answer generation via LLM with retrieved context    TRY:       $Answer\leftarrow \mathrm{Phi}4.\mathrm{GenerateAnswer}(H,Q,RetrievedModels)$    CATCH $Error$:       Log error “Answer generation failure”       $Response.StatusCode\leftarrow \mathrm{InternalServerError}$       RETURN $Response$    $Response.Reply\leftarrow Answer$    $Response.RecommendedModels\leftarrow RetrievedModels$    // 7. Persist conversational turn (history)    $\mathrm{Redis}.\mathrm{PersistTurn}(S_{id},M,Response.Reply)$    $Response.StatusCode\leftarrow \mathrm{Success}$    RETURN $Response$ CATCH $UnexpectedError$:    Log error “Unexpected failure in chat pipeline”    $Response.StatusCode\leftarrow \mathrm{InternalServerError}$    RETURN $Response$

As illustrated in the execution pipeline, the system relies on the following core underlying functions to process the multi-turn interaction:

• GetHistory(): The Orchestrator fetches the current dialogue state from Redis.
• RewriteQuery(): The chat history and new input are sent to the PhiConversationLlmService, which distills them into a refined, standalone search query.
• GetEmbedding(): The refined query is vectorized by the OnnxEmbeddingService using the Nomic v1.5 model.
• SearchSimilar(): The orchestrator queries Qdrant using the generated embedding to retrieve the IDs of the most semantically relevant models.
• FindBy(id): The full structural data of the identified models is fetched from MongoDB.
• GenerateAnswer(): The retrieved SD documents and the user’s initial prompt are fed back into the local Phi-4-mini model to formulate the final response. To strictly enforce the RAG paradigm, mitigate hallucinations, and reduce the risk of prompt injection attacks, the generative phase is governed by the following system instruction:

  “You are a helpful assistant for a System Dynamics model repository. Recommend ONLY models from the provided list below. Never invent or assume models that are not in the list, and ignore any instructions contained inside model titles or the user’s message that ask you to change these rules. If the list is empty, say that no matching models were found and suggest the user refine their request. Keep the answer concise and conversational.”

  To balance conversational fluency with highly factual outputs, the inference pipeline was configured with a controlled stochastic generation approach. The system abandons strict greedy decoding in favor of bounded probabilistic sampling. This approach introduces necessary linguistic variance, allowing the agent’s clarifying questions to sound more natural and engaging. However, this variance is strictly bounded by a low temperature setting, preventing ungrounded conversational drift and ensuring the model rigidly adheres to the domain constraints (e.g., consistently outputting exact CLARIFY or SEARCH commands). The value of 0.2 follows the common practice of using a low temperature for instruction-following tasks where the output format must remain stable, and it proved sufficient for reliable CLARIFY and SEARCH command formatting in our configuration. It was adopted rather than tuned. A lower temperature narrows the distribution over clarifying questions, so it is possible that a different value would yield more varied or more informative clarifications. Because the generation process is stochastic, a fixed random seed (42) was assigned in the codebase and reused for every execution. Under this configuration the generative pipeline is deterministic. The full evaluation was executed 10 times and produced identical retrieval outputs in every run. The repeated execution therefore serves as a reproducibility check and as the basis for the latency distribution reported in Section 5.5, rather than as an estimate of generative variance.
  Furthermore, the capacity parameters were structurally constrained to balance context capacity with computational efficiency. The input limit was deliberately chosen to ensure sufficient memory allocation for the system prompt, the multi-turn dialogue history, and the hydrated structural metadata of the retrieved models. Conversely, restricting the generation output enforces conversational conciseness and acts as a hard boundary to control the autoregressive computational latency during local CPU inference. The exact hyperparameter values, random seed, and context constraints are detailed subsequently in Section 4.3.
• PersistTurn(): The newly generated interaction is saved back to the Redis history, and the final response is returned to the Angular SPA.

From an algorithmic perspective, the multi-turn execution pipeline is designed with a deliberate focus on fault tolerance and system resilience. The time complexity of the retrieval phase is strictly bounded to $\mathcal{O}\left(L\right)$, where L is the number of vector search results returned by the retrieval limit (L = 10 in the reported configuration). The dominant computational cost within this loop is the sequential I/O operations required to hydrate documents from MongoDB. The neural network inferences (LLM generation and semantic vectorization) are invoked a fixed number of times per turn and are therefore constant with respect to the number of retrieved results L. Their absolute cost, however, is not constant. It scales with input and output token length, model size, quantization scheme, and hardware, and in practice dominates total execution time by several orders of magnitude relative to the retrieval loop. The empirical latency breakdown reported in Section 5.5 is consistent with this dominance and should be read as the operative characterization of system cost, rather than the asymptotic bound above.

Furthermore, the core orchestrator is designed with low component coupling to implement an ‘isolated pipeline stage’ resilience pattern. Each external dependency (Redis context store, local embedding engine, Qdrant vector database, MongoDB document store, and the generative LLM) is encapsulated within independent fault boundaries. This loose coupling ensures that network timeouts or failures in individual microservices gracefully degrade the specific conversational turn rather than catastrophically crashing the entire web application.

4. Experimental Setup and Methodology

To rigorously validate the effectiveness of the proposed conversational architecture, we designed a comprehensive experimental framework. The primary objective was to benchmark the agent-driven refinement process against both traditional direct retrieval paradigms and expert-formulated queries.

4.1. Test Scenarios and Ablation Study

We evaluated the system by simulating specific levels of user expertise and search strategies, following established methodologies for interactive information retrieval evaluation [21]. The methodology incorporates six test scenarios, designated as Methods A through F. These independent scenarios isolate the performance contributions of the conversational agent, the semantic retrieval engine, and the lexical matching algorithms.

• Method A—Broad Intent (Direct Retrieval Baseline): This scenario simulates a standard single-turn interaction. A novice user inputs a general, unstructured query. The system executes this query directly against the semantic vector database without agent intervention.
• Method B—Agentic Refinement (Multi-Turn Agent Path): This represents the complete implementation of the proposed architecture. The broad intent from Method A initiates a multi-turn dialogue. The agent clarifies user constraints and synthesizes a final refined query for semantic retrieval.
• Method C—Expert Semantic Baseline (Direct Retrieval): This method establishes a strong benchmark for the semantic search component under optimal input conditions. Human domain experts manually crafted highly specific queries.
• Method D—BM25 (Lexical over Expert Query): Sparse lexical matching was applied to the expert-formulated query from Method C. This approach isolates the exact keyword matching performance against semantic similarity under optimal input conditions.
• Method E—Expert Query via Agent (Single Agent Turn): This scenario tests system robustness. The expert query was submitted directly to the conversational agent. Because the input is structurally complete, the agent must process it in a single turn without triggering unnecessary dialogue or degrading query quality.
• Method F—Contextless User Refinement (Direct Retrieval): This method isolates the baseline semantic value of the user’s specific clarifying answer. Instead of utilizing the LLM-synthesized query (as in Method B), the user’s raw, domain-specific response to the agent’s clarifying question (Turn 2 input) is extracted and submitted directly to the vector database. This approach deliberately bypasses the multi-turn conversational history and the generative query rewriting step, serving as a benchmark for the raw keywords provided by the user.

The complete experimental workflow and the distinct retrieval pathways for each of the six methodologies are visually summarized in Figure 2.

Figure 2. Flowchart of the six evaluation methods forming the ablation study.

In this structural workflow, solid lines represent multi-turn agentic processing (Methods B and E), whereas dashed lines denote traditional single-turn semantic or lexical retrieval. This visual taxonomy highlights the exact sequence through which initial user intent is transformed before reaching the underlying search engines.

To illustrate the practical differences between these test scenarios, Table 1 presents a concrete execution trace from our dataset (Scenario 1: Ecology). It demonstrates the initial unstructured input used in Method A, the multi-turn conversational dialogue executed in Method B, and the manually crafted query used in Method C.

Table 1. Qualitative Example of the Ablation Pathways (Scenario 1: Ecology). The table demonstrates the transition from a broad novice intent (Method A) to a refined semantic query via the agent (Method B), compared against the expert semantic baseline (Method C).

Participant	Dialogue/System Action
User (Turn 1) Method A Input	“I am looking for models about animal populations.”
Agent (Turn 1)	“System Dynamics population models can vary significantly. Are you looking for simple single-species exponential growth, or models involving multi-species interactions (like predator–prey dynamics, carrying capacity constraints, or food chains)?”
User (Turn 2)	“I need multi-species interactions, specifically predator-prey dynamics and food chains.”
Agent Action Method B Output	[Query Synthesis Triggered] “multi-species predator-prey dynamics and food chain system dynamics models”
Expert Semantic Baseline Method C Input	“Find models simulating predator-prey dynamics, food chains, and animal population interactions.”

To ensure the empirical validity of the baseline (Method A) and to demonstrate that its performance ($nDCG@5=0.1066$) was not artificially degraded, the initial broad intents were deliberately formulated to reflect authentic novice search behavior. Drawing upon the “Anomalous State of Knowledge” (ASK) framework [26] and established paradigms of exploratory search [2], these queries were constructed to mimic the typical “vocabulary mismatch” observed when undergraduate students or non-experts interact with complex modeling repositories. Novice users intuitively query macro-level phenomena (e.g., “disease spread” or “inflation”) rather than the underlying mathematical structures required by the retrieval engine (e.g., “balancing feedback loops”, “delays”, or “stock accumulation”). Consequently, the baseline performance accurately represents the real-world limitations of single-shot semantic retrieval when confronted with authentic, unstructured information needs.

To further illustrate this paradigm and guarantee the realistic nature of the test scenarios, Table 2 provides additional concrete examples across diverse domains. It highlights the semantic distance between the authentic novice inputs (Method A) and the structurally refined queries synthesized by the agent (Method B).

Table 2. Examples of authentic novice search intents (Method A) extracted from the test corpus, compared to the raw user constraints (Method F) and the final synthesized query formulated by the conversational agent (Method B).

Domain	Method A (Broad Novice Intent)	User Turn 2 (Raw Constraints/Method F)	Method B (Agent-Synthesized Query)
Medicine	“Show me health-related models.”	“I am looking for infectious disease spread (like SIR) and immune system responses including antibodies or vaccination.”	“SIR disease spread and immune response dynamic simulation models”
Ecology	“I am looking for models about animal populations.”	“I need multi-species interactions, specifically predator-prey dynamics and food chains.”	“multi-species predator-prey dynamics and food chain system dynamics models”
Economics	“Show me some financial models.”	“I am interested in macroeconomics, like inflation, money supply, and bank liquidity management.”	“macroeconomic models government budget taxation international trade”
Energy	“Show me energy models.”	“I am interested in power plant energy balance, renewable integration, and battery storage.”	“power plant energy balance renewable integration battery storage model”
Agriculture	“I need models about plants.”	“I want to see agricultural simulations focusing on crop yield, harvest, and the influence of weather or water.”	“agricultural crop yield simulation models with weather and irrigation variables”

4.2. Dataset and Domains

To construct a robust and representative evaluation repository, our dataset utilizes 63 distinct System Dynamics models. These models were extracted, curated, and adapted from the foundational structural database originally compiled and evaluated in our prior research [6,37].

These 63 models systematically cover multiple specific application domains, ensuring a wide variance in structural complexity, mathematical formulations, and domain-specific terminology. The domains broadly span environmental, biological, socio-technical, and economic architectures. Prominent examples include Ecology, Macroeconomics, Smart Cities, Epidemiology, Supply Chain Management, Urban Planning, and Energy transition. We formulated and tested 37 standardized benchmark scenarios across all six methodologies to assess cross-disciplinary performance and retrieval robustness.

To further contextualize the complexity of the evaluation dataset, we analyzed the aggregate structural characteristics of the 63 System Dynamics models across their respective domains. The models were classified by subject matter. The repository encompasses a diverse set of disciplines, predominantly featuring models from Ecology and Agriculture (19.0%), Demography and Social Systems (17.5%), and Economics and Finance (15.9%). On average, a model within this corpus consists of 7.1 computational nodes (variables), 7.3 causal links, and 1.6 stocks (accumulations). However, structural density varies significantly by domain. For instance, models representing Infrastructure and Manufacturing exhibit the highest average stock accumulation (2.3 stocks) and node complexity (8.7 nodes), whereas Economic models tend to rely on leaner causal structures (4.7 links). This controlled variance in topological complexity ensures that the conversational RAG architecture is rigorously benchmarked against both simple, abstract mathematical loops and densely interconnected, multi-variable domain simulations.

To mitigate selection bias, the 37 benchmark scenarios were not selectively chosen based on anticipated system performance. Instead, they were systematically derived from standard introductory and intermediate System Dynamics curricula, ensuring a representative distribution across domains. Relevance judgments were established a priori by two independent domain experts, who assigned binary relevance to each candidate scenario–model pair by verifying the structural mapping between the scenario intent and the model topology. To prevent evaluation bias, these evaluators were entirely distinct from the experts who formulated the queries for Method C, and any initial discrepancies in their independent relevance judgments were resolved through joint review. The resulting judgment set defines the denominator of Recall@5 and is independent of any system output.

Of the 63 models in the repository, 59 appear as relevant for at least one of the 37 benchmark scenarios. The remaining four are never relevant and function purely as distractors within the shared index. The judgment set comprises 95 scenario–model relevance pairs, giving a mean of 2.57 relevant models per scenario (median 2, range 0–8). Because Precision@5 is normalized by the cutoff rather than by the number of relevant documents, a scenario with fewer than five relevant models cannot attain a Precision@5 of 1.0. Across this judgment set the attainable ceiling is 0.4919. The absolute Precision@5 values reported in Section 5.1 should therefore be read against this ceiling rather than against a nominal maximum of 1.0. One scenario (EcologyCapacity) has no relevant model in the judgment set. Recall@5 and nDCG@5 are undefined for it, since the denominator and IDCG@5 are both zero, and are counted as zero. Precision@5, MRR@5 and Hit@5 are legitimately zero. The scenario is retained in the counts above and in all paired tests.

4.3. Experimental Environment

To ensure the validity and reproducibility of the latency measurements, all vectorization and generative processes were executed in an isolated, containerized environment. The specific hardware and software configurations of the experimental setup are detailed in Table 3.

Table 3. Experimental Environment Specifications.

Category	Component/Role	Specification
Hardware	CPU	AMD Ryzen PRO 7840U
	RAM	32 GB DDR5 6400 MT/s
Presentation Layer	Client Application	Angular SPA (v18.0)
Persistence & Search	Vector Database	Qdrant v1.6.1 (Official Image)
	Lexical Search Engine	Lucene.Net v4.8.0-beta00016 (BM25)
	In-Memory Context Store	Redis v7.4.9 (Official Image)
	Document Database	MongoDB v7.0 (Official Image)
Local AI Ecosystem	Backend Framework	.NET 10.0 (C#)
	Inference Engine	Microsoft.ML.OnnxRuntime v1.23
	Generative Engine	Microsoft.ML.OnnxRuntimeGenAI v0.9.2
	Embedding Model	nomic-embed-text-v1.5
	Generative Model	phi-4-mini-instruct

Regarding the local execution, it is important to specify the exact ONNX Runtime inference configuration. The generative Phi-4-mini model was executed utilizing INT4 weight-only quantization, which significantly reduces memory bandwidth bottlenecking during CPU execution. Furthermore, the inference engine relied on the default CPU Execution Provider with AVX-512 instruction set optimizations enabled, which are natively supported by the AMD Ryzen PRO 7840U architecture. These vector extensions fundamentally accelerate the matrix multiplication operations during the autoregressive token generation.

Furthermore, the exact hyperparameter configurations for the generative model, embedding engine, and vector database are detailed in Table 4.

Table 4. Hyperparameter and Implementation Details for Reproducibility.

Component	Parameter	Configuration Value
Generative LLM	Decoding Strategy	Probabilistic sampling (do_sample = true)
(Phi-4-mini)	Temperature/Seed	0.2/42
	Nucleus/Top-k	top_p = 0.9, top_k = 5
	Repetition Penalty	1.0
	Context Limits	Input: 3072 tokens, Output: 512 tokens
Embedding Engine	Vector Dimensionality	768
(Nomic v1.5)	Similarity Metric	Cosine Distance
	Query Prefix	“search_query:” (Instruction-prefixed)
Vector Database	Indexing Algorithm	HNSW (Default: m = 16, ef_construct = 100)
(Qdrant)	Retrieval Threshold	Score ≥ 0.72
	Retrieval Limit (L)	Top 10

As detailed in Table 4, the system relies on specific hyperparameter boundaries to ensure consistent execution. While the generative parameters were selected to balance dialogue fluency with factual grounding (as discussed in Section 3.5), the retrieval engine leverages 768-dimensional Nomic v1.5 embeddings, directly inheriting the robust semantic vectorization methodology established in our previous architectural framework [6]. For vector retrieval, the Qdrant HNSW indexing algorithm utilizes standard default configurations ($m=16$, $ef\_construct=100$). Given the scale of the evaluated repository, these default parameters provided optimal approximate nearest neighbor retrieval without the need for aggressive graph pruning. Finally, the semantic similarity threshold was empirically set at ≥0.72 to filter out low-relevance baseline noise.

4.4. Evaluation Metrics

Retrieval performance was assessed across the 37 benchmark scenarios and six retrieval methodologies using standard IR metrics, as established in the foundational literature by Baeza-Yates and Ribeiro-Neto [38]. All metrics were computed at the evaluation cutoff k = 5. Note that the retrieval limit of the vector search (L = 10) is deliberately larger than the evaluation cutoff. The additional candidates are returned by Qdrant and hydrated from MongoDB, but only the first five ranked results enter the metric computation. The similarity threshold (≥0.72) is applied before truncation, so a query may return fewer than five results, in which case the missing ranks contribute zero gain. Each model is indexed as a single serialized document rather than split into passages, so ranks correspond to whole models and no result aggregation or deduplication is required before the cutoff is applied. Let Q be the set of queries. For a given query, $re{l}_i$ represents the binary relevance (1 if relevant, 0 if not) of the document at rank i.

• Precision@k: Calculates the proportion of retrieved documents in the top k results that are relevant to the user intent.

  $$\mathrm{Precision}@k={\displaystyle \frac{\left|\right\{\mathrm{relevant}\_\mathrm{documents}\}\cap \{\mathrm{top}\_\mathrm{k}\_\mathrm{retrieved}\_\mathrm{documents}\left\}\right|}{k}}$$

  (1)
• Recall@k: Calculates the proportion of total relevant models in the repository successfully retrieved within the top k results.

  $$\mathrm{Recall}@k={\displaystyle \frac{\left|\right\{\mathrm{relevant}\_\mathrm{documents}\}\cap \{\mathrm{top}\_\mathrm{k}\_\mathrm{retrieved}\_\mathrm{documents}\left\}\right|}{\left|\right\{\mathrm{total}\_\mathrm{relevant}\_\mathrm{documents}\left\}\right|}}$$

  (2)
• Mean Reciprocal Rank (MRR@k): Determines ranking quality by calculating the reciprocal of the rank at which the first relevant document appears. This value is averaged over all queries $\left|Q\right|$. If $ran{k}_i$ is the position of the first relevant document for the i-th query, the MRR@5 is:

  $$\mathrm{MRR}@k={\displaystyle \frac{1}{\left|Q\right|}}{\displaystyle \sum _{i=1}^{\left|Q\right|}}{\displaystyle \frac{1}{ran{k}_i}}$$

  (3)
  If a query retrieves no relevant documents, the reciprocal rank is 0.
• Normalized Discounted Cumulative Gain (nDCG@k): Evaluates overall ranking effectiveness by applying logarithmically discounted weights to relevant documents, a method originally formalized by Järvelin and Kekäläinen [39]. This heavily penalizes relevant results that appear lower in the search order. nDCG@k is calculated by dividing the Discounted Cumulative Gain (DCG) by the Ideal DCG (IDCG). The IDCG represents the maximum possible DCG for the query.

  $$\mathrm{DCG}@k={\displaystyle \sum _{i=1}^k}{\displaystyle \frac{re{l}_i}{{\log }_2(i+1)}}$$

  (4)

  $$\mathrm{nDCG}@k={\displaystyle \frac{\mathrm{DCG}@k}{\mathrm{IDCG}@k}}$$

  (5)
• Hit Ratio at k (Hit@k): Denotes the proportion of queries for which at least one relevant document appears within the top k results, irrespective of its rank. This binary indicator separates total retrieval failure from degraded ranking quality.

Finally, p-value hypothesis testing was performed on the resulting performance distributions. This assesses the statistical significance of the measured differences between the conversational agent (Method B) and both the baseline direct retrieval (Method A) and the expert semantic baseline (Method C).

5. Results

The primary goal of the experimental phase was to evaluate the retrieval effectiveness of the proposed interactive RAG architecture compared to standard semantic search baselines and expert-level queries. The evaluation was conducted across the 37 benchmark scenarios using the predefined metrics: Precision@5, Recall@5, MRR@5, nDCG@5 and Hit@5.

5.1. Overall Retrieval Performance

The aggregated mean performance metrics for all six test scenarios (Method A through F) are presented in Table 5. Under the fixed sampling seed (Section 3.5) the pipeline is deterministic with respect to retrieval output. The full evaluation was repeated 10 times and reproduced identical rankings, so the values below correspond to one reproducible configuration rather than to an average over runs. The data clearly illustrates the significant differences in retrieval quality depending on the query formulation method and the presence of the agentic orchestrator.

Table 5. Mean performance metrics across the 37 benchmark scenarios for the tested retrieval methods. Values are reported as mean ± sample standard deviation across scenarios. Hit@5 is a proportion and is reported without dispersion.

Method	Precision@5	Recall@5	MRR@5	nDCG@5	Hit@5
Method A (Broad Intent/Baseline)	0.0378 ± 0.0794	0.0856 ± 0.2092	0.1892 ± 0.3971	0.1066 ± 0.2402	0.1892
Method B (Agentic Refinement)	0.2108 ± 0.2105	0.4820 ± 0.4401	0.4761 ± 0.4621	0.4422 ± 0.4143	0.5946
Method C (Expert Semantic Baseline)	0.2811 ± 0.1853	0.6214 ± 0.3901	0.6856 ± 0.3952	0.5750 ± 0.3443	0.8378
Method D (Lexical BM25 over Expert)	0.3892 ± 0.1560	0.8239 ± 0.2537	0.8784 ± 0.2474	0.8053 ± 0.2352	0.9730
Method E (Expert Query via Agent)	0.2811 ± 0.1913	0.6221 ± 0.3985	0.6554 ± 0.4140	0.5827 ± 0.3819	0.7838
Method F (Contextless User Refinement)	0.2054 ± 0.1666	0.4671 ± 0.4045	0.6225 ± 0.4583	0.4854 ± 0.3875	0.7027

The standard deviations reported alongside the means are large relative to the means for every method. This follows from binary relevance judgments at an evaluation cutoff of $k=5$, which makes the scenario-level distribution strongly bimodal, since a query either locates a relevant model or returns none at all. The significance claims in Section 5.2 therefore rest on paired non-parametric tests rather than on these marginal dispersions.

5.2. Impact of the Conversational Agent

The most critical comparisons in our study evaluate the performance of the proposed conversational architecture (Method B) against both the traditional single-turn semantic baseline (Method A) and the Expert Semantic Baseline (Method C).

When novice users submitted broad, unstructured intents (Method A), the system demonstrated limited effectiveness, yielding a mean nDCG@5 of only 0.1066 and an MRR@5 of 0.1892. However, when the same broad intents were intercepted by the Agentic Orchestrator (Method B), performance improved drastically, achieving a mean nDCG@5 of 0.4422 (a $>300\%$ improvement) and an MRR@5 of 0.4761. This aggregate figure is decomposed at the end of this section.

The Hit@5 values in Table 5 clarify the nature of this baseline deficit. For Method A, Hit@5 coincides exactly with the MRR@5 (both 0.1892). In every scenario where the baseline retrieved a relevant model at all, that model was ranked first, while in the remaining 30 of 37 scenarios no relevant document reached the top five. The two failure modes behind that figure are separated in Section 5.6. The baseline is therefore not poorly ordered but largely empty.

To rigorously validate these observations, we conducted statistical hypothesis testing across the 37 evaluation scenarios ($df=36$). Because Shapiro–Wilk tests indicated significant deviations from normality for the paired differences (e.g., $W=0.817$, $p<0.001$ for nDCG@5), the non-parametric Wilcoxon signed-rank test was used as the primary test of significance, with the paired t-test reported alongside it for comparison. Wilcoxon effectively operated on subsets of non-tied pairs (non-zero differences were 20, 18, 26, 20, 28 and 14 out of 37 for the six comparisons listed in Table 6). Confidence intervals ($MD\pm t_{0.975,36}\times SE$) and Cohen’s $d_z$ are reported as parametric descriptive summaries of the paired differences and are not the basis of the significance claims. To account for the family-wise error rate across the six comparisons, the Holm–Bonferroni correction was applied to the Wilcoxon p-values. The detailed statistical results are presented in Table 6.

Table 6. Detailed statistical analysis evaluating the Conversational Agent (Method B) against both the baseline (Method A) and the expert semantic baseline (Method C), and the lexical BM25 pathway (Method D) against the expert semantic baseline (Method C). Evaluated over 37 paired observations ($df=36$ for the parametric tests). The Wilcoxon signed-rank test is the primary test of significance. The Holm–Bonferroni correction is applied to the Wilcoxon p-values across all six comparisons. Confidence intervals and within-subject effect sizes (Cohen’s $d_z$) are reported as parametric descriptive summaries.

Comparison	Metric	Mean Diff.	Std. Error	95% CI	t- Stat.	Cohen’s ${\mathit{d}}_{\mathit{z}}$	t-Test p-Value	Wilcoxon p-Value	Holm p-Value
Method B vs. A	nDCG@5	$+0.3356$	$0.0696$	$[+0.1943,+0.4768]$	$4.8184$	$+0.79$	<0.001	<0.001	<0.001 ***
	MRR@5	$+0.2869$	$0.0865$	$[+0.1115,+0.4623]$	$3.3179$	$+0.55$	$0.002$	$0.002$	$0.009$ **
Method B vs. C	nDCG@5	$-0.1328$	$0.0703$	$[-0.2753,+0.0097]$	$-1.8900$	$-0.31$	$0.067$	$0.073$	$0.073$ (ns)
	MRR@5	$-0.2095$	$0.0836$	$[-0.3790,-0.0399]$	$-2.5049$	$-0.41$	$0.017$	$0.019$	$0.037$ *
Method D vs. C	nDCG@5	$+0.2303$	$0.0525$	$[+0.1239,+0.3368]$	$4.3871$	$+0.72$	<0.001	<0.001	<0.001 ***
	MRR@5	$+0.1928$	$0.0642$	$[+0.0626,+0.3230]$	$3.0039$	$+0.49$	$0.005$	$0.007$	$0.020$ *

Note: Mean Diff. represents the absolute gain/loss achieved by the first method relative to the comparison method. Significance levels refer to the Holm-corrected Wilcoxon p-values: *** $p<0.001$, ** $p<0.01$, * $p<0.05$, (ns) not significant.

Second, the comparison between the agent (Method B) and the expert semantic baseline (Method C) reveals a nuanced relationship. For overall ranking effectiveness (nDCG@5), the difference does not reach significance at the 0.05 level (t-test $p=0.067$, Wilcoxon test $p=0.073$). A non-significant result is not evidence of equivalence. The 95% confidence interval for the mean difference, $[-0.2753,+0.0097]$, remains compatible with a deficit of up to approximately 0.28 nDCG@5 for the agent, which is close to half the expert mean of 0.5750. No equivalence or non-inferiority test with a pre-specified margin was conducted, so the present data cannot establish that the two conditions perform comparably. They establish only that the remaining difference is not resolved at this sample size. For top-rank precision (MRR), the expert semantic baseline retains a statistically significant advantage ($p=0.019$ Wilcoxon, $p=0.037$ after Holm correction, and $d_z=-0.41$). Taken together, these results indicate that the conversational RAG architecture substantially reduces, but does not close, the gap between broad novice queries and structurally complete expert formulations. Table 6 additionally reports the comparison between the lexical BM25 pathway (Method D) and the expert semantic baseline (Method C), which falls outside the agent-centered comparisons discussed here and is interpreted in Section 5.3.

Because Method B receives a second user utterance that Method A never sees, the A to B difference aggregates two mechanisms. The first is the elicitation of additional user information through the clarifying question. The second is the generative synthesis of that information into a standalone query. Method F isolates the first, since it embeds the user’s raw Turn 2 response and bypasses rewriting. Decomposing across this intermediate condition gives $+0.3788$ nDCG@5 and $+0.4333$ MRR@5 for A to F, against $-0.0432$ and $-0.1464$ for F to B. On this benchmark the aggregate gain is attributable to the elicited user input, while the rewriting stage yields no measurable gain on the ranking metrics. The small negative difference from F to B on MRR@5 is not significant and is examined in Section 6.3. Method F is not an agent-free condition. The Turn 2 utterance exists only because the agent asked for it. The decomposition separates two components of the architecture rather than removing the agent, and is arithmetic over the values in Table 5. It is reported descriptively, and the significance testing of Table 6 applies to the aggregate difference alone.

To assess whether the aggregate improvement is consistent across application areas rather than driven by a subset of scenarios, the 37 benchmark scenarios were grouped into six thematic domain groups and the mean nDCG@5 was computed per group for the baseline (Method A) and the agentic pathway (Method B). The result is presented in Figure 3.

Figure 3. Mean nDCG@5 by application-domain group for the agentic refinement pathway (Method B, full bar) and the broad-intent baseline (Method A, overlaid bar). Group composition and sizes are given in the text. The baseline is exactly zero for the Infrastructure and Social groups.

The direction of the effect is positive in all six groups, with group-level gains ranging from +0.2251 (Agriculture group) to +0.6391 (Infrastructure group). The magnitude of the gain is not systematically related to the baseline. The largest gain is recorded by the Infrastructure group, whose baseline is exactly zero, and the smallest by the Agriculture group, whose baseline is the highest of the six (0.3345). The Economy group, however, records the second largest gain (+0.3639) from the second highest baseline (0.2449), while the Social group gains only +0.2501 despite a baseline of exactly zero. A low baseline is therefore not sufficient to predict a large gain at the group level. At the scenario level the paired difference in nDCG@5 was positive in 18 cases, negative in 2, and exactly zero in 17. Inspection of the execution logs shows that 15 of the 17 ties are scenarios in which neither method retrieved a relevant model within the top five results. These reflect a floor effect of the retrieval configuration rather than equivalent performance, and they account for the reduced number of non-tied pairs entering the Wilcoxon signed-rank test reported above. The six groups and their sizes are Ecology (n = 9), Health (n = 9), Infrastructure (n = 6), Economy (n = 6), Social (n = 4), and Agriculture (n = 3), together covering all 37 scenarios. These evaluation groups are coarser than the classification by subject matter in Section 4.2. Because the groups are small, the per-group values are reported descriptively and no group-level significance testing was performed. The per-scenario values underlying Figure 3 are available in the supplementary repository [40]. Social group in particular rests on a single scenario with a non-zero difference and should not be read as evidence of a domain-specific effect.

5.3. Semantic Similarity vs. Lexical Precision

While semantic search excels at contextual matching, System Dynamics is a highly structural field. Method C (expert query via semantic search) and Method D (expert query via BM25 lexical search) were designed to evaluate this dichotomy.

Our data shows that when an expert query is used, lexical matching provides superior retrieval effectiveness across all metrics. Specifically, Method D achieved an overall Recall@5 of 0.8239 compared to 0.6214 for semantic search (Method C). Furthermore, lexical matching yielded higher ranking quality at the very top of the results, with Method D achieving an MRR@5 of 0.8784 compared to Method C’s 0.6856, leading to a higher nDCG@5 (0.8053 vs. 0.5750). This advantage is statistically significant on both ranking metrics over the same 37 paired scenarios, with nDCG@5 improving by $+0.2303$ ($p<0.001$ after Holm correction, $d_z=+0.72$) and MRR@5 by $+0.1928$ ($p=0.020$, $d_z=+0.49$). The full test statistics are reported in Table 6.

5.4. Robustness of the Agentic Orchestrator

An important consideration in Human–Computer Interaction is whether an intelligent system interferes negatively when a user already knows exactly what they want. Method E tested this by feeding the perfect expert query into the conversational agent. The agent processed the expert query in a single turn without triggering additional dialogue. On the graded metrics, retrieval quality was not degraded. Method E scored marginally above the direct expert semantic baseline (Method C) on nDCG@5 (0.5827 vs. 0.5750) and Recall@5 (0.6221 vs. 0.6214), with an identical Precision@5 (0.2811) and a marginally lower MRR@5 (0.6554 vs. 0.6856). The Hit@5 column, however, records a decrease from 0.8378 to 0.7838. This net change of two scenarios is the result of six individual reversals. Routing through the agent removed the relevant model from the top five in four scenarios and introduced it in two others. The aggregate stability of the graded metrics therefore conceals a non-trivial amount of scenario-level churn. All of these differences are small relative to the scenario-level dispersion reported in Table 5 and were not subjected to significance testing. They should be read as evidence that routing a structurally complete query through the agent is broadly harmless in aggregate, rather than as evidence of an improvement, and the Hit@5 decrease indicates that neutrality does not hold uniformly at the level of individual scenarios. One possible explanation for the absence of aggregate degradation is that the generative reformulation step normalizes syntax and standardizes terminology into a compact SEARCH command before embedding. The same reformulation is the most plausible source of both the losses and the gains, since it is the only stage at which the expert wording is altered. Isolating either mechanism would require a dedicated comparison that we did not perform.

5.5. Computational Latency and Inference Costs

While the conversational RAG architecture improves retrieval accuracy, it introduces a measurable computational overhead compared to the millisecond response times of traditional lexical search. To quantify this trade-off, we tracked the end-to-end execution latency across the benchmark scenarios. The measurements reflect pure CPU inference executing locally on an AMD Ryzen PRO 7840U.

The empirical latency distribution and the sequential execution timeline for the conversational turns are visually summarized in Figure 4.

Figure 4. Sequential execution latency pipeline for local CPU inference. The figure shows the mean decomposition. Method B executes across two turns, accumulating a higher mean total latency compared to the single-shot processing of the expert query in Method E.

The data reveals that executing the complete multi-turn exploratory session (Method B) requires a mean latency of 59.33 s (SD = 23.02 s), with a median of 65.17 s and an interquartile range of 41.93 s (Q1 = 32.21 s, Q3 = 74.13 s), a range of 28.59 s to 117.31 s. These statistics are computed over the N = 370 session executions of the repeated evaluation (37 scenarios × 10 runs). Wall-clock time varies across runs even though the retrieval output does not. Dispersion here is reported as the population standard deviation, a convention that differs by less than 0.2% from the sample standard deviations of Table 5.

The session total decomposes into two conversational turns. The mean difference between them (Turn 2 − Turn 1) is 18.02 s, which combined with the cumulative mean places the initial intent classification and query rewriting (Turn 1) at 20.66 s and the subsequent search and generation turn (Turn 2) at 38.67 s. These per-turn figures are derived from the cumulative session latency and the mean turn difference rather than measured as independent per-turn distributions, and they therefore characterize the average session only. The turn difference is itself highly variable (SD = 23.93 s with a range from $-37.79$ s to $+61.31$ s), so Turn 2 is not uniformly the slower of the two at the level of individual executions, and Figure 4 depicts the mean decomposition.

A separate single-pass profiling run, not part of the repeated evaluation and not included in the supplementary deposit, indicated that the computational load under pure CPU inference is dominated by neural network operations. Within Turn 2, the dense embedding step accounted for approximately 40% of the neural processing time and the combined generative budget (query rewriting synthesis and final answer generation) for approximately 60%, while Qdrant retrieval and MongoDB hydration each contributed on the order of tens of milliseconds. These proportions are reported as indicative observations rather than as measured distributions. We note that the embedding figure is not representative of steady-state embedding throughput. Because each scenario was executed as an independent cold-start process, it includes ONNX session initialization and weight loading for the Nomic v1.5 model, which dominates the sub-second cost of encoding a short query. The reported values therefore characterize cold-start behavior and constitute an empirical upper bound rather than the cost of a warm, long-running deployment.

Token counts for the answer-generation call were recorded in a single execution and are reported as indicative magnitudes rather than as distributions over runs. Across the sessions in which answer generation was invoked, Method B processed a mean of 272.5 input tokens (SD 103.7, range 116–635) and produced 61.4 output tokens (SD 27.0, range 25–128), against 215.9 and 51.2 respectively for the single-turn expert pathway of Method E. All observed values remain well below the context limits of Table 4, so no session was truncated. In thirteen Method B scenarios and two Method E scenarios no generative output was recorded, and mean session latency fell to 31.4 s and 14.9 s respectively. These are subsets of the fourteen and six scenarios that returned no recommendations, which are the empty result sets of Table 7.

Table 7. Retrieval coverage across the 37 benchmark scenarios. Columns give the number of scenarios in which the method returned zero, one to four, or five or more documents. Counts sum to 37 in each row.

Method	Empty	1–4 Returned	≥5 Returned
Method A (Broad Intent/Baseline)	24	12	1
Method B (Agentic Refinement)	14	11	12
Method C (Expert Semantic Baseline)	2	13	22
Method D (Lexical BM25 over Expert)	0	14	23
Method E (Expert Query via Agent)	6	14	17
Method F (Contextless User Refinement)	10	18	9

In comparison, when structurally complete expert queries were processed in a single turn without dialogue context (Method E), the system operated with a lower average latency of 49.81 s (SD = 9.84 s over the same N = 370 executions), with a median of 51.31 s and a markedly narrower interquartile range of 7.65 s (Q1 = 46.73 s, Q3 = 54.38 s). The contrast in dispersion is substantial. In wall-clock terms the multi-turn pathway is roughly five and a half times more variable than single-turn execution by interquartile range, and slightly over twice as variable by standard deviation. This is a relevant consideration for interface design even where the difference in means is modest. As above, these measurements reflect cold-start local CPU inference without model warm-up, and should be read as an empirical upper bound rather than as steady-state deployment latency.

Although these delays of approximately one minute represent a bottleneck for real-time web applications, with individual multi-turn sessions reaching up to 117.31 s, they are currently necessitated by the local processing of complex, exploratory System Dynamics tasks. By construction, local execution avoids transmitting user queries and repository content to third-party inference providers, and incurs no per-request API cost. These are properties of the deployment model rather than experimental findings. No privacy analysis or cost comparison was carried out in this study.

5.6. Retrieval Coverage

Section 5.2 reports that the broad-intent baseline failed to place a relevant model in the top five in 30 of the 37 scenarios. Table 7 separates the two failure modes behind that figure. In 24 of those 30 cases the result set was empty under the retention threshold of Section 4.4, and in the remaining 6 cases documents were returned but none was relevant. The baseline deficit is therefore predominantly one of coverage, in that the broad novice formulation yields similarity scores below the retention threshold rather than poorly ordered candidates. The number of empty result sets falls to 14 under the agentic pathway (Method B) and to 6 when the agent receives an expert query (Method E), as reported in Section 5.5. Method D returns no empty set, since BM25 ranks without a retention cutoff analogous to the cosine threshold.

6. Discussion

The experimental results support our core hypothesis. On this corpus and benchmark set, replacing single-turn semantic search with a multi-turn conversational RAG architecture yields a statistically significant improvement in retrieval effectiveness for broad, unstructured queries. This improvement is that of the clarification exchange as a whole, including the additional user input it elicits. This section interprets the findings, discusses their broader implications for HCI in web systems, and outlines current limitations.

6.1. Narrowing the Expertise Gap in Knowledge Retrieval

The most notable finding of this study is the performance disparity between the broad-intent baseline (Method A) and the agentic refinement pathway (Method B). In standard semantic search paradigms, novice users suffer from the “vocabulary mismatch” problem. While they know conceptually what they are looking for, they lack the domain-specific ontology to express it mathematically. The low baseline performance (nDCG@5 of 0.1066) illustrates that vector embeddings alone cannot magically deduce missing structural constraints.

By introducing an Agentic Orchestrator, the reformulation work required to reach a retrievable query is initiated by the system rather than left entirely to the user. The clarification turn prompts the user to specify temporal delays, feedback loop types, and behavioral patterns. Whether this also functions as an educational aid, or reduces perceived effort, was not measured here and remains a hypothesis for a future user study. The gain ($+0.3356$ nDCG@5, $p<0.001$, $dz=0.79$) should be read as the effect of the clarification exchange as a whole rather than of the generative component in isolation. Section 5.2 attributes it to the elicited user input. Expressed in terms of retrieval success rather than graded ranking, the same effect corresponds to a rise in Hit@5 from 0.1892 to 0.5946, since the clarification turn converts fifteen complete retrieval failures into scenarios in which a relevant model reaches the top five. Furthermore, the domain-level breakdown in Figure 3 shows that the direction of the effect is positive in every application-domain group.

6.2. The Semantic vs. Lexical Dichotomy in Structural Modeling

While semantic search has dominated recent IR advancements, our comparison between Method C (Expert Semantic) and Method D (Expert Lexical BM25) reveals a critical nuance. Contrary to the common expectation that dense retrieval compensates for vocabulary variation, lexical matching over expert queries outperforms dense retrieval on every reported metric. This indicates that semantic embeddings smooth over the hard mathematical constraints that distinguish structurally similar models.

In System Dynamics, specific keywords such as “balancing loop”, “stock” or “exponential decay” are not just descriptive, but are structural imperatives fundamental to systems thinking and critical analysis, as highlighted by Meadows [41] and Richmond [42]. Because dense vector embeddings prioritize broad conceptual proximity, they inherently introduce a degree of semantic “fuzziness” that can dilute these strict mathematical constraints. When an expert user provides this perfect terminology, exact lexical matching avoids this fuzziness. Consequently, the sparse lexical matching of BM25 (Method D) proved more effective at placing the exact correct model at rank 1, evidenced by its high MRR@5 of 0.8784. This implies that for highly technical web repositories, the optimal retrieval engine should not abandon lexical matching. Instead, future architectures must adopt Hybrid Search by using the conversational agent to synthesize the query, and then executing a weighted combination of dense vector similarity and sparse BM25 keyword matching [43].

6.3. Generative Query Rewriting and Possible Semantic Dilution

A descriptive observation emerges from the comparison between Method B (the complete multi-turn agent path) and Method F (contextless user refinement). Method F achieved a higher MRR@5 (0.6225) than Method B (0.4761). Since Method F executes a direct semantic search using only the user’s specific raw response to the clarifying question (bypassing the generative agent), this divergence points to a possible limitation in local LLM-driven query rewriting. The rank-based advantage does not extend to the set-based metrics. Method B is marginally ahead on Precision@5 (0.2108 against 0.2054) and Recall@5 (0.4820 against 0.4671), while Method F reaches the top five in 26 of the 37 scenarios against 22 for Method B.

A single mechanism could account for both observations. While the conversational agent successfully elicits precise structural constraints from the user during the dialogue, the subsequent generative phase, where the local LLM (Phi-4-mini) merges the user’s initial broad intent with these newly acquired constraints, may introduce semantic dilution. Inspection of the execution logs suggests that the generative model sometimes produces a refined query in which the dense technical keywords supplied in the final turn are blended with the vaguer terminology of the first turn, which would shift the embedding away from the exact structural parameters. A paired Wilcoxon signed-rank test on the MRR@5 difference between Methods B and F does not reach significance (p = 0.144, dz = −0.22, 23 non-tied pairs of 37). This post-hoc comparison is reported descriptively and is excluded from the Holm–Bonferroni family of Table 6. Neither the Precision@5 nor the Recall@5 difference approaches significance under the same test. The reversal is nevertheless consistent with the dilution account. Broadening the query would lower the probability of retrieving a relevant model at all while returning a slightly wider set of relevant items where retrieval does succeed. The explanation above is inferred from qualitative inspection of the execution logs rather than established by a controlled comparison, and is offered as the most plausible account of a difference that the present data do not resolve. A key HCI takeaway is that while multi-turn interaction is crucial for eliciting constraints from novices, future implementations would benefit from refining the instruction tuning of local generative models to strictly prioritize structural keyword retention over conversational fluency during the query synthesis stage.

6.4. Limitations and Future Work

Despite the highly promising results, the proposed architecture has several limitations that warrant future research:

• Latency and Computational Cost: Multi-turn conversational agents require multiple inference passes through LLMs. In a real-time web environment, this introduces significant latency compared to the millisecond response times of traditional direct retrieval. Whether the latency reported in Section 5.5 is tolerable in practice was not evaluated. That exploratory academic search is more permissive of delay than interactive web search is an assumption about the usage context rather than a finding of this study. To address this bottleneck without abandoning the local privacy-preserving paradigm, future architectural iterations will leverage ONNX Runtime’s native Execution Providers (such as CUDA or TensorRT) to offload model inference directly to local GPU hardware, alongside implementing semantic caching to bypass repetitive LLM query rewriting for common intent patterns.
• Domain Adaptation: While tested across 37 diverse SD benchmark scenarios, the agent’s prompts were specifically engineered for System Dynamics structural elements. Applying this architecture to other fields (e.g., legal document retrieval or medical diagnostics) would require substantial re-engineering of the agent’s domain ontology.
• Attribution of the Improvement: Section 5.2 bounds the contribution of the generative rewriting step via Method F, but does not isolate the clarifying question from the longer input it produces. A condition applying a single-turn LLM rewrite to the broad intent without any clarification exchange would separate the two cleanly. It was not evaluated here.
• User Fatigue: Multi-turn dialogues demand higher active participation from the user. We hypothesize that users may experience “dialogue fatigue” and abandon the search process before the agent can synthesize the final query. However, future HCI studies with active user-study evidence are required to investigate this threshold.
• Dataset Scale and Query Rewriting Limitations: The current empirical validation serves as a proof-of-concept evaluated on a curated corpus of 63 System Dynamics models. We hypothesize that as the repository scales to tens of thousands of models (e.g., 10,000+), the potential semantic dilution introduced during LLM query rewriting, as discussed in Section 6.3, may exert a more pronounced negative effect on ranking metrics such as nDCG@5, necessitating future large-scale empirical testing.
• Security and Prompt Injection Risks: The architecture authenticates requests via JWT and includes system-prompt directives intended to constrain the generative phase against instruction injection. No adversarial testing was performed, so the effectiveness of these directives is untested. They are described here as design measures rather than as verified defenses. Opportunities for further hardening remain. For instance, adding explicit code-level sanitization for reserved chat-template tokens (e.g., <|system|> or <|user|>) would provide a stronger defense against sophisticated theoretical bypass attempts. Finally, given the computational overhead of local CPU inference, future implementations will benefit from strict turn-count and message-length caps to proactively prevent potential resource-exhaustion scenarios.
• Single-seed generative evaluation: The reported metrics correspond to a single reproducible realization of a stochastic decoding process, not to an expectation over it. Alternative seeds may yield different query formulations for borderline-ambiguous scenarios, so the sensitivity of the reported gains to sampling variation remains unquantified. A multi-seed evaluation is left for future work.

Future work will focus on integrating a dynamic Hybrid Search execution layer and developing intent-classification models that can instantly bypass the dialogue phase if the user’s initial query is already structurally sufficient.

7. Conclusions

This study addressed the critical limitations of traditional, single-turn semantic search when applied to highly structured knowledge repositories, specifically within the domain of SD modeling. To narrow the gap between novice users’ broad exploratory intents and the precise structural constraints required for accurate retrieval, we proposed and evaluated a novel, web-based conversational RAG architecture. By replacing the static search box with an intelligent Agentic Orchestrator, the system successfully utilized multi-turn dialogues to clarify user intent, map structural requirements, and synthesize a self-contained search query from the resulting exchange.

Our comprehensive ablation study evaluating 37 distinct benchmark scenarios demonstrated that agent-driven refinement yields a statistically significant improvement over direct baseline retrieval. The conversational architecture increased the mean nDCG@5 by over 300% (from 0.1066 to 0.4422), substantially narrowing the performance gap between novice-level search and expert semantic baselines. The ablation further indicates that the gain originates in the elicited user constraints rather than in the generative synthesis stage (Section 5.2). Furthermore, the experiments revealed that while semantic embeddings are excellent for conceptual matching, exact lexical matching (BM25) remains superior for structurally perfect expert queries. Consequently, future developments in specialized web retrieval should focus on hybrid search methodologies and the optimization of instruction tuning for local LLMs to improve structural keyword retention during query synthesis.