Mitigating LLM Hallucinations Using a Multi-Agent Framework

Abstract

The rapid advancement of Large Language Models (LLMs) has led to substantial investment in enhancing their capabilities and expanding their feature sets. Despite these developments, a critical gap remains between model sophistication and their dependable deployment in real-world applications. A key concern is the inconsistency of LLM-generated outputs in production environments, which hinders scalability and reliability. In response to these challenges, we propose a novel framework that integrates custom-defined, rule-based logic to constrain and guide LLM behavior effectively. This framework enforces deterministic response boundaries while considering the model’s reasoning capabilities. Furthermore, we introduce a quantitative performance scoring mechanism that achieves an 85.5% improvement in response consistency, facilitating more predictable and accountable model outputs. The proposed system is industry-agnostic and can be generalized to any domain with a well-defined validation schema. This work contributes to the growing research on aligning LLMs with structured, operational constraints to ensure safe, robust, and scalable deployment.

1. Introduction

Many applications and use cases empowered by Large Language Models (LLMs) have recently been published [1]. Multi-agent systems leveraging conversational LLMs are widely used for tasks that require creativity and analytical thinking, such as data analysis or code generation. In these systems, multiple agents are instantiated as needed, each with predefined tasks and specialized skill sets, as demonstrated in frameworks like AutoGen [2]. This approach has proven effective for relatively simple creative tasks. However, when scaled to production-level use, these applications often suffer from inconsistency due to LLM hallucinations, i.e., generating responses beyond the model’s factual knowledge [3]. As a result, confidence in deploying LLMs at scale is reduced.

To mitigate these issues, we propose a solution that uses a hybrid combination of rule-based and LLM-based generation techniques. We first generate an initial response from the LLM using a relevant prompt. This response is then passed to a second stage focused on verification and fact-checking. In this stage, we use a custom logic module to help a reviewer agent assess the factuality of the initial response. In addition, we developed a performance scoring function to evaluate factual consistency based on this logic. Figure 1 shows an overview of the proposed framework combining rule-based verification and LLM-based generation to improve factuality and coherence in noisy dialogue transcripts in call centers.

The proposed framework is general-purpose and was applied in a call center platform where call transcripts are often incoherent due to filler words and interruptions between agents and clients. Furthermore, in some instances, both speakers talk over each other, resulting in garbled content. The transcribed text is then input into an LLM, which is tasked with refining the conversation into a well-structured, punctuated dialogue free of filler words without rephrasing and while maintaining the original tone and wording of the speaker.

The LLM’s refined output is subsequently passed to another LLM agent, which is restricted to executing a predefined evaluation function and reporting its result only. More details about the evaluation function can be found in Appendix A. In this framework, we define the similarity function to precisely compare the source and modified dialogue, detecting any rephrasing, or speaker swaps using a sequence-matching score. This function evaluates the initial LLM output, and if the similarity score falls below a set threshold, the system raises a flag for revision, or a concern is raised.

To refine the output, a controlled loop is initiated between the responsible LLM agent and the evaluator to tweak the response until it reaches a fixed number of iterations or a desired accuracy score within a specified tolerance. In our case, the stop criterion is either reaching zero mistakes in the consultant’s response or completing the second iteration, whichever comes first. Figure 1 shows a process flowchart of the stop criteria used to control the consultant-reviewer iterations.

Then, a benchmark evaluation metric is calculated to evaluate the process performance over all processed transcripts. This evaluation is scalable and helps to detect any odd behavior of the LLM during deployment and take the necessary action. The evaluation matrix was primarily constructed using the similarity score between generated responses and reference answers. The mean similarity score was calculated for each model and iteration to provide an overall measure of average response quality. Alongside the mean, key descriptive statistics such as the standard deviation, minimum, 25th percentile, median (50th percentile), 75th percentile, and maximum similarity scores were computed to capture the distribution and variability of the model outputs. These metrics reveal not only the average quality but also the consistency and spread of the responses. Additionally, 95% confidence intervals (CIs) around the hallucination rates were calculated to quantify the uncertainty in these estimates, providing a statistical range within which the actual hallucination rate is expected to lie with 95% confidence. This combination of metrics enables a comprehensive and statistically robust assessment of model behavior across iterations. The contributions of this work include:

• Design and implementation of a general hybrid framework that combines rule-based and LLM-based generation techniques. This framework is used for verifying and fact-checking initial LLM-generated responses and was applied to enhance the factuality and coherence of outputs in a call center transcript use case.
• Proposal of a performance scoring function that assesses the factuality of LLM responses using custom-defined logic, supported by a controlled feedback loop between the generation and evaluation agents. This loop continues refining the responses until a set number of iterations is completed (two iterations in our case) or until the target accuracy threshold is achieved.

2. Related Work

A wide range of strategies have been proposed to mitigate hallucination in large language models (LLMs), each exhibiting distinct strengths and limitations.

Retrieval-Augmented Generation (RAG) techniques [4,5,6] ground LLM outputs in external sources such as Wikipedia or proprietary databases to improve factuality. While effective in significantly reducing hallucinations and increasing trust [5], they suffer from high implementation complexity, infrastructure dependency, latency issues, and diminished efficacy when relevant retrieval is sparse or noisy.

Prompt-based self-refinement and reflection [7,8] involve multi-step generation and confidence-based answer re-ranking, showing model-agnostic gains in tasks involving factual consistency and negation. However, their reliance on prompt engineering and repetitive LLM calls makes them computationally expensive and brittle in general settings.

Multi-agent debate and verification frameworks [2,9,10,11,12] introduce LLM ensembles to cross-verify or debate generated content. These methods have demonstrated success in post hoc hallucination detection, but primarily act as filters rather than preventative techniques and lack scalability for real-time applications.

Tagged-context prompting [13] embeds explicit source markers into input prompts, enabling precise factual grounding. While highly interpretable and effective in constrained QA settings, the method relies heavily on curated and structured input data, limiting applicability in open-ended generation. On the evaluation side, benchmark frameworks like Q² [14], HaluEval [15], and ANAH [16] have been proposed to better assess hallucination through question-answering consistency, large-scale annotations, and human–AI agreement metrics. Nevertheless, these tools primarily focus on static evaluation and lack real-time adaptivity or generalization across domains.

Analytical studies and surveys [3,17,18,19] offer comprehensive taxonomies and categorization of hallucination types, but generally stop short of providing deployable solutions. Collectively, while these methods advance understanding and detection of hallucination, many face limitations in scalability, latency, generalization, and reliance on external resources.

A complete comparison between related work in terms of strengths, weaknesses, and limitations can be found in Appendix A.

3. Materials and Methods

3.1. Dataset

The dataset used in our evaluation consists of real customer service calls from a commercial call center in the automotive industry. The data was collected through several stages. (1) Inbound and outbound calls were recorded following the call center’s privacy policy. (2) These recordings were transcribed into text with speaker diarization using Nova-2 with a median word error rate (WER) of 8.4% by Deepgram [20]. (3) The transcripts and relevant attributes were stored in a database. Figure 2 illustrates the data transformation process applied to our dataset. The transcripts consist of text chunks from either the agent or the customer. Each chunk is defined using a fixed window of length 150 words and annotated with a speaker ID identifying the speaker, and a chunk ID indicating its position in the call sequence. Additional metadata includes call length, call index, call timestamp, and agent ID.

The dataset comprises approximately 9 million recorded calls, with durations ranging from a few seconds to over an hour, depending on the nature of the interaction. On average, each call contains around 50 dialogue chunks with a fixed window of 150 words, providing a rich and detailed corpus for analysis.

This dataset is commercially private, including personal information from real customers and agents. To protect the privacy and confidentiality of the participants, it cannot be publicly distributed. Access for research purposes may require a stringent approval process and adherence to strict data protection protocols in compliance with applicable privacy regulations.

Nevertheless, anonymized data samples are presented in Figure 3 to illustrate some of the challenges associated with the dataset. Also, anonymized data supporting this study’s findings will be available for reproducibility. The implementation framework is hosted in a private GitHub repository and can be accessed upon reasonable request, subject to approval.

3.2. Framework

Although multi-agent debate can enhance the factuality and reasoning abilities of LLMs [2], empirical evidence shows that existing libraries for multi-agent conversational tasks offer limited control over agent transitions [12]. These systems often exhibit divergent behavior or veer into off-topic discussions, and typically provide only minimal tuning options, such as limiting the maximum number of transitions, to manage such dynamics [11].

Most multi-agent libraries provide only basic mechanisms for controlling agent behavior, such as step limits or termination logic, and lack built-in safeguards against semantic drift or off-topic dialogue beyond prompt engineering. LangChain [21] includes parameters like max_iterations to cap the number of steps and early_stopping_method to terminate loops. AutoGen [2] offers finer control at the message level through parameters such as max_consecutive_auto_reply and termination_condition. In contrast, LlamaIndex lacks native loop control mechanisms and instead relies on external orchestration or custom wrappers to impose such constraints. This behavior is demonstrated in Figure 3.

As shown in Figure 1, the proposed approach follows a strict mechanism that controls the agent’s conversation as needed. We employ nested chats, with child agents inheriting from the parent leader agent. That is, we divide the complex task into several smaller child tasks, each involving a maximum of two agents: the task agent and the evaluator agent. The parent agent orchestrates the execution sequence as planned by triggering the desired child agent, and the responses from the nested chats are passed back to the parent agent for further coordination. The proposed approach aims to simplify complex tasks by breaking them down into smaller, manageable sub-tasks, thereby promoting hallucination-free responses. Furthermore, we monitor the performance of LLM agents for each sub-task at every stage (Figure 4) using a consultant-evaluator agent framework.

In this framework, we employ the open-source LLaMA-3-8B model by Meta [22] as the consultant agent responsible for performing the main tasks and potentially prone to hallucinations. Following the consultant agent, we utilize the open-source Mistral 7B [23] model as an evaluator agent to assess the consultant’s performance. The evaluator is a function caller with code execution capabilities, invoking a rule-based function that operates similarly to an employee performance index in organizations. The function caller agent extracts the required arguments in the desired format from the consultant’s response and invokes the function. Thus, each sub-task is managed by a pair of agents, one for task execution and the other for evaluation, both equipped with code execution capabilities.

Figure 1 illustrates a statically configured interaction between the consultant and evaluator agents, designed to control the number of iterations before concluding. Each stage is assessed through a predefined, rule-based numerical function. All data are then pushed to a performance database that logs execution behavior and facilitates the detection of anomalies or divergence.

This database also serves to diagnose and troubleshoot cases of LLM hallucination. In complex use cases involving multiple consultant agents, a detailed performance matrix can be generated for each agent to evaluate task-specific LLM behavior and prompt effectiveness. In our showcase, we have used manual prompt optimization and observed the impact on the model’s performance and response quality.

The consultant-evaluator interaction is typically constrained to a single or, at most, two iterations. Empirical evaluations indicate that additional iterations offer minimal benefit, as most hallucinations are identified and resolved within the first two cycles. Residual minor inaccuracies generally stem from the inherent limitations of the LLM or suboptimal prompt design.

Appendix B provides a detailed explanation of the algorithm used by the rule-based function to evaluate the model’s response.

3.3. Performance Evaluation

The evaluation strategy is structured around a numerical scoring function tailored explicitly to the use case and task characteristics. In the context of dialogue restructuring tasks, we identify two predominant types of hallucinations: (1) incorrect sentence swaps between speakers, and (2) rephrasing of specific words or phrases. To address these issues, we evaluate each chunk in the LLM-generated output by comparing it against its closest matching chunk from the original transcript.

The evaluation begins by concatenating all original transcript segments into a single document per speaker. For each chunk in the modified dialog, we locate the closest matching sentence from the corresponding source using a sliding window equal to the chunk length. A sequence-matching algorithm computes a similarity score, which serves as the basis for quantifying alignment between the original and revised content.

The sequence-matching method employed is based on Gestalt pattern matching, specifically the Ratcliff and Obershelp algorithm [24]. Gestalt pattern matching is preferred for our use case due to its ability to capture holistic structural similarities, aligning closely with human perceptual organization. This approach is more robust to noise and variation, making it effective for analyzing call center transcript datasets. The main idea behind this algorithm is to identify the longest contiguous matching subsequence between two input sequences. Once this subsequence is identified, the algorithm recursively applies the same strategy to the remaining segments on either side of the match. The similarity metric, Dro, is calculated using the formula:

$$\mathrm{Dro}={\displaystyle \frac{2K_m}{|S_1|+|S_2|}},$$

(1)

where $K_m$ is the number of matching characters, and $S_1$ and $S_2$ are the lengths of the two strings. This algorithm identifies the longest common substrings, applying the process recursively to the non-matching regions on both sides of these substrings.

Manual inspection of similarity scores and input sentence pairs revealed that scores greater than 80% generally do not involve speaker swaps or rephrasing. Most hallucination cases were associated with scores below 80%, with severity ranging from minor, close to 80% SS, to major, closer to 0% SS. Consequently, any sentence with a matching score below 80% is flagged as potentially problematic.

In some cases, the consultant agent generates a single chunk that combines multiple segments of dialogue, leading to overly large chunks. When this occurs, we subdivide the chunk into smaller overlapping sub-chunks of no more than 20 words, with a three-word overlap to preserve context. The matching score of each sub-chunk is then evaluated individually. Following this process, the evaluator agent aggregates all sub-chunks whose matching scores fall below an 80% threshold and returns them to the consultant agent with the chat history flag set to True, prompting correction of these discrepancies. This comprehensive assessment helps ensure the accuracy of dialog reconstruction by identifying and correcting substantial deviations, thereby minimizing hallucinations.

4. Experiment and Results

We applied our framework to 308 transcripts between speaker 0 (Call Center Agent) and speaker 1 (Client), each containing between 15 and 25 dialogue chunks.

We then evaluated the performance of the LLM consultant agent and benchmarked the similarity scores against a reference threshold of 80%.

Of the 3680 total fragments across the 308 transcripts, 60.2% of Mistral-7B outputs and 32.6% of LLaMA-3-8B outputs fell below the 80% similarity threshold as shown in Figure 5. The 95% confidence intervals indicate these differences are statistically robust (

Table 1. Hallucination rates and corresponding 95% confidence intervals for LLaMA-3-8B and Mistral-7B across initial response and subsequent refinement iterations.

Iteration	Model	Halluc. Rate (%)	95% CI Low (%)	95% CI Up (%)
Initial Response	LLaMA-3-8B	32.61	31.11	34.13
Initial Response	Mistral-7B	60.27	58.91	61.63
1st iteration	LLaMA-3-8B	11.36	10.33	12.46
1st iteration	Mistral-7B	31.09	29.60	32.60
2nd iteration	LLaMA-3-8B	4.73	4.06	5.51
2nd iteration	Mistral-7B	19.48	18.16	20.86

). Overall, iterative refinement substantially improves factual accuracy, with LLaMA-3-8B demonstrating more effective reduction in hallucinations than Mistral-7B.

In the first iteration, we asked the same consultant agent to correct the mistakes in the initial response. The hallucination rate was reduced to 31.1% and 11.4% for Mistral-7B and LLaMA-3-8B, respectively. This rate was reduced to 19.5% and 4.7% in the last iteration with Mistral-7B and LLaMA-3-8B, respectively.

Table 2. The average similarity score (SS) for each model across different stages.

	LLaMA3-3-8B	Mistral-7B	SS (avg.)
Initial response	78.84%	70.35%	73.92%
1st iteration	89.31%	86.70%	87.45%
2nd iteration	89.56%	86.98%	87.54%
SS (avg.)	84.03%	81.33%	82.20%

shows the average similarity score (SS) for each model across different stages.

Despite the reduction in hallucination, LLMs may still produce minor hallucinations due to their reliance on learned patterns rather than factual grounding and limited internal knowledge. Prompt instructions are not always enforceable since the model lacks true understanding and instead predicts the most likely next token based on prior training data.

Table 3. Number of chunks with similarity score below 80%.

	LLaMA-3-8B	Mistral-7B	Total
Initial response	1200	2217	3417
1st iteration	418	1144	1562
2nd iteration	174	717	891

displays the performance of the LLaMA-3-8B and Mistral-7B agents in terms of similarity scores across different iterations. As shown in the table, the number of hallucinating chunks with similarity scores below 80% was reduced from 1200 and 2217 to 174 and 717 for LLaMA-3-8B and Mistral-7B, respectively. The number of corrected chunks was 1,026 for LLaMA-3-8B and 1,500 for Mistral-7B, corresponding to reductions of 85.5% and 67.7% in hallucination rates after applying two-step iteration refinements.

Figure 6 shows that observations reveal notable trends, including consistently high median scores for LLaMA-3-8B and Mistral-7B in the first and second iterations. The initial responses from Mistral and LLaMA exhibit greater variability and more outliers, indicating less consistent performance compared to their refined responses. In contrast, the first and second iterations show improved consistency, with reduced variability and fewer outliers.

Table 4. Descriptive statistics of SS produced by the two LLM models across different stages.

Model	Type	Std	Min	25%	50%	75%	Max
LLaMA-3-8B	Initial response	30.05	0.00	47.79	100.00	100.00	100.00
	1st iteration	24.07	0.00	98.31	100.00	100.00	100.00
	2nd iteration	25.13	0.00	100.00	100.00	100.00	100.00
Mistral-7B	Initial response	25.34	0.00	49.49	70.44	97.14	100.00
	1st iteration	24.27	0.00	72.42	100.00	100.00	100.00
	2nd iteration	24.87	0.00	74.61	100.00	100.00	100.00

presents the descriptive statistics of SS produced by the two LLM models across different stages: initial response, first iteration, and second iteration. For both models, the central tendency (median) rapidly converges to 100% from the first iteration onward, indicating highly similar or near-identical outputs relative to reference answers. The inter-quartile ranges shrink with each iteration, particularly for Mistral-7B, suggesting increasing consistency in model behavior. While initial responses show more variability, especially in the lower quartiles, both models demonstrate a marked improvement in alignment and output similarity as iterative refinement progresses. The standard deviation decreases correspondingly, underscoring reduced variability in later responses.

The initial response shows relatively low similarity scores, with LLaMA-3-8B scoring 78.84% and Mistral-7B scoring 70.35%. After the first iteration, both agents demonstrated significant improvement, with LLaMA-3-8B reaching 89.31% and Mistral-7B achieving 86.70%. The second iteration led to a slight additional improvement, with LLaMA-3-8B scoring 89.56% and Mistral-7B scoring 86.98% (Figure 7). This trend indicates that our framework effectively improves the agents’ ability to generate more coherent and accurate responses.

Chunks are categorized into three main groups based on text length: (1) fewer than 50 words, (2) between 50 and 100 words, and (3) more than 100 words.

Table 5. Average similarity score per chunk size category. Long (more than 100 words), medium (50–100 words) and short (less than 50 words).

Chunk Size	Model	Initial	1st	2nd	Avg.
		Response	Iteration	Iteration
Long	LLaMA-3-8B	74.45	81.42	84.71	77.42
	Mistral-7B	68.06	76.68	75.16	72.59
Medium	LLaMA-3-8B	77.20	86.06	86.30	80.88
	Mistral-7B	67.37	82.45	82.71	76.56
Short	LLaMA-3-8B	81.65	91.98	92.06	87.61
	Mistral-7B	74.07	90.53	90.69	86.07
Avg.		73.92	87.45	87.54	82.20

shows the improvement in similarity score per chunk size category. The observed improvement appears to be independent of text length. Figure 8 displays each model’s similarity score per chunk size category.

The progression from the Initial Response to subsequent iterations within the LLaMA and Mistral models demonstrates iterative improvements in performance, characterized by increased median similarity scores and reduced variability.

Based on these results, it is evident that our framework has a significant impact on the factual consistency of LLMs and the reduction in hallucinations. Moreover, the framework’s utility extends to a variety of scenarios. Additionally, the performance evaluation database facilitates the assessment of LLM agents across diverse use cases and datasets.

The framework was executed on a Linux-based machine equipped with an NVIDIA RTX 3090 GPU (24 GB VRAM) and an AMD Ryzen 9 5950X 16-core CPU (32 threads), completing the full dataset processing in approximately 52 min.

5. Discussion

While this work has made a significant contribution to improving the factual consistency of LLMs and reducing hallucinations, several promising avenues for future research remain. One such direction involves expanding the rule-based functions to incorporate a wider range of error detection and scoring algorithms, thereby enhancing the evaluation process. The evaluation function employed in our study is designed to detect hallucinations, which we define, within the context of our use case, as instances of speaker swaps or rephrasings. However, the notion of hallucination is inherently domain-dependent, and thus, rule-based evaluation functions should be adapted accordingly. For example, in code generation tasks, hallucinations may manifest as syntactically invalid outputs, which can be identified using executor LLMs or compilers. In machine translation, hallucinations may be assessed through deviations in grammatical correctness or fidelity to the source text. As a general principle, domain-specific rules grounded in expert knowledge can be formulated to measure the frequency with which generated outputs violate factual, structural, or functional constraints relevant to the target application.

In sentence restructure tasks, models with stronger language understanding and contextual reasoning generally yield more accurate and fluent outputs. LLaMA-3 8B demonstrates superior performance over Mistral-7B, as evidenced by its higher scores on benchmarks such as MMLU (68.4% vs. 60.1%) and HumanEval (62.2% vs. 36.6%), indicating better generalization and semantic control in generative tasks [22,23]. These differences are partly due to LLaMA-3’s larger parameter count and broader pretraining corpus, spanning 15 trillion tokens. While Mistral-7B is optimized for efficiency through Grouped Query Attention (GQA), its tradeoff is reflected in lower performance on tasks requiring nuanced paraphrasing. Therefore, LLaMA-3 8B is generally better suited for sentence restructuring where preservation of meaning and fluency is critical.

The proposed framework was executed on a Linux-based system equipped with an NVIDIA RTX 3090 GPU (24 GB VRAM) and an AMD Ryzen 9 5950X 16-core CPU (32 threads), completing the full dataset processing in approximately 52 min. To enhance computational efficiency, the system can be deployed on higher-performance hardware or optimized further by adjusting convergence parameters to reduce the number of correction iterations. However, such modifications may introduce a trade-off between processing speed and the effectiveness of hallucination mitigation.

Limitations

While the dataset offers valuable insights, it may suffer from limited diversity, primarily reflecting a specific market segment, customer base, or regional communication style. These limitations constrain the generalizability of models trained on the data to broader domains or geographic contexts. Although the transcripts are written in English, they exhibit considerable variation in dialects, accents, technical terminology, and informal speech patterns, introducing linguistic complexity that challenges robust model training. Additionally, the dataset may encode latent biases, such as the overrepresentation of certain customer demographics or service categories, leading to models that underperform for underrepresented groups. Structural biases may also exist, for example, a disproportionate focus on specific vehicle types, customer profiles, or support scenarios. These imbalances risk skewing model behavior and reducing performance in edge cases, such as interactions with first-time car owners or non-standard service requests.

This study is also limited to an open-domain dataset; other domains such as medical and legal remain unexplored. Furthermore, the current approach has only been tested on English-language conversations. Extending the evaluation to other languages will be constrained by the varying capabilities of LLMs in multilingual settings.

Future work could also benefit from the development of visualization dashboards and monitoring tools, which would offer deeper insights into model performance and aid in anomaly detection and tracking.

6. Conclusions

This paper addresses the inconsistent and unpredictable performance of LLMs in large-scale production environments. We propose a hybrid mechanism that combines rule-based techniques with generative methods. This framework effectively enhances the factual consistency of LLMs by simplifying complex use cases into pairs of single-task consultant-evaluator agents, where the consultant agent generates responses and the evaluator agent scores them using a predefined rule-based function.

Furthermore, we introduce a performance database that logs the performance score of each consultant agent every time it responds to a specific prompt. This database serves as a robust tool for continuous monitoring and optimization, enabling real-time detection and correction of anomalous behavior.

When implemented in a call center setting, the framework led to an 85.5% improvement in consistency for the Meta-LLaMA-3-8B model and 67.7% for the Mistral-7B model. This was achieved through a structured interaction protocol between agents, in which the consultant LLM refines its response based on feedback scores from the evaluator agent in a dual-iteration process.

Finally, this work advances the current literature by providing a practical and reliable solution applicable to domains that demand hallucination-free LLMs. This helps mitigate concerns surrounding LLMs at scale and enhances their utility in solving real-world problems. Future research can expand on this foundation to explore additional rule-based functions and refine evaluation criteria for a broader range of applications and datasets. The systematic and quantifiable strategy we present lays the groundwork for the more precise deployment of LLMs across industries.