Context Is King: Large Language Models’ Interpretability in Divergent Knowledge Scenarios

Abstract

Large language models (LLMs) have revolutionized the field of artificial intelligence in both academia and industry, transforming how we communicate, search for information, and create content. However, these models face knowledge cutoffs and costly updates, driving a new ecosystem for LLM-based applications that leverage interaction techniques to extend capabilities and facilitate knowledge updates. As these models grow more complex, understanding their internal workings becomes increasingly challenging, posing significant issues for transparency, interpretability, and explainability. This paper proposes a novel approach to interpretability by shifting the focus to understanding the model’s functionality within specific contexts through interaction techniques. Rather than dissecting the LLM itself, we explore how contextual information and interaction techniques can elucidate the model’s thought processes. To this end, we introduce the Context-Driven Divergent Knowledge Evaluation (CDK-E) methodology, along with the Divergent Knowledge Dataset (DKD), for evaluating the interpretability of LLMs in context-specific scenarios that diverge from the model’s inherent knowledge. The empirical results demonstrate that advanced LLMs achieve high alignment with divergent contexts, validating our hypothesis that contextual information significantly enhances interpretability. Moreover, the strong correlation between LLM-based metrics and semantic metrics confirms the reliability of our evaluation framework.

1. Introduction

Large language models (LLMs) have ushered in a new era of artificial intelligence (AI), fundamentally transforming our interaction with technology and unlocking unprecedented capabilities. The introduction of models such as BERT [1] and the subsequent release of ChatGPT, based on the GPT-3 model [2], acted as a catalyst, sparking widespread excitement and revolutionizing the LLM landscape. Following these releases, a new wave of LLMs flooded the field, with models such as Llama [3,4], the Gemini family [5], Mistral [6], and Phi [7]. Built on massive datasets and sophisticated architectures, these models have an unparalleled ability to understand, reason, and generate human-like text.

Leading tech companies like Google, Meta, and Microsoft have developed their own models, integrating and deploying them into their commercial products to enhance their functionalities. The emergence of the first open-source models has further democratized technology access, enabling the broader community to integrate and utilize these models in a wide range of applications. From facilitating natural language understanding to enabling creative content generation, LLMs have become indispensable tools across academia and industry. They are being used for tasks such as automated customer support [8], personalized recommendations [9], real-time translation [10], summarization of complex documents [11], and even aiding in scientific research by analyzing large datasets, generating hypotheses and better representations [12]. The versatility and robustness of these models make them critical assets in driving innovation and efficiency across diverse sectors.

However, as these models have grown more powerful, they have also become increasingly complex, resembling opaque black boxes whose inner workings are difficult to interpret. This lack of transparency poses significant challenges for model interpretability and accountability [13,14]. The intricate architectures and vast parameter spaces make it challenging to understand how these models arrive at their decisions. This opacity can lead to several negative consequences, such as the generation of fictitious information (typically called hallucinations) [15], where the model produces plausible-sounding but incorrect or nonsensical outputs. In addition, without a clear understanding of the decision-making process, it becomes difficult to identify and mitigate biases embedded in the models [16], potentially leading to unfair or discriminatory outcomes [17,18,19]. The lack of transparency not only undermines user trust but also raises ethical and security concerns, particularly when LLMs are used in critical applications such as healthcare [20], legal advice [21], education [22], and autonomous systems [23]. Addressing these issues is crucial to ensuring the responsible and effective use of LLMs and highlights the need for robust explainability frameworks and methods that illustrate the decision-making processes of these models.

In terms of artificial intelligence, explainability refers to the ability to understand, explain, and interpret the decisions and behavior of AI systems in human terms [24,25,26]. Within LLMs, explainability is particularly crucial due to the models’ inherent complexity and the high stakes of their applications. The ability to provide clear, understandable explanations for the decisions made by LLMs can significantly enhance user trust and facilitate the adoption of these technologies in sensitive areas.

Given the opaque nature of LLMs, achieving explainability requires innovative approaches that go beyond traditional model introspection. One promising direction is the use of contextual information to illuminate the decision-making processes of these models, enhancing interpretability. Such techniques have proven to be effective in dealing with knowledge cutoffs and the need to access updated content without retraining the model, as well as mitigating issues such as biases and hallucinations [27]. Furthermore, the widespread adoption of these techniques is driven by the ease of creating LLM-based applications and gaining greater control over data. By situating LLMs within specific controlled contexts, we can better interpret and understand the decision-making processes.

In this work, we propose a novel approach to interpretability by leveraging interaction techniques that provide additional context to LLMs. Rather than dissecting the LLMs themselves, we focus on how contextual information can be utilized to interpret and explain the models’ reasoning, chains of thoughts, and outcomes. The aim of this study is to evaluate this capability, shedding light on the reliability and interpretability of LLM outputs in relation to the popular interaction techniques used today. This complements, rather than replaces, the existing explainability and interpretability techniques in the literature.

To test our approach, we introduce the Context-Driven Divergent Knowledge Evaluation (CDK-E) methodology, designed to evaluate the interpretability of large language models within context-divergent scenarios. The novelty of our contribution lies in how we incorporate fabricated contexts to systematically test a model’s ability to adapt its reasoning process. This is achieved through the Divergent Knowledge Dataset (DKD), which includes a series of questions based on fabricated historical events and facts that act as context, diverging from the model’s inherent knowledge. The CDK-E evaluates whether the model can produce accurate and contextually aligned responses, providing insights into how its reasoning adapts to the given context.

Our methodology establishes a baseline using state-of-the-art LLMs through prompt engineering and a robust performance assessment module. By structuring the experiment around divergent contexts, we directly address the research question of how well LLMs can align their outputs with external information, offering a more thorough assessment of their interpretability in complex scenarios. The empirical results show that advanced LLMs achieve strong alignment with the provided divergent contexts, confirming our hypothesis that contextual information significantly enhances interpretability. Furthermore, the strong correlation between LLM-based metrics and semantic evaluations affirms the reliability of our assessment module.

The main contributions of this paper are as follows:

• The introduction of the Context-Driven Divergent Knowledge Evaluation (CDK-E) methodology along with the Divergent Knowledge Dataset (DKD), a novel methodology and dataset for evaluating the interpretability of LLMs in context-specific scenarios that diverge from the model’s inherent knowledge.
• The presentation of empirical results demonstrating the effectiveness of contextual information in achieving interpretability and explainability.
• The provision of analysis and discussion for the integration of LLMs, focusing on improving interpretability and enhancing the understanding of their decision-making processes, along with a framework for future research in explainable AI (XAI).

The rest of the paper is organized as follows: Section 2 discusses the related work and context-based interaction techniques, Section 3 presents the CDK-E methodology and the DKD, Section 4 describes the experimental setup, Section 5 details the results and discussion, and Section 6 provides the conclusions of our work.

2. Related Work

In order to address the challenges posed by the increasing complexity and opacity of large language models, significant research has been dedicated to enhancing their interpretability, explainability, and trustworthiness. These efforts have led to the development of a range of methodologies aimed at making LLMs more transparent and reliable. In this section, we explore the most prominent approaches in explainable AI for LLMs, with a particular focus on the context-based interaction techniques relevant to our work.

2.1. Context-Based Interaction Techniques

The field of prompt engineering has emerged as a new way to explore the potential of LLMs by developing and refining natural language prompts. These techniques enable us to exploit the full potential of LLMs while addressing their inherent limitations [28]. Through effective prompt engineering, we can enhance the performance of LLMs in a wide range of tasks, from question answering to complex reasoning.

Prompt engineering involves more than just designing complex prompts; it encompasses a variety of techniques, skills, and methods to extract information, process it, and interact with LLMs. By crafting robust and efficient prompts, we can significantly improve the safety, reliability, and overall performance of these models [29]. Moreover, the approaches to providing information and interacting with LLMs—such as zero-shot, multi-turn dialogues, chain-of-thought prompting, and self-consistency—are crucial for achieving the desired results. These techniques are particularly important for integrating LLMs with specific knowledge, external tools, or to sequence their tasks, enabling the models to provide relevant and updated information that extends beyond their training.

The essence of many of these techniques is based on the use of context together with the original request. Context-based interaction techniques leverage the additional information provided within the original request to guide the LLMs’ responses. This context can include specific instructions, examples, or any pertinent information that helps to shape the model’s output in a desired manner. By embedding context, we can make the models’ responses more coherent, accurate, and aligned with human understanding [30].

One of the primary advantages of context-based techniques is their ability to mitigate the limitations of LLMs’ knowledge cutoffs. Given that these models can only consider a limited amount of text at a time, crafting prompts that efficiently utilize this window is essential. Effective prompt design ensures that the most critical information is included within the context window, thereby improving the model’s ability to generate meaningful and accurate responses.

In the following, we list several advanced interaction techniques that have been developed to further enhance the capabilities of LLMs:

• Few-shot Prompting: In this technique, a few examples are provided to the model as part of the prompt. This approach enables in-context learning to help the model understand the desired output format and the type of responses expected [31].
• Chain-of-Thought Prompting: Chain-of-thought (CoT) prompting encourages the model to break down complex problems into smaller manageable steps [32]. By prompting the model to “think aloud” and generate intermediate reasoning steps, users can improve the model’s ability to handle tasks that require logical reasoning and multi-step problem solving. This technique enhances the transparency and interpretability of the model’s decision-making process.
• Retrieval-Augmented Generation: One of the most popular techniques nowadays due to its ability to connect LLMs with external information sources, Retrieval-Augmented Generation (RAG) [33] combines the LLMs’ understanding and generation capabilities with external information retrieval. These retrieval mechanisms fetch relevant and up-to-date information from external sources, enabling LLMs to access an authoritative knowledge base beyond their training data before generating a response. RAG enhances the already powerful capabilities of LLMs by integrating specific domain knowledge or an organization’s internal knowledge base, all without the need for model retraining. It is a cost-effective approach that addresses the limitation of knowledge cutoffs, ensuring that outcomes are relevant, accurate, and useful in various contexts.
• Prompt Chaining: This technique breaks down complex tasks requiring iterative processing or multi-step reasoning into smaller manageable tasks. By connecting multiple prompts in a sequence, each building on the previous one, prompt chaining guides the model through a series of steps. This approach helps to improve the reliability, performance, and understanding of LLMs’ interactions, resulting in more accurate and comprehensive responses.
• ReAct: ReAct (synergizing reasoning and acting) explores the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner [34]. By prompting the model to not only reason through a problem but also suggest specific actions, the system can induce, track, and update action plans while handling exceptions. Additionally, ReAct enables the model to integrate external sources such as knowledge bases or environments to inform its actions.

These are just some of the most commonly used context-based interaction techniques, but there are many others, such as self-consistency [35], Active-Prompt [36], Directional Stimulus Prompting [37], and Reflexion [38], or more complex techniques that integrate multimodal inputs, such as Graph Prompting [39] and Multimodal CoT [40]. All these techniques rely on the effective use of context and how it is extracted, prepared, and integrated into prompts.

Despite the potential improvements that context-based techniques offer, it is crucial to be aware of their limitations. Irrelevant context or noisy ground-truth labels can degrade the model’s performance [41,42]. Ensuring the quality and relevance of the context is essential. This means not only selecting the appropriate information but also structuring it in a way that aligns well with the task at hand. By meticulously curating context, we can mitigate these risks and maximize the effectiveness of LLMs, harnessing their full potential to generate high-quality context-aware responses.

In summary, context-based interaction techniques through prompt engineering are key for maximizing the utility of LLMs and integrating them into diverse and specific contexts. These techniques offer structured methods to interact with LLMs, addressing their limitations and unlocking new capabilities. Their effectiveness in addressing LLM limitations and extending their capabilities has led to widespread adoption, underscoring the importance of basing model explainability on context-based techniques.

2.2. Explainable AI and Interpretability in Large Language Models

The transformative impact of LLMs opens up new possibilities for XAI research, posing a major challenge in terms of their explainability, interpretability, and ethical and reliable use. The traditional methods and strategies for interpreting machine learning models, such as feature importance analysis and decision tree visualization, are not directly applicable to LLMs due to their massive size and complex architecture. Therefore, researchers have explored various approaches tailored to the unique characteristics of LLMs to provide insights into their decision-making processes [43].

One prominent approach to enhancing the interpretability of LLMs involves the use of attention mechanisms [44]. Attention mechanisms, which are integral to the architecture of many LLMs, have been leveraged to provide insights into which parts of the input data the model is focusing on when making predictions. For instance, studies have shown that visualizing attention weights can help to trace the reasoning paths taken by the model, thereby offering a glimpse into its decision-making process [45]. However, the complexity of attention patterns often requires sophisticated methods to decode meaningful explanations.

Another significant area of research has been the development of post hoc explanation techniques [26]. These methods aim to provide insights into the decision-making processes of LLMs without altering their underlying structure. One common approach is the use of Local Interpretable Model-agnostic Explanations (LIMEs) [46], which approximate the model locally with a simpler interpretable model to explain individual predictions. Another technique, Shapley Additive Explanations (SHAPs) [47,48], leverages cooperative game theory to attribute the contribution of each feature to the final prediction, offering a detailed understanding of feature importance. Additionally, counterfactual explanations have gained appeal [49], where slight modifications are made to the input data to observe how these changes affect the output, thereby highlighting the critical factors influencing the model’s decisions. These post hoc methods are essential for uncovering biases, identifying potential errors, and enhancing user trust in LLMs, especially in high-stakes applications.

Despite these advancements, challenges remain to ensure that the explanations provided by XAI techniques are faithful, understandable, and meaningful. The inherent complexity of LLMs means that, even with advanced methods like attention visualization and post hoc techniques, it is difficult to guarantee that the explanations accurately reflect the true reasoning process of the model. These techniques can sometimes oversimplify the complex operations within LLMs, posing the risk of misrepresentation. Additionally, research challenges persist, such as providing explanations without ground-truths, developing metrics to measure the quality of explanations, mitigating biases, addressing hallucinations, and adapting traditional interpretability techniques to the scale and complexity of LLMs.

There is also ongoing exploration into understanding model decisions through contextual explanations and explainable prompting [50,51]. A key differentiator between LLMs and traditional artificial intelligence models is their ability to accept input data in the form of natural language during model inference [52]. This characteristic offers more intuitive and user-friendly insights into these powerful models’ decision-making processes, enabling us to analyze, understand, and draw conclusions using natural language. Indeed, such techniques are being widely adopted and have proven to be effective in addressing the problems of knowledge cutoffs and accessing up-to-date information [53,54].

Building on these foundations, our work proposes a novel approach to interpretability through context-based interaction techniques. By focusing on the contextual information inherent in the inputs and outputs of LLMs, we aim to offer explanations and interpretations that are more intuitive and user-friendly. This approach not only enhances transparency but also aligns closely with how humans understand and generate language, making the explanations more accessible and actionable for users. To the best of our knowledge, this is the first time that the performance of LLMs has been evaluated using these interaction techniques within divergent content. This evaluation highlights context as a powerful means to interpret and understand the outputs of the models.

3. CDK-E: Context-Driven Divergent Knowledge Evaluation

In this section, we introduce the Context-Driven Divergent Knowledge Evaluation (CDK-E) methodology, the principal contribution of our work. CDK-E is designed to evaluate the interpretability of large language models in scenarios that rely on external context rather than the model’s internal knowledge. Specifically, the methodology tests LLMs using contexts that diverge from the inherent knowledge, meaning that the models can only generate correct responses based on the provided information. By assessing the accuracy of these context-dependent responses, CDK-E offers a more interpretable and reliable evaluation of the models’ decision-making processes. This approach complements other XAI techniques by establishing a framework for leveraging context-based methods to interpret model outputs under controlled knowledge conditions.

To support the implementation of this methodology, we also introduce the Divergent Knowledge Dataset (DKD), a dataset specifically designed to provide these divergent contexts necessary for testing the approach. It contains fabricated contexts, questions, and ground-truths that challenge the models’ inherent knowledge, forcing them to rely on the provided context to produce accurate responses.

The CDK-E methodology enables evaluating the interpretability of an LLM by obtaining answers to the DKD questions given a specific context, assessing their correctness, and, therefore, evaluating the model’s interpretability under controlled knowledge conditions. Figure 1 illustrates a diagram with the CDK-E methodology’s key components and process flow.

In this process, the fabricated contexts and DKD questions are integrated with the prompt in the inference pipeline, forming the query sent to the LLM under evaluation. In this study, we assess multiple models to compare their performance in these divergent-context scenarios. Then, the generated responses are analyzed within the performance assessment module, which utilizes independent evaluator LLMs to compute key performance and interpretability metrics. This module compares the generated answers with ground-truth answers to calculate the metrics, thus providing insights into the model’s ability to interpret and adapt to specific contexts while addressing key challenges in XAI for LLMs, such as model reliability, transparency, and alignment with provided contexts.

Specifically, CDK-E methodology addresses the following:

• Biases and Hallucinations: By using divergent contexts, CDK-E enhances the detection of hallucinations and biases in LLM responses as these issues become more apparent when the model is exposed to information that conflicts with its internal knowledge.
• Contextual Misalignment: By evaluating LLM outputs against the provided context, CDK-E helps to identify any misalignment between generated and ground-truths.
• Scalability: The model-independent pipeline, combined with prompt engineering, enables scalable and flexible evaluation across various models and datasets.

In the following sections, the key components are outlined, including dataset fabrication and the performance assessment module.

3.1. DKD: Divergent Knowledge Dataset

As part of the CDK-E methodology, we introduce the Divergent Knowledge Dataset (DKD) to evaluate the interpretability of LLMs in scenarios that deviate from their internal knowledge. The DKD consists of contexts intentionally modified to differ from information the models inherently possess. For each context, we created corresponding questions and ground-truth answers that require the models to rely solely on the provided divergent information to produce accurate responses. This dataset is a foundational component of our approach, ensuring rigorous and contextually relevant evaluation.

Figure 2 illustrates the dataset fabrication process. First, real contexts are altered to create divergent scenarios, and questions are generated based on these modified contexts. This is accomplished using a data generator LLM, which automates the process. This dataset generation is conducted only once, and the resulting data will be used consistently across all evaluated models, ensuring consistency.

Next, ground-truth answers are created for each question. These answers are carefully reviewed by human annotators to ensure their accuracy and relevance to the modified contexts. Finally, a comprehensive human review is conducted to validate the quality of the fabricated contexts, the generated questions, and the ground-truth answers, ensuring that each element aligns with the intended divergent knowledge conditions. Each of these steps is described in detail in the following sections.

3.1.1. Divergent Context Fabrication

We create divergent contexts by systematically altering existing information to challenge the inherent knowledge of the models. These contexts involve modifications to elements such as dates, names, locations, logical sequences, and events across various topics, including history, literature, popular culture, and basic mathematics. The modifications are crafted to be plausible yet absent from the training data of the LLMs, ensuring that models rely solely on the altered information to generate accurate responses.

To construct these contexts, articles from reliable and comprehensive databases can serve as source material. Using a single LLM, we modify key details in the contexts to create coherent yet divergent scenarios. This process involves altering data points—such as dates, names, and locations—to ensure that the contexts are sufficiently challenging and novel for the models. The prompts guiding the LLM in this task are specifically designed to yield contexts that are both coherent and contextually rich.

Figure 3 provides examples of divergent contexts included in the DKD, illustrating modifications to dates, places, and the narratives surrounding key events. These are limited excerpts from the complete set of generated divergent contexts.

3.1.2. Question Design

From the divergent contexts, a series of questions are formulated that require models to rely exclusively on the provided information to produce accurate and coherent responses. These questions vary in complexity, from moderate to high, to ensure they can be answered solely based on the altered context. A single LLM is used to generate a substantial number of questions for each context, with prompts carefully crafted to create questions that are both challenging and directly relevant to the specific alterations introduced. Figure 3 includes two examples of questions generated for the divergent context excerpts.

3.1.3. Validation and Annotation

The divergent contexts and generated questions were manually reviewed by two human annotators who established specific criteria to guide the annotation process. This validation involved each annotator independently checking each generated question for logical coherence, alignment with the modified context, and manually providing answers to establish the ground-truth dataset. The use of human annotators ensures high-quality contextually relevant ground-truths that automated systems might not provide as human judgment captures nuanced contexts and assesses the logical coherence of the questions and answers. When discrepancies arose, annotators discussed and refined their criteria to reach consensus, ensuring consistency and reliability in the annotated data.

3.2. Performance Assessment Module

The CDK-E incorporates a robust performance assessment module based on RAGAS [55] and metrics analyzed by evaluator LLMs. The RAGAS metrics measure the alignment of LLM responses with ground-truth data and contextual information, capturing semantic alignment. Furthermore, in order simulate a human-based assessment and to complement this, we introduce three additional metrics derived from evaluator LLM interpretations. These additional metrics are essential for assessing aspects such as response accuracy and completeness beyond mere semantic alignment, offering a more comprehensive view of model performance in relation to the altered context.

Thus, our assessment employs two primary RAGAS metrics—Answer Semantic Similarity and Answer Correctness—to evaluate correctness and relevance based on embeddings and factual accuracy. These are complemented by our three proposed LLM-based evaluator metrics, which are detailed in the following sections.

3.2.1. Answer Semantic Similarity

The Answer Semantic Similarity (ASS) metric evaluates how closely the generated response matches the ground-truth answer in terms of meaning and context. This metric uses semantic similarity measures to compare the two texts, ensuring that the generated response contains the same information as the ground-truth, even if the wording is different. The score ranges from 0 to 1, with higher values indicating greater semantic alignment.

To calculate the ASS metric, embeddings (vectorized representations) of both the generated answer and the ground-truth answer are obtained using a language model encoder. The embeddings are then compared using cosine similarity, which is calculated as follows:

$$\mathrm{ASS}=cos\left(\theta \right)={\displaystyle \frac{A·B}{\parallel A\parallel \parallel B\parallel }}={\displaystyle \frac{{\sum }_{i=1}^n{A}_i{B}_i}{\sqrt{{\sum }_{i=1}^n{A}_i^2}\sqrt{{\sum }_{i=1}^n{B}_i^2}}}$$

(1)

where

• A represents the embeddings (vectorized form) of the generated answer.
• B represents the embeddings (vectorized form) of the ground-truth answer.
• $A_i$ and $B_i$ are the components of vectors A and B, respectively.
• n is the number of dimensions in the vector space.

3.2.2. Answer Correctness

The Answer Correctness (AC) metric measures the accuracy of the generated answer when compared to the ground-truth, with scores ranging from 0 to 1. A higher score indicates a closer alignment between the generated answer and the ground-truth.

The calculation of the AC is based on the use of two measures: the already introduced Answer Semantic Similarity (ASS) (1) and Factual Correctness ($FC$), also defined by RAGAS [55]. The Factual Correctness quantifies the factual overlap between the generated answer and the ground-truth answer and is calculated as the F1 score as follows:

$$FC={\displaystyle \frac{TP}{TP+\frac{1}{2}(FP+FN)}}$$

(2)

where

• $TP$ (true positive) represents the number of facts or statements that are present in both the ground-truth and the generated answer.
• $FP$ (false positive) represents the number of facts or statements that are present in the generated answer but not in the ground-truth.
• $FN$ (false negative) represents the number of facts or statements that are present in the ground-truth but not in the generated answer.

With the Factual Correctness and Answer Semantic Similarity obtained, the Answer Correctness is calculated as

$$\mathrm{AC}=(1-w)\mathrm{ASS}+w\mathrm{FC}$$

(3)

where $(1-w)$ is the weight assigned to the ASS, and w is the weight assigned to the FC, with $0\le w\le 1$.

3.2.3. LLM-Score-Based Metrics for Accuracy and Completeness

We propose a set of metrics based on a large language model’s scoring system to evaluate the accuracy and completeness of generated answers relative to the ground-truth:

• Accuracy (Acc): Assesses if the LLM-generated answer reflects the factual information presented, calculated by prompting “Does the LLM-generated answer correctly reflect the facts presented in the ground-truth?”.
• Completeness (Cm): Evaluates if the answer includes all key points by prompting “Does the LLM-generated answer cover all key points mentioned in the ground-truth?”.

The key points of the evaluation criteria for both metrics are shown in

Table 1. LLM-score-based evaluation criteria for accuracy and completeness.

Score	Description
0.0	Completely incorrect or irrelevant answer.
0.25	Partially correct answer, but largely incomplete or includes irrelevant information.
0.5	Somewhat correct answer that covers some key points but may miss others or include irrelevant details.
0.75	Mostly correct answer covering most key points with minor omissions or irrelevant details.
1.0	Completely correct answer covering all key points accurately and concisely.

. The complete prompts used for these metrics can be found in Appendix A.6.

These metrics provide a detailed assessment of both the accuracy and completeness of LLM-generated responses, enabling a nuanced evaluation of the model’s alignment with the ground-truth. Each evaluation considers the context in the prompt, the factual content of the ground-truth, and the LLM’s generated response.

3.2.4. LLM Answer Validation

The LLM Answer Validation (LAV) metric uses a large language model to determine, in a binary fashion, whether the generated response is correct when compared to the ground-truth. As the third metric proposed in this study, it ensures consistency with human judgments by leveraging the advanced validation capabilities of language models.

$$\mathrm{LAV}=\left\{\begin{array}{cc}1\hfill & \mathrm{if} \mathrm{LLM} \mathrm{considers} \mathrm{the} \mathrm{response} \mathrm{valid}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

This metric provides straightforward binary validation, enhancing the robustness of the evaluation framework by ensuring that the responses are not only semantically and factually correct but also validated by a sophisticated language model. LAV is the main metric in our framework because it most closely aligns with human evaluative processes, making it capable of discerning subtle differences that determine the correctness of a response. Unlike similarity-based metrics, which can produce high scores for minimal changes, LAV provides a more nuanced assessment of correctness. The prompt used for the evaluation can be found in Appendix A.7.

4. Experimental Setup

In this section, we describe the experimental setup used to evaluate the interpretability of LLMs using the CDK-E. The primary objective is to assess the alignment of LLM responses with the provided divergent contexts and the correctness of these responses based on the ground-truth. We begin by detailing the configuration and creation of the Divergent Knowledge Dataset, followed by a presentation of the baseline models and the evaluation pipeline used in the experiments.

4.1. Configuration and Creation of DKD

The Divergent Knowledge Dataset was meticulously constructed to evaluate the interpretability of LLMs under conditions that challenge their inherent knowledge. This dataset includes 50 contexts covering a broad range of topics, such as history and historical figures, literature and authors, popular culture and entertainment, and science and mathematics. Each context was modified by altering dates, names, locations, logical sequences, and events to create scenarios that are sufficiently challenging for the models.

The contexts were sourced from Wikipedia articles and range from 4000 to 8000 characters in length (approximately 1200 to 2000 tokens). Modifications were performed using GPT-4o [56], transforming the original information into coherent divergent scenarios. The specific prompt used to guide the LLM in this task is detailed in Appendix A.1, and the details of the fabricated context can be found in Appendix B.

For question generation, GPT-4o [56] was also used to create 20 questions per context, resulting in a total of 1000 questions across the dataset. These questions are designed to be answerable solely based on the provided contexts, asking specifically for the altered information. The prompt used to generate these questions is detailed in Appendix A.2.

Finally, a manual annotation and review of the contexts, questions, and ground-truths was conducted by two annotators, who established specific criteria before beginning the review process. This involved verifying the logical coherence and relevance of the questions, as well as manually annotating the ground-truth answers for all 1000 questions. This rigorous validation process ensures the reliability and accuracy of the dataset, providing a robust foundation for evaluating the interpretability of LLMs.

4.2. Evaluated Models and Configuration

The baseline models and pipeline were established using prompt engineering techniques to directly create the queries and integrate the models through APIs provided by OpenAI (for GPT models) and Amazon Web Services (AWS) Bedrock (for the other models). This approach enables precise control over the prompt construction process, ensuring that the appropriate context from the Divergent Knowledge Dataset is injected into each question and that additional instructions can be included as needed.

By leveraging the APIs from OpenAI and AWS Bedrock, we ensure seamless integration of different LLMs without requiring modifications to the rest of the pipeline. This flexibility facilitates the dynamic construction of contextually enriched prompts, guiding the models to generate accurate and relevant responses. In this work, several state-of-the-art models were evaluated using this setup, ensuring a diverse representation of model architectures and capabilities. The models evaluated include the following:

• GPT-3.5 Turbo [2]: An enhanced version of GPT-3, known for its improved performance and efficiency in generating human-like text. It has 175 billion parameters and is trained on a diverse range of internet text. Inference for GPT-3.5 Turbo was conducted using the OpenAI API.
• GPT-4o [56]: The latest iteration in the GPT series, featuring even more parameters and fine-tuned capabilities. GPT-4o offers superior accuracy and a deeper understanding of context compared to its predecessors. Inference for GPT-4o was conducted using the OpenAI API.
• Llama 3 (70B) [57]: Developed by Meta, this model features 70 billion parameters and is designed for large-scale language tasks. Llama 3 demonstrates state-of-the-art performance on a wide range of industry benchmarks and includes new capabilities such as improved reasoning and code generation. Inference for Llama 3 was conducted using AWS Bedrock.
• Mixtral 8x7B [58]: Developed by Mistral AI, this is a Sparse Mixture-of-Experts (SMoE) model that utilizes only 12.9 billion active parameters out of a total of 46.7 billion. It is designed for efficiency and performance, with strong capabilities in multiple languages and coding tasks. Inference for Mixtral was conducted using AWS Bedrock. It is the smallest model we are testing, aiming to determine if our interpretability thesis holds with less powerful models.

For all models, the inference configuration included a temperature of 0.2, a top-k value of 50 (used only for Llama 3 and Mixtral), and a top-p of 0.9. A low temperature setting, such as 0.2, reduces randomness in the output, encouraging the model to produce more consistent and focused responses by prioritizing high-probability words. The top-k sampling, set to 50, limits the choice in next words to the 50 most probable options, allowing some variability while preventing unlikely words from being considered. Lastly, a top-p value of 0.9, or nucleus sampling, dynamically narrows the selection to a subset of words whose cumulative probability reaches 90%, balancing precision with slight diversity. This consistent configuration ensures a fair and controlled comparison across different models, optimizing for accuracy and coherence in responses.

By utilizing these diverse models, we aim to provide a comprehensive evaluation of the CDK-E methodology across different architectures and capabilities. This approach ensures that our findings are robust and generalizable across various state-of-the-art LLMs.

4.3. Contextual Prompt Injection, Evaluator LLMs, and Metric Configuration

For each question, the context is dynamically injected into the prompt, incorporating the full context to instruct the model in generating a concise response. Prompts were crafted to emphasize the relevance of the given context, ensuring models rely solely on the injected information rather than prior knowledge. Specific prompt formats for OpenAI models, LLama 3, and Mixtral 8x7B are detailed in Appendix A.3, Appendix A.4 and Appendix A.5.

To evaluate LLM performance, we calculate Answer Semantic Similarity (ASS), Answer Correctness (AC), LLM-based accuracy (Acc), completeness (Cm), and LLM Answer Validation (LAV). For ASS, embeddings are obtained using OpenAI’s embedding model.

Metrics are calculated using GPT-4 Turbo [59] and Claude 3 Haiku [60] as evaluator LLMs. For AC, facts are retrieved using both models, with $w=0.25$, the default in RAGAS, prioritizing semantic alignment over ground-truth factual overlap.

5. Results and Discussion

In this section, we present and discuss the results obtained from applying the CDK-E methodology to our version of the DKD employing a baseline with different LLMs. Our evaluation focused on the overall correctness of these responses based on a predefined ground-truth as well as on how well these models align their responses with the provided divergent contexts.

5.1. Empirical Results

Table 2. Empirical results in percentages (%) for the CDK-E methodology, using GPT-4 Turbo ($\gamma$) and Claude 3 Haiku ($\kappa$) as evaluator LLMs. It shows the LAV precision and the averages for Answer Correctness (AC), accuracy (Acc), and completeness (Cm) metrics for each evaluated model, with a single value per model for Answer Semantic Similarity (ASS) as it is unaffected by the evaluator.

Model	LAV		Acc		Cm		ASS	AC
	$\mathbf{\gamma }$	$\mathbf{\kappa }$	$\mathbf{\gamma }$	$\mathbf{\kappa }$	$\mathbf{\gamma }$	$\mathbf{\kappa }$	-	$\mathbf{\gamma }$	$\mathbf{\kappa }$
GPT-3.5 Turbo	94.2	95.2	92.1	93.6	90.3	93.3	95.7	76.1	86.3
GPT-4o	96.6	96.9	93.1	95.3	91.4	95.2	97.1	80.8	88.2
Llama 3—70B	92.8	92.9	88.8	90.5	62.3	88.1	87.3	52.2	81.4
Mixtral 8x7B	84.8	91.9	82.4	89.3	77.7	89.7	92.1	66.0	81.0

shows the empirical results of our work using the DKD from 50 contexts with a total of 1000 questions, using GPT-4 Turbo ($\gamma$) and Claude 3 Haiku ($\kappa$) as evaluators. The table presents the LAV precision, which is the total number of correct answers among the total number of questions, and the averages obtained for Answer Semantic Similarity (ASS), Answer Correctness (AC), and the LLM-based accuracy (Acc) and completeness (Cm) metrics for 50 divergent contexts.

LAV precision will be the reference measure in our analysis as it is the most human-like metric that determines whether the answer is correct and thus whether it has aligned with divergent knowledge regardless of whether it is accurate, complete, or lies within the same vector space as the ground-truth.

As shown, the high values of measures such as LAV precision, ASS, or accuracy for both evaluator models indicate that it is indeed possible to interpret, understand, and explain the outputs of these models in terms of the divergent contexts provided, which confirms the main thesis of our methodology, CDK-E.

Table 2 also shows that our interpretability thesis is best fulfilled when the model performs better. GPT-4o shows the highest adherence to the provided divergent contexts with LAVs of 96.6% and 96.9% for GPT-4 Turbo and Claude 3 Haiku as the evaluators, respectively. This indicates that it was highly effective in generating responses that aligned with the altered contexts, reflecting its advanced understanding and reasoning capabilities. This performance proves that leveraging contextual information significantly enhances the interpretability, understanding, and explainability of model outcomes, even when such contexts diverge significantly from inherent knowledge.

GPT-3.5 and Llama 3 also exhibit strong performance in LAV precision, with GPT-3.5 surpassing all the metrics and being the second-best model. Llama 3 obtains a particularly low result in completeness and Answer Correctness (AC). Upon analyzing the answers provided during inference, we observe that this model tends to respond much more tersely than the rest of the models analyzed. Although it adheres well to the divergent information, it often fails to include all the details present in the ground-truth. Such short but correct answers affect both metrics significantly, distorting the real performance of the model.

Mixtral 8x7B, the smallest model analyzed, demonstrates slightly lower, yet still impressive, adherence to divergent contexts, with LAV scores of 84.8% and 91.9%, and with high values for ASS, outperforming Llama 3 in completeness and Answer Correctness. These results confirm that even models with fewer parameters can effectively leverage context to generate accurate and coherent responses.

Regarding the evaluative models, GPT-4 Turbo and Claude 3 Haiku yield generally similar trends, reinforcing the observation that higher-performing models tend to align better with the divergent context provided in CDK-E. However, Claude 3 Haiku tends to assign slightly higher scores to models like Llama 3 and Mixtral compared to GPT-4 Turbo, particularly on LLM-based metrics such as accuracy and completeness. This suggests a slight difference in sensitivity between the evaluators, although the overall patterns remain consistent. Full tables of results for both evaluators are presented in Appendix C, and the estimated costs of using LLMs in the study, along with the exact versions used for each model, are provided in Appendix D.

5.2. Detailed Response Analysis

A detailed analysis of the models’ responses and errors reveals noteworthy findings. GPT-4o demonstrates the strongest understanding and interpretation of divergent contexts, with no hallucinations or major misinterpretations observed. Instances of misalignment with divergent knowledge are rare; however, these typically occur when answers require deeper interpretative reasoning or involve strongly ingrained facts in the model’s internal knowledge. In our analysis, three main types of errors were identified:

• Hallucinations errors: These occur when models generate information that is entirely unrelated to the provided context or divergent knowledge. Those models with inferior performance, especially Mixtral 8x7B, exhibited a relatively high frequency of hallucination errors. For example, in response to a question about a fictional historical event, Mixtral 8x7B introduced unrelated historical facts not included in the prompt, likely drawing from its internal knowledge base.
• Context misalignment: This type of error occurs when models fail to use the divergent context correctly even though relevant information is available. This is the most common error and occurs mainly when the answer to the question is not explicitly found in the divergent context but the model needs to assimilate some of that knowledge to achieve an indirect answer. For example, when a question involves calculating a celebrity’s age based on an altered date of birth, the model may default to its internal knowledge instead of processing the divergent information. Such cases highlight opportunities for improvement, particularly through refined prompting techniques. Approaches like chain-of-thought [32], task decomposition, and sequential reasoning could be employed to guide the model’s focus toward divergent information more effectively. These methods are increasingly utilized in the most advanced models, such as OpenAI’s GPT-o1 series, and show promise in enhancing alignment with context-specific knowledge.
• Verbose and extraneous information: Predominantly found in the GPT models, this error involves providing overly detailed or verbose responses that include information not directly relevant to the question. For instance, when asked about a simple fictional fact, GPT-3.5 added unnecessary background information, diluting the focus of the answer. On the other hand, Llama 3 and Mixtral 8x7B were more concise in their responses, sometimes to the point of omitting important information.

Regarding model performance by content type, the analysis shows that complex topics such as scientific subjects and historical contexts tend to yield more errors across all the models. In particular, questions on topics like deep learning, relativity, and mathematics showed a greater incidence of both hallucination and misalignment errors, likely due to the higher cognitive demand these topics place on the models to override inherent knowledge with fabricated information. In contrast, the models performed more reliably with fictional contexts (e.g., literature and fantasy), showing stronger alignment with divergent knowledge and fewer hallucinations, likely due to the more flexible and creative nature of these contexts.

5.3. Correlations Between Metrics

In this section, we analyze the correlations between various evaluation metrics to validate the effectiveness of our proposed methodology. First, we focus on how LLM-based evaluation metrics (LAV, Acc, and Cm) correlate with objective metrics, such as ASS, which is based on cosine similarity. By understanding these correlations, we aim to identify any alignment or divergence between subjective model-based assessments and objective vector-space metrics, thereby evaluating the overall usefulness of the proposed measures.

Figure 4 shows the correlation between a weighted combination of the LLM-based metrics (0.5 for LAV, 0.25 for Acc, and 0.25 for Cm) and ASS, employing GPT-4 Turbo (Figure 4a) and Claude 3 Haiku (Figure 4b) as evaluators. The weighting was chosen to reflect the relative importance of each metric in capturing core aspects of model alignment with the context: LAV, as the primary indicator of correct answers in our analysis, is assigned the highest weight (0.5), while Acc and Cm each contribute 0.25 to balance precision with thoroughness.

The figure presents the results for each of the 50 contexts in the Divergent Knowledge Dataset (DKD) for the four models analyzed, highlighting the correlation area between ASS and the weighted LLM-based metrics. These results illustrate the alignment between ASS and the combined LLM-based metrics (LAV, accuracy, and completeness), with GPT-4o demonstrating the highest overall alignment. The strong correlation observed between ASS and the weighted LLM-based metrics supports the validity of our proposed evaluation framework, suggesting that models performing well in generating semantically similar answers (high ASS) also score highly in the combined subjective metrics.

In particular, the figure shows that GPT-4o and GPT-3.5 Turbo exhibit the highest correlation between ASS and the weighted metrics with both evaluators. This indicates that models that excel in adhering to divergent contexts are also effective at producing semantically aligned responses. GPT-4o’s high scores across both measures underscore its advanced capability in handling context-specific information, while GPT-3.5 also achieves strong alignment, albeit with slightly less consistency than GPT-4o.

Analyzing the performance for Llama 3 and Mixtral 8x7B, we observe that both models exhibit lower ASS, reflecting a tendency for more concise or less contextually aligned responses. However, GPT-4 Turbo as the evaluator tends to assign lower scores in the LLM-based metrics, while Claude 3 Haiku generally provides higher, although more variable, scores. This contrast suggests that Haiku may be overestimating subjective metrics (or GPT-4 Turbo undervaluing them), highlighting slight differences between the evaluation models.

Furthermore, if we analyze the correlation between LAV precision and LLM-based accuracy, also employing GPT-4 Turbo as the evaluator, we obtain the diagram shown in Figure 5. This diagram shows the correlation between both measures for the 50 contexts for the model that committed the most errors, Mixtral 8x7B. The figure clearly shows how the analysis of response accuracy correlates highly with response validity, confirming the sense of the proposed metrics.

Some of these metrics rely on the evaluative LLM’s ability to respond or score accurately, making them somewhat subjective. Therefore, we decided to manually review the results, especially those of LAV. Additionally, to confirm the consistency of the metrics, we tested the model outputs using the real context while keeping the altered ground-truth and confirmed that the metrics yielded very poor values in every context.

In summary, the high correlation between LAV, ASS, AC, and the LLM-based metrics indicates that models that perform well in maintaining context accuracy also tend to produce responses that are semantically similar, factually correct, and complete. This relationship suggests that improving a model’s ability to adhere to provided contexts can enhance its overall performance across multiple evaluation metrics.

5.4. Discussion

The empirical results obtained confirm that our CDK-E methodology effectively achieves interpretability and explainability for LLMs in context-divergent interactions. This supports the extension of these conclusions to many of the widely used context-based interaction techniques prevalent today. The study contributes valuable insights into whether it is feasible to trust, interpret, and understand model responses when using common context-based interaction techniques.

Unlike the existing XAI methods, which often focus on examining model internals or analyzing feature importance, CDK-E focuses on the interpretability of outputs in scenarios where the context provided diverges from the model’s inherent knowledge. This approach does not aim to replace the current interpretability techniques but rather to complement them by evaluating the models’ ability to adapt to and align with new context-specific information. Given the unique scope of our methodology, it was not feasible to directly compare it with traditional interpretability approaches as CDK-E’s primary goal is not to probe model internal mechanisms but to assess contextual adaptability and coherence.

Models with superior performance, such as GPT-4o, align better with divergent knowledge, committing fewer errors overall. This high alignment underscores the potential of advanced LLMs to provide reliable and accurate explanations for their outputs. The correlations observed between LLM-based metrics and cosine similarity metrics, along with tests using real contexts and meticulous manual validations, confirm the reliability of the metrics used for evaluating the methodology.

While CDK-E demonstrates significant strengths, it is worth noting certain challenges and considerations inherent to this approach. The methodology relies heavily on the construction and quality of divergent contexts, which requires a detailed and resource-intensive annotation process. Additionally, given the focus on context divergence, this study emphasizes controlled static contexts and may require further adaptation to evaluate real-time dynamic scenarios. Nonetheless, these challenges do not detract from the primary contribution of CDK-E as a practical and complementary tool for interpretability.

The results underscore the significance of context in enhancing the interpretability of LLMs. The ability of these models to align their responses with divergent contexts illustrates that contextual information can be effectively used to mitigate issues like hallucinations and biases. However, CDK-E alone does not fully address all the challenges related to explainability, particularly in cases where contextual misalignment leads to subtle biases that may go unnoticed. Additional layers of analysis, such as user studies or error analyses focused on specific biases, could provide a more holistic understanding of how models interact.

In conclusion, the CDK-E methodology offers a robust framework for evaluating the interpretability of LLMs in context-divergent scenarios. The high performance of models across multiple metrics reaffirms the effectiveness of leveraging contextual information to enhance model transparency and trustworthiness. Nonetheless, CDK-E should be viewed as one component of a broader XAI toolkit rather than a standalone solution.

5.4.1. Implications for Future Research

The development of more sophisticated context-based techniques can further improve the interpretability of LLMs, making them more trustworthy for critical applications. Investigating the performance of smaller models, such as Mixtral 8x7B, can provide insights into optimizing resource usage while maintaining high explainability standards. Furthermore, exploring additional evaluation frameworks, domain dependency, and biases can help to refine and ensure comprehensive and consistent assessments of LLM interpretability.

5.4.2. Challenges and Limitations

Although CDK-E demonstrates strong potential for enhancing the interpretability of LLM outputs, certain limitations inherent to the approach must be acknowledged. One key limitation is the dependence on the DKD, which currently consists of a relatively small sample—50 contexts and 1000 questions. Expanding the DKD with additional contexts and question types across a wider range of domains could enhance the robustness of CDK-E, providing a more comprehensive testbed for evaluating LLMs in context-divergent scenarios. However, scaling the dataset involves a resource-intensive process as each context must be carefully curated and validated to ensure that it accurately diverges from general knowledge while maintaining internal coherence.

Additionally, while CDK-E effectively addresses contextual misalignment in a controlled setting, it does not capture all the dimensions of explainability. For instance, incorrect responses that superficially align with the context can still carry subtle biases or inaccuracies that CDK-E alone may not detect. Further layers of analysis, such as targeted bias detection or user-centered studies, could provide deeper insights into model behavior in cases where simple context alignment might mask underlying issues.

6. Conclusions

This study presents the Context-Driven Divergent Knowledge Evaluation (CDK-E) methodology, demonstrating its effectiveness in enhancing the interpretability of large language models through divergent contexts. By leveraging the Divergent Knowledge Dataset (DKD), we evaluated how well LLMs can align their responses with provided contexts that diverge from their inherent knowledge.

The empirical results show that advanced models like GPT-4o and GPT-3.5 Turbo achieve the highest alignment with divergent contexts, with LAV precision scores of 96.6% and 94.2%, respectively, using GPT-4 Turbo as the evaluator. These findings validate our hypothesis that contextual information can significantly improve the interpretability of LLM outputs. The strong correlation between LLM-based metrics (LAV, Acc, and Cm) and Answer Semantic Similarity (ASS) further supports the reliability of our evaluation framework.

Our results also indicate that models with superior performance in traditional metrics (e.g., GPT-4o) are better suited to maintaining contextual accuracy, thereby providing more reliable and transparent responses. This suggests that improving a model’s alignment with provided contexts can enhance its performance across multiple evaluation dimensions. Importantly, CDK-E is intended to complement existing XAI techniques by focusing on the model’s adaptability to context rather than replacing other methods.

In summary, the CDK-E methodology offers a robust complementary framework for evaluating and enhancing the interpretability of LLMs, reinforcing their suitability for high-stakes applications where trust and transparency are critical. Regarding future research, we envision three key directions for expanding CDK-E and advancing explainable AI research:

• Extended Datasets: Expanding the DKD to include more diverse and complex contexts, thereby increasing the robustness of the evaluation framework.
• Advanced Metrics: Developing and integrating new metrics that capture additional dimensions of explainable AI, such as causal inference and user interpretability.
• Interactive Interpretability: Exploring interactive approaches where users can engage with LLMs to understand and refine their responses, enhancing the transparency and usability of AI systems.