Testing Explainability of Chain of Thought for Large Language Models

Abstract

Large Language Models (LLMs) have demonstrated superior abilities in complex tasks such as text generation, reasoning, and question answering. However, the explainability of LLMs becomes weak as the parameters and complexity of LLMs increase. Chains of Thought (CoTs) guide the model to perform step-by-step reasoning and effectively enhance its reasoning ability. The multi-step rationales verbalized in a CoT are widely regarded as the explanation of the model itself. This paper proposes an automated approach to testing the behavioral sensitivity of responses to self-cited evidence in CoTs from sufficiency and necessity perspectives under context intervention. Specifically, we intervene in the reasoning chain by changing the input context and measure the behavioral consistency as a proxy for the faithfulness of the CoT. We test the CoT rationales of mainstream open-source LLMs on multi-hop question-answering tasks. The experimental results show that the self-stated reasoning chain is insufficient and unnecessary. The CoT cannot fully explain the behavior of LLMs.

1. Introduction

Large Language Models (LLMs) have been widely applied and have demonstrated superior capabilities in many complex tasks. However, the large parameter scale and complex internal calculations of LLMs make their decision-making process like a black box. The lack of transparency seriously hinders the application of LLMs in high-risk fields such as healthcare, finance, and justice. Explainable AI (XAI) aims to uncover this black box. XAI utilizes explainability technology to create artificial intelligence systems that users can understand and trust, while maintaining a high level of performance [1]. Explainability refers to explaining the decision-making basis and process of a model [2].

Unlike traditional machine learning models that mainly focus on classification and prediction, LLMs can handle complex tasks, including text generation, reasoning, and question answering. This brings challenges and opportunities for the development of explainability technology. The challenge is that LLMs with large scales and parameters are more difficult to explain. Traditional feature attribution methods, such as SHAP [3], may generate huge computational costs in the process of explaining language models. The opportunity is that the text generation and common-sense reasoning capabilities of LLMs can drive the innovation of explainability technologies. Chain-of-Thought (CoT) prompting guides models to perform logical reasoning step by step and generate prediction results [4]. CoT prompting can be achieved by using direct instructions such as “think step by step” or providing examples of step-by-step reasoning and has become an effective method for improving the reasoning ability of LLMs. The CoT is the step-by-step reasoning process output by LLMs before they provide the final answer. The CoT rationales are widely interpreted as explanations for model decisions [5].

Based on the CoT, users can understand the thinking process of the model and then carry out model optimization and trust building. However, the premise is that the CoT truly reflects the decision-making behavior of the model. If this assumption is not valid, the CoT will mislead users. The model might get the answer first, while the CoT is just a seemingly reasonable explanation fabricated afterwards. Turpin et al. [5] find that CoTs may systematically distort the true reasons for model predictions by biasing model inputs. They ensure that the answers in the few-shot prompt are always “(A)” by reordering the options of multiple-choice questions. When the model is guided to incorrect answers, it frequently generates a CoT that rationalize these answers. Although this CoT seems reasonable, it is actually misleading. Such explanations can trigger a trust crisis for LLMs among users and are more likely to cause security risks and serious consequences in high-risk and sensitive fields.

It is unconvincing to claim the successful construction of explainability merely because the explanation seems reasonable without scientific testing and evaluation [6]. At present, there are few studies on evaluating explainability methods, and the standardization level is low. Vilone and Longo [7] investigated 406 papers related to explainability from 1975 to 2020, of which only 70 focused on explainability assessment. Only by systematically testing explainability and comprehensively verifying the faithfulness of explanation can we discover the defects in explanation. Explainability testing can guide researchers to optimize algorithms in a targeted manner. This will further promote the continuous development of explainability research and ultimately help artificial intelligence models achieve safe applications.

The purpose of explanation is to enable humans to understand the basis for model decisions. The difficulty in explaining LLMs lies in the lack of ground truth, which also poses challenges to the credibility and accuracy of testing explanations. There is no formal definition of the correct explanation in many application scenarios. The method of evaluating an explanation is to capture the subjective perception of the user or observe the impact of the explanation on user behavior [8]. For example, Chen et al. [9] evaluate the counterfactual simulatability of natural language explanations by observing whether explanations can enable humans to accurately infer the results of models under different counterfactual inputs. Although subjective evaluation can directly reflect the degree of human understanding and trust in explanations, costs are high, and the results are subjective. Researchers may unconsciously influence the experimental procedures to meet user expectations. In addition, subjective evaluation is the comparison of the similarity between the explanation and human reasoning, which often ignores the principle of faithfulness. Faithfulness refers to the degree of closeness between the explanation and the decision-making process of the explained model [9]. Even if the explanation is similar to the human reasoning process, it may not be faithful to the true reasoning process of the model. We adopt a behavioral notion of faithfulness in this paper. We frame faithfulness in terms of behavioral alignment: a CoT is faithful if the model’s output is dependent on its self-cited evidence. Objective evaluation does not involve humans. Evaluating the quality of explanation through indicators is beneficial for achieving repeatable, automated, and low-cost explainability testing. The automated explainability testing method proposed in this paper aims to evaluate the behavioral faithfulness of CoTs generated by LLMs through quantifiable indicators.

In this paper, we consider a CoT as the self-description explanation of the model and test whether this explanation is consistent with the actual behavior of the model. We propose an automated approach to test whether the self-cited evidence in CoT is behaviorally influential on the output through input perturbation. This approach does not directly capture the internal computational processes of LLMs. Rather, it provides a scalable, black-box method for assessing behavioral consistency as a proxy for faithfulness: the model’s output should depend on the evidence it reports. Figure 1 shows an overview of the proposed approach. Based on the two assumptions of necessity and sufficiency, this paper constructs a test framework comprising questioning, explanation extraction, reasoning chain intervention, and answer comparison. We change the inputs and observe the changes in the outputs to test the explanatory quality of the CoT. We test whether the model’s output is robust to input perturbations by measuring behavioral consistency. The testing process includes two-step questioning and consistency determination. Different strategies are used to intervene in the reasoning chain, and the answers of LLMs to “complete context” and “context after reasoning chain intervention” are compared to test the explainability of the CoT.

The main contributions of this paper are as follows:

• We propose an automated framework to test the behavioral sensitivity of responses to self-cited evidence in CoTs to help users analyze explainability.
• We design testing indicators to quantitatively measure the behavioral faithfulness of CoTs based on consistency judgment.
• We reveal that the CoT explanation of mainstream open-source LLMs cannot fully reflect the decision-making basis and behavior of the model. There are insufficient and unnecessary issues.

The rest of this paper is organized as follows. Section 2 shows the background and motivation of testing the explainability of CoTs. Section 3 introduces our proposed method. Section 4 presents the experimental setup, including three research questions and the test process. Section 5 describes the experimental results. Section 6 discusses the threats to validity. Section 7 lists the related work, and Section 8 concludes.

2. Background and Motivation

We present the background and the motivation of our work.

2.1. Explainability

There is no generally accepted standard definition for explainability currently. It is difficult to unify the definition of explainability due to the different demands in various application fields [10,11]. Biran and Cotton [12] define explainability as the extent to which an observer can understand the reasons for a decision. Kim et al. [13] believe that explainability refers to the ability of users to accurately and effectively predict the outcome of a method. Doshi Velez and Kim [6] define explainability as the ability to explain or present in terms that are understandable to humans. They believe that explainability can help qualitatively determine whether other demands, such as fairness, reliability, availability, etc., are being met.

Explainability can be divided into ante hoc explainability and post hoc explainability. Ante hoc explainability refers to the explainability of the model itself, such as decision trees, linear models, naive Bayes, etc. Post hoc explainability refers to the generation of explanations for black-box models with opaque decision-making mechanisms. Post hoc explainability can be divided into global explainability and local explainability. Global explainability explains the overall behavior of a model, while local explainability explains individual decisions. In real-world applications, the global distributions of samples cannot be obtained. Thus, the explainability in this paper only focuses on local explainability.

For LLMs trained with the pre-training and fine-tuning paradigms, local explainability approaches are mainly divided into feature attribution-based explanation, attention-based explanation, example-based explanation, and natural language explanation [14]. Among them, natural language explanation refers to the process of generating text to explain the decision-making process of a model. The language model is trained jointly with raw data and manually labeled explanations to enable the model to automatically generate explanations in the form of natural language [15]. Additionally, LLMs can also be used to explain other models. For example, a study by OpenAI shows that GPT-4 can generate natural language explanations for single neuron activation in GPT-2 XL [16].

For LLMs based on the prompting paradigm, the significant increase in model size makes traditional computationally intensive explainability technologies no longer applicable. In addition, the overly complex internal decision-making mechanism of the model cannot be represented by a simplified model. The CoT provides a new way and perspective for understanding the behavior of large language models.

2.2. Mechanistic vs. Behavioral Interpretability

Mechanism interpretability aims to reveal the causal role of specific model components (such as attention heads, neurons, or circuits) in generating output. These approaches often involve techniques like attention pattern visualization [17] or activation patching [18]. While powerful, mechanistic methods require access to internal states and do not scale easily to black-box models.

Behavioral interpretability treats the model as a black box and relies on input–output manipulations to infer its reasoning. Behavioral consistency verifies the matching degree between the external explanation (such as CoT) and the decision behavior, focusing only on the input and output behaviors of the model. The common strategy is to measure the change in prediction by interfering with the input [19,20].

Given the distributed, probabilistic nature of transformer-based LLMs, there is no single ground-truth reasoning chain encoded in weights or activations. We focus on behavioral interpretability in this paper. The CoT is regarded as the self-reporting explanation of the model. We test whether the model’s behavior depends on these self-reporting paths by intervening in the input. Black-box testing avoids the difficulty of accessing internal representations and provides a practical, replicable evaluation framework.

2.3. Chain of Thought

Wei et al. [21] and Wang et al. [22] proposed the chain-of-thought prompting technique. The CoT can guide the language model to generate a reasoning path that breaks down complex reasoning into multiple simple steps by inputting step-by-step reasoning examples. Kojima et al. [4] demonstrated that large language models perform equally well in zero-shot reasoning by adding a simple prompt “Let’s think step by step” to guide the model to think step by step before answering each question.

Prompt-based approaches may not trigger LLMs to provide explanations. DeepSeek-R1 [23] trains the model to provide explanations before making the final prediction and only gives positive rewards if the prediction is correct. Even without human-labeled explanations as ground truths to guide training, LLMs can still learn to provide human-understandable explanations and maintain high accuracy when answering reasoning questions.

The CoT effectively improves the accuracy of LLMs in solving complex problems by decomposing complex problems into a series of sub-problems [21]. Outputting intermediate steps also helps users adjust prompts when they observe abnormal model behavior [24,25]. In addition, the step-by-step reasoning process of LLMs can also serve as a fine-tuning dataset for small LLMs to help improve their reasoning capabilities [26].

2.4. Multi-Hop Question Answering

Multi-hop question answering (Multi-hop QA) focuses on complex reasoning capabilities. Unlike single-hop QA, which can directly match answers through a single document or knowledge fragment, multi-hop QA requires the model to integrate information from multiple data sources or information fragments, and indirectly deduce the answer through logical associations and reasoning chains [27,28,29]. The multi-hop QA task aims to break through the limitations of traditional retrieval-based matching in QA systems. It is closer to the cognitive process of gradually associating and deducing when humans solve complex problems. Multi-hop QA has become a key benchmark for evaluating the deep semantic understanding and logical reasoning ability of LLMs.

The complexity and step-by-step nature of the multi-hop QA reasoning process make it an ideal scenario for testing the CoT. In multi-hop QA tasks, the CoT guides the LLM to output reasoning steps and visually presents the decision-making path of the model from information retrieval to answer generation. For example, when answering the question “The establishment time of the institution where a key figure of a certain historical event once worked”, the LLM needs to first identify the key figure, then associate it with the institution where he or she worked, and finally deduce the establishment time of the institution.

2.5. Motivation

A CoT seems intuitive. However, there is a key question that needs to be confirmed: Does the CoT truly describe the reasoning process of LLMs? Studies have shown that large language models still rely on sentence-level memory capabilities and corpus-level statistical patterns rather than having strong reasoning abilities [30,31]. In multi-hop QA tasks, there may be deviations between the CoT and the actual inference of the model. It is necessary to explore whether the CoT reflects the decision-making behavior of the model, which can provide a key research direction for further optimizing the explainability of LLMs. We propose an automated explainability testing framework to quantitatively test the behavioral consistency of CoTs from the dimensions of sufficiency and necessity.

3. Proposed Approach

The approach proposed in this paper is based on context intervention and answer consistency measurement. We evaluate the behavioral robustness of model responses under input perturbation and measure the degree of behavioral correlation between the information cited in CoT explanations and the outputs. We design testing indicators to measure the impact of CoT reasoning steps on model behavior.

3.1. Assumptions of Explainability Testing

Assumption 1 (Sufficiency Assumption).

If the CoT conforms to the actual behavior of the model, and only the context cited in the CoT is retained, the regenerated answers of the model will maintain high consistency with the original answers.

Assumption 2 (Necessity Assumption).

If the CoT conforms to the actual behavior of the model, removing the context cited in the CoT will significantly reduce the consistency between the regenerated answers and the original answers generated by the model.

3.2. Explainability Testing

The testing framework includes questioning, explanation extraction, reasoning chain intervention, and answer comparison, to test the behavioral consistency of the CoT. Figure 2 shows the test process and the strategies for intervening in the reasoning chain. This proposed approach changes the input of the model and observes the changes in the outputs to determine whether the CoT conforms to the actual decision-making behavior.

In the multi-hop question-answering task, we provide background knowledge paragraphs to the LLM and ask it questions. Our framework is designed to test whether the paragraphs cited in the CoT of the LLM are behaviorally influential on its final answer. The testing process consists of questioning and consistency judgment. We compare the responses of the LLM to “complete context” and “context after reasoning chain intervention” to test the explainability. Firstly, we provide a complete context (the background knowledge in Figure 1) to the model and require the model to think step by step to answer the question. Secondly, we extract the set of background paragraphs cited in the CoT. Then, we carry out targeted intervention on the context, including paragraph filtering or deletion. Thirdly, we question the model again and obtain an answer. Finally, we compare the original answer with the answer after interventions to determine whether there is a significant change in the model outputs, to test the behavioral consistency of the CoT.

3.3. Reasoning Chain Intervention Strategies

The background knowledge (complete input context) consists of multiple paragraphs: $P=\{p_1,p_2,\dots ,p_n\}$. The background knowledge cited in the CoT is referred to as the supporting paragraphs and are sorted according to the reasoning steps: $P^{\prime }=\{p_{s_1},p_{s_2},\dots ,p_{s_m}\}$. $PI$ is the background knowledge set obtained after applying various strategies for the reasoning chain intervention.

Strategy 1 (Keep only supporting paragraphs cited in CoT).

We retain supported paragraphs cited in the CoT and remove all paragraphs not cited in the CoT from the original background knowledge set:

$$PI=P^{\prime },$$

(1)

Strategy 1 verifies the sufficiency of the supporting paragraphs set (information claimed by the CoT for inference). Sufficiency measures whether the information declared by the CoT alone is sufficient for the model to generate the original answer. If the model fails to output a consistent answer while supporting paragraphs are retained, it indicates that the information declared by the CoT is insufficient to support the decision, and the model actually relies on other undeclared information.

Strategy 2 (Remove the supporting paragraph cited in the first step of the CoT).

We remove the supporting paragraph cited in the first step of the CoT from the original background knowledge set, while all other paragraphs (including the supporting paragraphs cited later in the CoT and those not cited) are retained:

$$PI=P\setminus \left\{p_{s_1}\right\}.$$

(2)

Strategy 3 (Remove all supporting paragraphs cited in the CoT).

We remove all supported paragraphs cited in CoT from the original background knowledge set and only retain paragraphs not cited by the CoT:

$$PI=P-P^{\prime }.$$

(3)

A fundamental characteristic of transformer-based LLMs is that they operate as probabilistic next-token predictors. Language generation is inherently stochastic: the same input may result in different computational pathways depending on the sampling choices made at each step. LLMs represent information in superposition across parameters and layers. The same input can activate different combinations of attention heads and neurons, leading to multiple computational pathways that yield the same output distribution. Therefore, the test framework proposed in this paper does not assume that the CoT represents the only possible reasoning path. This paper assesses whether the cited information in the CoT is behaviorally relevant to the model’s output. For situations where there may be multiple reasoning chains that can reach the final answer, we adopt a multi-round removal strategy. After removing the supporting paragraphs cited in the CoT for the first time, we ask the LLM again and remove the supporting paragraphs again based on the CoT to intervene in the reasoning chain multiple times.

Strategy 2 and Strategy 3 are used to test the necessity of CoT explainability. Whether we remove the supporting paragraph involved in the first step of the CoT or all the cited supporting paragraphs, it will affect the model behavior. A significant change in the answer after removing cited paragraphs suggests that the model’s output is sensitive to those paragraphs, i.e., the model behaviorally relies on them. Otherwise, the model may either not depend on those paragraphs or rely on external information. It is easier to test the impact of input perturbations on model behavior in Strategy 2 than in Strategy 3. Part of the information related to answering the question is still retained, while the self-described reasoning chain in the CoT cannot be formed. This makes it easy to test whether the inference of the LLM relies on matching ability while measuring the correlation between the CoT and decision-making behavior of the LLM. Strategy 3 defines relatively lenient requirements for the explainability of LLMs than Strategy 2, as the model is prone to output different answers when all relevant information is removed.

3.4. Test Indicators

We name the original background knowledge set P and the background knowledge set after intervention $PI$. Y is the set of answers output by the LLM based on the original background knowledge set P. $Y^{\prime }$ is the set of answers output by the LLM based on the background knowledge set after intervention $PI$. When the CoT mentions $P_i\subseteq P$, the answer of the LLM is ${\widehat{y}}_i\in Y$. After intervening in the input context, the answer of the LLM is ${\widehat{y}}_i^{\prime }\in Y^{\prime }$ when mentioning $P{I}_i\subseteq PI$ in the CoT. The consistency score measures the average degree of consistency between the answer pairs, while the inconsistency score is the opposite:

$$\begin{array}{cc}\hfill consistency\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)& ={\displaystyle \frac{1}{\left|Y\right|}}\sum _{\begin{array}{c}{\widehat{y}}_i\in Y,\\ {\widehat{y}}_i^{\prime }\in Y^{\prime }\end{array}}\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill inconsistency\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)& =1-consistency\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right),\hfill \end{array}$$

(5)

where $\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$ measures the degree of consistency of the answers. The definition of $\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$ can be found later in Equations (9) and (10). Based on the F1 score and Exact Match (EM), which are universally adopted to assess performance of models on multi-hop QA tasks [32,33], we design two evaluation metrics for calculating $\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$: F1 score and Contain. Specifically, we take F1 as a string-based similarity measure that quantifies the token-level information overlap. Contain measures consistency based on containment relationship and refers to the EM for measuring a strict match. The definition of $\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$, which is calculated using F1 and Contain, is as follows:

• After standardizing the text, we perform word segmentation, then calculate the precision and recall based on word-level overlap to derive the F1 score. In this case, we assign the value of F1 to the $\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$:

  $$\begin{array}{cc}\hfill precision& ={\displaystyle \frac{same\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)}{len\left({\widehat{y}}_i^{\prime }\right)}},\hfill \end{array}$$

  (6)

  $$\begin{array}{cc}\hfill recall& ={\displaystyle \frac{same\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)}{len\left({\widehat{y}}_i\right)}},\hfill \end{array}$$

  (7)

  $$\begin{array}{cc}\hfill F1& =2·{\displaystyle \frac{precision·recall}{precision+recall}},\hfill \end{array}$$

  (8)

  $$\begin{array}{cc}\hfill \delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)& =F1,\hfill \end{array}$$

  (9)

  where $same\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$ counts the number of word matches between the answers, $len\left({\widehat{y}}_i\right)$ measures the total number of words of answer ${\widehat{y}}_i$, and $len\left({\widehat{y}}_i^{\prime }\right)$ measures the total number of words of answer ${\widehat{y}}_i^{\prime }$.
• The calculation rule of Contain in this paper is that after standardizing the text, if ${\widehat{y}}_i^{\prime }$ contains ${\widehat{y}}_i$ or ${\widehat{y}}_i$ contains ${\widehat{y}}_i^{\prime }$, it is regarded as a match. In this case, we define $\delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$ as follows:

  $$\begin{array}{c}\hfill \delta \left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)=contain\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right),\end{array}$$

  (10)

  where $contain\left({\widehat{y}}_i,{\widehat{y}}_i^{\prime }\right)$ measures whether ${\widehat{y}}_i$ is a subsequence of ${\widehat{y}}_i^{\prime }$ or whether ${\widehat{y}}_i^{\prime }$ is a subsequence of ${\widehat{y}}_i$.

Consistency reflects whether the answer after the intervention is consistent with the original answer. When retaining supporting paragraphs, a high consistency score generally denotes satisfactory behavioral relevance between the CoT and the model’s behavior. In the setting where supporting paragraphs are excluded, a low consistency score implies superior relevance; in other words, a high inconsistency score is indicative of strong relevance.

Additionally, we use the unknown rate to reflect the proportion of the model that directly answers “unknown” after removing supporting paragraphs. We define the unknown rate as an indicator of sensitivity to contextual intervention behavior. The reasons for the abstention behavior include not only the removal of supporting paragraphs but also alignment tuning, conservative response strategies, or safety mechanisms. In addition, it is related to format compliance. Therefore, we only consider it as a referential indicator of the behavioral correlation between the cited paragraph and the decision of the model. The high rate of answering “unknown” means sensitivity to the cited paragraphs:

$$\begin{array}{cc}\hfill \mathrm{unknown}\mathrm{rate}& ={\displaystyle \frac{1}{\left|Y^{\prime }\right|}}\sum _{\begin{array}{c}{\widehat{y}}_i^{\prime }\in Y^{\prime }\end{array}}u\left({\widehat{y}}_i^{\prime }\right),\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill u\left({\widehat{y}}_i^{\prime }\right)& =\left\{\begin{array}{cc}1,\hfill & {\widehat{y}}_i^{\prime }\mathrm{is}“\mathrm{unknown}”\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right..\hfill \end{array}$$

(12)

The metrics defined above are operational indicators designed to capture different facets of behavioral dependence. They are not direct measures of semantic equivalence or causal alignment. Rather, they provide reproducible, scalable signals that support inferences about model behavior.

4. Experimental Setup

To test the explainability of mainstream open-source LLMs, we conducted experiments to verify the two assumptions of sufficiency and necessity by applying three strategies (see Section 3). We propose three Research Questions (RQs) to test the behavioral consistency of CoTs for LLMs.

4.1. Research Questions

RQ1. Will the LLM output an answer consistent with the original answer when Strategy 1 is applied? The purpose of this research question is to test the sufficiency of the CoT explanation. Specifically, if the CoT is consistent with the behavior of the model, the model can infer an answer consistent with the original one only based on the supporting paragraphs used in the CoT.

RQ2. Will the LLM output an answer inconsistent with the original answer when Strategy 2 and Strategy 3 are applied? This research question aims to test the necessity of CoT explanation. Removing the supporting paragraphs cited in the CoT will interfere with reasoning behavior. If the model can still return a result consistent with the original answer, it indicates that the self-stated explanation of the LLM is not a necessary reasoning basis of its decision-making.

RQ3. How is the consistency between the model outputs after multiple rounds of Strategy 2 and Strategy 3? Since there may be multiple reasoning chains that can lead to the answer, the model may still be able to obtain the answer after interfering with the reasoning chain for the first time. Therefore, we remove the supporting paragraphs again based on the second reasoning chain, and ask the model to regenerate the answer. Then, we compare the consistency between the last answer and the original answer.

4.2. Data Preparation

We employed an open-source dataset to evaluate the test approach proposed in this paper. The MuSiQue dataset [27] is a multi-hop question-answering dataset with 25K 2–4 hop questions. Hop refers to the number of steps for reasoning. k-Hop means that the model needs to break down the question into k sub-questions and go through k steps of reasoning before the answer can be obtained. When answering the questions in this dataset, the answer to each sub-question must rely on the answer of the previous sub-questions. Each sample includes 20 background paragraphs marked with an index, questions, and reference answers, which are suitable for verifying the dependency relationship between paragraphs and answers. We used the dev set with a total of 2417 samples, including 1252 two-hop questions, 760 three-hop questions, and 405 four-hop questions.

4.3. Implementation

We selected three mainstream open-source large language models from the Open LLM Leaderboard (https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, accessed on 16 March 2026) as test models. Qwen3-Next-80B-A3B-Instruction (https://www.aliyun.com/, accessed on 16 March 2026) and Llama-4-Maverick (https://ai.meta.com/llama/, accessed on 16 March 2026) have reliable general capabilities, and DeepSeek-V3.2-Exp (https://www.deepseek.com/, accessed on 16 March 2026) achieves strong performance on the long context reasoning task. We explicitly report the model call parameters in Appendix A.1.

4.4. Test Process

Step 1. We input background knowledge, questions and format constraints into the model. We designed a prompt to explicitly instruct the model to indicate which paragraphs (by their index) were used to support the answer. The model was required to think step by step, declare supporting paragraph indexes in the order of inference, and give the final answer.

Step 2. Based on different intervention strategies, we filtered or removed background paragraphs according to the support paragraph indexes obtained in the first step. Then, we questioned the model again and recorded the answer. We prompted the model to directly answer “unknown” if it could not infer the answer from the provided background knowledge.

Step 3. We measured test indicators such as consistency between two answers and the unknown rate in the second step. Before calculating consistency, we performed text standardization, including converting the text to lowercase letters, removing all punctuation marks, filtering stop words, and standardizing spaces.

We specify the full prompt template used in Step 1 and Step 2 in Appendix A.2 and the preprocessing procedures with the tokenization rules and the stop-word list in Appendix A.3. If there was a call error or if the answer or the supporting paragraph indexes could not be extracted, the retry mechanism was triggered, with a maximum of 3 retries. If the retries failed, the sample was marked as an error sample. To ensure that extraction failures did not bias our comparisons, we quantified the frequency of errors. We describe the parsing rules and report the error rate in Appendix A.4. Errors were mainly due to the security checks on the input data. The extraction failed mainly because the API returned an empty response. Error samples were excluded from quantitative analysis. The low error rate and extraction failure rate indicated that the extraction mechanism was robust and that the vast majority of samples were successfully processed.

5. Experimental Results and Analysis

We tested the CoT explainability of LLMs by answering research questions. We quantitatively tested the quality of the supporting indexes cited in CoTs by comparing them with the ground-truth supporting facts provided in the MuSiQue dataset. We calculated the average Jaccard similarity and recall of the five experiments in RQ1–3 to observe the reliability of supporting paragraph indexes. Recall refers to the proportion of ground-truth indexes covered by cited indexes.

Table 1. Degree of overlap between the paragraphs cited by CoT and the ground-truth paragraphs.

Metric	DeepSeek-V3.2-Exp (%)	Qwen3-Next-80B-A3B-Instruction (%)	Llama 4 Maverick (%)
Jaccard similarity	79.11	64.45	75.93
Recall	83.89	79.39	82.33

shows the degree of overlap. We evaluated the model performance on the original complete context against MuSiQue ground-truth answers. DeepSeek-V3.2-Exp obtained 68.51%, Llama 4 Maverick 66.46%, and Qwen3-Next-80B-A3B-Instruction 62.69%.

We added matched control baselines for context manipulation. For Strategy 1, we compared “keep only supporting paragraphs” to “keep the same number of randomly chosen paragraphs” and to “keep the same number of non-supporting paragraphs”. For Strategy 2 and Strategy 3, we added “remove the same number of random paragraphs” and “remove the same number of non-supporting paragraphs” as baselines.

Table 2. Baseline of Strategy 1.

Model	Keep Randomly Chosen Paragraphs		Keep Non-Supporting Paragraphs
	Consistency-Contain (%) ↑	Consistency-F1 (%) ↑	Consistency-Contain (%) ↑	Consistency-F1 (%) ↑
DeepSeek-V3.2-Exp	7.37	6.69	4.14	3.77
Qwen3-Next-80B-A3B-Instruction	8.57	9.45	4.89	5.55
Llama 4 Maverick	12.78	12.11	5.10	5.12

↑ indicates a higher value means better performance.

,

Table 3. Baseline of Strategy 2.

Model	Remove a Random Paragraph		Remove a Non-Supporting Paragraph
	Inconsistency-Contain (%) ↑	Inconsistency-F1 (%) ↑	Inconsistency-Contain (%) ↑	Inconsistency-F1 (%) ↑
DeepSeek-V3.2-Exp	56.26	56.14	52.94	52.75
Qwen3-Next-80B-A3B-Instruction	45.40	44.72	40.92	40.54
Llama 4 Maverick	30.46	31.86	26.95	28.45

↑ indicates a higher value means better performance.

and

Table 4. Baseline of Strategy 3.

Model	Remove Randomly Chosen Paragraphs		Remove Non-Supporting Paragraphs
	Inconsistency-Contain (%) ↑	Inconsistency-F1 (%) ↑	Inconsistency-Contain (%) ↑	Inconsistency-F1 (%) ↑
DeepSeek-V3.2-Exp	58.78	59.48	52.20	52.75
Qwen3-Next-80B-A3B-Instruction	53.22	52.95	43.44	43.54
Llama 4 Maverick	36.24	37.82	29.00	30.74

↑ indicates a higher value means better performance.

shows the baselines. The test results all exceeded the baselines, which proved that changes in consistency were specific to CoT-cited evidence rather than generic context reduction.

5.1. Will the LLM Output an Answer Consistent with the Original Answer When Strategy 1 Is Applied?

To answer RQ1, we input the original background knowledge into the LLM and asked it to answer the question. We retained the supporting paragraphs cited in the CoT based on the index extracted from the CoT and removed the others. We re-input the filtered background knowledge and questioned the LLM again. Then, we compared whether the two answers output by the LLM were consistent.

Table 5. Consistency test results of retaining the supporting paragraphs cited in the CoT from the background knowledge.

Model	Consistency-Contain (%) ↑				Consistency-F1 (%) ↑
	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg
DeepSeek-V3.2-Exp	69.01	59.34	56.54	63.96	68.47	58.28	53.46	62.75
Qwen3-Next-80B-A3B-Instruction	60.86	52.24	41.98	55.56	62.11	55.11	40.47	56.29
Llama 4 Maverick	70.13	67.76	52.59	66.64	69.84	64.93	49.25	64.85

↑ indicates a higher value means better performance. Bootstrap 95% confidence intervals (CIs) for key results: DeepSeek-V3.2-Exp: Avg Consistency-Contain [0.621, 0.657]; Avg Consistency-F1 [0.61, 0.645]; Qwen3-Next-80B: Avg Consistency-Contain [0.53, 0.571]; Avg Consistency-F1 [0.61, 0.645]; Llama 4 Maverick: Avg Consistency-Contain [0.646, 0.684]; Avg Consistency-F1 [0.631, 0.666]. Bootstrap CIs were computed with 1000 resamples (seed = 42) for reproducibility.

shows the test results when Strategy 1 was applied. We calculated the consistency for two-hop, three-hop and four-hop questions by using Contain and F1 evaluation metrics. Avg represents the average scores. The reason we used the containment rule is to make the metric more robust to superficial variations in model outputs which do not change the semantic content. We assessed the extent of potential overestimation. We conducted manual validation on a stratified sample of instances, where Contain judged the answers as consistent, while EM judged them as inconsistent. These were the high-risk false positive samples. The samples were drawn proportionally by hop count: 90 from two-hop, 45 from three-hop, and 15 from four-hop, for a total of 150 instances. We manually determined whether the answers were semantically equivalent. The results showed that 119 out of 150 instances (79.33%) were semantically equivalent, which indicated that Contain captured semantic equivalence in most cases compared to EM. Based on this analysis, we adopted Contain as the primary metric. We report the EM results in Appendix B.1.

The average consistency-Contain of the three large language models was around 60%. Llama 4 Maverick had the highest value (66.64%), followed by DeepSeek-V3.2-Exp (63.96%). Qwen3-Next-80B-A3B-Instruction was the lowest (55.56%). The trend of consistency-F1 was consistent with consistency-Contain, ranging from 56% to 65%. DeepSeek performed better when the number of hops was high. We performed statistical analyses on inconsistency-Contain scores to assess the model differences. Between-model comparisons via one-way ANOVA with Tukey HSD post hoc tests revealed significant overall differences at each hop (all p < 0.001). At two hops, DeepSeek and Llama significantly outperformed Qwen3 (p < 0.001). At three hops, all pairwise differences were significant (p < 0.05), with Llama leading. At four hops, DeepSeek and Llama both exceeded Qwen3 (p < 0.01). If the CoT is related to the decision-making behavior of the model, retaining only supporting paragraphs should enable the model to generate consistent answers, with a consistency rate close to 100%. However, the actual result was far lower than this expectation. The inconsistent responses observed after intervention suggest behavior resembling ex-post rationalization, which appears most prominently in the CoT of Qwen3-Next-80B-A3B-Instruction. From a behavioral perspective, the information declared by the CoT is insufficient to support complete inference. In addition, the consistency rate decreases as the number of hops increases. We performed paired t-tests to evaluate the hop-related degradation. The results showed that Qwen3-Next-80B-A3B-Instruction exhibited significant declines between all consecutive hops (two hops → three hops: p < 0.001; three hops → four hops: p = 0.003). As the complexity increases, the proportion of models that could derive the original answer based on supporting paragraphs decreased. The model could not fully reproduce the original reasoning process while retaining only the supported paragraphs cited in the CoT, possibly because it relied on other implicit information that was not cited in the CoT. This phenomenon indicates that CoTs can only partially reflect the decision-making basis of the model and cannot fully cover all key information. There is a significant defect in the sufficiency of the explanation.

To reduce confounds from context length changes, we conducted a length-controlled replacement experiment. We replaced non-supporting paragraphs with an equal-length ellipsis placeholder while keeping the supporting paragraphs unchanged. This ensured the total context length was the same as that of the original input. We re-ran the consistency evaluation under this length-controlled setting on Qwen3-Next-80B-A3B-Instruction. We conducted a paired Wilcoxon signed-rank test between the non-supporting removal setting and the length-controlled replacement setting. Results showed no significant differences across inconsistency-Contain, inconsistency-F1, and unknown rate (all p > 0.05). This confirmed that context length did not confound our main findings.

Conclusion. When only the supporting paragraphs cited in the CoT were retained, the average consistency of the model answers was below 70%, which means the CoT explanation was insufficient. In more than 30% of cases, the supporting paragraphs declared by the CoT were not sufficient conditions for the decision behavior of the model.

5.2. Will the LLM Output an Answer Inconsistent with the Original Answer When Strategy 2 and Strategy 3 Are Applied?

To answer RQ2, after questioning the LLM, we removed the supporting paragraphs involved in the CoT. We re-input the modified background knowledge and questioned the LLM again. Then, we compared whether the two answers output by the LLM were inconsistent.

Table 6. Test results of removing the first supporting paragraph cited in the CoT from the background knowledge.

Model	Inconsistency-Contain (%) ↑				Inconsistency-F1 (%) ↑				Unknown rate (%) ↑
	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg
DeepSeek-V3.2-Exp	77.32	69.21	70.12	73.53	77.54	68.51	70.69	73.55	72.12	57.24	67.16	66.69
Qwen3-Next-80B-A3B-Instruction	76.44	73.82	76.54	75.37	76.21	70.57	77.66	74.68	70.53	67.11	77.04	71.31
Llama 4 Maverick	64.06	48.82	58.02	58.24	64.34	50.84	59.14	59.22	44.33	29.34	33.33	37.79

↑ indicates a higher value means better performance. Bootstrap 95% confidence intervals (CIs) for key results: DeepSeek-V3.2-Exp: Avg Inconsistency-Contain [0.718, 0.752]; Avg Inconsistency-F1 [0.718, 0.752]; Avg Unknown rate [0.647, 0.686]; Qwen3-Next-80B: Avg Inconsistency-Contain [0.739, 0.773]; Avg Inconsistency-F1 [0.73, 0.763]; Avg Unknown rate [0.695, 0.732]; Llama 4 Maverick: Avg Inconsistency-Contain [0.562, 0.601]; Avg Inconsistency-F1 [0.572, 0.61]; Avg Unknown rate [0.359, 0.397]. Bootstrap CIs were computed with 1000 resamples (seed = 42).

shows the test results when Strategy 2 was applied.

The average inconsistency-Contain of all three models was below 76%, with Qwen3-Next-80B-A3B-Instruction having the highest (75.37%), DeepSeek-V3.2-Exp following closely (73.53%), and Llama 4 Maverick having the lowest (58.24%). The trend of inconsistency-F1 was consistent with inconsistency-Contain. Llama 4 Maverick returned consistent answers after removing the first supporting paragraph in the CoT rationales in nearly half of the cases. Even if the first supporting paragraph was removed, the model could still give an answer consistent with the original answer based on other information (such as the remaining paragraphs and prior knowledge obtained during training). The reasoning steps described by the CoT were not necessary conditions for decision-making.

The average unknown rate of Qwen3-Next-80B-A3B-Instruction was the highest, reaching 71.31%. After removing the first supporting paragraph, the model was unable to generate clear answers on over 70% of the samples based on the remaining context. This indicates that its behavioral correlation between the cited paragraphs and the model decisions was relatively high. Llama 4 Maverick performed the worst, with an average unknown rate of only 37.79%. This indicates that there was a gap between the model’s behavior and its self-described thinking path. Even though the supporting paragraph was removed, this model still provided an answer.

We selected a case to introduce the situation when the answer in the second step of Llama 4 Maverick was not “unknown”. In that case, the model made inference errors after removing the first supporting paragraph in the CoT and returned an unrelated answer, as shown in Figure 3. The red part in the figure represents key information. The blue part represents irrelevant interference information, while the green part represents incorrect reasoning. When the background paragraph of “Philippe, Duke of Orléans, was the younger son of Louis XIII of France” is removed, the model assumes that “Philippe, Duke of Orléans” refers to “Louis Philippe I”. The model first derives “Louis Philippe II and his wife are the grandparents of François d’Orléans” based on the interference information. Then, the model makes its first inference error. Françoise Marie de Bourbon is the wife of Philippe II. According to the previous analysis of the model, it should be concluded that Françoise Marie de Bourbon is the grandmother of François d’Orléans. However, the model concludes that “Françoise Marie de Bourbon is the grandmother of Louis Philippe I”. Moreover, the subsequent reasoning of the model does not use this information. Instead, it directly concludes “For François d’Orléans, his paternal grandmother would be the mother of Louis Philippe I. Louis Philippe I’s mother was Louise Marie Adélaïde de Bourbon”. It is worth noting that “Louise Marie Adélaïde de Bourbon” does not appear in the context, which is directly dependent on memory retrieval by the LLM. Then, the model makes another inference error and concludes that “Louis Philippe I…, his grandmother on the father’s side is Louise Marie Adélaïde de Bourbon”, which directly changed the “mother” in the previous conclusion to “grandmother”. Finally, the wrong answer was given: “Louise Marie Adélaïde de Bourbon”. From this case, we can find that the large language model relied on the matching and memory capabilities of similar texts to provide a possible guessed answer. The CoT provided by the LLM was not reliable.

As shown in

Table 7. The proportion that model answers correctly, incorrectly, and as “unknown” under Strategy 2.

Model	Correct Rate (%)	Incorrect Rate (%)	Unknown Rate (%)
DeepSeek-V3.2-Exp	21.75	11.68	66.69
Qwen3-Next-80B-A3B-Instruction	17.61	12.55	71.31
Llama 4 Maverick	35.68	26.57	37.79

, we report how often models produced an incorrect but confident answer versus a correct answer versus “unknown” under Strategy 2. It can be seen from the results that Llama 4 Maverick is inclined to produce an incorrect but confident answer versus “unknown”. The relatively high correct rate compared to the other two models also suggests that Llama 4 Maverick may answer based on prior knowledge or shortcuts.

The test results for removing all supported paragraphs are shown in

Table 8. Test results of removing all supporting paragraphs cited in the CoT from the background knowledge.

Model	Inconsistency-Contain (%) ↑				Inconsistency-F1 (%) ↑				Unknown Rate (%) ↑
	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg
DeepSeek-V3.2-Exp	93.13	94.61	92.84	93.54	93.33	93.69	93.36	93.45	90.42	83.55	92.10	88.65
Qwen3-Next-80B-A3B-Instruction	92.25	93.95	92.10	92.68	92.38	90.83	92.82	91.97	88.82	89.08	96.05	91.05
Llama 4 Maverick	87.22	82.37	85.68	85.43	87.04	83.30	86.39	85.76	68.85	60.66	61.73	65.11

↑ indicates a higher value means better performance. Bootstrap 95% confidence intervals (CIs) for key results: DeepSeek-V3.2-Exp: Avg Inconsistency-Contain [0.926, 0.945]; Avg Inconsistency-F1 [0.925, 0.943]; Avg Unknown rate [0.873, 0.899]; Qwen3-Next-80B: Avg Inconsistency-Contain [0.916, 0.938]; Avg Inconsistency-F1 [0.908, 0.931]; Avg Unknown rate [0.899, 0.921]; Llama 4 Maverick: Avg Inconsistency-Contain [0.84, 0.868]; Avg Inconsistency-F1 [0.845, 0.87]; Avg Unknown rate [0.632, 0.669]. Bootstrap CIs were computed with 1000 resamples (seed = 42).

. The average inconsistency-Contain of all three models exceeded 85%, with DeepSeek-V3.2-Exp having the highest (93.54%), followed by Qwen3-Next-80B-A3B-Instruction (92.68%), and Llama 4 Maverick having the lowest (85.43%). The results of applying Strategy 3 were significantly higher than those of applying Strategy 2. As shown in Figure 4, after removing all the support paragraphs cited in the CoT rationales, although the models still output consistent answers with the original ones on some samples, the overall consistency performed better compared with the results of removing the first support paragraph. This indicates that when the complete information basis cited in the CoT is removed, the ability of the model to generate original answers is limited. When the first supporting paragraph is removed, although the reasoning chain is affected, the LLM may guess the answer based on the context related to the question. After removing all supporting paragraphs, almost all relevant contexts are discarded, and the difficulty for models to obtain the same answer through matching significantly increases. However, the CoT does not fully cover the necessary basis for model decision-making. Although the deficiency in the necessity of explanation has been alleviated, it has not been eliminated.

The average unknown rate of Qwen3-Next-80B-A3B-Instruction is the highest when removing all supporting paragraphs, reaching 91.05%. The model cannot generate clear answers on over 90% of the samples based on remaining knowledge. DeepSeek-V3.2-Exp ranks second, with an average unknown rate of 88.65%, and its overall performance is similar to that of Qwen3-Next-80B-A3B-Instruction. The abstention behavior might be related to the conservative response strategy or security mechanism of these models. Llama 4 Maverick still performs the worst, with an average unknown rate of only 65.11%. Even if all supporting paragraphs are removed, the model can generate clear answers on over 30% of the samples. Although the inconsistency rate is high after removing all supporting paragraphs, the low unknown rate reflects that the model is more inclined to guess an incorrect answer rather than answer “unknown”. There will be many flawed reasoning steps in CoT rationales, which will severely affect the quality of the CoT as an explanation.

Conclusion. After removing supporting paragraphs cited in the CoT rationales, all three models had outputs that were consistent with the original answer, which indicates that there is a gap between CoT explanation and model behavior. The self-described CoT appears to rationalize after the fact and only serves as the reasoning basis for the model to choose to present, rather than necessarily as the basis that the model must rely on. The necessity of the CoT explanation is limited.

5.3. How Is the Consistency Between the Model Outputs After Multiple Rounds of Strategy 2 and Strategy 3?

The distributed representation of LLMs allows for multiple reasonable reasoning chains. A correct answer may be obtained through different combinations of evidence. The faithfulness, as defined behaviorally, is not about identifying a unique ground-truth chain but about measuring the influence of cited evidence on the output. When multiple paths exist, deleting one set of evidence may not change the answer if another path remains accessible. Our multi-round removal strategy was precisely designed to address this multi-path situation. For example, as shown in Figure 5, when the question is “What type of animal is Xiao Liwu’s mother”, the standard problem decomposition step given by the dataset is to deduce that Xiao Liwu’s mother is Bai Yun based on the 11th paragraph. Then, according to the 14th paragraph, it is known that Bai Yun is a panda. However, in reality, there is other implicit information that can also lead to the answer. According to the first paragraph, Xiao Liwu is Zhen Zhen’s full sibling, and Zhen Zhen’s mother is Bai Yun. It can be concluded that Xiao Liwu’s mother is also Bai Yun. Then, based on the 14th paragraph, it can be inferred that Bai Yun is a panda. The final answer can be deduced: Xiao Liwu’s mother is a panda. For the situation where the model can obtain the answer after the first disruption of the thinking chain, we adopted a multi-round removal strategy. We removed the context again based on the CoT of the second response of the model and compared the consistency between the final answer and the original answer to test explainability to a loose extent. The experimental results are shown in

Table 9. Test results of removing the first supporting paragraph cited in the CoT from the background knowledge twice.

Model	Inconsistency-Contain (%) ↑				Inconsistency-F1 (%) ↑				Unknown Rate (%) ↑
	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg
DeepSeek-V3.2-Exp	89.22	79.61	78.52	84.38	89.00	79.42	79.13	84.33	87.38	72.37	75.56	80.78
Qwen3-Next-80B-A3B-Instruction	86.26	84.21	86.17	85.42	86.46	80.84	87.53	84.87	85.22	78.42	88.89	84.75
Llama 4 Maverick	79.63	67.89	67.90	73.87	79.94	68.64	70.10	74.74	61.34	42.76	41.23	52.35

↑ indicates a higher value means better performance. Bootstrap 95% confidence intervals (CIs) for key results: DeepSeek-V3.2-Exp: Avg Inconsistency-Contain [0.83, 0.858]; Avg Inconsistency-F1 [0.829, 0.857]; Avg Unknown rate [0.791, 0.824]; Qwen3-Next-80B: Avg Inconsistency-Contain [0.842, 0.869]; Avg Inconsistency-F1 [0.835, 0.862]; Avg Unknown rate [0.834, 0.861]; Llama 4 Maverick: Avg Inconsistency-Contain [0.722, 0.756]; Avg Inconsistency-F1 [0.73, 0.764]; Avg Unknown rate [0.504, 0.542]. Bootstrap CIs were computed with 1000 resamples (seed = 42).

and 

Table 10. Test results of removing all supporting paragraphs cited in the CoT from the background knowledge twice.

Model	Inconsistency-Contain (%) ↑				Inconsistency-F1 (%) ↑				Unknown Rate (%) ↑
	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg	2-Hop	3-Hop	4-Hop	Avg
DeepSeek-V3.2-Exp	95.21	96.05	95.31	95.48	95.56	95.85	96.18	95.76	96.33	93.42	95.31	95.36
Qwen3-Next-80B-A3B-Instruction	94.17	95.39	94.81	94.59	94.04	92.37	95.56	93.77	95.93	93.55	98.27	96.94
Llama 4 Maverick	93.29	88.03	89.38	90.96	92.93	89.56	89.52	91.30	75.24	70.26	63.95	71.93

↑ indicates a higher value means better performance. Bootstrap 95% confidence intervals (CIs) for key results: DeepSeek-V3.2-Exp: Avg Inconsistency-Contain [0.947, 0.963]; Avg Inconsistency-F1 [0.95, 0.965]; Avg Unknown rate [0.945, 0.961]; Qwen3-Next-80B: Avg Inconsistency-Contain [0.938, 0.955]; Avg Inconsistency-F1 [0.928, 0.946]; Avg Unknown rate [0.962, 0.976]; Llama 4 Maverick: Avg Inconsistency-Contain [0.899, 0.921]; Avg Inconsistency-F1 [0.902, 0.923]; Avg Unknown rate [0.701, 0.736]. Bootstrap CIs were computed with 1000 resamples (seed = 42).

.

From the experimental results, we can see that when the first supporting paragraph was removed twice, Llama 4 Maverick performed the worst, with an average inconsistency-Contain of 73.87%, and the average unknown rate (52.35%) was much lower than that of the other two models. This further confirmed its highest dependence on undeclared information from the CoT. Even after multiple rounds of removing the first supporting paragraph, the model still provided consistent answers on over 25% of the samples. After multiple rounds of reasoning chain intervention, all three large language models still had some outputs that were consistent with the original answers.

The intervention effect of removing all supported paragraphs twice was significantly enhanced. The average inconsistency-Contain of the three models exceeded 90%, with DeepSeek-V3.2-Exp having the highest (95.48%), followed by Qwen3-Next-80B-A3B-Instruction (94.59%). The average unknown rates of DeepSeek-V3.2-Exp and Qwen3-Next-80B-A3B-Instruction both exceeded 95%, which means that the models were unable to generate answers based on the remaining knowledge on over 90% of the samples. Although the average inconsistency-Contain of Llama 4 Maverick increased to 90.96%, the average unknown rate (71.93%) was still lower than the other two models. This suggests that Llama 4 Maverick may be more inclined to fabricate even the wrong answer based on relevant information or relying on memory abilities.

We compared test results under multiple rounds of removal and a single round of removal. The comparison results can be found in Appendix B.2. To evaluate the impact of multi-round removal of supporting paragraphs on model performance, we conducted paired t-tests and Wilcoxon signed-rank tests ($\alpha$ = 0.05) on inconsistency-Contain, inconsistency-F1, and unknown rate. For all three LLMs, multi-round removal led to significantly higher inconsistency and unknown rate compared to single-round removal (all p < 0.05), which confirms that repeated removal of supporting paragraphs exacerbates model inconsistency and reduces its ability to generate valid answers. As more relevant contexts are discarded, the difficulty for the model to output the same answer increases. Removing all background paragraphs cited in the CoT leaves less information related to the answer compared to removing the first paragraph cited in the CoT. Multiple rounds of removal delete more relevant information compared to a single round of removal. When removing a small amount of relevant information, the self-narrative inference chain in the CoT is broken, while some information related to the answer is still retained. In this case, it is easier to test the CoT of the LLM. If the LLM can provide consistent answers, it means that it may rely on matching and not conform to the reasoning behavior described in the CoT.

Conclusion. Considering that there are multiple reasoning chains that can lead to the final answer, we conducted multiple rounds of removal to repeatedly intervene in the self-stated reasoning chains in CoT to test the explainability. Compared with simple removal, the LLMs performed better in the case of multiple rounds of paragraph removal. Because a large amount of relevant information was discarded, the difficulty for LLMs to obtain consistent answers through matching increased.

5.4. Summary of Answers to RQs

Based on the experimental results and the analysis of the three research questions, we can summarize the following two conclusions:

• The sufficiency of the CoT explanation is flawed. When only the supporting paragraphs cited in CoT rationales are retained, the LLMs cannot reproduce the original answer in a relatively high proportion of cases. This indicates that CoT rationales are not sufficient conditions for decision-making.
• The necessity of CoT explanation is flawed. After removing the supporting paragraphs cited in CoT rationales, the model can generate consistent answers in a higher proportion. This indicates that CoT rationales are not necessary conditions for decision-making.

These conclusions indicate that the CoT explanation of current mainstream LLMs cannot fully reflect the behavior of the model. There are insufficient and unnecessary problems. The explainability of CoTs can be optimized based on the goals of sufficiency and necessity in the future.

6. Threats to Validity

We now discuss the threats to the validity of our results from two dimensions.

6.1. Internal Validity

The reasoning process that humans consider correct can be obtained based on experience, while the true reasoning process that is faithful to the decision-making mechanism of the LLM is unknown. Our proposed testing method avoids the threat of exploring the ground truth by observing the changes in the results of LLMs to indirectly test whether the CoT is consistent with the model’s behavior.

Transformer-based LLMs operate via distributed representations and probabilistic next-token prediction. There is no clearly defined, single symbolic reasoning chain internally encoded in the model. We dealt with the situation where there are multiple effective reasoning paths from two aspects. First, faithfulness was defined as behavioral consistency rather than path consistency; second, we introduced a multi-round removal strategy for testing multiple reasoning paths.

The test indicators proposed in this paper are operational measures designed to capture different facets of behavioral dependence, not direct measures of semantic equivalence or causal alignment. The metrics have limitations related to answer formulation, semantic matching, and abstention behavior. The F1-based indicator is a function of string normalization choices related to the preprocessing pipeline. We employed it as a similarity measure quantifying the token overlap. Although it is not a principled measure, it can reflect behavioral consistency in a practical and reproducible manner.

The interventions operated at the paragraph level, which means we could only remove or replace full supporting paragraphs. This limited the granularity of the analysis, as we could not isolate individual sentences or facts within paragraphs.

Behavioral consistency does not measure direct causal alignment between internal states and explanations. Despite this, we believe it can offer valuable insights. We evaluated the behavioral robustness under context intervention as a proxy for explanatory faithfulness. It tested a minimal necessary condition for faithfulness: if the model’s answer depended on the evidence it cited, the CoT was at least behaviorally relevant. Future work could combine behavioral interventions with finer-grained analyses, such as attention attribution or causal tracing.

6.2. External Validity

Test-case generation is challenging [34,35,36]. Due to time-cost limitations, we could not conduct detailed testing on all LLMs and datasets. As a preliminary step toward broader generalization, we added experiments on the commercial model GPT-4o-mini (see Appendix B.3). We re-ran the experiment with the temperature set to zero to improve reproducibility and report the results in Appendix B.4. We will extend the testing framework to other prominent models in future work. The testing concept proposed in this paper can be extended to other scenarios, as long as the explainability technology can provide explanations for a single sample and the context dependencies can be extracted from the explanation. Based on the testing schema proposed in this paper, users can test explainability by changing the inference preconditions and comparing the consistency of the results.

7. Related Work

We list related work in three categories: explainability based on input perturbations, evaluation of traditional feature attribution algorithms, and evaluation of CoTs.

7.1. Explainability Based on Input Perturbations

The research on using input perturbations to explain models is constantly increasing. The core idea is to infer which features are influential by modifying inputs and observing changes in outputs. LIME [37] perturbs input instances to learn a local proxy model. SHAP [3] uses Shapley values from cooperative game theory to attribute importance. Goldberg et al. [38] propose an approach to creating a Bi-directional Decision Support System (DSS) as an intermediary between an expert and a machine learning system to select the optimal solution, where the user can change some initial inputs and see the changes in the output results. DSS is conceptually similar to the approach proposed in this paper. DSS aims to provide explanations to help users understand the decisions of machine learning systems. This paper aimed to test the CoT explanation by perturbing the input and observing the impact on the model output behavior.

7.2. Evaluation of Traditional Feature Attribution Algorithms

The common strategy for explainability evaluation is to remove features from the input and observe the drop in model performance. For example, Shah and Sheppard [39] design a pair of experiments to evaluate the explanations generated by LIME on a Convolutional Neural Network (CNN). Warnecke et al. [40] remove the most relevant features sequentially from a sample and measure the drop in classification scores to evaluate the descriptive accuracy of explanations. However, the training data and test data come from different distributions. Without retraining, these methods cannot determine whether the drop in model performance is due to a distribution shift. Hooker et al. [41] evaluate the explainability by observing the decrease in the performance of the retrained model after the important features are removed. However, although the model structure is the same, the evaluated model is different from the model used to obtain feature importance.

In this paper, we proposed a universal framework of explainability testing that can be adapted to other indicators. Unlike the common strategies that focus on performance degradation, we only paid attention to whether the results of the model were consistent. Based on consistency judgment, we avoided the problems existing in the current common strategies and tested the explainability of CoTs for LLMs.

7.3. Evaluation of CoTs

A CoT guides models to think step by step and generate results. The CoT rationales have been widely regarded as explanations for the LLM. However, a CoT may not be faithful to the LLM. Turpin et al. [5] reorder the options of multiple-choice questions in the few-shot prompt to ensure the answers are always “(A)”. They find that CoT may systematically distort the true reasons for model predictions. When the model is guided to incorrect answers, the CoT frequently rationalizes these answers. Chen et al. [9] evaluate the counterfactual simulatability of explanations generated by the LLM. They observe whether explanations can enable humans to accurately infer the results of models under different counterfactual inputs.

However, there is no unified testing framework up to now. The standardization level of testing is relatively low. This paper proposed an automated testing framework and designed test indicators to quantitatively evaluate the behavioral faithfulness of CoTs.

8. Conclusions

This paper constructed a testing framework, which included questioning, explanation extraction, reasoning chain intervention, and answer comparison, based on the two assumptions of necessity and sufficiency. We used reasoning chain intervention strategies to change the input of the model and compared the consistency of answers to test whether responses of LLMs depended on self-cited evidence in CoT explanations. The experiment results showed that the CoT explanation of current LLMs generally had deficiencies in sufficiency and necessity. When the supporting paragraphs of the CoT rationales were retained, the information cited in the CoT could not independently support the model to reproduce the original reasoning process. When the support paragraphs were removed to interfere with the self-stated reasoning chain in the CoT, the models could generate an answer consistent with the original answer to a large extent through undeclared implicit paths. The CoT had a tendency to rationalize afterwards and relied on implicit information.

In future work, we plan to optimize the CoT generation mechanism and promote the evolution of explanations for LLMs from surface rationality to intrinsic trustworthiness. Researchers can also apply the test framework to other metrics to expand the coverage of explainability tests. In addition, subjective evaluation can be added based on the testing approach proposed in this paper to improve the explainability testing. This automated testing approach can be considered to be combined with interactive approaches to test the quality of the explanations by observing whether the user can reproduce the decision of the model based on the explanations.