Cognitive Biases in Large Language Models: A Systematic Quantitative Assessment and Debiasing Analysis

Abstract

Large Language Models (LLMs) are increasingly deployed in decision-support systems across high-stakes domains, yet their susceptibility to cognitive biases—systematic deviations from rational judgment well-documented in human psychology—remains poorly understood in quantitative terms. Existing studies typically examine a narrow set of biases, test a single model family, and rely on qualitative assessments of bias presence. In this work, we present a rigorous experimental framework, inspired by the methodology of experimental physics, for the systematic quantitative measurement of cognitive biases in LLMs. We introduce the Bias Strength Index (BSI), a normalized metric with associated confidence intervals that quantifies the magnitude of bias on a continuous scale, and we decompose the total uncertainty into statistical and systematic components—the latter arising from prompt reformulation. We evaluate a comprehensive taxonomy of eleven cognitive biases (including anchoring, framing effect, confirmation bias, availability heuristic, sunk cost fallacy, bandwagon effect, status quo bias, and others) across eight state-of-the-art LLMs from seven families: GPT-4.1 Mini, Claude 3.5 Sonnet, Gemini 2.5 Flash, Llama 3.3 70B, Llama 3.1 8B, Mistral Large (mistral-large-2411), DeepSeek V3, and MiniMax M2.5. Each bias is probed through multiple semantically equivalent prompt variants, with N = 100 independent trials per configuration, yielding a dataset of over 70,000 model responses. Our results reveal that all tested models exhibit non-zero bias effects for multiple bias categories, though with markedly different profiles. A trial-level Generalized Linear Mixed-Effects Model (GLMM) analysis finds statistically significant bias effects in 27 of 43 testable bias–model combinations (62.8%) after multiple-comparison correction, while a more conservative variant-level test—which requires effects to generalize across prompt formulations—yields only one significant result, highlighting the dominant role of prompt-induced systematic uncertainty. Framing and primacy/recency effects are near-universal, while susceptibility to other biases varies substantially across model families. We further evaluate three debiasing strategies—zero-shot chain-of-thought, adversarial counter-prompting, and role-based prompting—applied at inference time without modifying model weights. Our findings provide a quantitative foundation for auditing cognitive biases in LLMs and highlight the bias-dependent effectiveness of prompt-based debiasing techniques.

1. Introduction

Large Language Models (LLMs) have rapidly transitioned from research curiosities to foundational components of decision-support systems in medicine [1], law [2], finance [3], and public policy [4]. Their ability to generate fluent, contextually appropriate text has led to widespread adoption not only as information retrieval tools, but increasingly as autonomous or semi-autonomous decision-making agents. This growing reliance raises a fundamental question: do LLMs exhibit systematic cognitive biases analogous to those documented in human cognition?

Cognitive biases—systematic patterns of deviation from normative rationality in judgment and decision-making—have been extensively studied in humans since the seminal work of Tversky and Kahneman [5,6]. These biases, including anchoring [5], framing effects [7], confirmation bias [8], and the availability heuristic [9], arise from the use of mental shortcuts (heuristics) that, while often efficient, can lead to predictable errors. If LLMs—trained on vast corpora of human-generated text—have internalized these biased patterns of reasoning, their deployment in critical applications could systematically distort decisions at scale, amplifying rather than mitigating human cognitive limitations.

Recent work has begun to explore this question. Sumita et al. [10] surveyed cognitive biases in LLMs and tested mitigation strategies on GPT-3.5 and GPT-4, finding that the AwaRe method partially reduces bias effects. Lou and Sun [11] demonstrated that GPT-4 and GPT-4o are susceptible to anchoring bias, particularly when anchors are attributed to perceived experts, and that standard mitigation techniques (chain-of-thought, explicit instructions to ignore anchors) are largely ineffective. Echterhoff et al. [12] systematically evaluated cognitive biases in LLM decision-making across multiple scenarios and tested several mitigation strategies. Cheung et al. [13] showed that LLMs exhibit amplified cognitive biases in moral decision-making compared to humans. These studies have revealed that the problem is real and consequential, yet several critical gaps remain.

First, existing studies typically evaluate a narrow set of biases (three to six), making it difficult to construct a comprehensive profile of a model’s cognitive bias landscape. Second, most studies test models from a single family (predominantly OpenAI’s GPT series), leaving open the question of whether biases are universal properties of autoregressive language models or idiosyncratic to specific training procedures and data. Third, and most importantly, the field lacks a rigorous quantitative framework for measuring bias strength. Most studies report bias as a binary phenomenon (present or absent) or as a difference in accuracy under biased versus unbiased conditions, without formal metrics, confidence intervals, or uncertainty decomposition. Fourth, the sensitivity of bias measurements to prompt formulation—a form of systematic uncertainty analogous to instrumental systematics in experimental physics—has not been systematically characterized.

In this paper, we address these gaps through the following contributions:

• We define the Bias Strength Index (BSI), a normalized, continuous metric for quantifying cognitive bias magnitude, with associated statistical and systematic uncertainty estimates. The systematic component captures the sensitivity of bias measurements to semantically equivalent prompt reformulations, providing a measure of measurement robustness.
• We construct a comprehensive benchmark of eleven cognitive biases, each probed through multiple prompt variants designed to isolate the target bias while controlling for confounds. The benchmark comprises over 70,000 individual model responses.
• We conduct the first multi-family comparison of cognitive biases across eight LLMs from seven families (GPT-4.1 Mini, Claude 3.5 Sonnet, Gemini 2.5 Flash, Llama 3.3 70B, Llama 3.1 8B, Mistral Large, DeepSeek V3, and MiniMax M2.5), enabling the identification of both universal and model-specific bias patterns, as well as a controlled analysis of the effect of model scale within the Llama family.
• We evaluate three inference-time debiasing strategies—zero-shot chain-of-thought, adversarial counter-prompting, and role-based prompting—on the model exhibiting the strongest baseline biases, and characterize their bias-dependent effectiveness.
• We release our complete benchmark, prompts, raw responses, and analysis code as an open-source resource to facilitate reproducibility and future research.

The remainder of this paper is organized as follows: Section 2 reviews the relevant background on cognitive biases in human psychology and AI systems. Section 3 presents our experimental methodology, including the bias taxonomy, the BSI metric, and the debiasing strategies. Section 4 reports the experimental results. Section 5 discusses the implications for AI safety and the comparison with human cognitive biases. Section 6 concludes the paper.

2. Background and Related Work

2.1. Cognitive Biases in Human Decision-Making

The systematic study of cognitive biases was pioneered by Tversky and Kahneman [5], who demonstrated that human judgment under uncertainty relies on a limited number of heuristics that, while computationally efficient, produce systematic and predictable errors. Their research program, later synthesized in Kahneman’s dual-process theory [6], distinguishes between fast, automatic, intuitive reasoning (System 1) and slow, deliberate, analytical reasoning (System 2). Cognitive biases arise predominantly from System 1 processing, where rapid pattern matching and associative memory substitute for careful probabilistic reasoning.

The literature has identified over 180 distinct cognitive biases [14], though many are closely related or overlapping. For the purposes of this study, we focus on biases that (a) have well-established experimental paradigms in human psychology, (b) can be operationalized through text-based prompts, and (c) are relevant to the decision-support applications of LLMs. The eleven biases selected for our taxonomy are detailed in Section 3.1.

Among the most robust and well-replicated biases are anchoring [5], in which exposure to an initial numerical value systematically shifts subsequent estimates toward that value; the framing effect [7], in which logically equivalent descriptions of a decision problem lead to different choices depending on whether outcomes are framed as gains or losses; confirmation bias [8], the tendency to seek, interpret, and recall information that confirms pre-existing beliefs; and the availability heuristic [9], in which the perceived probability of an event is influenced by the ease with which examples come to mind. These biases have been replicated across cultures, expertise levels, and experimental paradigms, establishing them as fundamental features of human cognition rather than artifacts of specific experimental designs. Notably, cognitive biases have also been shown to influence aggregate market behavior: Pagliaro [15] demonstrated that processing fluency biases (the alphabet effect and ticker fluency) produce measurable pricing anomalies in US equity markets, underscoring the real-world consequences of systematic irrationality.

2.2. Cognitive Biases in AI Systems

The question of whether AI systems can exhibit cognitive biases has gained urgency with the deployment of LLMs in decision-critical applications. Unlike traditional algorithmic biases—which arise from biased training data or discriminatory features and have been extensively studied in the fairness literature [16]—cognitive biases in LLMs represent a distinct phenomenon: the reproduction of systematic reasoning errors that mirror those observed in human psychology.

Several mechanisms have been proposed for how cognitive biases emerge in LLMs. First, since LLMs are trained on human-generated text, they may learn biased reasoning patterns directly from the training corpus [17]. Second, the autoregressive generation process—in which each token is conditioned on all preceding tokens—may create path dependencies analogous to anchoring effects. Third, reinforcement learning from human feedback (RLHF), used to align models with human preferences, may inadvertently reinforce biased patterns if human annotators themselves exhibit cognitive biases during the feedback process [18].

Empirical evidence for cognitive biases in LLMs has accumulated rapidly. Jones and Steinhardt [19] demonstrated that language models exhibit systematic biases in probability estimation tasks. Hagendorff et al. [20] showed in Nature Computational Science that LLMs reproduce human-like intuitive (System 1) reasoning errors that diminish with model scale and instruction tuning. Itzhak et al. [21] found that instruction-tuned models can be more susceptible to certain biases than their base counterparts. In the clinical domain, Mahajan et al. [22] demonstrated that LLMs exhibit anchoring and framing biases in diagnostic reasoning tasks, with implications for medical AI systems. Echterhoff et al. [12] systematically evaluated cognitive biases in LLM decision-making across multiple scenarios.

However, the existing literature is fragmented. Studies differ in their choice of biases, models, experimental designs, and evaluation criteria, making cross-study comparisons difficult. There is no standardized benchmark or metric for cognitive bias evaluation, unlike other areas of LLM assessment such as factual accuracy (TruthfulQA [23]) or mathematical reasoning (MATH [24]).

2.3. LLM Evaluation and Benchmarking

The evaluation of LLMs has evolved from simple perplexity measurements to comprehensive multi-task benchmarks. General-purpose benchmarks such as MMLU [25], HellaSwag [26], and BIG-bench [27] assess knowledge, reasoning, and language understanding across diverse domains. More specialized benchmarks target specific capabilities: TruthfulQA [23] for factual accuracy, MATH [24] for mathematical reasoning, and APPS [28] for code generation.

For bias evaluation specifically, the landscape is less mature. The CoBBLEr benchmark [29] evaluates six cognitive biases in LLM-as-judge settings, focusing on evaluation biases rather than decision-making biases. StereoSet [30] and WinoBias [31] target social stereotypes rather than cognitive biases. The BBQ benchmark [32] addresses social biases through ambiguous question answering. None of these benchmarks provide a comprehensive, quantitative assessment of cognitive biases with formal metrics and uncertainty quantification.

Our work fills this gap by proposing a benchmark that combines the breadth of bias coverage with the quantitative rigor of measurement science, drawing on concepts from experimental physics—such as systematic versus statistical uncertainty decomposition and controlled experimental design—to establish a robust framework for cognitive bias assessment in LLMs.

3. Methodology

3.1. Taxonomy of Cognitive Biases

We select eleven cognitive biases for evaluation, organized into three functional categories based on the cognitive mechanism involved: judgment under uncertainty (biases affecting estimation and probability assessment), decision-making (biases affecting choices between alternatives), and belief updating (biases affecting how new information is integrated with prior beliefs). Table 1 summarizes the taxonomy.

Table 1. Taxonomy of eleven cognitive biases evaluated in this study, organized by functional category. Each bias is tested through K semantically equivalent prompt variants with $N=100$ independent trials per variant.

Category	Bias	Operational Definition	K
Judgment under uncertainty	Anchoring	Numerical estimates shift toward an arbitrary reference value	5
	Availability heuristic	Probability estimates are influenced by the ease of recall of examples	4
	Representativeness	Probability judgments are based on similarity to stereotypes rather than base rates	4
Decision-making	Framing effect	Choices change depending on whether outcomes are framed as gains or losses	5
	Sunk cost fallacy	Continued investment is driven by irrecoverable past costs	4
	Status quo bias	Preference for the current state of affairs over alternatives	3
	Decoy effect	Preference between two options changes when a dominated third option is introduced	4
Belief updating	Confirmation bias	Selective seeking or endorsement of information consistent with a prior belief	5
	Bandwagon effect	Opinions shift toward the perceived majority position	4
	Authority bias	Judgments shift toward positions attributed to authority figures	4
	Primacy/recency	Evaluation is disproportionately influenced by items presented first or last	3

For each bias, we design a set of K prompt variants (see Table 1), where each variant is semantically equivalent but differs in surface form: lexical choice, sentence structure, domain context, and ordering of presented information. Crucially, the prompt variants span multiple application domains (e.g., medical, financial, social, and technical scenarios) to test whether bias effects generalize across contexts rather than being artifacts of a single domain. While this multi-domain design improves ecological validity, it does not allow us to disentangle domain-specific bias magnitudes; a systematic investigation of domain effects is left for future work. This design serves a dual purpose: it tests the robustness of bias measurements across formulations (analogous to using multiple independent measurement techniques in physics) and provides the basis for estimating systematic uncertainty (Section 3.3).

3.2. Experimental Design

For each of the eleven biases, we adopt a controlled experimental design consisting of a control condition and one or more treatment conditions. The control condition presents the decision or estimation problem without any bias-inducing element. The treatment condition(s) introduce the bias-inducing manipulation while keeping all other aspects of the prompt identical. This paired design allows us to isolate the effect of each bias from baseline model behavior.

3.2.1. Prompt Structure

Each prompt follows a standardized three-part structure:

• Context: A brief scenario description providing the necessary background for the task.
• Manipulation (treatment only): The bias-inducing element, e.g., an irrelevant anchor value, a gain/loss framing, or a majority opinion statement.
• Query: A question requiring either a numerical estimate (for judgment biases) or a choice among alternatives (for decision biases), followed by an instruction to provide a brief justification.

3.2.2. Sampling Protocol

For each prompt variant under each condition, we query the model $N=100$ times with temperature $T=0.7$ (balancing response diversity with coherence). We record both the model’s answer and its justification. For models that support it, we also record token-level log-probabilities. The total number of API calls per model is:

$$N_{\mathrm{total}}={\displaystyle \sum _{b=1}^{11}}{K}_b\times C_b\times N={\displaystyle \sum _{b=1}^{11}}{K}_b\times C_b\times 100,$$

(1)

where $K_b$ is the number of prompt variants for bias b and $C_b$ is the number of conditions (typically 2 or 3). Across all biases, this yields approximately 9000 calls per model and over 70,000 calls in total across eight models.

3.2.3. Response Parsing

Model responses are parsed using a combination of regular expressions and a secondary LLM call (using a smaller model) to extract structured answers from free-text responses. Responses that cannot be reliably parsed are flagged and excluded from quantitative analysis; we report the exclusion rate for each model.

3.2.4. Implementation

The experimental pipeline is implemented in Python 3.11 (Python Software Foundation, Wilmington, DE, USA; https://www.python.org, accessed on 27 May 2026) and orchestrates all API calls, response collection, parsing, and statistical analysis. Each model is accessed through its provider’s official Python SDK or REST API: OpenAI’s openai library for GPT-4.1 Mini, Anthropic’s anthropic SDK for Claude 3.5 Sonnet, Google’s google-generativeai SDK for Gemini 2.5 Flash, Groq’s API for both Llama models, Mistral’s API for Mistral Large (mistral-large-2411), and Together AI’s API for DeepSeek V3 and MiniMax M2.5. The pipeline runs fully automated: for each bias–model–strategy combination, it sequentially dispatches $N=100$ API calls per prompt variant, respects per-provider rate limits, and stores raw JSON responses for reproducibility. All experiments were conducted over a two-week period in April–May 2026.

3.3. Bias Strength Index

A central contribution of this work is the Bias Strength Index (BSI), a normalized metric that quantifies the magnitude of a cognitive bias on a continuous scale in the interval $[0,1]$. The BSI is defined differently for numerical estimation biases and categorical choice biases, but in both cases it captures the degree to which the model’s behavior shifts from the control condition in the direction predicted by the bias.

3.3.1. BSI for Numerical Estimation Biases

For biases involving numerical estimates (e.g., anchoring), the BSI is defined as:

$${\mathrm{BSI}}_{\mathrm{num}}=\left|{\displaystyle \frac{{\overline{x}}_{\mathrm{treatment}}-{\overline{x}}_{\mathrm{control}}}{x_{\mathrm{anchor}}-{\overline{x}}_{\mathrm{control}}}}\right|,$$

(2)

where ${\overline{x}}_{\mathrm{treatment}}$ is the mean response under the treatment condition, ${\overline{x}}_{\mathrm{control}}$ is the mean response under the control condition, and $x_{\mathrm{anchor}}$ is the bias-inducing reference value. A BSI of 0 indicates no bias effect (the model ignores the anchor), while a BSI of 1 indicates complete anchoring (the model’s response equals the anchor). Values above 1 indicate over-adjustment beyond the anchor.

3.3.2. BSI for Categorical Choice Biases

For biases involving choices among alternatives (e.g., framing effect), the BSI is defined as:

$${\mathrm{BSI}}_{\mathrm{cat}}=\left|p_{\mathrm{treatment}}-p_{\mathrm{control}}\right|,$$

(3)

where $p_{\mathrm{treatment}}$ and $p_{\mathrm{control}}$ are the proportions of responses selecting the bias-predicted option under the treatment and control conditions, respectively. A BSI of 0 indicates identical choice distributions across conditions (no bias), while higher values indicate stronger bias influence.

3.3.3. Uncertainty Decomposition

We decompose the total uncertainty on the BSI into two components, following the standard practice in experimental physics.

Statistical Uncertainty

For each prompt variant k, the statistical uncertainty on the BSI arises from the finite sample size N and is estimated via bootstrapping (10,000 resamples):

$${\sigma }_{\mathrm{stat}}^{\left(k\right)}=\mathrm{std}\left[{\mathrm{BSI}}^{*\left(1\right)},{\mathrm{BSI}}^{*\left(2\right)},\dots ,{\mathrm{BSI}}^{*\left(\mathrm{10,000}\right)}\right],$$

(4)

where ${\mathrm{BSI}}^{*\left(j\right)}$ is the BSI computed on the j-th bootstrap resample.

Systematic Uncertainty

The systematic uncertainty captures the sensitivity of the BSI to the choice of prompt formulation. It is estimated as the standard deviation of the BSI values obtained across the K prompt variants:

$${\sigma }_{\mathrm{sys}}=\sqrt{{\displaystyle \frac{1}{K-1}}\sum _{k=1}^K{\left({\mathrm{BSI}}^{\left(k\right)}-\overline{\mathrm{BSI}}\right)}^2},$$

(5)

where $\overline{\mathrm{BSI}}={\displaystyle \frac{1}{K}}{\sum }_{k=1}^K{\mathrm{BSI}}^{\left(k\right)}$ is the mean BSI across prompt variants.

Total Uncertainty

The total uncertainty is obtained by combining the two components in quadrature, under the assumption of independence:

$${\sigma }_{\mathrm{tot}}=\sqrt{{\overline{\sigma }}_{\mathrm{stat}}^2+{\sigma }_{\mathrm{sys}}^2},$$

(6)

where ${\overline{\sigma }}_{\mathrm{stat}}$ is the average statistical uncertainty across prompt variants. The final result for each bias–model combination is reported as $\overline{\mathrm{BSI}}\pm {\sigma }_{\mathrm{tot}}$.

3.3.4. Statistical Significance

We employ two complementary approaches to statistical significance testing, each addressing a different question:

Variant-Level Test (Conservative)

For each bias–model combination, we test the null hypothesis $H_0$, BSI = 0 (no bias), using a one-sample t-test across the K prompt variants. This tests whether the bias effect generalizes across prompt formulations. We apply the Benjamini–Hochberg (BH) correction for multiple comparisons across the 78 tests with sufficient data (out of 11 biases × 8 models = 88 nominal combinations; 10 combinations are excluded because of insufficient parseable responses, see Section 4.1). This test is deliberately conservative: with K as low as 3–5 (and K = 1 for primacy/recency), statistical power is limited, and a non-significant result indicates that the effect is not robustly detectable across prompts—not that the bias is absent.

Trial-Level Test (GLMM)

To assess whether bias effects are detectable at the level of individual model responses, we fit a Generalized Linear Mixed-Effects Model (GLMM) for each bias–model combination [33]. For categorical biases, we model the binary response (bias-consistent choice vs. not) as a function of condition (treatment vs. control) with a logit link and a random intercept for prompt variant. For numerical biases (anchoring), we use a linear mixed-effects model with the same structure. This approach pools information across all $N\times K$ trials while accounting for variant-level heterogeneity, providing substantially greater statistical power. We apply BH correction across all GLMM tests and report odds ratios as effect sizes.

3.4. Models Under Study

We evaluate eight LLMs spanning seven distinct model families, selected to represent the current state of the art across both commercial and open-source ecosystems. Crucially, we include two models from the same family (Meta’s Llama) at different scales (8B and 70B parameters), enabling a controlled analysis of the relationship between model size and bias susceptibility. Table 2 summarizes the models.

Table 2. Models evaluated in this study.

Model	Family	Access	Parameters	Inference
GPT-4.1 Mini	OpenAI	Commercial API	Undisclosed	OpenAI API
Claude 3.5 Sonnet	Anthropic	Commercial API	Undisclosed	Anthropic API
Gemini 2.5 Flash	Google	Commercial API	Undisclosed	Google AI Studio
Llama 3.3 70B	Meta	Open weights	70B	Groq API
Llama 3.1 8B	Meta	Open weights	8B	Groq API
DeepSeek V3	DeepSeek	Open weights	671B (MoE)	Together API
Mistral Large	Mistral AI	Commercial API	Undisclosed	Mistral API
MiniMax M2.5	MiniMax	Commercial API	Undisclosed	Together API

All models are accessed through their respective APIs with consistent hyperparameters: temperature $T=0.7$, maximum output length of 512 tokens, and—for the baseline experiments—no system prompt beyond the experimental prompt itself (the role-based debiasing strategy D3 in Section 3.5 deliberately prepends a system-level instruction as part of the intervention being tested). For Llama models, we use the Groq-hosted inference endpoint; DeepSeek V3 and MiniMax M2.5 are accessed via the Together API. MiniMax M2.5 is reported in Table 3 for completeness but is omitted from Figure 1 and from most of the cross-model analyses below because its parse-failure rate exceeds 66% for the majority of biases (see Section 4.1); only authority bias, availability heuristic, sunk cost fallacy, and confirmation bias have analyzable data for this model. All experiments were conducted over a two-week period to minimize potential confounds from model updates. The total API cost for the full experimental campaign (baseline plus debiasing, approximately 75,000 calls) was approximately USD 85, with per-model costs ranging from USD 0 (Groq and Together’s free tiers for Llama and DeepSeek) to approximately USD 25 (Anthropic for Claude 3.5 Sonnet). These modest costs reflect the use of relatively short prompts and the availability of competitive pricing across providers as of mid-2026.

Table 3. Bias Strength Index (BSI) with total uncertainty (Part B: open-weight models). Bold indicates significance after Benjamini–Hochberg correction ($\alpha$ = 0.05). Dashes indicate insufficient data due to high parse failure rates.

Bias	DeepSeek	Llama 8B	Llama 70B	MiniMax
Anchoring	0.044 ± 0.069	0.241 ± 0.450	0.006 ± 0.013	–
Availability	0.019 ± 0.031	0.200 ± 0.297	0.000 ± 0.000	0.000 ± 0.000
Representat.	0.113 ± 0.194	0.145 ± 0.133	0.208 ± 0.417	–
Framing	0.637 ± 0.154	0.700 ± 0.247	0.667 ± 0.577	–
Sunk cost	0.076 ± 0.096	0.025 ± 0.037	0.000 ± 0.000	0.000 ± 0.000
Status quo	0.293 ± 0.273	0.322 ± 0.191	0.000 ± 0.000	–
Decoy	0.367 ± 0.301	0.892 ± 0.118	0.500 ± 0.577	–
Confirmation	0.034 ± 0.066	0.113 ± 0.128	0.000 ± 0.000	0.000 ± 0.000
Bandwagon	0.128 ± 0.105	0.111 ± 0.146	0.000 ± 0.000	–
Authority	0.013 ± 0.021	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000
Primacy/rec.	0.590 ± 0.050	0.966 ± 0.034	1.000 ± 0.000	–

3.5. Debiasing Strategies

We evaluate three debiasing strategies, applied at inference time without modifying model weights:

3.5.1. (D1) Zero-Shot Chain-of-Thought (CoT)

We append “Let’s think step by step.” to the query, following the standard zero-shot CoT protocol [34]. This strategy aims to engage deliberate (System 2) reasoning.

3.5.2. (D2) Adversarial Counter-Prompting

We explicitly warn the model about the specific bias being tested: “Be aware that this question may contain elements designed to trigger [bias name]. Ensure your answer is based solely on the relevant information.” This tests whether explicit awareness mitigates bias.

3.5.3. (D3) Role-Based Prompting

We prepend a system-level instruction: “You are a rational decision analyst trained to identify and avoid cognitive biases. Provide your most objective assessment.” This tests whether identity framing improves reasoning quality.

We evaluate all three debiasing strategies on Llama 3.1 8B—the model exhibiting the strongest baseline bias effects—for the three biases with the highest baseline BSI (decoy, framing, and primacy/recency), repeating the full experimental protocol ($N=100$ trials per prompt variant) and computing the debiased BSI.

4. Results

We present the results from a total of over 70,000 API calls across eight models and eleven cognitive biases. All unparseable responses are excluded from quantitative analysis; the full per-model and per-bias parse failure rates are released as part of the open-source archive cited in the Data Availability Statement. The full raw responses are publicly available.

4.1. Baseline Bias Measurements

Table 3 and Table 4 present the BSI with total uncertainty for all 88 bias–model combinations. Figure 1 provides a visual overview.

Table 4. Bias Strength Index (BSI) with total uncertainty for all bias–model combinations (Part A: commercial models). Dashes indicate insufficient data due to high parse failure rates.

Bias	Claude 3.5	Gemini 2.5	GPT-4.1m	Mistral
Anchoring	0.018 ± 0.031	–	0.002 ± 0.004	0.007 ± 0.016
Availability	0.000 ± 0.000	–	0.115 ± 0.230	0.000 ± 0.000
Representat.	0.465 ± 0.476	–	0.000 ± 0.000	0.080 ± 0.136
Framing	0.393 ± 0.511	0.478 ± 0.427	0.703 ± 0.514	0.667 ± 0.577
Sunk cost	0.242 ± 0.485	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000
Status quo	0.021 ± 0.037	0.143 ± 0.145	0.000 ± 0.000	0.100 ± 0.174
Decoy	0.275 ± 0.478	0.370 ± 0.398	0.253 ± 0.498	0.670 ± 0.375
Confirmation	0.000 ± 0.000	0.076 ± 0.170	0.000 ± 0.000	0.000 ± 0.000
Bandwagon	0.472 ± 0.537	0.420 ± 0.447	0.005 ± 0.011	0.023 ± 0.040
Authority	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000
Primacy/rec.	1.000 ± 0.000	1.000 ± 0.000	0.950 ± 0.022	0.000 ± 0.000

Figure 1. Heatmap of Bias Strength Index (BSI) across eleven cognitive biases and the seven models with sufficient parseable data (MiniMax M2.5 omitted owing to very high parse-failure rates across most biases; see Table 3). Cell colour encodes BSI magnitude (darker = stronger bias); white cells indicate bias–model combinations with insufficient parseable responses for analysis. BSI magnitude alone does not imply statistical significance: in the conservative variant-level test (Section 4.1) only the decoy effect for Llama 3.1 8B survives Benjamini–Hochberg correction at $\alpha$ = 0.05, whereas the trial-level GLMM analysis (Section 4.4 and Table 5) identifies 27 of 43 testable combinations as significant.

Figure 1. Heatmap of Bias Strength Index (BSI) across eleven cognitive biases and the seven models with sufficient parseable data (MiniMax M2.5 omitted owing to very high parse-failure rates across most biases; see Table 3). Cell colour encodes BSI magnitude (darker = stronger bias); white cells indicate bias–model combinations with insufficient parseable responses for analysis. BSI magnitude alone does not imply statistical significance: in the conservative variant-level test (Section 4.1) only the decoy effect for Llama 3.1 8B survives Benjamini–Hochberg correction at $\alpha$ = 0.05, whereas the trial-level GLMM analysis (Section 4.4 and Table 5) identifies 27 of 43 testable combinations as significant.

Of the 78 bias–model combinations with sufficient parse rates for analysis, nine (11.5%) showed statistically significant bias before multiple-comparison correction, but only one—the decoy effect in Llama 3.1 8B (BSI = 0.892 ± 0.118, p < 0.001)—survived Benjamini–Hochberg correction at $\alpha$ = 0.05. The most pervasive biases in terms of raw BSI magnitude were primacy/recency (mean BSI = 0.787 across models), the framing effect (mean BSI = 0.606), and the decoy effect (mean BSI = 0.475). Conversely, authority bias (mean BSI = 0.002), confirmation bias (mean BSI = 0.028), and the sunk cost fallacy (mean BSI = 0.043) showed weak or absent effects. However, for many bias–model combinations, the large systematic uncertainty (from prompt sensitivity) prevents the effects from reaching statistical significance despite non-trivial BSI magnitudes.

4.1.1. Judgment Under Uncertainty

Anchoring effects were weak across models, with BSI values ranging from $0.002\pm 0.004$ (GPT-4.1 Mini) to $0.241\pm 0.450$ (Llama 3.1 8B). No anchoring result reached statistical significance, suggesting that current LLMs are relatively resistant to arbitrary numerical anchors, particularly for factual questions where models have strong parametric knowledge. The availability heuristic showed a similar pattern of weak, non-significant effects (mean BSI = 0.048), with only GPT-4.1 Mini (BSI = 0.115) and Llama 3.1 8B (BSI = 0.200) showing non-negligible effects. Representativeness bias was moderate but variable: Claude 3.5 Sonnet showed the highest BSI ($0.465\pm 0.476$), though the large systematic uncertainty prevented significance.

4.1.2. Decision-Making

The framing effect was one of the strongest biases observed, with BSI values exceeding 0.60 for five of seven models tested. GPT-4.1 Mini showed the highest raw BSI (0.703 ± 0.514), followed closely by Llama 3.1 8B (0.700 ± 0.247, p = 0.018) and Llama 3.3 70B (0.667 ± 0.577). The large systematic uncertainties for most models reflect high sensitivity to prompt formulation for this bias. The sunk cost fallacy was largely absent across models (mean BSI $=0.043$), with only Claude 3.5 Sonnet showing a non-trivial effect (BSI = 0.243 ± 0.485). The decoy effect was the strongest decision-making bias: Llama 3.1 8B exhibited a BSI of $0.892\pm 0.118$ (p < 0.001; significant after BH correction), the only result in the entire study to survive multiple-comparison correction. Status quo bias was moderate and model-dependent, with Llama 3.1 8B (BSI = 0.322 ± 0.191, p = 0.041) and DeepSeek V3 (BSI = 0.293 ± 0.273) showing the strongest effects.

4.1.3. Belief Updating

Primacy/recency effects were the strongest bias category overall (mean BSI = 0.787), with Claude 3.5 Sonnet, Gemini 2.5 Flash, and Llama 3.3 70B all exhibiting BSI = 1.0 (complete primacy dominance), though these results are based on a single prompt template (K = 1) and thus lack systematic uncertainty estimates. Confirmation bias was weak (mean BSI $=0.028$), with only Llama 3.1 8B showing a marginally significant effect (BSI = 0.113, p = 0.048). The bandwagon effect was highly variable across models: Claude 3.5 Sonnet (BSI = 0.472 ± 0.537) and Gemini 2.5 Flash (BSI = 0.420 ± 0.447) showed strong effects, while GPT-4.1 Mini was nearly immune (BSI = 0.005). Authority bias was essentially absent across all models (mean BSI = 0.002), indicating that current LLMs do not shift their judgments based on attributed authority.

4.2. Cross-Model Comparison

Figure 2 compares the mean BSI across all eleven biases for each model.

Figure 2. Mean BSI with total uncertainty (statistical ⊕ systematic) for GPT-4.1 Mini across all eleven biases. Error bars represent ${\sigma }_{\mathrm{tot}}$. All bars are shown in grey because no bias reaches statistical significance for this model after Benjamini–Hochberg correction at $\alpha$ = 0.05; the red colour entry in the legend, reserved for significant cases, therefore does not appear in the figure body.

Several patterns emerge from the cross-model comparison:

• No model is bias-free. All models with sufficient parse rates exhibit non-zero BSI values for multiple bias categories. Llama 3.1 8B shows the highest mean BSI across biases (0.338), followed by Gemini 2.5 Flash (0.311) and Claude 3.5 Sonnet (0.262). Mistral Large shows the lowest mean BSI (0.141).
• Bias profiles are model-specific. While the framing effect and primacy/recency bias are near-universal, the overall bias profile differs substantially across model families. For instance, GPT-4.1 Mini shows the strongest framing effect (BSI = 0.703) but is nearly immune to the bandwagon effect (BSI = 0.005), while Claude 3.5 Sonnet shows a strong bandwagon effect (BSI = 0.472) but relatively weak framing (BSI = 0.393).
• Model size matters, but not uniformly. Comparing Llama 3.1 8B with Llama 3.3 70B—which share a similar architecture and training methodology but differ by an order of magnitude in parameter count—reveals that the smaller model is substantially more biased overall (mean BSI of 0.338 vs. 0.216). However, this comparison is confounded by the much higher parse failure rate of Llama 3.3 70B (66.7% for several biases), which reduces the number of analyzable bias–model combinations. Where both models have usable data, the larger model shows lower BSI for anchoring, framing, and status quo bias, but similar levels for representativeness and availability.
• Open-source vs. commercial models. No systematic difference emerges between commercial (GPT-4.1 Mini, Claude, Gemini) and open-weight (Llama, DeepSeek, Mistral) models. Both groups contain models with high and low overall bias susceptibility. The most biased model (Llama 3.1 8B, open weights) and the least biased model (Mistral Large, commercial) span the full range, suggesting that bias profiles depend more on specific training procedures than on the open/closed distinction.

4.3. Prompt Sensitivity Analysis

A key contribution of this work is the decomposition of measurement uncertainty into statistical and systematic components. Figure 3 shows the relative contribution of each component for a representative model.

Figure 3. Decomposition of total BSI uncertainty into statistical (blue) and systematic (orange) components for GPT-4.1 Mini. Statistical uncertainty arises from finite sampling (N = 100 trials); systematic uncertainty arises from variation across semantically equivalent prompt formulations.

For most biases, the systematic uncertainty (prompt sensitivity) substantially exceeds the statistical uncertainty, indicating that how a bias is probed matters more than the stochastic variability in model responses. This is a central finding of this work: many bias–model combinations show non-trivial BSI values that fail to reach statistical significance precisely because the systematic uncertainty is large. For example, the framing effect in GPT-4.1 Mini has BSI = 0.703 but ${\sigma }_{\mathrm{sys}}=0.514$, yielding a non-significant result despite the large effect size. This finding has important implications: studies that test a single prompt formulation per bias may report results that are strongly dependent on arbitrary wording choices. The biases most sensitive to prompt formulation were the framing effect, sunk cost fallacy, and decoy effect, where ${\sigma }_{\mathrm{sys}}/{\sigma }_{\mathrm{stat}}$ ratios commonly exceeded 10. The most robust measurements (lowest prompt sensitivity) were observed for primacy/recency, though this is partly due to the use of a single template (K = 1) for this bias in most models.

4.4. Trial-Level Statistical Analysis (GLMM)

The variant-level t-test (Section 4.1) is conservative by design, testing whether bias effects generalize across prompt formulations with only K = 3–5 observations per test. To complement this analysis, we fit GLMMs at the individual-trial level (Section 3.3), pooling all $N\times K$ responses per bias–model combination while accounting for prompt-variant heterogeneity via random intercepts.

The GLMM analysis covers nine of the eleven biases in our taxonomy: representativeness and the availability heuristic admit neither a binary bias-consistent outcome (categorical GLMM, Equation (3)) nor a canonical anchor value (numerical GLMM, Equation (2)), so they are excluded from the trial-level analysis (see also Section 5.3, item “BSI Operationalization for Non-Anchoring Numerical Biases”). Within the remaining nine biases, the “Tested” column of Table 5 counts the bias–model combinations for which the GLMM converged: combinations with zero outcome variance or complete separation between the control and treatment conditions are listed as analyzable for BSI computation in Table 3 and Table 4 but cannot support a non-degenerate mixed-effects fit and are therefore excluded from the GLMM denominator. The 43 testable combinations reported below decompose as 36 categorical plus 7 numerical (anchoring).

Table 5 summarizes the results. Of 36 categorical bias–model combinations with sufficient data, 23 (63.9%) show statistically significant treatment effects after BH correction at $\alpha$ = 0.05. Of seven numerical (anchoring) tests, four (57.1%) are significant. In total, 27 of 43 testable bias–model combinations (62.8%) show significant bias effects at the trial level—a stark contrast with the single significant result from the variant-level t-test.

Table 5. Summary of GLMM trial-level significance results by bias. For each bias, we report the number of models showing a significant effect (after BH correction) out of the total number of models with sufficient data.

Bias	Sig./Tested	Representative OR or d
Anchoring (numerical)	4/7	d = 0.001 (Mistral)
Framing effect	6/7	OR = 1.35 (GPT-4.1m)
Decoy effect	6/7	OR = 2.43 (Llama 8B)
Bandwagon effect	5/7	OR = 1.56 (Gemini)
Status quo bias	3/5	OR = 1.24 (Llama 8B)
Confirmation bias	2/4	OR = 1.08 (Gemini)
Primacy/recency	1/3	OR = 72.0 (DeepSeek)
Sunk cost fallacy	0/2	—
Authority bias	0/1	—

Table 5. Summary of GLMM trial-level significance results by bias. For each bias, we report the number of models showing a significant effect (after BH correction) out of the total number of models with sufficient data.

Bias	Sig./Tested	Representative OR or d
Anchoring (numerical)	4/7	d = 0.001 (Mistral)
Framing effect	6/7	OR = 1.35 (GPT-4.1m)
Decoy effect	6/7	OR = 2.43 (Llama 8B)
Bandwagon effect	5/7	OR = 1.56 (Gemini)
Status quo bias	3/5	OR = 1.24 (Llama 8B)
Confirmation bias	2/4	OR = 1.08 (Gemini)
Primacy/recency	1/3	OR = 72.0 (DeepSeek)
Sunk cost fallacy	0/2	—
Authority bias	0/1	—

These results demonstrate that cognitive biases in LLMs are statistically detectable when analyzed with appropriate trial-level methods. The discrepancy between the variant-level t-test (1/78 significant) and the GLMM (27/43 significant) arises because the t-test conflates two distinct questions: (1) does the bias exist? and (2) is the bias stable across prompt formulations? The GLMM answers question (1) by leveraging full trial-level data with random effects absorbing prompt-variant heterogeneity, while the t-test answers the stricter question (2). Both perspectives are informative: biases that are only significant in the GLMM (e.g., framing in Claude 3.5) are real but prompt-sensitive, whereas the sole bias surviving both tests (decoy in Llama 3.1 8B) represents a robust, prompt-invariant effect. One important caveat concerns primacy/recency: because this bias is operationalized with a single prompt template (K = 1) in our benchmark, the random-intercept-on-variant component of its GLMM is degenerate, and the very large odds ratio reported for DeepSeek V3 (OR ≈ 72) reflects near-complete separation between the control and treatment conditions on that single variant rather than a robust population-level estimate; the primacy/recency GLMM rows of Table 5 should therefore be interpreted with the same caution discussed in Section 5.3 (item “Primacy/Recency with K = 1”).

4.5. Effectiveness of Debiasing Strategies

Table 6 and Figure 4 summarize the effectiveness of the three debiasing strategies.

Table 6. BSI values under baseline and three debiasing strategies for Llama 3.1 8B. Percentage reduction from baseline shown in parentheses. Best result per bias in bold.

Bias	Baseline	D1: CoT	D2: Counter	D3: Role
Decoy effect	0.892	0.195 (−78%)	0.340 (−62%)	0.311 (−65%)
Framing effect	0.700	0.467 (−33%)	0.530 (−24%)	0.360 (−49%)
Primacy/recency	0.966	0.760 (−21%)	0.520 (−46%)	0.599 (−38%)

Figure 4. Relative BSI reduction for each debiasing strategy on Llama 3.1 8B, restricted to the three biases with the strongest baseline effects (decoy, framing, primacy/recency; cf. Table 6). The ordinate extends below zero so that any strategy that worsened bias would be visible as a downward bar; the horizontal dashed line at zero marks the no-effect reference (positive bars indicate a reduction in BSI relative to baseline, negative bars would indicate an increase). All three strategies produced positive reductions across all three biases.

Debiasing was evaluated on Llama 3.1 8B for the three biases with the strongest baseline effects: decoy (baseline BSI = 0.892), framing (baseline BSI = 0.700), and primacy/recency (baseline BSI = 0.966). The key findings are:

• D1 (chain-of-thought) was the most effective strategy for the decoy effect, reducing BSI from 0.892 to 0.195 (a 78% reduction). For framing, it achieved a moderate reduction (BSI from 0.700 to 0.467, $-33\%$). For primacy/recency, the reduction was more modest (BSI from 0.966 to 0.760, $-21\%$). These results suggest that engaging deliberate reasoning is most effective for biases involving explicit choice manipulation.
• D2 (counter-prompting) was most effective for primacy/recency (BSI from 0.966 to 0.520, $-46\%$) but showed limited effectiveness for framing (BSI from 0.700 to 0.530, $-24\%$) and decoy (BSI from 0.892 to 0.340, $-62\%$). Explicitly warning about the bias appears most effective when the bias mechanism is transparent.
• D3 (role-based) achieved moderate reductions across all three biases: decoy (BSI from 0.892 to 0.311, $-65\%$), framing (BSI from 0.700 to 0.360, $-49\%$), and primacy/recency (BSI from 0.966 to 0.599, $-38\%$). Adopting a “rational analyst” persona provided the most consistent (though not always the strongest) debiasing effect.

Overall, all three strategies reduced bias for all three tested biases, but no single strategy dominated across the board. The effectiveness varied substantially by bias type, confirming that debiasing is inherently bias-specific.

5. Discussion

5.1. Implications for AI Safety and Alignment

Our findings have several implications for the safe deployment of LLMs in decision-support systems.

First, the universality of cognitive biases across all tested models—including state-of-the-art commercial systems—suggests that these biases are not artifacts of a specific training procedure but rather emergent properties of learning from human-generated text at scale. This has direct consequences for applications in medicine [1,22], law [2], and finance [3], where biased recommendations could lead to systematic errors affecting large populations.

Second, the finding that single-prompt debiasing strategies are partially effective but highly bias-dependent echoes recent concerns raised by Lou and Sun [11] and Echterhoff et al. [12], and suggests that robust debiasing may require either bias-specific prompt engineering or architectural/training-time interventions rather than a one-size-fits-all approach.

Third, the large systematic uncertainties observed for several biases (Section 4.3) highlight a fundamental challenge for LLM evaluation: bias assessments based on a single prompt formulation may not generalize. This prompt sensitivity is not unique to BSI; it reflects a broader instability in LLM evaluation documented across diverse tasks [35,36]. The BSI inherits this instability because it is defined over model outputs that are themselves prompt-sensitive. Our explicit decomposition of uncertainty into statistical and systematic components (and the complementary GLMM analysis in Section 4.4) is designed precisely to quantify—rather than hide—this fundamental challenge. We recommend that future studies adopt multi-prompt evaluation protocols and report systematic uncertainties alongside statistical ones.

5.2. Comparison with Human Cognitive Biases

A natural question is whether LLMs are more or less biased than humans. Direct comparison is complicated by differences in experimental conditions, but we can make several qualitative observations.

For anchoring, Lou and Sun [11] reported a human anchoring index of approximately 0.61, compared to 0.45 for GPT-4 on the same task. Our results for GPT-4.1 Mini show BSI = 0.002 ± 0.004, suggesting substantially weaker anchoring compared to both earlier GPT-4 results and human baselines. This may reflect improvements in model training or differences in our experimental protocol.

For the framing effect, Tversky and Kahneman [7] found that approximately 72% of human subjects chose the risk-averse option in the gain frame versus 22% in the loss frame (a shift of 50 percentage points). Our BSI for the framing effect in GPT-4.1 Mini was $0.703\pm 0.514$, indicating a comparable or even stronger framing effect, though the large systematic uncertainty reflects high sensitivity to prompt formulation.

Cheung et al. [13] found that LLMs show amplified cognitive biases in moral decision-making compared to humans. Our results partially support this finding: the framing effect and primacy/recency bias show BSI values comparable to or exceeding human baselines, while other biases (anchoring, authority, confirmation) are substantially weaker in LLMs.

An important qualitative difference is that LLM biases show low statistical uncertainty but high systematic uncertainty: while individual model responses cluster tightly (low ${\sigma }_{\mathrm{stat}}$), the measured bias strength varies dramatically across prompt formulations (high ${\sigma }_{\mathrm{sys}}$). This suggests that LLM biases, unlike human biases, are more of a property of the specific prompt–model interaction than a stable trait, with implications for how bias audits should be designed.

5.3. Limitations

Several limitations should be noted.

5.3.1. Single-Turn Protocol

Our prompt-based evaluation framework necessarily tests biases through text-based scenarios. Biases that manifest primarily through interactive, multi-turn dialogs (e.g., anchoring that builds over a conversation) may not be fully captured by our single-turn protocol.

5.3.2. Snapshot of a Moving Target

While we test eight models from seven families, the LLM landscape evolves rapidly, and our results represent a snapshot of model behavior at a specific point in time; future model versions may exhibit different bias profiles.

5.3.3. Bias Coverage

Our benchmark covers eleven of the over 180 documented cognitive biases [14]; many biases remain untested, including overconfidence, which requires a different experimental setup (confidence intervals rather than point estimates).

5.3.4. Statistical Inference over Prompt Variants

Our primary variant-level significance test uses a one-sample t-test over K prompt variants, which has inherently low statistical power when K is small (3–5, or K = 1 for primacy/recency). This is a deliberate design choice that tests a strict question—whether the effect generalizes across prompt formulations—but it means that many real effects (confirmed by the trial-level GLMM; see Section 4.4) appear non-significant at this level. We acknowledge that the variant-level analysis alone would understate the prevalence of cognitive biases; the GLMM provides the complementary trial-level perspective. The variants are researcher-crafted rather than randomly sampled from a population of prompts. A hierarchical mixed-effects model with crossed random effects for variant, item, and model—akin to the Total Evaluation Error (TEE) framework [36]—would provide more appropriate variance estimates and hypothesis tests. Our GLMM analysis (Section 4.4) is a step in this direction, though it does not include crossed model effects.

5.3.5. Primacy/Recency with K = 1

The very high BSI values observed for primacy/recency (≥0.95 for several models) are based on a single prompt template, which precludes systematic uncertainty estimation. These results should be interpreted with caution; adding structurally diverse templates (e.g., varying item order, content domain, and response format) is needed to verify robustness.

5.3.6. BSI Operationalization for Non-Anchoring Numerical Biases

The numerical BSI definition (Equation (2)) is natural for anchoring but less well-specified for other numerical biases (e.g., availability heuristic) where a canonical anchor value is absent. A per-bias mapping from operationalization to BSI components would improve transparency and is provided only implicitly through our code release.

5.3.7. Parse Failures and Selection Bias

Model responses that could not be reliably parsed were excluded from quantitative analysis. Parse failure rates vary substantially across models (from <1% for GPT-4.1 Mini to >66% for several biases in Llama 3.3 70B and MiniMax M2.5), potentially introducing selection bias if failures correlate with specific conditions or treatments. A formal missingness analysis and sensitivity checks are warranted.

5.3.8. Option Ordering and Position Confounds

For categorical biases (framing, decoy, status quo, bandwagon), option labels and positions were not counterbalanced across trials. This means that observed BSI values may partly reflect position or label preferences rather than the target cognitive bias, a concern supported by recent evidence that LLMs exhibit significant position bias in multiple-choice settings [37]. Future work should adopt Latin-square or randomized option-order designs to disentangle these effects.

5.3.9. Debiasing Scope

The debiasing evaluation was conducted on a single model (Llama 3.1 8B) for three biases with the strongest baseline effects. While this provides proof-of-concept evidence, the generalizability of debiasing effectiveness across models and biases remains uncharacterized. Furthermore, we did not systematically measure potential side effects of debiasing interventions. Debiasing prompts (particularly CoT and role-based strategies) increase response length and may alter response quality, factual accuracy, or verbosity in ways that are not captured by BSI reduction alone. In safety-critical applications, verifying that debiasing does not degrade task performance is essential before deployment. A comprehensive side-effect analysis—measuring accuracy on held-out tasks, response calibration, and output verbosity under each debiasing strategy—is needed and is left for future work.

5.3.10. The Prompt-Sensitivity Literature

Recent work has formalized prompt sensitivity in LLM evaluation [35,36] through metrics such as PromptSE (prompt stability for coding tasks), JudgeSense (paraphrase stability with JSS), and TEE (mixed-effects variance budgeting). Our systematic uncertainty decomposition resonates with these frameworks but was developed independently; a formal comparison and integration with TEE-style variance budgeting would strengthen the conclusions about optimal evaluation-budget allocation (more items vs. more variants vs. more trials).

5.3.11. Training Data Contamination

We cannot rule out that models may have encountered our specific test scenarios (or similar ones) during training, which could confound our measurements by eliciting trained responses rather than genuine reasoning.

5.3.12. Temperature and Decoding Sensitivity

All experiments used a single temperature (T = 0.7) and default decoding. Sensitivity analyses over temperatures and decoding schemes (e.g., top-p variations) would help assess the robustness of BSI and ${\sigma }_{\mathrm{sys}}$ estimates.

6. Conclusions

We have presented a systematic, quantitative framework for measuring cognitive biases in Large Language Models, introducing the Bias Strength Index (BSI) as a normalized metric with formal uncertainty decomposition into statistical and systematic components. Our evaluation of eleven cognitive biases across eight state-of-the-art LLMs from seven model families yields several key findings:

• Cognitive biases are pervasive in current LLMs. All tested models—commercial and open-source alike—exhibit non-zero BSI values for multiple bias categories. Trial-level GLMM analysis confirms statistically significant bias effects in 27 of 43 testable combinations (62.8%), while the more conservative variant-level t-test yields only one significant result (decoy effect in Llama 3.1 8B), reflecting the dominant role of prompt-induced systematic uncertainty.
• Bias profiles are heterogeneous. Framing and primacy/recency effects are near-universal (mean BSI > 0.6), while anchoring, authority, and confirmation biases are weak or absent across models. The relationship between model size and bias susceptibility is nuanced, as demonstrated by the Llama 8B vs. 70B comparison, where the smaller model shows higher overall bias but the comparison is confounded by differential parse failure rates.
• Prompt sensitivity is a dominant uncertainty. For most biases, the systematic uncertainty from prompt reformulation exceeds the statistical uncertainty from finite sampling by an order of magnitude, underscoring the need for multi-prompt evaluation protocols in future work. Many substantial BSI values fail to reach significance due to this large systematic component.
• Prompt-based debiasing has bias-dependent effectiveness. The three inference-time debiasing strategies tested (CoT, counter-prompting, role-based) all reduce bias, but their effectiveness varies substantially by bias type (reductions ranging from 21% to 78%), confirming that no single debiasing approach is universally effective.

These findings provide a quantitative foundation for auditing cognitive biases in LLMs before deployment in high-stakes applications. We release our complete benchmark—including all prompts, raw responses, analysis code, and the BSI computation framework—as an open-source resource to facilitate reproducibility, standardized evaluation, and future research.

Future directions include extending the benchmark to additional biases and languages, investigating training-time debiasing approaches, developing multi-turn bias evaluation protocols, and establishing a public leaderboard for cognitive bias assessment in LLMs.