Large Language Model Benchmarks: A Taxonomy of Capabilities, Scientific Quality Assessment, and Saturation Analysis

Abstract

The rapid evolution of Large Language Models (LLMs) has exposed limitations of static, accuracy-oriented benchmarks and increased the need for evaluation frameworks that distinguish among capabilities and benchmark quality. This survey analyzes 63 LLM benchmarks spanning 2012–2026 and organizes them into a taxonomy of six capability dimensions and 20 operational subcategories. We also propose the Benchmark Quality Assurance Index (BQAI), an AHP-weighted composite framework for assessing the scientific quality of benchmarks across seven dimensions related to annotation, clarity, standardization, reproducibility, robustness, coverage, and fairness. The BQAI is applied to 30 representative benchmarks, corresponding to 48% of the 63-benchmark corpus, with three-evaluator blinded scoring, formal inter-rater reliability validation $\left(\mathrm{ICC}(2,k)\right.$ and quadratic-weighted Cohen’s $\left.\kappa \right)$, and Monte Carlo sensitivity analysis $\left(n=1000\mathrm{trials},\pm 10\%\mathrm{to}\pm 50\%\mathrm{weight}\mathrm{perturbation}\right)$. In addition, we synthesize public performance results for 16 models across 10 benchmarks to examine saturation trends and reporting gaps. The analysis indicates that benchmark usefulness varies substantially across evaluation settings, that several established benchmarks are becoming less discriminative for frontier models, and that important gaps remain in safety, agentic, and cross-cultural assessment. Together, the taxonomy, BQAI, and saturation analysis provide a structured perspective on the current LLM benchmark landscape and on priorities for more rigorous evaluation.

1. Introduction

Measuring machine intelligence has long been central to artificial intelligence research. Turing’s “Imitation Game,” later known as the Turing Test, provided an early conceptual lens for assessing machine behavior [1]. However, modern scientific evaluation requires standardized protocols, quantifiable metrics, and reproducible experimental settings. In this context, contemporary views increasingly emphasize learning efficiency, generalization, and trustworthiness as operational desiderata for measuring intelligence [2,3].

The emergence of statistical natural language processing (NLP) enabled the use of quantitative metrics, including accuracy, BLEU, and F1-score, to systematically compare systems [4]. With deep neural architectures, evaluation shifted toward shared benchmark suites and leaderboards, accelerating convergence in language understanding. For example, sentiment analysis datasets such as the Stanford Sentiment Treebank [5] and reading comprehension benchmarks such as SQuAD [6] have established widely adopted reproducible testbeds for empirical progress. In parallel, early taxonomies of autonomous behavior helped formalize what it means for a system to act as an agent [7], a notion that has become increasingly relevant as models interact with tools and environments.

A major paradigm shift arrived with Large Language Models (LLMs), enabled by scalable Transformer architectures and massive pre-training corpora [8,9,10]. LLMs quickly expanded from classification to instruction following, multi-turn dialog, and code generation [11,12]. As performance improved, many established benchmarks became less discriminatory, motivating increasingly challenging evaluations such as SuperGLUE [13], WinoGrande [14], and MATH [15]. This iterative cycle of benchmark saturation and renewal remains a defining dynamic of AI evaluation [16].

Despite rapid progress, the evaluation ecosystem for LLMs remains fragmented and, in some cases, difficult to interpret. Key challenges include inconsistent reporting practices, limited reproducibility, and vulnerability to data contamination or benchmark-specific artifacts, which can undermine the validity of reported gains [17]. Moreover, as LLMs move toward multimodal and interactive settings, evaluation must extend beyond static linguistic accuracy to probe reasoning, factual grounding, reliability, and performance in tool-augmented environments. Broad multi-task suites such as GLUE [18] and MMLU [19] provide broad coverage of language understanding and knowledge-intensive reasoning, whereas contamination-aware datasets such as GPQA [20] are designed to reduce shortcut solutions and encourage genuine reasoning. Recent interactive and multimodal frameworks, including MMKE-Bench [21] and GUI-World [22], further reflect the growing ambition to evaluate intelligence-in-context, that is, how models behave under dynamic inputs, perceptual signals, and tool access.

At the same time, evidence suggests that scaling and increased instructability do not necessarily translate into improved reliability in all settings [23]. As LLMs increasingly mediate real-world decisions, there is a pressing need for standardized, interpretable, and robust evaluation practices, including domain-relevant assessments for enterprise deployment [24]. Consequently, the field is shifting from purely performance-centric benchmarking to capability-centric evaluation that emphasizes robustness, transparency, and reliability.

Rather than attempting to resolve the broader theoretical question of how to define or measure intelligence in artificial systems, this review focuses on the benchmark-based operationalization of LLM evaluation. Specifically, we examine how current benchmarks represent different capability dimensions, how their scientific quality can be assessed, and how their discriminative value changes as frontier model performance advances.

Our main contributions are threefold:

• We develop a taxonomy of LLM benchmark capabilities comprising six dimensions and 20 operational subcategories.
• We propose the Benchmark Quality Assurance Index (BQAI), an AHP-weighted framework for assessing benchmark scientific quality, and apply it to 30 representative benchmarks (48% of the corpus) using three-evaluator blinded scoring with formal inter-rater reliability (ICC, weighted Cohen’s $\kappa$) and multi-regime sensitivity analysis.
• We synthesize public performance evidence across 16 models and 10 benchmarks to characterize saturation patterns, reporting gaps, and differences across benchmark families.

The remainder of this paper is organized as follows: Section 2 reviews related surveys and positions our contributions; Section 3 presents the proposed taxonomy of cognitive dimensions and their operational subcategories; Section 4 defines the benchmark inclusion criteria and corpus composition; Section 5 describes the reference models and their performance matrix; Section 6 introduces the Benchmark Quality Assurance Index (BQAI); Section 7 applies and validates the BQAI across 30 representative benchmarks (48% of the 63-benchmark corpus, with ≥4 benchmarks per taxonomy dimension) using a three-evaluator blinded protocol with formal inter-rater reliability; Section 8 identifies open challenges and research gaps; and Section 9 concludes with limitations and future directions.

2. Related Work

The rapid proliferation of Large Language Models (LLMs) has motivated several survey efforts aimed at systematizing the evaluation landscape. In this section, we position our work relative to six major surveys, summarized comparatively in

Table 1. Comparison with prior surveys on LLM evaluation. ✓: covered; ✗: not covered; ∼: partially covered.

Feature	Zhao [11]	Minaee [25]	Chang [26]	Guo [27]	Laskar [28]	Ni [29]	Ours
Benchmarks cataloged	∼30	∼20	147	∼50	–	283	63
Formal taxonomy (dimensions → axes)	✗	✗	✗	∼	✗	✗	✓
Benchmark quality metric (BQAI)	✗	✗	✗	✗	✗	✗	✓
Sensitivity/robustness analysis	✗	✗	✗	✗	✗	✗	✓
Model × benchmark performance matrix	✗	✓	∼	✓	✗	✗	✓
Post-2024 benchmarks (HLE, BBEH, etc.)	✗	✗	✗	✗	✗	✓	✓
Reasoning-mode models (o1, R1)	✗	✗	✗	✗	✗	∼	✓
Contamination & reproducibility critique	∼	✗	∼	∼	✓	✓	✓

.

2.1. General LLM Surveys

Zhao et al. [11] provide a broad overview of LLM training, alignment, and use, and dedicate a section to evaluation that covers benchmarks such as MMLU, HumanEval, and BIG-bench. Similarly, Minaee et al. [25] survey three major model families (GPT, LLaMA, and PaLM) and compile a performance comparison between representative benchmarks. Although both works offer valuable panoramic views, their evaluation treatment is secondary to their focus on architecture and training methodology. Neither proposes a framework for assessing benchmark quality itself.

2.2. Evaluation-Focused Surveys

Chang et al. [26] present the most comprehensive evaluation-focused survey to date, organizing their analysis around “what, where, and how to evaluate.” They catalog 147 datasets across tasks ranging from NLP fundamentals to ethics and agent applications. Guo et al. [27] structure their review similarly around capability dimensions (knowledge, reasoning, alignment) and cover both automated and human evaluation methods. However, neither survey addresses the quality of the benchmarks themselves; their analyses implicitly treat all benchmarks as equally valid instruments.

2.3. Evaluation Methodology Critiques

Laskar et al. [28] provide a critical review focused specifically on reproducibility failures in LLM evaluation, documenting how inconsistent evaluation set ups lead to unreliable comparisons. McIntosh et al. [17] similarly argue that current benchmarks are inadequate for assessing general-purpose LLMs, highlighting issues of contamination, format sensitivity, and reporting bias. These works identify critical problems, but they do not propose a quantitative remedy.

2.4. Benchmark-Centric Surveys

The most recent benchmark-focused survey was by Ni et al. [29] (August 2025), which catalogs 283 benchmarks and identifies data contamination, cultural bias, and static evaluation as systemic problems. Their classification into general, domain-specific, and target-specific categories provides breadth, but treatment remains descriptive without a formal quality assessment framework. Complementary work has begun to formalize the quality of benchmarks and the validity of evaluations from adjacent perspectives. In educational and psychological testing, established assessment standards emphasize validity, reliability, fairness, and transparent interpretation as central properties of sound measurement instruments [30]. More recently, Salaudeen et al. proposed a validity-centered framework for AI evaluation that links benchmark design and measurement evidence to the claims that evaluation results can legitimately support [31]. Reuel et al. introduced BetterBench, a structured framework for assessing AI benchmarks through a broad set of lifecycle-oriented quality criteria and applied it quantitatively across multiple benchmark families [32]. In software engineering evaluation, Koohestani et al. reviewed benchmarking practices and proposed a unified approach to improve benchmark quality, with particular emphasis on transparency, maintainability, and methodological rigor [33]. These studies align closely with our motivation to move beyond performance-only comparisons and to examine the quality of evaluation instruments themselves. The BQAI complements this emerging literature by providing an AHP-weighted composite metric specifically tailored to the scientific-quality assessment of LLM benchmarks, operationalized through seven evidence-based dimensions and applied to a representative benchmark sample.

2.5. Positioning of the Present Work

Our survey builds on these foundations but differs in three key respects. First, we propose the Benchmark Quality Assurance Index (BQAI), an AHP-weighted composite metric to assess the scientific quality of evaluation benchmarks. Within this emerging line of work, the BQAI provides a complementary operationalization of benchmark quality through seven evidence-based dimensions, AHP-derived weights, three-evaluator blinded scoring with formal inter-rater reliability $\left(\mathrm{ICC}\right.$ and quadratic-weighted Cohen’s $\left.\kappa \right)$, and Monte Carlo sensitivity analysis. Second, we introduce a unified two-level taxonomy $\left(\mathrm{D}1–\mathrm{D}6\mathrm{cognitive}\mathrm{dimensions}\to 20\mathrm{operational}\mathrm{subcategories}\right)$ that explicitly bridges cognitive theory with measurable evaluation targets. Third, our coverage extends to post-2024 benchmarks, including HLE, FrontierMath, LiveBench, and BBEH, as well as reasoning-mode models, including DeepSeek-R1, GPT-5 with extended thinking, and Kimi K2 Thinking, which were absent from previous surveys, except for Ni et al. [29], which appeared concurrently.

3. Dimensions of Machine Intelligence and Evaluation Taxonomy

Defining “intelligence” in artificial systems remains a challenging and evolving problem. Early AI evaluation emphasized behavioral indistinguishability and task-level accuracy. However, the rapid evolution of Large Language Models (LLMs)—from pattern-matching systems to models capable of reasoning, code synthesis, tool use, long-horizon planning, and multimodal integration—has rendered single-score evaluation increasingly inadequate. A modern evaluation framework must therefore distinguish among different families of capabilities rather than collapsing them into a unified notion of performance.

To address this, we propose a structured taxonomy composed of six analytical dimensions, each operationalized through specific subcategories that define measurable evaluation targets. This distinction is important: dimensions represent broad conceptual families of machine intelligence, while subcategories constitute the granular level at which benchmarks are classified and compared.

A central design principle of our taxonomy is avoiding conflation of heterogeneous concepts. In particular, modality (e.g., vision-language integration), interaction format (e.g., dialog), and normative properties (e.g., safety) should not be treated as equivalent forms of cognition. Instead, each is positioned within a dimension reflecting its functional role in intelligent behavior. Additionally, several properties—including calibration, multilingual coverage, contamination resistance, and ecological validity—are treated as cross-cutting evaluation properties rather than standalone dimensions, yielding a taxonomy that is both conceptually coherent and aligned with the contemporary benchmark ecosystem.

3.1. Dimension 1: Reasoning and Problem-Solving

The first dimension captures the capacity for abstract logical inference, mathematical derivation, causal discovery, and scientific reasoning—capabilities central to fluid intelligence and compositional generalization. These abilities form a coherent family because they assess how effectively models process, organize, and transform symbolic information to solve problems requiring multi-step reasoning beyond surface-level pattern matching.

This dimension comprises four subcategories:

• Logical and Abstract Reasoning: Commonsense inference, coreference resolution, multi-step reasoning chains, and physical intuition (e.g., WinoGrande [14], HellaSwag [34], BBH [35]).
• Mathematical Reasoning: Arithmetic competence, symbolic manipulation, formal derivation, and competition-level problem solving spanning elementary word problems to research-frontier proofs (e.g., GSM8K [36], MATH [15], FrontierMath [37]).
• Causal Reasoning: Reasoning across Pearl’s causal hierarchy (association, intervention, counterfactuals) and statistical causal inference from observational data (e.g., CLadder [38], QRData [39]).
• Scientific Reasoning: Application of domain knowledge in physics, chemistry, and biology, ranging from grade-school concepts to graduate-level problems designed to resist search-engine solvability (e.g., AI2 ARC [40], GPQA Diamond [20]).

3.2. Dimension 2: Knowledge and Comprehension

This dimension evaluates crystallized knowledge acquisition, reading comprehension, and the ability to ground textual or visual information within domain-specific contexts. It encompasses both the breadth of factual knowledge and the depth of understanding required to extract, integrate, and apply information from diverse sources.

Four subcategories operationalize this dimension:

• General Knowledge: Factual recall and understanding across academic disciplines, standardized human exams, and expert-level questions (e.g., MMLU [19], AGIEval [41], HLE [42]).
• Reading Comprehension: Extractive and abstractive understanding, natural language inference, and discrete reasoning over textual passages (e.g., SQuAD [6], DROP [43], SuperGLUE [13]).
• Multimodal Knowledge: Integration of visual perception with textual reasoning across mathematical diagrams, educational materials, and general visual question answering (e.g., MathVista [44], MMMU [45], VQA [46]).
• Specialized Domain Knowledge: Deep expertise in specific professional or academic domains, including legal reasoning, interdisciplinary frontier science, and structured knowledge formats (e.g., GreekBarBench [47], OneEval [48]).

3.3. Dimension 3: Generation and Creativity

This dimension measures the capacity to produce syntactically correct, functionally accurate, and contextually appropriate outputs. Unlike comprehension or reasoning, generation requires models to synthesize coherent artifacts—whether code, natural language responses, or instruction-compliant outputs—under diverse constraints and evaluation criteria.

Three subcategories define this dimension:

• Code Generation: Synthesis of executable code ranging from single-function completion to repository-scale software engineering, evaluated through functional correctness and test-based verification (e.g., HumanEval [49], SWE-bench [50], DS-1000 [51]).
• Language Generation: Open-ended dialog quality, instruction-following in conversational contexts, and helpfulness as judged by human preference or LLM-as-judge protocols (e.g., Chatbot Arena [52], MT-Bench [53], AlpacaEval 2.0 [54]).
• Instruction Following: Adherence to explicit, verifiable compositional constraints, including length restrictions, keyword requirements, and format specifications (e.g., IF-Eval [55]).

3.4. Dimension 4: Interaction and Agency

This dimension evaluates capabilities essential for autonomous systems: multi-step planning, tool integration, environment navigation, and document processing in realistic interactive settings. The central assessment is whether models can coordinate actions over time in partially open environments rather than merely generating isolated outputs.

Three subcategories operationalize this dimension:

• Complex Agentic Tasks: Autonomous planning, tool selection, web interaction, and goal-directed behavior in simulated operating systems, databases, or functional websites (e.g., GAIA [56], WebArena [57], AgentBench [58]).
• Information Retrieval: Retrieval-augmented generation requiring integration of external knowledge sources, multimodal document corpora, and visual grounding (e.g., Visual-RAG [59]).
• Document Processing: Optical character recognition and structured data extraction from complex real-world documents, including invoices, forms, and degraded scans (e.g., OmniAI OCR [60]).

3.5. Dimension 5: Alignment and Safety

This dimension measures adherence to human values, resistance to harmful requests, factual accuracy, and fairness across demographic groups—critical considerations for real-world deployment. Unlike generation quality (Dimension 3), this dimension evaluates whether system behavior is trustworthy and socially robust.

Four subcategories define this dimension:

• Safety and Refusal Behavior: Recognition and refusal of harmful instructions across risk categories, including misinformation, illegal activities, hate speech, and privacy violations, including adversarial jailbreak resistance (e.g., HarmBench [61], AgentHarm [62]).
• Truthfulness: Resistance to generating plausible-sounding falsehoods and common misconceptions, prioritizing factual accuracy over imitating human text patterns (e.g., TruthfulQA [63]).
• Bias and Fairness: Evaluation of reliance on demographic stereotypes and moral reasoning in socially sensitive contexts (e.g., BBQ [64], MFTCXplain [65]).
• General Alignment: Holistic assessment of helpfulness, honesty, and harmlessness as interconnected alignment properties (e.g., HHH [66]).

3.6. Dimension 6: Holistic Evaluation

This dimension encompasses comprehensive evaluation frameworks that aggregate multiple capabilities and benchmarks designed for cross-linguistic or culturally diverse contexts. Rather than targeting a single narrow capability, these benchmarks provide integrated assessment across multiple dimensions or ensure evaluation validity beyond English-only settings.

Two subcategories define this dimension:

• Comprehensive Frameworks: Multi-task benchmark suites providing standardized evaluation across diverse scenarios with unified metrics for accuracy, robustness, fairness, and contamination resistance (e.g., HELM [67], BIG-bench [16], LiveBench [68]).
• Multilingual and Cross-Cultural: Benchmarks addressing evaluation validity across languages and cultural contexts, including Chinese academic exams, Greek legal reasoning, and multilingual moral assessment (e.g., C-Eval [69], GreekBarBench [47], MULTI [70]).

3.7. Cross-Cutting Evaluation Properties

Not all important evaluation aspects should be represented as standalone dimensions. Four properties cut across multiple dimensions and are better treated as transversal descriptors of benchmark quality:

• Calibration and Uncertainty Awareness: Whether models recognize when they are likely wrong, relevant across knowledge, reasoning, dialog, and safety benchmarks.
• Temporal Freshness and Contamination Resistance: Mechanisms to prevent test set leakage through rolling updates, unpublished problems, or expert validation (e.g., LiveBench [68], FrontierMath [37]).
• Execution-Based Verification: Functional correctness testing through unit tests, executable patches, or interventional accuracy rather than string-matching heuristics.
• Ecological Validity: Whether benchmark evaluation contexts reflect real-world deployment scenarios, including tool use, multi-step planning, and external knowledge integration.

These properties are incorporated into our benchmark selection criteria (Section 4) and quality analysis.

3.8. Taxonomy Summary

Our six-dimensional taxonomy with 20 subcategories provides a structured framework for benchmark classification:

• Reasoning and Problem-Solving (four subcategories): Logical/Abstract, Mathematical, Causal, Scientific;
• Knowledge and Comprehension (four subcategories): General Knowledge, Reading Comprehension, Multimodal Knowledge, Specialized Domains;
• Generation and Creativity (three subcategories): Code Generation, Language Generation, Instruction Following;
• Interaction and Agency (three subcategories): Agentic Tasks, Information Retrieval, Document Processing;
• Alignment and Safety (four subcategories): Safety/Refusal, Truthfulness, Bias/Fairness, General Alignment;
• Holistic Evaluation (two subcategories): Comprehensive Frameworks, Multilingual/Cross-Cultural.

This structure offers three key advantages: (i) conceptual coherence by ensuring each dimension represents a capability family rather than mixing capabilities, modalities, and normative criteria; (ii) operational practicality by allowing benchmarks to receive primary subcategory assignments while acknowledging multi-dimensional coverage where appropriate; and (iii) richer annotation supporting secondary descriptors such as multimodal, live, interactive, or multilingual.

4. Benchmark Inclusion Criteria and Corpus Composition

Having established our taxonomic framework, we now define the criteria governing benchmark selection and enumerate the resulting corpus of 63 benchmarks spanning our six-dimensional taxonomy.

4.1. Two-Track Inclusion Policy

To represent the high-stakes evaluation landscape of 2026 while accommodating rapid advances in reasoning-heavy and multimodal architectures, we adopt a two-track inclusion policy with a temporal cut-off of March 2026:

• Core Track (Gold Standards): Benchmarks satisfying all mandatory quality indicators, including resistance to saturation (state-of-the-art performance <92%) and documented contamination controls. These represent primary instruments for model rankings in frontier system reports and academic literature.
• Emerging Track (Frontier Probes): Recent or specialized benchmarks addressing emergent capabilities (research-level mathematics, causal discovery, multimodal reasoning) that may waive strict citation requirements if providing: (i) verifiable ground-truth validation protocols, (ii) expert human baselines, and (iii) mechanisms resisting training data leakage through “live” updates or unpublished test sets. Denoted with ^†.

4.2. Selection Criteria (C1–C7)

Our seven criteria prioritize cognitive depth, methodological rigor, and contamination resistance over historical popularity alone:

• Thematic Alignment (C1): Benchmarks must map coherently to our six-dimensional taxonomy, emphasizing capabilities beyond surface-level pattern matching—specifically distinguishing statistical recall from genuine reasoning, causal inference, and compositional generalization.
• Adoption and Expert Validation (C2): Core benchmarks require widespread adoption on public leaderboards (Chatbot Arena, Papers with Code) or rigorous expert validation (PhD-level verification in HLE [42], professional Bar exam questions in GreekBarBench [47]). Emerging benchmarks require open-source evaluation scripts and ≥2 independent replication studies.
• Metacognitive Transparency (C3): Priority given to benchmarks evaluating intermediate reasoning steps, source citations, or solution trajectories beyond final answer accuracy (process supervision in MATH-500 [15,71], legal citation accuracy in GreekBarBench [47]).
• Resistance to Saturation (C4): State-of-the-art performance as of March 2026 must remain <92% accuracy to prevent ceiling effects. Saturated benchmarks are retained only if providing foundational baselines or historical continuity (SQuAD [6], MMLU [19]).
• Execution-Based Metrics (C5): Functional correctness and behavioral validation prioritized over string-matching heuristics: unit-test-verified code (Pass@k), executable patches (SWE-bench [50]), interventional accuracy (CLadder [38]).
• Dynamic Openness and Anti-Contamination (C6): Benchmarks must provide either: (i) open evaluation code and data partitions, (ii) rolling test sets refreshing monthly/quarterly (LiveBench [68], LiveCodeBench [72]), or (iii) unpublished held-out problems with expert verification (FrontierMath [37], AIME 2025 [73]).
• Scaffolding Compatibility (C7): Benchmarks must permit or explicitly measure integration of external tools (Python interpreters, web search, RAG, vision encoders), distinguishing parametric knowledge from procedural intelligence in real-world deployment scenarios (GAIA [56], WebArena [57]).

4.3. Corpus Overview

Applying these criteria yields 63 benchmarks spanning six dimensions and 20 subcategories. Table 2 presents the complete taxonomy-aligned classification. Table 3 provides quantitative distribution across dimensions. Below, we summarize key corpus characteristics:

• Temporal Coverage. Spans 14 years (WSC 2012 [74] to AIME 2025 [73]), with 35% from 2024 to 2026, enabling both longitudinal capability analysis and contemporary relevance.
• Modality Distribution. Forty-two text-only, nine multimodal (vision-language), eight code-specialized, six agentic/interactive benchmarks, reflecting expansion beyond pure language modeling toward multimodal and embodied intelligence.
• Linguistic Diversity. English-dominant (58 benchmarks) with targeted Chinese (C-Eval [69], CLUEWSC [75], MULTI [70]), Greek (GreekBarBench [47]), and multilingual coverage (MFTCXplain [65]: Portuguese, Italian, Persian, English).
• Difficulty Spectrum. Basic reasoning tasks (83.7% human accuracy on SimpleBench [76]) through undergraduate knowledge (MMLU [19]) and competition mathematics (AIME [73]) to research-frontier problems (<1% expert accuracy on FrontierMath [37]).
• Anti-Contamination Mechanisms. Fourteen benchmarks (22%) employ active contamination resistance: rolling updates (LiveBench [68]), unpublished problems (FrontierMath [37]), expert validation (HLE [42], SWE-bench Verified [77]), or adversarial filtering (WinoGrande [14]).
• Multi-Dimensional Coverage. Nine benchmarks span multiple dimensions: MathVista [44] (mathematical reasoning + multimodal knowledge), CLadder [38] (logical + causal reasoning), OlympiadBench [78] (mathematical + scientific reasoning), reflecting increasingly multifaceted frontier evaluation.

Detailed benchmark descriptions with task examples, input formats, and evaluation criteria are provided in Appendix A.

Table 2. Taxonomy-Aligned Classification of 63 Benchmarks Across Six Dimensions and 20 Subcategories.

Dimension/Subcategory	Benchmarks
D1. Reasoning and Problem-Solving
Logical and Abstract	BBH [35], BIG-bench [16], CLadder [38], HellaSwag [34], PIQA [79], WinoGrande [14], WSC [74]
Mathematical	† AIME 2025 [73], † FrontierMath [37], GSM8K [36], MATH [15], MATH-500 [15,71], MathVista [44], OlympiadBench [78]
Causal	† CLadder [38], † QRData [39]
Scientific	AI2 ARC [40], GPQA Diamond [20], OlympiadBench [78]
D2. Knowledge and Comprehension
General Knowledge	AGIEval [41], C-Eval [69], GPQA Diamond [20], HLE [42], MMLU [19], MMLU-Pro [80]
Reading Comprehension	DROP [43], GLUE [18], SQuAD [6], SuperGLUE [13]
Multimodal Knowledge	MathVista [44], MME [81], MMMU [45], † MULTI [70], VQA [46]
Specialized Domains	† GreekBarBench [47], HLE [42], † OneEval [48]
D3. Generation and Creativity
Code Generation	CodeXGLUE [82], DS-1000 [51], HumanEval [49], LiveCodeBench [72], MBPP [83], SWE-bench [50], † SWE-bench Multimodal [84], SWE-bench Verified [77]
Language Generation	AlpacaEval 2.0 [54], ArenaHard [85], Chatbot Arena [52], MT-Bench [53]
Instruction Following	IF-Eval [55]
D4. Interaction and Agency
Complex Agentic Tasks	AgentBench [58], GAIA [56], VisualWebArena [86], WebArena [57]
Information Retrieval	† Visual-RAG [59]
Document Processing	† OmniAI OCR [60]
D5. Alignment and Safety
Safety and Refusal	AgentHarm [62], HarmBench [61], † SafetyBench [87]
Truthfulness	TruthfulQA [63]
Bias and Fairness	BBQ [64], † MFTCXplain [65]
General Alignment	HHH [66]
D6. Holistic Evaluation
Comprehensive Frameworks	† BBEH [88], HELM [67], LiveBench [68], † SimpleBench [76]
Multilingual/Cross-Cultural	C-Eval [69], CLUEWSC [75], † GreekBarBench [47], † MFTCXplain [65], † MULTI [70]

^† Emerging Track benchmarks (Frontier Probes). Some benchmarks appear in multiple subcategories due to multi-dimensional evaluation targets.

Table 2. Taxonomy-Aligned Classification of 63 Benchmarks Across Six Dimensions and 20 Subcategories.

Dimension/Subcategory	Benchmarks
D1. Reasoning and Problem-Solving
Logical and Abstract	BBH [35], BIG-bench [16], CLadder [38], HellaSwag [34], PIQA [79], WinoGrande [14], WSC [74]
Mathematical	† AIME 2025 [73], † FrontierMath [37], GSM8K [36], MATH [15], MATH-500 [15,71], MathVista [44], OlympiadBench [78]
Causal	† CLadder [38], † QRData [39]
Scientific	AI2 ARC [40], GPQA Diamond [20], OlympiadBench [78]
D2. Knowledge and Comprehension
General Knowledge	AGIEval [41], C-Eval [69], GPQA Diamond [20], HLE [42], MMLU [19], MMLU-Pro [80]
Reading Comprehension	DROP [43], GLUE [18], SQuAD [6], SuperGLUE [13]
Multimodal Knowledge	MathVista [44], MME [81], MMMU [45], † MULTI [70], VQA [46]
Specialized Domains	† GreekBarBench [47], HLE [42], † OneEval [48]
D3. Generation and Creativity
Code Generation	CodeXGLUE [82], DS-1000 [51], HumanEval [49], LiveCodeBench [72], MBPP [83], SWE-bench [50], † SWE-bench Multimodal [84], SWE-bench Verified [77]
Language Generation	AlpacaEval 2.0 [54], ArenaHard [85], Chatbot Arena [52], MT-Bench [53]
Instruction Following	IF-Eval [55]
D4. Interaction and Agency
Complex Agentic Tasks	AgentBench [58], GAIA [56], VisualWebArena [86], WebArena [57]
Information Retrieval	† Visual-RAG [59]
Document Processing	† OmniAI OCR [60]
D5. Alignment and Safety
Safety and Refusal	AgentHarm [62], HarmBench [61], † SafetyBench [87]
Truthfulness	TruthfulQA [63]
Bias and Fairness	BBQ [64], † MFTCXplain [65]
General Alignment	HHH [66]
D6. Holistic Evaluation
Comprehensive Frameworks	† BBEH [88], HELM [67], LiveBench [68], † SimpleBench [76]
Multilingual/Cross-Cultural	C-Eval [69], CLUEWSC [75], † GreekBarBench [47], † MFTCXplain [65], † MULTI [70]

^† Emerging Track benchmarks (Frontier Probes). Some benchmarks appear in multiple subcategories due to multi-dimensional evaluation targets.

Table 3. Quantitative Distribution of Benchmarks Across Taxonomy Dimensions.

Dimension	Subcategories	Benchmarks	Core/Emerging
D1. Reasoning and Problem-Solving	4	17	13/4
D2. Knowledge and Comprehension	4	17	14/3
D3. Generation and Creativity	3	13	12/1
D4. Interaction and Agency	3	6	4/2
D5. Alignment and Safety	4	7	5/2
D6. Holistic Evaluation	2	9	4/5
Total (unique)	20	63	49/14

Per-dimension counts sum to 69 because six benchmarks span two dimensions: C-Eval (D2, D6), GPQA Diamond (D1, D2), GreekBarBench (D2, D6), MathVista (D1, D2), MFTCXplain (D5, D6), and MULTI (D2, D6). The total row reports globally unique counts.

Table 3. Quantitative Distribution of Benchmarks Across Taxonomy Dimensions.

Dimension	Subcategories	Benchmarks	Core/Emerging
D1. Reasoning and Problem-Solving	4	17	13/4
D2. Knowledge and Comprehension	4	17	14/3
D3. Generation and Creativity	3	13	12/1
D4. Interaction and Agency	3	6	4/2
D5. Alignment and Safety	4	7	5/2
D6. Holistic Evaluation	2	9	4/5
Total (unique)	20	63	49/14

Per-dimension counts sum to 69 because six benchmarks span two dimensions: C-Eval (D2, D6), GPQA Diamond (D1, D2), GreekBarBench (D2, D6), MathVista (D1, D2), MFTCXplain (D5, D6), and MULTI (D2, D6). The total row reports globally unique counts.

5. Reference Models for Saturation Analysis

Just as rigorous criteria were established for selecting benchmarks, the selection of models for this review is also systematic. Our framework ensures comprehensive, balanced, and comparable analysis across the full benchmark corpus compiled in Section 4 (cut-off: March 2026). Rather than focusing solely on the latest systems, we include representative families that (i) cover diverse architectural choices and licensing regimes, (ii) expose trade-offs between scale and efficiency, and (iii) can be evaluated consistently across the six dimensions of our taxonomy (Section 3).

5.1. Evaluation Protocols

Unless a benchmark mandates otherwise, we adopt standardized evaluation settings: 0-shot or 5-shot prompting (benchmark-specific), temperature $\in \{0.0,0.2\}$ with fixed seed, nucleus sampling $p=0.95$, and no external tools by default. For reasoning-intensive models (marked with ^†), we utilize their specific “thinking” or extended test-time compute modes. Where available, we report scores at the model’s highest reasoning effort setting. For agentic benchmarks (D4: Interaction and Agency), we permit tool use as specified in benchmark protocols.

5.2. Model Selection

Our corpus includes 16 representative models across three categories:

• Frontier Proprietary Models (Nine Models).

GPT-5.4 [89], GPT-5.2 [90], GPT-5 (High) [91], Gemini 3.1 Pro [92], Gemini 3.0 Pro [93], Claude Opus 4.6 [94], Claude Sonnet 4.6 [95], Claude Sonnet 4.5 [96], and Grok 4.1 [97]. These define the current state-of-the-art and serve as upper-bound references.

• Leading Open-Source Models (Six Models).

Llama 3.1 (405B) [98], DeepSeek-V3.2 [99], DeepSeek-R1 0528 [100], Qwen 3 235B [101], Qwen 3.5 [102], and Kimi K2 Thinking [103]. Critical for reproducible science and demonstrating open-source/proprietary parity.

• Longitudinal Baseline (One Model).

Claude 3.7 Sonnet serves as a longitudinal reference point for capability evolution over the past 12 months.

5.3. Performance Matrix Across Six Dimensions

Table 4. Performance Matrix: Frontier Proprietary Models Across 6 Dimensions (Part 1 of 3).

Model	D2: Know.		D1: Reas.		D1: Math	D3: Code Gen.			D2: Expert	D3
	MMLU	MMLU-Pro	GPQA Dia.	MATH-500	AIME ’25	HumanEval	LiveCodeB.	SWE-b Ver.	HLE	Arena Rank
Frontier Proprietary Models
GPT-5.4 †	–	87.5	94.0	–	–	–	–	–	41.6	6
GPT-5.2 †	–	87.4	90.3	–	99.0	–	88.9	80.0	35.4	36
Gemini 3.1 Pro †	–	91.2	94.1	–	–	–	–	80.6	45.9	3
Gemini 3.0 Pro †	–	90.1	90.8	–	95.7	–	91.7	76.2	37.2	5
Claude Opus 4.6 †	–	89.1	89.6	–	–	–	–	80.8	36.7	1
Claude Son. 4.6 †	–	87.3	87.5	–	–	–	–	79.6	30.0	17
Claude Son. 4.5 †	–	87.4	–	–	88.0	93.7	71.4	–	17.3	24
Grok 4.1 †	–	84.2	87.7	–	89.3	–	82.2	–	17.6	11
GPT-5 (High) †	–	87.1	85.4	99.4	89.3	93.4	84.6	74.9	26.5	41

Bold indicates the best (highest) score in each benchmark column. Italic rows are model-category groupings. ^† Evaluated in extended test-time compute (“thinking”) configurations. Scores are percentages except Arena Rank. “–” indicates a score not publicly reported as of the cut-off date; per-cell source traceability is provided in Appendix B.

,

Table 5. Performance Matrix: Open-Source and Baseline Models (Part 2 of 3).

Model	D2: Know.		D1: Reas.		D1: Math	D3: Code Gen.			D2: Expert	D3
	MMLU	MMLU-Pro	GPQA Dia.	MATH-500	AIME ’25	HumanEval	LiveCodeB.	SWE-b Ver.	HLE	Arena Rank
Longitudinal Baseline
Claude 3.7 Sonnet †	85.3	84.0	77.2	94.7	56.3	–	47.3	70.3	10.3	106
Open-Source Models
Llama 3.1 405B	84.5	61.6	72.7	88.9	69.7	89.0	68.6	–	10.3	162
DeepSeek-V3.2 †	–	85.0	84.0	–	92.0	–	86.2	73.1	22.2	52
Qwen 3 235B †	–	84.5	77.2	98.4	88.3	–	64.6	–	10.1	65
Qwen 3.5 †	–	87.8	89.3	–	–	–	–	76.4	27.3	27
Kimi K2 Think. †	88.3	81.0	83.8	97.1	94.7	93.3	85.3	71.3	22.3	45
DeepSeek-R1 0528 †	90.5	83.4	81.3	–	76.0	–	77.0	44.6	14.6	56

Italic rows are model-category groupings. ^† Evaluated in extended test-time compute (“thinking”) configurations. Scores are percentages except Arena Rank. “–” indicates a score not publicly reported as of the cut-off date; per-cell source traceability is provided in Appendix B.

and

Table 6. Performance Matrix: Summary Statistics and Best Scores (Part 3 of 3).

Benchmark	Best Score	Best Model	Range	Models Rep.
MMLU	90.5	DeepSeek-R1	84.5–90.5	4/16
MMLU-Pro	91.2	Gemini 3.1 Pro	61.6–91.2	16/16
GPQA Diamond	94.1	Gemini 3.1 Pro	72.7–94.1	15/16
MATH-500	99.4	GPT-5 (High)	88.9–99.4	5/16
AIME 2025	99.0	GPT-5.2	56.3–99.0	11/16
HumanEval	93.7	Claude Son. 4.5	89.0–93.7	4/16
LiveCodeBench	91.7	Gemini 3.0 Pro	47.3–91.7	11/16
SWE-bench Ver.	80.8	Claude Opus 4.6	44.6–80.8	11/16
HLE	45.9	Gemini 3.1 Pro	10.1–45.9	16/16
Arena Elo (Rank)	1 (best)	Claude Opus 4.6	1–162	16/16

Range shows minimum to maximum reported scores. Models Reporting indicates a fraction of 16 models with available data. MMLU severely underreported (only 4 models) as frontier systems prioritize MMLU-Pro. HLE shows widest performance spread (10.1–45.9%), confirming status as most discriminative benchmark.

synthesize publicly reported scores for 16 models across 10 key benchmarks spanning our six-dimensional taxonomy. Benchmark selection prioritizes discriminative power: MMLU/MMLU-Pro (D2: General Knowledge), GPQA Diamond (D1: Scientific Reasoning), MATH-500/AIME 2025 (D1: Mathematical Reasoning), HumanEval/LiveCodeBench/SWE-bench Verified (D3: Code Generation), HLE (D2: Specialized Expert Knowledge, most discriminative), and Arena Elo Rank (D3: Language Generation, holistic proxy).

Scores compiled from official technical reports, independent leaderboards (Artificial Analysis, LMArena, HELM, LLM-Stats), and peer-reviewed publications. Blank cells indicate unavailable verified scores at the cut-off date (March 2026). All scores are reported as percentages except Arena Elo Rank. Models marked with ^† were evaluated in extended test-time compute configurations.

5.4. Key Observations from Performance Data

Analysis of the 16-model performance matrix reveals structural patterns across our six-dimensional taxonomy, informing both benchmark saturation status and evaluation priorities:

• D1: Reasoning and Problem-Solving—Bifurcated Saturation.

Scientific reasoning (GPQA Diamond) exhibits ceiling effects approaching saturation: Gemini 3.1 Pro (94.1%) and GPT-5.4 (94.0%) now exceed the average PhD-level human baseline. Four models cluster above 90% (Gemini 3.0 Pro: 90.8%, GPT-5.2: 90.3%), with 15 of 16 models reporting scores. This benchmark may lose discriminative power within 6–12 months as additional reasoning-specialized models reach this tier.

Mathematical reasoning shows extreme bifurcation, with only 5 of 16 models reporting MATH-500 scores. Among those reporting, performance is tightly compressed at the ceiling: GPT-5 (High) achieves 99.4%, followed by Qwen 3 235B (98.4%) and Kimi K2 (97.1%). This >97% clustering indicates MATH-500 saturation for frontier reasoning models. AIME 2025 exhibits similar patterns (GPT-5.2: 99.0%, Gemini 3.0 Pro: 95.7%, Kimi K2: 94.7%), with 11 models reporting but top performers compressed within five percentage points.

The mathematical reasoning bifurcation becomes starkest when considering FrontierMath (not shown due to limited public reporting), where even top models remain <50%. This creates a problematic gap: intermediate-difficulty benchmarks targeting 75–90% frontier performance are absent, preventing fine-grained capability profiling.

• D2: Knowledge and Comprehension—MMLU Obsolescence, HLE Dominance.

MMLU obsolescence: Only 4 of 16 models report MMLU scores (DeepSeek-R1: 90.5%, Kimi K2: 88.3%, Claude 3.7: 85.3%, Llama 3.1: 84.5%), reflecting industry abandonment of this saturated benchmark in favor of MMLU-Pro. This underscores the importance of benchmark replacement strategies as evaluation instruments approach ceilings.

MMLU-Pro saturation trajectory: All 16 models report MMLU-Pro, making it the most comprehensively evaluated benchmark in our matrix. However, frontier performance clustering at 87–91% (Gemini 3.1 Pro: 91.2%, Gemini 3.0 Pro: 90.1%, Claude Opus 4.6: 89.1%) indicates approaching saturation. The 91.2% ceiling leaves less than two percentage points to the estimated annotation-error bound (∼93%), suggesting MMLU-Pro will require successor benchmarks within 12–18 months.

HLE discriminative dominance: HLE achieves the widest performance spread in the entire corpus (10.1–45.9%, range: 35.8 points), with all 16 models reporting. The top score (Gemini 3.1 Pro: 45.9%) remains below 50%, confirming no model has crossed the halfway threshold on this expert-vetted interdisciplinary benchmark. This spread is 3× larger than any other benchmark and validates HLE’s status as the premier instrument for frontier model ranking. The gap between top open-source (Qwen 3.5: 27.3%) and top proprietary (Gemini 3.1 Pro: 45.9%) on HLE (17.4 points) is also the largest observed, suggesting expert-level interdisciplinary reasoning remains a frontier proprietary advantage.

• D3: Generation and Creativity—Code Convergence, Arena Compression.

Code generation maturation: HumanEval shows a narrow spread among reporting models (89.0–93.7%, four models reporting), with Claude Sonnet 4.5 (93.7%), GPT-5 High (93.4%), and Kimi K2 (93.3%) clustered within 0.4 percentage points. This saturation at synthetic code benchmarks contrasts with the broader spread on repository-scale engineering.

LiveCodeBench exhibits wider variance (47.3–91.7%, 11 models reporting), with Gemini 3.0 Pro (91.7%) substantially ahead of the next tier (GPT-5.2: 88.9%, DeepSeek-V3.2: 86.2%). The large gap between frontier (91.7%) and longitudinal baseline (Claude 3.7: 47.3%)—44.4 points—demonstrates rapid code generation improvement over 12 months.

SWE-bench Verified (11 models reporting) shows convergence at 76–81%: GPT-5.2 and Claude Opus 4.6 both achieve 80%+, with Gemini 3.1 Pro (80.6%), Claude Sonnet 4.6 (79.6%), and Qwen 3.5 (76.4%) forming a tight cluster. The gap between top frontier (80.8%) and top open-source (Qwen 3.5: 76.4%) has narrowed to just 4.4 percentage points, indicating repository-scale engineering parity emerging between proprietary and open-source models. Notably, DeepSeek-R1’s lower performance (44.6%) suggests reasoning-specialized architectures may sacrifice repository understanding for mathematical capability.

Arena Elo compression: Chatbot Arena rankings show extreme compression at the top: the best five models (Claude Opus 4.6 rank 1, Gemini 3.1 Pro rank 3, Gemini 3.0 Pro rank 5, GPT-5.4 rank 6, Grok 4.1 rank 11) cluster within the top 11 positions. This contrasts with the >100-rank spreads observed in 2024, indicating human preference evaluation has reduced discriminative power as frontier dialog capabilities converge.

• D4: Interaction and Agency—Proxy-Only Assessment.

No dedicated agentic benchmarks (GAIA, WebArena, AgentBench) appear in the performance matrix due to sparse public reporting. SWE-bench Verified serves as the sole agentic proxy, but this measures only code-specific repository navigation rather than general-purpose planning, tool use, or environment manipulation. The 11-model reporting rate on SWE-bench Verified (69%) is higher than dedicated agentic benchmarks (typically <30% in frontier model technical reports), suggesting evaluation infrastructure for D4 capabilities lags substantially behind other dimensions.

• D5: Alignment and Safety—Complete Data Absence.

Zero safety benchmarks (TruthfulQA, HarmBench, SafetyBench, BBQ) appear in the performance matrix, reflecting critical transparency gaps in frontier model documentation. This absence is particularly concerning given the increasing deployment in high-stakes applications. The complete lack of systematic safety reporting prevents assessment of whether frontier models maintain safety properties as general capabilities advance, or whether capability–safety trade-offs emerge.

• D6: Holistic Evaluation—Open-Source Parity Confirmed.

Arena Elo rankings demonstrate open-source competitive positioning: Qwen 3.5 (rank 27), Kimi K2 (rank 45), and DeepSeek-V3.2 (rank 52) achieve top-100 positions, with Qwen 3.5 outperforming multiple proprietary models, including GPT-5 High (rank 41). However, the best open-source (Qwen 3.5 rank 27) still trails the best proprietary (Claude Opus 4.6 rank 1) by 26 positions.

Cross-benchmark analysis confirms parity on specific capabilities: Kimi K2 matches or exceeds proprietary models on AIME 2025 (94.7%, competitive with Gemini 3.0 Pro’s 95.7%), and Qwen 3.5 achieves near-parity on GPQA Diamond (89.3% vs. Gemini 3.1 Pro’s 94.1%, gap: 4.8 points). The gap narrows to single digits on most technical benchmarks, though proprietary advantages persist on expert-level interdisciplinary reasoning (HLE: 17.4-point gap) and repository-scale engineering (SWE-bench Verified: 4.4-point gap to Qwen 3.5).

• Cross-Cutting: Data Sparsity and Reporting Heterogeneity.

Performance matrix completeness varies dramatically by benchmark: MMLU-Pro (16/16 models, 100%), HLE (16/16, 100%), and GPQA Diamond (15/16, 94%) achieve near-universal reporting, while MMLU (4/16, 25%), HumanEval (4/16, 25%), and MATH-500 (5/16, 31%) suffer severe underreporting. This heterogeneity prevents systematic capability comparison across the full model corpus.

The reporting pattern reveals benchmark lifecycle dynamics: MMLU’s 25% reporting rate confirms industry abandonment post-saturation, while MMLU-Pro’s 100% adoption demonstrates successor benchmark uptake. However, critical gaps persist: no safety benchmarks (D5) appear despite deployment importance, and agentic benchmarks (D4) remain absent from frontier model reporting despite increasing real-world applications requiring autonomous task execution.

This data sparsity validates our benchmark quality assessment (Section 7): Tier A benchmarks with strong reproducibility and contamination resistance (MMLU-Pro, HLE, SWE-bench Verified) achieve high reporting rates, while benchmarks lacking standardized evaluation infrastructure (safety, agentic) remain underreported even when methodologically sound.

• Implications for Benchmark Selection.

The performance data validate our inclusion criteria (Section 4): (i) saturated benchmarks like MMLU face industry abandonment (25% reporting), confirming the need for dynamic replacement strategies; (ii) discriminative benchmarks like HLE maintain universal reporting adoption (100%) and provide widest performance spreads (35.8 points), justifying prioritization in evaluation protocols; (iii) contamination-resistant instruments (LiveCodeBench, SWE-bench Verified) achieve strong reporting rates (69%), confirming that anti-contamination mechanisms enhance rather than hinder benchmark adoption; and (iv) systematic gaps in safety (D5: 0% reporting) and agentic (D4: proxy-only) evaluation demand urgent infrastructure development to match capability deployment patterns.

These patterns inform both the BQAI quality analysis (Section 7) and the research gap identification (Section 8) in subsequent sections.

6. Benchmark Quality Assurance Index (BQAI)

Benchmarks for Large Language Models vary widely in scientific rigor, annotation quality, and methodological robustness. Many suffer from saturation, inconsistent annotation protocols, vulnerability to contamination, or inadequate documentation—factors that compromise their utility for frontier model evaluation. As model capabilities advance and training corpora increasingly subsume public test sets, systematic assessment of benchmark quality becomes critical for maintaining scientific validity in LLM evaluation.

To address this need, we introduce the Benchmark Quality Assurance Index (BQAI), a composite metric quantifying the intrinsic scientific quality of evaluation instruments independent of community adoption patterns or transient performance trends. The BQAI provides an evidence-based framework for identifying high-quality benchmarks that will remain discriminative and methodologically sound as the field evolves.

The BQAI aggregates seven quality dimensions ($Q_1$–$Q_7$), each scored in the interval $[0,1]$ where 1 indicates full compliance with best practices. The index is computed as a weighted sum:

$$\mathrm{BQAI}={\displaystyle \sum _{i=1}^7}{w}_i{Q}_i,{\displaystyle \sum _{i=1}^7}{w}_i=1,$$

(1)

where weights $w_i$ reflect the relative importance of each dimension, derived via the Analytic Hierarchy Process (AHP) [104] under the principle of Scientific Reproducibility Priority (Section 6.3).

The seven quality dimensions assess complementary aspects of benchmark validity:

• Annotation Quality and Consistency ($Q_1$): Rigor of annotation pipeline, expert validation protocols, inter-annotator agreement (IAA), and systematic quality control. High-quality benchmarks employ multi-stage expert annotation with documented IAA $\ge 0.80$ and verifiable quality assurance mechanisms.
• Instructional Clarity and Format Consistency ($Q_2$): Precision of task definitions, prompt templates, and input/output specifications. Evaluates whether instructions are unambiguous, standardized across instances, and resistant to format artifacts that enable superficial pattern matching.
• Standardization and Versioning Practices ($Q_3$): Existence of frozen test sets, semantic versioning, documented train/validation/test splits, and contamination-resistance mechanisms. Essential for longitudinal comparability and reproducible research across studies.
• Reproducibility and Baseline Implementations ($Q_4$): Availability of official evaluation scripts, deterministic scoring procedures, explicit metric definitions, and documented hyperparameters. A benchmark lacking reproducible evaluation protocols has limited scientific value regardless of other qualities.
• Robustness to Prompt Variation and Contamination ($Q_5$): Resistance to format sensitivity, prompt engineering artifacts, and training data leakage. Includes mechanisms such as rolling test set updates (LiveBench [68]), adversarial filtering (WinoGrande [14]), or unpublished held-out problems (FrontierMath [37]).
• Cognitive Skill Coverage and Task Diversity ($Q_6$): Breadth of cognitive abilities assessed, from single narrow tasks to comprehensive multi-dimensional evaluation spanning reasoning, knowledge integration, planning, and metacognitive skills. Broader coverage enables more informative capability profiling.
• Bias Mitigation and Cross-Cultural Validity ($Q_7$): Linguistic diversity, demographic representation, documented bias analyses, and cross-cultural evaluation validity. This dimension addresses whether benchmarks generalize beyond English-speaking, Western, Educated, Industrialized, Rich, and Democratic (WEIRD) populations [105].

6.1. Operational Scoring Rubric

To ensure reproducibility and minimize subjective judgment, each dimension is scored using observable evidence-based indicators organized into four performance tiers.

Table 7. Evidence-Based Scoring Rubric for BQAI Dimensions (Abbreviated).

Dimension	Scoring Criteria Summary
$Q_1$ Annotation	Low: No documentation. Moderate: Single annotator, minimal checks. High: Expert annotators, IAA $\ge 0.70$. Excellent: Multi-stage pipeline, IAA $\ge 0.80$, systematic validation.
$Q_2$ Clarity	Low: Ambiguous definitions. Moderate: Defined but not standardized. High: Clear I/O, minimal ambiguity. Excellent: Fully specified templates, comprehensive docs.
$Q_3$ Standardization	Low: No versioning, undocumented split. Moderate: Ad hoc splits, informal versioning. High: Frozen test set, documented partitions. Excellent: Semantic versioning, contamination-proof, public leaderboard.
$Q_4$ Reproducibility	Low: No code, undefined metrics. Moderate: Partial scripts, missing params. High: Official scripts, documented metrics. Excellent: Deterministic pipeline, variance $\pm 0.5\%$, containerized.
$Q_5$ Robustness	Low: Severe sensitivity, contaminated, saturated (>95%). Moderate: Format sensitivity documented. High: Adversarial filtering, SOTA <92%. Excellent: Rolling updates or unpublished, SOTA <90%, detection tools.
$Q_6$ Coverage	Low: Single narrow task. Moderate: 2–3 related subtasks. High: ≥3 cognitive skills/domains. Excellent: ≥5 dimensions or 10+ diverse tasks.
$Q_7$ Fairness	Low: English-only, no bias analysis. Moderate: Multilingual, no audit. High: Documented bias analysis. Excellent: Cross-cultural validation, demographic audits, mitigation protocols.

Four-tier scoring intervals: Low ($0.00$–$0.30$), Moderate ($0.31$–$0.60$), High ($0.61$–$0.80$), Excellent ($0.81$–$1.00$). Intermediate scores are assigned when evidence partially satisfies adjacent tiers.

presents the complete operational rubric applied consistently across all benchmarks in our corpus.

Scoring proceeds as follows. Each benchmark is independently assessed against the indicators in Table 7 by three blinded evaluators with expertise in LLM evaluation methodology (Section 7.1). Inter-rater reliability is reported on the raw pre-aggregation scores using the Intraclass Correlation Coefficient $\mathrm{ICC}(2,k)$ under a two-way random-effects absolute-agreement model and quadratic-weighted Cohen’s $\kappa$. Intermediate scores (e.g., $0.75$) are assigned when evidence partially satisfies adjacent tiers. For each benchmark, dimension-level scores are first averaged across the three evaluators. Final BQAI scores are then computed using the AHP-weighted sum in Equation (1), yielding a value in the range $[0,1]$.

The full operational rubric with evidence-anchored indicators per tier, together with the three-evaluator scoring sheets, AHP weight derivation, sensitivity analysis, and analysis scripts, is released as a public reproducibility repository (https://github.com/Magnusvron/llm-benchmark-quality-index, accessed on 19 May 2026).

6.2. Quality Tier Classification

Based on the BQAI distribution across our 63-benchmark corpus and established psychometric standards for assessment validity, we define three quality tiers:

• Tier A (High Quality): $\mathrm{BQAI}\ge 0.82$—Benchmarks meeting rigorous scientific standards across all dimensions. Suitable for primary model comparison, longitudinal capability tracking, and anchoring claims about frontier model capabilities.
• Tier B (Moderate Quality): $0.70\le \mathrm{BQAI}<0.82$—Benchmarks with partial methodological rigor. Useful for secondary evaluation, rapid prototyping, or specialized contexts, but requiring cautious interpretation and acknowledgment of limitations.
• Tier C (Low Quality): $\mathrm{BQAI}<0.70$—Benchmarks with substantial methodological limitations. May retain historical value for longitudinal analysis or serve niche purposes, but should not anchor primary claims about model capabilities without explicit caveats.

The tier thresholds are calibrated such that Tier A benchmarks satisfy the corpus inclusion criteria I1–I7 (Section 4) with high confidence, while Tier C benchmarks typically exhibit low scores on at least two critical BQAI dimensions, most commonly robustness ($Q_5$) and standardization ($Q_3$). The criteria I1–I7 governing inclusion in the 63-benchmark corpus are distinct from the criteria S1–S5 governing selection of the 30-benchmark subset to which the BQAI is applied (Section 7).

6.3. AHP Weight Derivation

To determine the relative importance weights $w_i$ in Equation (1), we employ the Analytic Hierarchy Process (AHP) [104], a structured technique for multi-criteria decision-making. The AHP constructs a $7\times 7$ pairwise comparison matrix where element $a_{ij}$ represents the relative importance of dimension $Q_i$ over $Q_j$ on Saaty’s fundamental scale: $1/9$ (much less important) through 1 (equal importance) to 9 (much more important).

Our comparison matrix (

Table 8. AHP Pairwise Comparison Matrix (Consistency Ratio = 0.023).

Dim.	${\mathit{Q}}_1$	${\mathit{Q}}_2$	${\mathit{Q}}_3$	${\mathit{Q}}_4$	${\mathit{Q}}_5$	${\mathit{Q}}_6$	${\mathit{Q}}_7$
$Q_1$ Annot.	1	3	2	1/2	2	5	4
$Q_2$ Clarity	1/3	1	1/2	1/5	1/3	3	2
$Q_3$ Standard.	1/2	2	1	1/3	1/2	4	3
$Q_4$ Repro.	2	5	3	1	3	7	6
$Q_5$ Robust.	1/2	3	2	1/3	1	5	4
$Q_6$ Coverage	1/5	1/3	1/4	1/7	1/5	1	1/2
$Q_7$ Fairness	1/4	1/2	1/3	1/6	1/4	2	1

) is constructed under the principle of Scientific Reproducibility Priority: reproducibility ($Q_4$) is weighted most heavily because irreproducible benchmarks cannot support cumulative scientific progress, followed by annotation quality ($Q_1$) and contamination resistance ($Q_5$) as second-order priorities essential for frontier evaluation validity.

The priority weights are computed as the normalized principal eigenvector of this matrix. Consistency is verified via the Consistency Ratio (CR):

$$\mathrm{CR}={\displaystyle \frac{\mathrm{CI}}{\mathrm{RI}}},\mathrm{where}\mathrm{CI}={\displaystyle \frac{{\lambda }_{\max }-n}{n-1}},$$

(2)

with ${\lambda }_{\max }=7.180$, the maximum eigenvalue, $n=7$ dimensions, and $\mathrm{RI}=1.32$ the random index for $n=7$ (standard AHP reference value [104]). Our matrix achieves $\mathrm{CR}=0.023$, well below the $0.10$ acceptability threshold, confirming judgment consistency.

The resulting weight distribution (

Table 9. Derived BQAI Dimension Weights via AHP.

Quality Dimension	Weight (${\mathit{w}}_{\mathit{i}}$)	Rank
$Q_4$ Reproducibility and Baseline Implementations	0.349	1
$Q_1$ Annotation Quality and Consistency	0.213	2
$Q_5$ Robustness to Prompt Variation/Contam.	0.167	3
$Q_3$ Standardization and Versioning Practices	0.117	4
$Q_2$ Instructional Clarity and Format Consistency	0.073	5
$Q_7$ Bias Mitigation and Cross-Cultural Validity	0.048	6
$Q_6$ Cognitive Skill Coverage and Task Diversity	0.033	7
Total	1.000	–

) prioritizes reproducibility most heavily ($w_4=0.349$), followed by annotation quality ($w_1=0.213$) and contamination robustness ($w_5=0.167$). This weighting reflects the growing evidence that benchmark contamination and benchmark-specific overfitting threaten the validity of frontier LLM evaluation [106,107], whereas fairness considerations ($w_7=0.048$), though important for equitable deployment, have less immediate impact on benchmark scientific validity.

The BQAI provides a standardized metric for comparing benchmark quality across diverse evaluation contexts, but several important caveats apply:

• Not a Measure of Importance.

BQAI scores reflect methodological quality, not practical importance or community adoption. A widely used benchmark like MMLU may score in Tier C on BQAI ($\mathrm{BQAI}=0.589$ in our three-evaluator assessment, due to severe contamination and approaching saturation; see Section 7) while remaining highly influential for historical comparison. Conversely, a methodologically rigorous but niche benchmark may score high on BQAI but see limited adoption.

• Context-Dependent Utility.

Benchmark selection should balance BQAI scores with task-specific requirements. For longitudinal capability tracking and anchoring claims about frontier model performance, high-BQAI benchmarks (Tier A) are essential. For rapid prototyping, ablation studies, or preliminary evaluation, moderate-BQAI benchmarks (Tier B) may suffice if contamination risks and methodological limitations are explicitly acknowledged.

• Weight Sensitivity.

While our AHP weights reflect current priorities in frontier LLM evaluation (reproducibility, contamination resistance, annotation quality), different research contexts may warrant alternative weighting schemes. We provide the complete scoring rubric (Table 7) to enable re-weighting for specialized applications such as safety-focused evaluation (higher $w_7$) or cognitive science research (higher $w_6$). A multi-regime sensitivity analysis confirming robustness of the main rankings under alternative weight choices is reported in Section 7.5.

• Evolving Standards.

As evaluation methodologies advance, the operational definitions in Table 7 should be periodically updated to reflect emerging best practices. For instance, the threshold for “excellent” contamination resistance ($Q_5$) may tighten as rolling update mechanisms and live evaluation platforms become standard practice, or the definition of adequate IAA ($Q_1$) may evolve with improved annotation protocols.

• Complementary to Performance Analysis.

The BQAI assesses benchmark quality but does not directly measure benchmark difficulty or discriminative power. A high-BQAI benchmark may still approach saturation (e.g., HELM, $\mathrm{BQAI}=0.769$, with rigorous holistic methodology but frontier-model performance clustering above 90% on most subtasks). Therefore, BQAI scores should be interpreted alongside saturation analysis (Section 5.4) and performance distribution analysis (Section 5) for a comprehensive benchmark assessment.

The BQAI framework enables systematic quality-aware benchmark selection, complementing performance-based saturation analysis with methodological rigor assessment. The following section applies this framework to a representative sample of 30 benchmarks from our corpus, identifying high-quality evaluation instruments suitable for frontier model assessment in 2026 and beyond.

7. BQAI Assessment of Benchmark Corpus

To assess the discriminative behavior and practical utility of the BQAI framework, we applied the weighted scoring methodology (Section 6) to a representative sample of 30 benchmarks (48% of the curated 63-benchmark corpus). The sample was selected under five explicit criteria designed to ensure both representativeness and methodological coverage:

• C1—Balanced dimensional coverage: At least four benchmarks per taxonomy dimension (D1–D6), correcting the under-representation of D4 (Agency) and D5 (Safety) that affected earlier evaluations of the BQAI on this corpus.
• C2—Documented adoption: Reported in at least two frontier model technical reports (2024–2026) or present on established public leaderboards (Chatbot Arena, HELM, Artificial Analysis, Papers with Code).
• C3—Temporal diversity: Coverage from 2019 (HellaSwag) through 2025 (HLE, AIME 2025), enabling longitudinal analysis of benchmark quality evolution.
• C4—Track diversity: A deliberate mixture of Core Track gold standards and Emerging Track frontier probes (20 Core/10 Emerging in the sample), reflecting the structure of the full corpus.
• C5—Expected variation in BQAI: Inclusion of candidates for each tier (A, B, C), enabling empirical validation of the framework’s discriminative power rather than mere assertion.

The per-benchmark selection rationale, mapping each benchmark to its motivating criteria, is documented in the supplementary reproducibility repository (https://github.com/Magnusvron/llm-benchmark-quality-index, accessed on 19 May 2026), alongside a representativeness analysis demonstrating that the sample preserves the structural properties of the full 63-benchmark corpus across nine dimensions (dimensional distribution, track ratio, temporal coverage, modality, linguistic coverage, anti-contamination prevalence, subcategory coverage, expected tier distribution, and modal year).

7.1. Scoring Methodology

Each benchmark was scored against the seven quality dimensions ($Q_1$–$Q_7$) using the operational rubric (Table 7). Scoring was performed through a systematic review of:

• Primary Sources: Official benchmark papers, technical reports, and documentation;
• Reproducibility Artifacts: Availability and quality of evaluation scripts, datasets, and leaderboards;
• Community Evidence: Independent replication studies, contamination analyses, and meta-reviews;
• Version Control: GitHub repositories, changelogs, and semantic versioning practices.

Scoring was performed by three independent evaluators with expertise in LLM evaluation methodology, working under a blinded protocol (no inter-evaluator communication during the scoring phase). Inter-rater reliability was assessed on the raw pre-aggregation scores using two complementary metrics: the Intraclass Correlation Coefficient $\mathrm{ICC}(2,k)$ under a two-way random-effects, absolute-agreement model—appropriate for continuous scores from independent raters—and the quadratic-weighted Cohen’s $\kappa$ computed pairwise after discretizing scores into the four operational tiers (Low/Moderate/High/Excellent).

Table 10. Inter-rater reliability of BQAI scoring across three independent evaluators ($n=30$ benchmarks). ICC interpretation: <0.50 poor, 0.50–0.74 moderate, 0.75–0.90 good, >0.90 excellent. Weighted $\kappa$ interpretation: <0.40 weak, 0.40–0.59 moderate, 0.60–0.79 substantial, ≥0.80 strong.

Dimension	$\mathbf{ICC}\mathbf{(}2,\mathit{k}\mathbf{)}$	95% CI	Interpretation	Weighted $\mathit{\kappa }$	Mean $|\Delta |$
$Q_1$ Annotation	0.744	[0.54, 0.87]	moderate	0.383	0.060
$Q_2$ Clarity	0.802	[0.64, 0.90]	good	0.461	0.046
$Q_3$ Standardization	0.828	[0.69, 0.91]	good	0.580	0.060
$Q_4$ Reproducibility	0.884	[0.79, 0.94]	good	0.543	0.064
$Q_5$ Robustness	0.852	[0.65, 0.93]	good	0.645	0.162
$Q_6$ Coverage	0.928	[0.86, 0.96]	excellent	0.671	0.087
$Q_7$ Fairness	0.943	[0.89, 0.97]	excellent	0.502	0.064
Mean	0.855	—	good	0.541	0.078

reports the per-dimension reliability metrics.

Average inter-rater reliability across the seven dimensions was $\mathrm{ICC}(2,k)=0.855$ (range $0.744$–$0.943$), indicating good agreement under standard psychometric thresholds [108], and quadratic-weighted Cohen’s $\kappa =0.541$ (range $0.383$–$0.671$), indicating moderate agreement under the more conservative ordinal-bin interpretation. The largest disagreement was concentrated in $Q_5$ (mean absolute difference $0.162$, maximum $0.78$), reflecting genuine conceptual variation among evaluators about how to penalize benchmark saturation—a methodological debate that we document transparently rather than attempting to reconcile post hoc.

Final BQAI scores reported in

Table 11. BQAI Assessment for 30 Representative Benchmarks (Part 1: Tier A and Tier B). Scores are means across three blinded evaluators.

Benchmark	Dim.	${\mathit{Q}}_1$	${\mathit{Q}}_2$	${\mathit{Q}}_3$	${\mathit{Q}}_4$	${\mathit{Q}}_5$	${\mathit{Q}}_6$	${\mathit{Q}}_7$	BQAI	Tier
Tier A: High Quality (≥0.82)
HLE	D2	0.81	0.88	0.82	0.84	0.93	0.82	0.43	0.831	A
Tier B: Moderate Quality ($0.70\le \mathrm{BQAI}<0.82$)
LiveBench	D6	0.55	0.90	0.93	0.88	0.87	0.82	0.29	0.786	B
LiveCodeBench	D3	0.58	0.86	0.91	0.89	0.82	0.60	0.23	0.772	B
HELM	D6	0.55	0.87	0.91	0.93	0.53	0.93	0.75	0.769	B
HarmBench	D5	0.61	0.81	0.81	0.84	0.67	0.65	0.34	0.727	B
SWE-bench Verified	D3	0.61	0.84	0.87	0.90	0.48	0.55	0.27	0.720	B
CLadder	D1	0.64	0.84	0.69	0.88	0.60	0.60	0.27	0.719	B
OlympiadBench	D1	0.59	0.85	0.76	0.78	0.71	0.70	0.53	0.716	B
MMMU	D2	0.57	0.84	0.84	0.83	0.60	0.85	0.33	0.714	B
MMLU-Pro	D2	0.59	0.88	0.81	0.91	0.39	0.85	0.28	0.710	B
BBQ	D5	0.67	0.86	0.74	0.81	0.49	0.56	0.68	0.707	B

Italic rows are tier groupings. $Q_1$–$Q_7$ are the seven quality-dimension scores defined in Section 6; Dim. indicates the primary taxonomy dimension.

and 

Table 12. BQAI Assessment for 30 Representative Benchmarks (Part 2: Tier C). Scores are means across three blinded evaluators. Benchmarks within $\pm 0.03$ of the B/C boundary are highlighted as threshold-adjacent.

Benchmark	Dim.	${\mathit{Q}}_1$	${\mathit{Q}}_2$	${\mathit{Q}}_3$	${\mathit{Q}}_4$	${\mathit{Q}}_5$	${\mathit{Q}}_6$	${\mathit{Q}}_7$	BQAI	Tier
Tier C: Compromised Quality ($\mathrm{BQAI}<0.70$)—Threshold-adjacent (within $-0.03$)
GPQA Diamond	D1/D2	0.75	0.88	0.76	0.84	0.33	0.62	0.27	0.695	C
WebArena	D4	0.53	0.78	0.84	0.78	0.69	0.72	0.27	0.694	C
MathVista	D2	0.58	0.82	0.82	0.80	0.53	0.66	0.32	0.682	C
SafetyBench	D5	0.55	0.81	0.77	0.81	0.45	0.68	0.58	0.675	C
AgentBench	D4	0.53	0.77	0.79	0.71	0.73	0.86	0.27	0.674	C
Tier C: Compromised Quality—Stable below threshold
GAIA	D4	0.60	0.81	0.87	0.72	0.49	0.84	0.29	0.665	C
C-Eval	D6	0.55	0.87	0.79	0.83	0.28	0.79	0.53	0.662	C
OmniAI OCR	D4	0.49	0.81	0.75	0.75	0.61	0.55	0.29	0.648	C
HumanEval	D3	0.53	0.91	0.77	0.90	0.15	0.40	0.25	0.634	C
AIME 2025	D1	0.57	0.95	0.66	0.72	0.47	0.28	0.24	0.618	C
TruthfulQA	D5	0.53	0.78	0.72	0.70	0.45	0.55	0.29	0.608	C
Chatbot Arena	D3	0.71	0.71	0.74	0.43	0.71	0.82	0.42	0.606	C
MT-Bench	D3	0.71	0.75	0.69	0.68	0.28	0.68	0.25	0.606	C
BIG-bench	D6	0.50	0.73	0.71	0.77	0.25	0.97	0.35	0.601	C
MMLU	D2	0.45	0.79	0.66	0.84	0.15	0.83	0.25	0.589	C
GSM8K	D1	0.53	0.91	0.52	0.85	0.17	0.35	0.23	0.587	C
HellaSwag	D1/D2	0.58	0.81	0.71	0.79	0.12	0.37	0.22	0.583	C
HHH	D5	0.63	0.73	0.60	0.65	0.32	0.52	0.32	0.571	C
FrontierMath	D1	0.42	0.59	0.63	0.51	0.85	0.58	0.27	0.558	C

Italic rows are tier groupings. $Q_1$–$Q_7$ are the seven quality-dimension scores defined in Section 6; Dim. indicates the primary taxonomy dimension.

are computed as the unweighted mean of the three evaluator scores, preserving each evaluator’s independent judgment without privileging any single perspective. Raw pre-aggregation scores per evaluator, the operational rubric with verifiable indicators per cell, and the full analysis code are released in the supplementary repository to enable independent verification.

The aggregated BQAI is computed using the AHP-derived weights (Table 9):

$$\mathrm{BQAI}=0.349Q_4+0.213Q_1+0.167Q_5+0.117Q_3+0.073Q_2+0.048Q_7+0.033Q_6$$

7.2. Benchmark Quality Assessment Results

Table 11 and Table 12 present the dimension-level scores (mean across three evaluators) and final weighted BQAI for all 30 benchmarks, organized by quality tier (Tier A: ≥0.82, Tier B: 0.70–0.82, Tier C: <0.70). Under the three-evaluator protocol, the BQAI yields a notably conservative tier distribution: 1 Tier A benchmark, 10 Tier B benchmarks, and 19 Tier C benchmarks. This stringent outcome reflects the rigor of multi-evaluator absolute-agreement scoring and demonstrates the BQAI’s discriminative power: even widely cited benchmarks rarely satisfy all seven quality criteria simultaneously when applied with consistency across independent raters.

7.3. Analysis by Quality Tier

The BQAI assessment reveals a quality hierarchy that diverges substantially from popularity-based rankings. The conservative tier distribution under the three-evaluator protocol underscores the stringency of the operational rubric: only HLE meets all seven quality criteria at the Tier A threshold. Widely cited foundational benchmarks (MMLU, GSM8K, HellaSwag, HumanEval) and even some recent instruments (FrontierMath, HHH, TruthfulQA) fall into Tier C due to specific methodological gaps documented below. Importantly, Tier C classification reflects unweighted absence of evidence across critical dimensions—often a documentation gap rather than a functional limitation—and does not imply the benchmark lacks scientific utility.

7.3.1. Tier A: Gold Standards ($\mathrm{BQAI}\ge 0.82$)

Only one benchmark achieves Tier A status under the three-evaluator protocol.

• HLE (0.831)—Humanity’s Last Exam.

Achieves Tier A through the unique combination of (i) expert annotation quality ($Q_1=0.81$), with PhD-level validation across ${100}^{+}$ disciplines; (ii) exceptional contamination resistance ($Q_5=0.93$), supported by an unpublished private test set and explicit anti-saturation design; (iii) strong cognitive coverage ($Q_6=0.82$), spanning specialized knowledge domains rarely covered in other instruments; and (iv) solid reproducibility ($Q_4=0.84$) through documented evaluation protocols and openly released grading scripts. HLE’s emergence as the sole Tier A benchmark is notable: it is one of the most recent instruments in the corpus (2025) and was explicitly designed to remain unsaturated as frontier models scale. The fairness dimension ($Q_7=0.43$) is the only meaningful gap, reflecting limited explicit attention to non-English contexts and demographic auditing.

• Tier A Characterization.

HLE’s profile suggests that Tier A status in 2026 requires the simultaneous presence of expert-vetted annotation, active anti-contamination design, and documented reproducibility—a combination that few benchmarks currently achieve. As discussed in Section 7.5, four additional benchmarks (LiveBench, LiveCodeBench, HELM, HarmBench) are positioned in upper Tier B and may reach Tier A in future releases as their fairness or coverage scores improve. The fact that only HLE crosses the Tier A threshold under multi-evaluator absolute-agreement scoring confirms the central thesis that benchmark quality assessment requires more nuanced criteria than community adoption alone.

7.3.2. Tier B: Moderate Quality ($0.70\le \mathrm{BQAI}<0.82$)

Ten benchmarks occupy Tier B, representing operationally useful evaluation instruments with one or two specific methodological limitations that prevent Tier A classification. These limitations cluster into three recurring patterns.

• Near-Tier-A: Strong Infrastructure with Fairness Gaps.

LiveBench (0.786), LiveCodeBench (0.772), HELM (0.769), and HarmBench (0.727) all combine high reproducibility ($Q_4\ge 0.84$) with strong robustness ($Q_5\in [0.53,0.87]$), but their fairness scores ($Q_7\in [0.23,0.75]$) reflect the documented English-centric bias of most evaluation infrastructure. HELM is the only benchmark in the entire corpus to achieve $Q_7\ge 0.70$ ($Q_7=0.75$), supported by its documentation of demographic bias and toxicity scenarios. LiveBench and LiveCodeBench achieve exceptional anti-contamination through rolling test set updates, but their multilingual coverage remains limited.

• The Contamination-Reproducibility Tension.

SWE-bench Verified (0.720) and CLadder (0.719) illustrate opposite poles of the contamination-reproducibility dilemma. SWE-bench Verified achieves excellent reproducibility ($Q_4=0.90$) through deterministic unit-test-based evaluation but exhibits only moderate robustness ($Q_5=0.48$) due to its static test set being vulnerable to future contamination. CLadder achieves strong reproducibility ($Q_4=0.88$) on its symbolic causal reasoning ground truth but only moderate robustness ($Q_5=0.60$), reflecting an absence of large-scale contamination audits despite its recent design. The tension between open verifiability and contamination resistance is fundamental to modern benchmark design (Section 8.7).

• Knowledge and Multimodal Benchmarks.

MMLU-Pro (0.710), MMMU (0.714), OlympiadBench (0.716), and BBQ (0.707) reach Tier B through balanced profiles. MMLU-Pro’s contamination-resistant redesign of MMLU (10 answer options, harder distractors) yields a substantial BQAI improvement over its predecessor (0.710 versus MMLU’s 0.589), though $Q_5=0.39$ indicates that its anti-contamination design is partial: format changes preserve the original questions’ general topic distribution. MMMU achieves strong coverage ($Q_6=0.85$) but lower robustness ($Q_5=0.60$). OlympiadBench is notable for its bilingual (English/Chinese) multimodal design, contributing positively to fairness ($Q_7=0.53$). BBQ is the only Tier B safety benchmark and has the second-highest fairness score in the corpus ($Q_7=0.68$), reflecting its explicit focus on social bias measurement.

7.3.3. Tier C: Compromised Quality ($\mathrm{BQAI}<0.70$)

Nineteen benchmarks fall into Tier C—a much larger group than initial estimates suggested, indicating that under rigorous multi-evaluator assessment, the majority of widely used benchmarks exhibit substantial methodological gaps. These cluster into four interpretive patterns.

• Saturation-Driven Tier C: Foundational Benchmarks That Defined the Field.

MMLU (0.589), GSM8K (0.587), HellaSwag (0.583), and HumanEval (0.634) exhibit a common quality profile: relatively strong reproducibility ($Q_4\in [0.79,0.91]$) coupled with a marked collapse in robustness ($Q_5\in [0.12,0.17]$). This pattern is consistent with documented contamination risks, benchmark saturation, and ceiling effects in widely used frontier-model evaluations [106,107]. Two representative cases provide direct empirical evidence. For GSM8K, Zhang et al. [107] introduced GSM1k, a held-out test set matched to GSM8K in difficulty and structure, and observed accuracy drops of up to $13\%$ for overfit models. They further reported an association between a model’s probability of regenerating GSM8K items and its performance gap on GSM1k, suggesting partial memorization. For MMLU, Zhao et al. [109] constructed MMLU-CF using a contamination-free test split and showed that GPT-4o drops from public MMLU scores above $88\%$ to $73.4\%$ on MMLU-CF, accompanied by ranking changes among the top-40 evaluated models. Accordingly, these benchmarks retain substantial longitudinal and historical value, but they should no longer serve as the primary basis for frontier capability claims. Their Tier C status supports a central thesis of this work: popularity is an unreliable proxy for current methodological rigor.

• Frontier Cases with Reproducibility Trade-offs.

FrontierMath (0.558) presents the most informative paradox in the assessment: it achieves the second-highest robustness in our corpus ($Q_5=0.85$, via expert-created unpublished problems) but the lowest reproducibility ($Q_4=0.51$) precisely because that protection prevents independent verification. This trade-off is fundamental in modern benchmark design—contamination resistance and open reproducibility are partially in tension. We discuss this dilemma further in Section 8.7. FrontierMath also exhibited the largest aggregate inter-evaluator disagreement in the corpus (appearing in five of the top ten pairwise dimensional disagreements), reflecting genuine epistemic uncertainty about how to score a benchmark whose primary methodological protection is opacity.

• Recent Benchmarks Approaching Tier B.

Five Tier C benchmarks sit within $-0.03$ of the B/C boundary and would cross it under modest weight perturbation or refined scoring: GPQA Diamond (0.695), WebArena (0.694), MathVista (0.682), SafetyBench (0.675), and AgentBench (0.674). GPQA Diamond’s high annotation quality ($Q_1=0.75$) and clarity ($Q_2=0.88$) are offset by its narrow scope ($Q_6=0.62$) and emerging-stage anti-contamination evidence ($Q_5=0.33$). WebArena’s strong standardization ($Q_3=0.84$) and reproducibility ($Q_4=0.78$) are offset by limited fairness assessment ($Q_7=0.27$). These threshold-adjacent cases warrant explicit acknowledgment: small variations in scoring or weighting can shift their tier classification, and they should be interpreted as functionally comparable to lower Tier B instruments.

• Stable Tier C: Underdeveloped Methodology.

HHH (0.571), Chatbot Arena (0.606), MT-Bench (0.606), and BIG-bench (0.601) reflect distinct methodological constraints. HHH is more a precursor evaluation suite within an alignment paper than a standardized benchmark; Chatbot Arena trades reproducibility ($Q_4=0.43$) for ecological validity through unstructured human preference, with the additional concern that top-5 Elo compression to within 40 points (versus >200 points in 2024) reduces its current discriminative power; MT-Bench depends on LLM-as-judge protocols with their own validation concerns ($Q_5=0.28$); BIG-bench’s enormous coverage ($Q_6=0.97$, the highest in the corpus) cannot offset substantial heterogeneity and partial saturation across its 200⁺ tasks.

• Tier C Interpretation.

Tier C classification does not imply complete invalidity. The 19 Tier C benchmarks retain value for: (i) longitudinal capability comparison against earlier model generations; (ii) rapid prototyping and regression testing; (iii) basic competence verification for non-frontier models; and (iv) targeted evaluation of specific narrow capabilities (e.g., HumanEval for synthetic code completion). However, they should not anchor primary capability claims about frontier models without explicit acknowledgment of the specific methodological gap each exhibits—saturation, contamination, narrow task focus, low fairness, or limited reproducibility infrastructure.

7.4. Dimension-Specific Patterns

Analysis of individual quality dimensions reveals systematic patterns across the benchmark corpus.

• Reproducibility ($Q_4$, $w=0.349$) Bimodal Distribution.

Benchmarks cluster at high ($Q_4\ge 0.85$: 8 benchmarks—HELM, MMLU-Pro, SWE-bench Verified, HumanEval, LiveCodeBench, CLadder, LiveBench, GSM8K) or moderate ($Q_4\in [0.65,0.84]$: 30 benchmarks) reproducibility. Only one benchmark (Chatbot Arena, $Q_4=0.43$) falls below 0.65, reflecting the inherent reproducibility cost of crowdsourced human-preference evaluation that depends on a live platform and millions of pairwise votes that cannot be re-executed deterministically.

• Annotation Quality ($Q_1$, $w=0.213$) Stricter Under Multi-Evaluator Protocol.

No benchmark in the 30-benchmark sample achieves $Q_1\ge 0.85$ under the three-evaluator mean, reflecting evaluator strictness regarding documented inter-annotator agreement reporting. The seven benchmarks with the highest annotation quality (GPQA Diamond, BBQ, Chatbot Arena, HHH, MT-Bench, OlympiadBench, HLE; $Q_1\in [0.59,0.81]$) do not uniformly achieve high robustness: GPQA Diamond ($Q_5=0.33$), HHH ($Q_5=0.32$), and MT-Bench ($Q_5=0.28$) demonstrate that annotation quality alone does not guarantee contamination resistance. The previously hypothesized strong correlation between $Q_1$ and $Q_5$ is partial rather than universal.

• Robustness ($Q_5$, $w=0.167$) Defines Tier A Membership and Drives Disagreement.

HLE achieves $Q_5=0.93$, and six additional benchmarks reach $Q_5\ge 0.70$ (LiveBench at 0.87, FrontierMath at 0.85, LiveCodeBench at 0.82, AgentBench at 0.73, Chatbot Arena at 0.71, OlympiadBench at 0.71)—a total of seven instruments with active contamination resistance. Most Tier C benchmarks score $Q_5\le 0.35$; the saturation-driven Tier C group (MMLU, GSM8K, HellaSwag, HumanEval) scores $Q_5\le 0.17$. Importantly, $Q_5$ exhibited the largest inter-evaluator disagreement (mean $|\Delta |=0.162$, maximum $0.78$ on GAIA, where one evaluator scored 0.10 and another 0.88), reflecting genuine conceptual variation among evaluators about how to penalize benchmark saturation. We retain this disagreement transparently in the aggregated scores rather than reconcile post hoc, in keeping with the blinded protocol.

• Coverage ($Q_6$, $w=0.033$) Low Weight but High Variance.

Despite the lowest AHP weight, coverage scores exhibit the widest range (0.28–0.97). Comprehensive benchmarks (BIG-bench at 0.97, HELM at 0.93, AgentBench at 0.86, MMMU and MMLU-Pro at 0.85, GAIA at 0.84) achieve $Q_6\ge 0.84$, while narrowly scoped benchmarks (AIME 2025 at 0.28, GSM8K at 0.35, HellaSwag at 0.37, HumanEval at 0.40) score $Q_6\le 0.40$. The low weight reflects the principle that narrow but rigorous evaluation is preferred over broad but methodologically weak assessment.

• Fairness ($Q_7$, $w=0.048$) Underdeveloped Dimension.

Only one benchmark in the sample achieves $Q_7\ge 0.70$: HELM ($Q_7=0.75$). Four additional benchmarks reach $Q_7\in [0.50,0.70]$: BBQ (0.68), SafetyBench (0.58), C-Eval (0.53), and OlympiadBench (0.53). The remaining 25 benchmarks (83%) score $Q_7\le 0.50$, indicating that fairness and cross-cultural validity remain a systemic underdeveloped dimension of LLM evaluation. This gap is even more pronounced under the three-evaluator strict protocol than in earlier estimates and represents a critical opportunity for the next generation of benchmark development as LLM deployment expands to global non-English contexts.

7.5. Sensitivity Analysis

To assess the robustness of BQAI rankings to weight uncertainty, we conducted a multi-regime sensitivity analysis. First, Monte Carlo perturbation with n = 1000 trials at three magnitudes ($\pm 10\%$, $\pm 25\%$, $\pm 50\%$) tested marginal-to-aggressive weight variation. Second, four realistic priority-shifted weighting schemes (fairness-emphasized, coverage-emphasized, annotation-emphasized, contamination-emphasized) tested alternative reasonable evaluation philosophies. Third, an equal-weights baseline ($w=1/7$) was included as an extreme contrast case representing the most neutral imaginable scheme. Fourth, Kendall’s $\tau$ rank correlation between AHP-derived and alternative rankings was computed for each scheme.

Table 13. Multi-regime sensitivity analysis of BQAI rankings. Panel A: Monte Carlo perturbation, where “stable” means ≥95% modal-tier consistency across 1000 trials. Panel B: alternative weighting schemes against AHP-derived control.

Panel A. Monte Carlo perturbation (n = 1000 trials per regime)
Perturbation	Benchmarks with ≥95% modal-tier consistency
$\pm 10\%$	29/30 (97%)
$\pm 25\%$	26/30 (87%)
$\pm 50\%$	17/30 (57%)
Panel B. Alternative weighting schemes
Scheme	Kendall $\tau$	Tier agreement
Fairness-emphasized ($Q_7$ doubled to 0.10)	0.903	80.0%
Coverage-emphasized ($Q_6$ doubled to 0.07)	0.908	96.7%
Annotation-emphasized ($Q_1$ elevated to 0.30)	0.899	80.0%
Contamination-emphasized ($Q_5$ elevated to 0.27)	0.821	83.3%
Equal-weights baseline ($w=1/7$, contrast case)	0.692	73.3%

Italic rows are panel titles. Panel A reports the number of benchmarks retaining ≥95% modal-tier consistency across 1000 Monte Carlo trials at each perturbation level. Panel B reports Kendall’s $\tau$ rank correlation and tier agreement of each alternative weighting scheme against the AHP-derived control.

reports the aggregate stability results, and

Table 14. Per-benchmark Monte Carlo sensitivity statistics (n = 1000 trials, $\pm 10\%$ weight perturbation), representative cases.

Benchmark	BQAI	Min	Max	$\mathit{\sigma }$
Tier A Benchmark
HLE	0.831	0.826	0.835	0.001
Upper Tier B Benchmarks
LiveBench	0.786	0.774	0.796	0.004
LiveCodeBench	0.772	0.759	0.781	0.004
HELM	0.769	0.756	0.783	0.005
HarmBench	0.727	0.718	0.735	0.003
Lower Tier B Benchmarks (B/C threshold-adjacent)
SWE-bench Verified	0.720	0.707	0.734	0.005
MMLU-Pro	0.710	0.696	0.726	0.005
BBQ	0.707	0.699	0.714	0.003
Upper Tier C Benchmarks (B/C threshold-adjacent)
GPQA Diamond	0.695	0.682	0.707	0.005
WebArena	0.695	0.685	0.703	0.003
Tier C Benchmarks (stable below threshold)
MMLU	0.589	0.571	0.608	0.007
GSM8K	0.587	0.570	0.604	0.007
HellaSwag	0.583	0.566	0.598	0.006
FrontierMath	0.558	0.549	0.568	0.004

Italic rows are tier groupings. BQAI is the aggregated quality score; Min and Max are the extreme values across 1000 Monte Carlo trials under $\pm 10\%$ weight perturbation; $\sigma$ is the standard deviation across trials.

reports per-benchmark Monte Carlo statistics for representative cases.

• Key Findings.

  – 

  HLE remains stably in Tier A across all perturbations and weighting schemes ($\sigma =0.001$ under $\pm 10\%$), validating its classification as robust to reasonable weight uncertainty.

  – 

  Upper Tier B is stable; near-threshold cases warrant explicit acknowledgment. LiveBench, LiveCodeBench, HELM, and HarmBench remain in Tier B with low variance ($\sigma \in [0.003,0.005]$). Thirteen benchmarks sit within $\pm 0.03$ of the B/C boundary: seven just above (HarmBench, SWE-bench Verified, CLadder, OlympiadBench, MMMU, MMLU-Pro, BBQ) and six just below (GPQA Diamond, WebArena, MathVista, SafetyBench, AgentBench). Small weight variations can shift these cases between tiers, and they should be interpreted as functionally adjacent in quality.

  – 

  Realistic priority-shifted schemes preserve main rankings. Across the four realistic alternative schemes, Kendall’s $\tau \ge 0.82$ and tier agreement ≥80%, indicating that reasonable variations in priority emphasis preserve the main conclusions. The fairness-emphasized and annotation-emphasized schemes shift the most benchmarks (both at 80% tier agreement), reflecting that the dimensions they elevate have the highest within-corpus variance.

  – 

  Equal-weight baseline provides instructive contrast. With $\tau =0.69$ and 73% tier agreement, the equal-weights scheme shifts more benchmarks than realistic alternatives—as expected, since equal weighting discards the AHP-derived priority structure entirely. Even under this extreme baseline, HLE remains Tier A, and the saturation-driven Tier C group (MMLU, GSM8K, HellaSwag) remains Tier C, demonstrating that the BQAI’s main discriminations are robust to weighting philosophy.

  – 

  Tier C benchmarks remain stably below threshold. Despite moderate variance ($\sigma \in [0.004,0.007]$), the saturation-driven Tier C group remains consistently below 0.65 even under fairness-emphasized weighting, confirming that their methodological limitations are dimension-independent.

Overall, the multi-regime sensitivity analysis indicates that BQAI rankings are robust to reasonable weight uncertainty within the AHP-derived priority structure. The highest-ranked benchmark (HLE) and the most clearly limited benchmarks (saturation-driven Tier C group) maintain their classification across all tested schemes, while the dense cluster of threshold-adjacent benchmarks reflects genuine ambiguity that we acknowledge transparently rather than obscure through artificially confident point estimates.

7.6. Implications for Benchmark Selection

The BQAI assessment provides actionable guidance for benchmark selection in frontier model evaluation.

• For Longitudinal Capability Tracking.

Prioritize HLE (Tier A) and upper Tier B benchmarks with strong robustness profiles—LiveBench, LiveCodeBench, HELM, and HarmBench—which combine high reproducibility ($Q_4\ge 0.84$) with active contamination-resistance mechanisms. These instruments are more likely to retain discriminative power and methodological validity as model capabilities continue to improve.

• For Comprehensive Model Profiling.

Combine HELM’s holistic coverage ($Q_6=0.93$) with dimension-specific complements: HLE for specialized expert knowledge ($Q_6=0.82$), MMMU for multimodal evaluation ($Q_6=0.85$), and AgentBench for agentic capabilities ($Q_6=0.86$). This combination provides breadth while maintaining methodological rigor. For multilingual coverage, supplement with C-Eval (Chinese, $Q_7=0.53$) and OlympiadBench (bilingual English-Chinese, $Q_7=0.53$).

• For Contamination-Resistant Evaluation.

Emphasize benchmarks with $Q_5\ge 0.80$: HLE ($Q_5=0.93$), LiveBench (0.87), FrontierMath (0.85), and LiveCodeBench (0.82). These represent the current frontier of anti-contamination methodology through rolling updates, unpublished problems, or expert verification. Note the trade-off: FrontierMath achieves a high $Q_5$ at the cost of low reproducibility ($Q_4=0.51$), while LiveBench and LiveCodeBench achieve both through their public-but-rolling design.

• For Rapid Prototyping and Ablation.

Tier C benchmarks with high reproducibility but moderate robustness—HumanEval (BQAI 0.634, $Q_4=0.90$), GSM8K (0.587, $Q_4=0.85$), and MMLU (0.589, $Q_4=0.84$)—remain useful for development iteration and ablation studies, provided contamination risks are explicitly acknowledged, and results are not used as the sole basis for frontier capability claims.

• For Historical Comparison.

Tier C foundational benchmarks (MMLU, GSM8K, HellaSwag, HumanEval) retain value for contextualizing current performance relative to earlier model generations. The full Tier C list (19 benchmarks) should be consulted with awareness of the specific methodological gap each exhibits—saturation, format sensitivity, narrow task focus, or limited cross-cultural validity. None should be used alone to support claims about state-of-the-art frontier performance.

The BQAI framework thus enables systematic, quality-aware benchmark selection tailored to different evaluation objectives, complementing the taxonomy-based organization (Section 3) and the model-comparison analysis presented elsewhere in this survey.

8. Open Challenges and Research Gaps

Our comprehensive analysis—spanning the six-dimensional taxonomy (Section 3), the 63-benchmark corpus (Section 4), the model performance matrix (Section 5), and the BQAI quality assessment (Section 7)—reveals seven critical gaps in the current LLM evaluation ecosystem that demand urgent research attention:

8.1. D1: Reasoning—Bifurcated Saturation Without Intermediate Difficulty

Mathematical reasoning benchmarks exhibit extreme bifurcation: competition-level problems (AIME 2025, OlympiadBench) approach complete saturation with three frontier models achieving 100% accuracy, while research-frontier mathematics (FrontierMath) remains below 50% even for the strongest systems. This binary distribution—trivial versus intractable—leaves a critical gap in the 70–90% difficulty range necessary for fine-grained capability profiling.

Causal reasoning remains severely underrepresented. Only two benchmarks in our corpus (CLadder, QRData) systematically evaluate reasoning across Pearl’s causal hierarchy (association, intervention, counterfactuals). Both are recent additions marked as emerging track (^†), and neither has achieved widespread adoption in frontier model reporting. Given the importance of causal inference for scientific discovery, policy analysis, and medical reasoning, this represents a fundamental limitation in current evaluation.

Recommendation: Develop intermediate-difficulty mathematical benchmarks targeting the 75–85% accuracy range for frontier models, and establish standardized causal reasoning protocols integrated into comprehensive evaluation suites like HELM.

8.2. D2: Knowledge—Saturation Without Discriminative Alternatives

General knowledge evaluation faces a crisis of diminishing returns. MMLU scores cluster at 88–92.5% for frontier models, approaching the benchmark’s annotation-error ceiling. MMLU-Pro, designed as a more discriminative successor, already reaches 90–92% for top systems (Gemini 3.1 Pro: 91.2%, Gemini 3.0 Pro: 90.1%), leaving less than 10 percentage points of dynamic range.

Specialized expert knowledge remains the sole discriminative dimension. HLE (Humanity’s Last Exam) achieves the widest performance spread in our entire corpus (10.1–45.9%, no model exceeding 50%), but its expert-validated, Google-proof design comes at the cost of limited public reproducibility (BQAI $Q_4=0.84$). This trade-off between contamination resistance and open evaluation represents an unresolved tension in frontier assessment.

Multimodal knowledge evaluation is critically underrepresented. Of 30 benchmarks in our BQAI assessment, only MMMU provides comprehensive multimodal knowledge evaluation. The performance matrix lacks systematic multimodal data because model providers report vision-language scores inconsistently across different benchmarks.

Recommendation: Prioritize development of contamination-resistant knowledge benchmarks with reproducible evaluation infrastructure. Establish standardized multimodal knowledge evaluation protocols with mandatory reporting requirements for frontier model releases.

8.3. D3: Generation—Dialog Quality Measurement Crisis

Code generation shows mature evaluation with Tier B instruments closest to Tier A (LiveCodeBench: BQAI 0.772, SWE-bench Verified: BQAI 0.720), but language generation quality measurement remains methodologically problematic. Chatbot Arena achieves only BQAI 0.606 (Tier C) due to fundamental reproducibility challenges (Q₄ = 0.43) inherent in human-preference-based evaluation. The discriminative power of crowdsourced preference also continues to compress (top five models within 40 Elo points in 2026 versus >200 in 2024).

The reproducibility-ecological validity dilemma. Deterministic dialog benchmarks (MT-Bench) sacrifice ecological validity for reproducibility, while human-preference platforms (Chatbot Arena) sacrifice reproducibility for realism. No current instrument achieves both. Furthermore, Chatbot Arena Elo compression (top five models separated by only 40 points vs. >200 points in 2024) indicates reduced discriminative power as frontier capabilities converge.

Instruction following evaluation remains narrow. IF-Eval provides objective compositional constraint verification but evaluates only syntactic adherence (length, keywords, format), not semantic instruction understanding or multi-step procedural following.

Recommendation: Develop a hybrid dialog evaluation combining deterministic rubrics for reproducibility with targeted human validation for quality. Expand instruction-following evaluation beyond surface constraints to semantic task decomposition and procedural reasoning.

8.4. D4: Agency—Severe Data Scarcity in Public Reporting

Agentic capabilities represent a critical frontier for autonomous systems, yet systematic evaluation data remains sparse. No agentic benchmark achieves Tier A status in our BQAI assessment, and all four fall into Tier C: WebArena (0.695), AgentBench (0.674), GAIA (0.665), and OmniAI OCR (0.648). The four share moderate reproducibility (Q₄ $\in [0.71,0.78]$) and limited fairness scores (Q₇ $\in [0.27,0.30]$). More critically, frontier model technical reports inconsistently include agentic benchmark scores.

SWE-bench Verified serves as an inadequate proxy. The performance matrix relies on SWE-bench Verified as the primary agentic capability indicator, but this measures only code-specific repository navigation rather than general-purpose tool use, planning, or environment manipulation. Comprehensive agentic benchmarks (WebArena, VisualWebArena, GAIA) receive minimal coverage in official model documentation.

Long-context agentic reasoning untested. Despite models supporting 128 k–2 M token contexts (GPT-5.4: 1 M, Gemini 3.1 Pro: 2 M, Claude Opus 4.6: 1 M), no benchmark in our corpus systematically evaluates information retrieval, synthesis, and reasoning over extended multi-document contexts in agentic settings.

Recommendation: Establish mandatory agentic evaluation reporting standards for frontier model releases. Develop contamination-resistant long-context agentic benchmarks requiring multi-step planning over >100 k token contexts.

8.5. D5: Safety—Critical Transparency Gap

Safety evaluation exhibits the most severe reporting gap across all dimensions. None of the five safety benchmarks (TruthfulQA, HarmBench, SafetyBench, BBQ, HHH) achieves Tier A status. The strongest is HarmBench (Tier B, 0.727) through standardized red-teaming methodology; BBQ (Tier B, 0.707) is notable for the highest fairness score among safety instruments (Q₇ = 0.68). TruthfulQA (Tier C, 0.608), once widely considered a gold standard, falls below the moderate-quality threshold due to evaluator strictness on its static-test-set vulnerability and English-only design. SafetyBench (Tier C, 0.675) achieves bilingual coverage but moderate robustness. More problematically, frontier model technical reports provide minimal safety benchmark coverage, preventing systematic comparison across systems.

Agentic harm scenarios are underexplored. AgentHarm addresses harmful behavior in agentic settings, but limited adoption in model reporting prevents assessment of whether agent-enabled models introduce novel safety vulnerabilities beyond conversational contexts.

Cross-cultural safety norms are unaddressed. Of 30 benchmarks in our BQAI assessment, only HELM achieves Q₇ $\ge 0.70$ (Q₇ = 0.75). Three additional benchmarks reach Q₇ $\in [0.50,0.70]$ (BBQ at 0.68, SafetyBench at 0.58, C-Eval at 0.53, OlympiadBench at 0.53). Most benchmarks (26/30, 87%) score Q₇ $\le 0.50$, evaluating only English-language, Western-cultural norms, leaving global deployment safety under-characterized.

Recommendation: Mandate comprehensive safety reporting (TruthfulQA, HarmBench, SafetyBench minimum) for all frontier model releases. Prioritize cross-cultural safety evaluation development, particularly for non-English and non-Western contexts.

8.6. D6: Holistic Evaluation—Methodological Fragmentation

Despite achieving a strong BQAI score (HELM: 0.769, the highest holistic-track score in our assessment), holistic evaluation suffers from fragmentation: multiple incompatible frameworks (HELM, BIG-bench, BBEH, LiveBench, SimpleBench) evaluate overlapping but non-identical capability sets with different metrics and reporting formats, preventing direct cross-benchmark comparison.

Open-source parity achieved, but under-documented. Our performance analysis demonstrates that leading open-weight models (DeepSeek-V3.2 Arena Elo 1421, Qwen 3 235B Elo 1422, Qwen 3.5 GPQA 88.4%) match or exceed many proprietary systems. However, this parity is poorly documented in systematic comparative evaluation across comprehensive benchmark suites.

Multilingual evaluation remains English-centric. Only five benchmarks in our corpus provide substantive multilingual coverage (C-Eval, CLUEWSC, GreekBarBench, MFTCXplain, MULTI), all marked as emerging track or specialized. Comprehensive frameworks like HELM lack systematic multilingual evaluation integration.

Recommendation: Establish unified holistic evaluation standards consolidating HELM-style comprehensive coverage with LiveBench-style contamination resistance. Mandate multilingual evaluation across at least five typologically diverse languages for all holistic frameworks.

8.7. Cross-Cutting: The Contamination-Reproducibility Dilemma

Our BQAI analysis reveals a fundamental tension: benchmarks achieving the highest robustness ($Q_5\ge 0.85$) through unpublished test sets or rolling updates (HLE: $Q_5=0.93$, LiveBench: $0.87$, FrontierMath: $0.85$, LiveCodeBench: $0.82$) trade off against reproducibility in different ways. FrontierMath illustrates the extreme: $Q_5=0.85$ but $Q_4=0.51$, the lowest reproducibility in the sample, because the protective opacity prevents independent re-execution.

Active contamination resistance remains rare. Only 14 of 63 benchmarks (22%) employ active anti-contamination mechanisms (rolling updates, unpublished problems, adversarial filtering, expert validation). This leaves 78% of the evaluation ecosystem vulnerable to training data leakage as model training corpora continue expanding.

Static benchmark lifetime shortening. The gap between benchmark release and frontier saturation continues shrinking: GSM8K (2021) saturated within 24 months, MMLU (2020) within 36 months. Recent benchmarks face even faster saturation as training data subsumes public test sets more rapidly.

Recommendation: Shift evaluation infrastructure toward mandatory rolling test set updates (quarterly minimum) or cryptographic commitment schemes for unpublished problems. Develop automated contamination detection tools integrated into evaluation pipelines.

8.8. Summary of Priority Research Directions

The gaps identified above suggest four urgent research priorities for robust frontier LLM evaluation:

• Contamination-resistant infrastructure: Transition from static benchmarks to dynamic evaluation platforms with monthly/quarterly test set rotation, cryptographic test set protection, or continuously generated problems from procedural generation frameworks.
• Causal and metacognitive evaluation: Develop standardized protocols for causal inference (intervention, counterfactuals) and calibration/uncertainty quantification, integrating these into comprehensive frameworks rather than isolated specialized benchmarks.
• Cross-cultural safety and fairness: Expand evaluation beyond English-speaking Western contexts to include systematic assessment across linguistic families, legal traditions, and cultural norms, with particular emphasis on safety evaluation in non-Western settings.
• Mandatory comprehensive reporting: Establish community standards requiring frontier model releases to report performance across all six taxonomy dimensions (D1–D6) with minimum benchmark coverage per dimension, ensuring systematic comparability and identifying capability gaps.

Addressing these gaps will be essential for maintaining evaluation validity as LLMs evolve toward more capable, agentic, multimodal, and globally deployed systems. The current evaluation ecosystem, while extensive, remains insufficiently robust to support the high-stakes decisions increasingly dependent on LLM capabilities.

9. Conclusions

This survey has presented a comprehensive, methodologically grounded analysis of the evaluation ecosystem for Large Language Models as of March 2026, examining 63 benchmarks organized across a six-dimensional taxonomy and assessed through 16 frontier, open-source, and longitudinal baseline models. Beyond cataloging evaluation instruments, this work argues that LLM assessment must transition from score-centered leaderboard reporting toward scientifically rigorous frameworks emphasizing reproducibility, contamination resistance, and ecological validity.

9.1. Principal Contributions

• A Unified Six-Dimensional Taxonomy.

Our framework organizes evaluation across six conceptually coherent dimensions—Reasoning and Problem-Solving (D1), Knowledge and Comprehension (D2), Generation and Creativity (D3), Interaction and Agency (D4), Alignment and Safety (D5), and Holistic Evaluation (D6)—operationalized through 20 distinct subcategories. This structure clarifies not only what current benchmarks measure but also exposes critical gaps: causal reasoning remains severely underrepresented (2/63 benchmarks), agentic capabilities suffer from sparse public reporting, and safety evaluation exhibits dangerous transparency deficits in frontier model documentation.

The taxonomy’s two-level structure—broad dimensions mapped to measurable subcategories—enables systematic analysis of evaluation coverage while preserving operational precision. Cross-cutting properties (calibration, contamination resistance, execution-based verification, ecological validity) are treated as transversal quality indicators rather than artificial dimensions, avoiding the conceptual conflation prevalent in earlier taxonomies that mixed modality, interaction format, and normative criteria.

• The Benchmark Quality Assurance Index (BQAI).

We introduced the BQAI as a quantitative instrument for assessing benchmark scientific quality independent of community adoption or model performance trends. The framework aggregates seven evidence-based dimensions (annotation quality, clarity, standardization, reproducibility, robustness, coverage, fairness) using AHP-derived weights prioritizing reproducibility ($w_4=0.349$) and contamination resistance ($w_5=0.167$).

Applying the BQAI to 30 representative benchmarks (48% of the curated corpus) under a three-evaluator blinded protocol reveals that popularity is an unreliable proxy for methodological rigor. Inter-rater reliability was strong across dimensions ($\mathrm{ICC}(2,k)=0.855$, range $0.744$–$0.943$; quadratic-weighted Cohen’s $\kappa =0.541$), and the resulting tier distribution is notably conservative: only HLE (BQAI 0.831) reaches Tier A, ten benchmarks occupy Tier B—including LiveBench (0.786), LiveCodeBench (0.772), HELM (0.769), HarmBench (0.727), and SWE-bench Verified (0.720)—and nineteen benchmarks fall into Tier C. Widely cited foundational instruments cluster in the bottom of Tier C: MMLU (0.589), GSM8K (0.587), HellaSwag (0.583), and HumanEval (0.634) all combine strong reproducibility with severe contamination vulnerabilities, validating the central thesis that scientific rigor and adoption are dissociable. Multi-regime sensitivity analysis—Monte Carlo perturbation at $\pm 10\%$, $\pm 25\%$, and $\pm 50\%$ plus four realistic priority-shifted weighting schemes—confirms that tier assignments remain robust to reasonable weight uncertainty (29/30 stable at $\pm 10\%$; Kendall’s $\tau \ge 0.82$ across realistic alternative schemes), validating the framework’s stability.

The BQAI assessment identifies a critical underdeveloped dimension: fairness and cross-cultural validity ($Q_7$), where only one of 30 benchmarks (HELM, $Q_7=0.75$) achieves a score ≥0.70. Most benchmarks (25/30, 83%) score $Q_7\le 0.50$, indicating systemic neglect of multilingual and demographic diversity in benchmark design. This gap becomes increasingly consequential as LLM deployment expands globally.

• Comprehensive Performance Analysis.

Our consolidated performance matrix synthesizes scores for 16 models across 10 key benchmarks, revealing five structural trends:

• Knowledge saturation (D2): MMLU-Pro clusters at 87–91% for frontier systems, approaching annotation-error ceilings. Only specialized expert knowledge (HLE: 10.1–45.9% range, no model >50%) maintains discriminative power.
• Reasoning bifurcation (D1): Mathematical benchmarks show extreme split—AIME 2025 approaches saturation (top score: 99.0%), MATH-500 compressed above 97%, while FrontierMath remains <50% for top systems. Scientific reasoning (GPQA Diamond) exhibits new ceiling effects (two models >94%).
• Generation convergence (D3): Code generation gaps are narrowing dramatically. Synthetic benchmarks (HumanEval) exceed 90% for most frontier models. Repository-scale engineering (SWE-bench Verified) shows four models within 1.2 percentage points (79.6–80.8%), contrasting with >50-point gaps observed 18 months earlier.
• Open-source parity (D6): Leading open-weight models match or exceed proprietary systems from previous generations. Performance gaps measured in single-digit percentage points rather than benchmark tiers.
• Data scarcity (D4, D5): Agentic and safety benchmarks are severely underreported in the frontier model documentation, preventing systematic capability assessment in these critical dimensions.

Chatbot Arena Elo compression (top five models separated by 40 points vs. >200 in 2024) indicates that capability convergence reduces human-preference evaluation discriminative power, necessitating more fine-grained assessment instruments.

• Contamination as Central Challenge.

Our analysis demonstrates that benchmark contamination has evolved from a peripheral concern to an existential threat. Only 14 of 63 benchmarks (22%) employ active resistance mechanisms (rolling updates, unpublished problems, adversarial filtering); within our 30-benchmark BQAI sample, only seven instruments achieve $Q_5\ge 0.70$ for sustained robustness. The contamination-reproducibility dilemma—unpublished test sets achieve robustness but sacrifice open evaluation—remains unresolved: only three benchmarks (HLE, LiveBench, LiveCodeBench) achieve both $Q_4\ge 0.80$ and $Q_5\ge 0.80$ simultaneously, with FrontierMath representing the extreme of the trade-off (highest $Q_5=0.85$ among public benchmarks but lowest $Q_4=0.51$ due to restricted test-set access).

9.2. Implications for Evaluation Practice

The evidence compiled in this survey supports several actionable recommendations:

• Prioritize methodologically rigorous benchmarks for primary claims. Longitudinal capability tracking and frontier model comparison should emphasize the Tier A benchmark (HLE) together with upper Tier B instruments combining high reproducibility and active contamination resistance: LiveBench, LiveCodeBench, HELM, HarmBench, and SWE-bench Verified. These six benchmarks collectively provide the strongest methodological foundation currently available for frontier evaluation.
• Mandate comprehensive dimensional reporting. Frontier model releases should report performance across all six taxonomy dimensions with minimum coverage: D1 (reasoning: GPQA + MATH-500 or FrontierMath), D2 (knowledge: HLE + MMLU-Pro), D3 (generation: SWE-bench Verified + Arena Elo), D4 (agency: GAIA or WebArena), D5 (safety: TruthfulQA + HarmBench), D6 (holistic: HELM or LiveBench).
• Transition to dynamic evaluation infrastructure. Replace static benchmarks with rolling test set platforms (LiveBench model) or procedurally generated problems. Quarterly test set rotation should become the minimum standard for knowledge-intensive and reasoning-heavy evaluation.
• Address the cross-cultural evaluation gap. Expand beyond English-centric Western evaluation to systematic multilingual and cross-cultural assessment, particularly for safety benchmarks where cultural norms vary substantially. With only HELM achieving $Q_7\ge 0.70$ in our assessment, fairness instrumentation represents the single largest methodological opportunity in current benchmark design.
• Develop intermediate-difficulty benchmarks. Fill the 70–90% accuracy range gap in mathematical and scientific reasoning, currently exhibiting binary saturation (trivial vs. intractable) without fine-grained capability profiling.

9.3. Future Research Directions

Four research priorities emerge as essential for maintaining evaluation validity:

• Contamination-Resistant Infrastructure.

Develop cryptographic commitment schemes for unpublished test sets, enabling contamination resistance without sacrificing reproducibility. Investigate procedural problem generation from formal specifications, particularly for mathematical and code generation domains. Establish community standards for contamination detection and disclosure.

• Causal and Metacognitive Evaluation.

Integrate causal reasoning protocols (intervention, counterfactuals) and calibration/uncertainty quantification into comprehensive frameworks rather than isolated specialized benchmarks. Current severe underrepresentation (2/63 benchmarks for causal reasoning) inadequately reflects the importance for scientific, policy, and medical applications.

• Agentic Capability Assessment.

Expand systematic evaluation beyond code-specific repository navigation (SWE-bench) to general-purpose tool use, multi-step planning, and long-context reasoning. The four agentic benchmarks in our BQAI sample (GAIA, AgentBench, WebArena, OmniAI OCR) all fall into Tier C with BQAI scores in the range $0.648$–$0.694$, reflecting both genuine methodological limitations and sparse public reporting that may artificially depress quality assessment. Develop benchmarks requiring information synthesis over >100 k token contexts with multi-document retrieval and reasoning under uncertainty.

• Cross-Cultural Safety and Fairness.

Move beyond English-language Western safety norms to comprehensive cross-cultural evaluation spanning linguistic families, legal traditions, and ethical frameworks. Establish demographic representation standards for benchmark construction and bias auditing protocols as quality requirements rather than optional enhancements.

9.4. Closing Perspective

Measuring machine intelligence is fundamentally a problem of scientific instrumentation. Just as physics demands increasingly sophisticated detectors as phenomena become more subtle, LLM evaluation requires progressively rigorous benchmarks as capabilities approach and exceed human expert performance. Our analysis demonstrates that the current evaluation ecosystem, while extensive, exhibits systematic vulnerabilities: saturation in foundational knowledge assessment, contamination in static test sets, sparse coverage of agentic and safety dimensions, and alarming neglect of cross-cultural validity. The conservative outcome of the BQAI assessment under multi-evaluator absolute-agreement scoring—only one benchmark in Tier A, the majority in Tier C—underscores how far the field remains from possessing a stable portfolio of methodologically sound instruments commensurate with the capabilities of frontier systems.

The BQAI framework provides a principled tool for identifying methodologically sound evaluation instruments, but quality assessment alone cannot compensate for fundamental capability gaps. The field requires not merely better benchmarks for existing dimensions, but entirely new evaluation paradigms for emergent capabilities: causal inference, long-context reasoning, cross-cultural safety, and agentic decision-making under uncertainty.

As LLMs transition from research artifacts to deployed systems mediating consequential decisions—scientific discovery, medical diagnosis, legal reasoning, policy analysis—evaluation validity becomes not merely an academic concern but a societal imperative. The gap between current evaluation capabilities and deployment realities demands urgent attention. Benchmark design must be recognized as central scientific infrastructure rather than a secondary artifact of model development.

This survey provides a structured foundation for that recognition: a taxonomy clarifying what we measure and what remains unmeasured, a quality framework distinguishing rigorous instruments from popular but flawed proxies, validated through formal inter-rater reliability under a transparent blinded protocol, and a gap analysis identifying priority research directions. All scoring data, the operational rubric, weight derivation, sensitivity analysis, and analysis code are released as a public reproducibility repository to enable independent verification and extension (https://github.com/Magnusvron/llm-benchmark-quality-index, accessed on 19 May 2026; per-cell source traceability for the performance matrix is documented in Appendix B). The next generation of LLM evaluation must build on this foundation, transitioning from static leaderboards to dynamic assessment ecosystems capable of tracking capabilities as models evolve toward more powerful, agentic, and globally consequential systems.