The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education

Abstract

Large language models (LLMs) are increasingly integrated into educational settings, yet their pedagogical reliability remains insufficiently understood. Beyond overt hallucinations, which informed users readily recognize, a subtler failure mode consists of explanations that are coherent, authoritative, and pedagogically plausible while harbouring hidden conceptual flaws, responses we term Socratic traps. This paper introduces SocraticTrap-CS, a publicly available benchmark that probes the capacity of open-weight LLMs to generate such strategic misconceptions on demand. A single structured prompt explicitly elicited three outputs per concept (a correct explanation, an overt hallucination, and a strategic misconception), yielding 735 expert-annotated response segments from seven open-weight models across 35 core concepts in algorithms and data structures, programming languages and paradigms, databases, computer networks, and operating systems. Three domain experts independently annotated each segment using a three-class schema, achieving near-perfect agreement (Fleiss’ $\kappa =0.9487$). Because models were explicitly instructed to produce the misconception, the central metric quantifies adversarial instruction-following capacity rather than the base rate of such errors in naturalistic use and should be read as a conservative upper bound on model capability. Under these conditions, compliance reached 91.7% overall (100% for three models; 57.1% for the smallest model, Mistral 7B, whose lower rate plausibly reflects weaker instruction-following rather than greater safety). Expert-judged persuasiveness was moderate to high, errors were predominantly conceptual rather than factual, models differed significantly, and no statistically significant domain-level differences were detected. The benchmark reframes the evaluation of educational LLMs around pedagogical trustworthiness rather than factual correctness alone.

1. Introduction

The rapid integration of Large Language Models (LLMs) into educational contexts has introduced a novel category of pedagogical risk that extends beyond simple factual errors. While overt hallucinations—responses that are demonstrably false or nonsensical—are easily identified and dismissed by informed users, a more insidious subset of LLM-generated content poses a disproportionate threat to learning. This content is characterized by logical coherence, an authoritative tone, and structural plausibility, yet contains subtle, domain-specific inaccuracies. We term this class of errors strategic misconceptions or Socratic traps.

A Socratic trap is defined as an LLM-generated explanation that satisfies three criteria: (1) it exhibits surface-level coherence, fluency, and an authoritative pedagogical tone; (2) it contains a domain-specific inaccuracy that is neither self-evident nor immediately detectable without domain expertise; and (3) it is structured in a manner that plausibly could reinforce or introduce a flawed mental model in a learner. This definition combines two dimensions: the nature of the error (subtle, domain-specific, and conceptually grounded rather than merely factual) and its pedagogical effect (the capacity to persuade and potentially mislead a learner).

For instance, when asked about Dijkstra’s shortest path algorithm, an LLM might confidently explain that it can handle negative edge weights as long as there are no negative cycles—a statement that sounds plausible but fundamentally misunderstands the greedy invariant upon which the algorithm’s correctness depends. In computer science (CS) education, where conceptual precision and the correct mental model of abstract systems are paramount, such misconceptions can fundamentally undermine the learning process [1].

The peril of strategic misconceptions lies in their ability to exploit the very cognitive biases that LLMs are designed to emulate. For instance, a student encountering a confident, well-articulated explanation is likely to accept it as truth, inadvertently constructing a flawed mental model of a core CS concept. This is particularly dangerous in foundational subjects like algorithms, programming paradigms, databases, and networking, where a single, seemingly minor misinterpretation can lead to cascading errors in more advanced coursework and professional practice. In other words, the problem is not that the LLM is wrong, but rather that the error is pedagogically optimized to be persuasive, thereby acting as a trap for the learner.

While recent literature has extensively surveyed the phenomenon of hallucination in natural language generation [2,3] and explored the alignment of LLMs with human feedback [4], these investigations often focus on the broad detection of factual inaccuracies or harmful content. Benchmarks for holistic LLM evaluation [5] provide valuable insights into general model capabilities but lack the granularity to detect domain-specific, conceptually coherent errors that are of greatest concern in specialized educational settings. Furthermore, existing research on LLMs in education [1,6] has largely centered on their potential as tutors, code generators, or assistants, rather than on systematically benchmarking their capacity to generate these harmful, plausible-sounding misconceptions. A critical gap remains in both the identification and automated detection of these strategic errors within the specific domain of CS education.

The technical contributions of this work are threefold. First, we introduce SocraticTrap-CS, a novel annotated benchmark dataset specifically designed for the detection and analysis of strategic misconceptions in the CS domain, comprising 735 expert-annotated response segments. Second, we propose a multidimensional evaluation framework that combines misconception frequency (SMR), error type analysis (factual vs. conceptual), and persuasiveness scoring to provide a comprehensive risk profile for LLMs in educational contexts. Third, we establish a rigorous annotation methodology with near-perfect inter-annotator agreement, demonstrating that strategic misconceptions constitute an operationalizable category of model behavior.

The capacity of LLMs to generate strategic misconceptions stems from their fundamental architecture as next-token prediction systems trained on vast corpora of human text. As Bender et al. [7] argue, LLMs are “stochastic parrots” that learn to reproduce statistical patterns in language without grounding in meaning or truth. This training objective naturally produces models that excel at generating linguistically coherent, stylistically authoritative text while having no mechanism for ensuring conceptual accuracy. The very features that make LLMs effective at language generation, fluency, coherence, and the ability to mimic expert discourse, are precisely what enable them to produce pedagogically persuasive yet conceptually flawed explanations. When these models are prompted with questions in technical domains, they draw upon patterns in their training data that may include misconceptions, errors, or flawed reasoning present in online sources, and combine these with their capacity for coherent exposition to produce what we term strategic misconceptions.

To address this gap, this paper introduces SocraticTrap-CS, the first publicly available, human-annotated benchmark corpus designed explicitly for the detection and analysis of strategic misconceptions generated by open-weight LLMs within the computer science domain. Our work makes three primary contributions. First, we present the design and construction of a novel corpus comprising 735 responses generated by seven open-weight models across 35 key concepts spanning five core CS subfields: Algorithms and Data Structures, Programming Languages and Paradigms, Databases, Computer Networks, and Operating Systems. For each concept, a single structured prompt was used to elicit three distinct responses: a correct explanation, an overt hallucination, and a strategic misconception. Each response was independently annotated by three domain experts using a novel three-class schema (Correct, Overt Hallucination, Strategic Misconception), with inter-annotator agreement measured via Fleiss’ kappa [8].

Second, we conduct a comprehensive benchmark analysis of these models, comparing their tendency to generate strategic misconceptions, the perceived plausibility of these errors, and the distribution of error types (factual, logical, conceptual) across the different CS domains. Our analysis employs rigorous statistical methods, including one-way ANOVA with Tukey HSD post-hoc correction [9] and chi-square tests, to provide a robust, quantitative profile of model risk.

The remainder of this paper is organized as follows. We first review related work on LLM hallucination and their role in education, situating our contributions within the broader research landscape. We then detail the design of the SocraticTrap-CS corpus, the annotation protocol, and our evaluation framework. Following this, we present a comparative benchmark analysis of the evaluated models and discuss the implications of our findings for computer science pedagogy and AI literacy. We conclude by outlining the limitations of our study and suggesting directions for future work.

2. Background and Related Work

Recent research on the intersection of Large Language Models (LLMs) and education has grown rapidly in recent years. The literature can broadly be grouped into two main areas: the pedagogical capabilities and limitations of LLMs and their application in demanding, conceptually precise domains such as computer science. While studies generally highlight their potential to improve access to information and support personalized learning, they also consistently point to important limitations in factual accuracy and in scaffolding effective learning processes.

2.1. LLM Hallucination: From Overt Errors to Strategic Misconceptions

The tendency of LLMs to generate plausible but factually incorrect content, commonly termed hallucination, has been recognized as a fundamental challenge to their reliable deployment, as extensively documented in recent surveys [2,3,10]. The term itself has been a subject of critique, most notably in the influential work by Bender et al., who characterize LLMs as “stochastic parrots”-systems that randomly stitch together linguistic forms from their training data without any reference to meaning [7]. This characterization underscores that LLMs are, at their core, text synthesis machines that operate solely on the formal side of language, with no grounding in communicative intent or veridicality. As such, the outputs of an LLM correspond to something true or useful only by chance, as the model is always just working with the statistical likelihood of word sequences.

The concept of a Socratic trap can be situated within existing taxonomies of hallucinations and misconceptions while representing a distinct category. Building on Huang et al.’s taxonomy [10], which distinguishes between intrinsic hallucinations (contradicting input context) and extrinsic hallucinations (inconsistent with world knowledge), Socratic traps primarily fall within the extrinsic category but are further characterized by their pedagogical nature. Unlike general hallucinations, which may be random or nonsensical, Socratic traps are deliberately constructed to mimic pedagogical discourse and exploit learners’ trust in authoritative sources. Similarly, compared to the misconception literature in education [11], Socratic traps differ from naturally occurring student misconceptions in that they are externally introduced by the AI system, yet they share the characteristic of being coherent, theory-like, and resistant to simple correction. The key distinction from existing concepts such as “misinformation” or “disinformation” lies in the absence of deliberate intent to deceive: a Socratic trap is an emergent property of LLM behavior arising from the tension between fluency optimization and factual grounding, rather than a deliberately crafted falsehood. This makes Socratic traps a unique category requiring dedicated detection and mitigation approaches.

Recent work has moved beyond treating hallucination as a monolithic phenomenon. Emerging taxonomies provide empirical evidence for at least two distinct failure modes: uncertainty-driven errors that occur at low model confidence and confidence-driven errors that manifest as detailed, self-assured fabrications that can override uncertainty cues [10]. This distinction is critical for educational contexts. While an uncertainty-driven error might be easily dismissed by a student due to the model’s expressed hesitation, a confidence-driven error, confidently presented and internally coherent, represents precisely the kind of strategic misconception that can mislead a learner. In addition, comprehensive taxonomies of hallucination further categorize these errors as intrinsic (contradicting input context) or extrinsic (inconsistent with world knowledge), and note that, given their theoretical inevitability in computable LLMs, future efforts must focus on robust detection and mitigation [10].

2.2. LLMs in Education: Promise and Peril

The integration of LLMs into educational settings has been met with both enthusiasm and caution. Studies across various disciplines report improvements in learner motivation, confidence, and access to information [6]. Surveys of faculty perceptions reveal a growing familiarity with tools like ChatGPT-3.5, but also persistent concerns about academic integrity and the changing role of the educator [12]. A significant body of work emphasizes that while LLMs can serve as valuable decision-support tools for educators, their inherent cognitive biases and lack of transparency pose substantial risks that necessitate human oversight and clear usage guidelines [1].

Within computer science education specifically, these concerns are magnified. CS is a discipline characterized by a strict hierarchical knowledge structure, where mastery of foundational concepts is a prerequisite for advanced topics. An error in a student’s mental model at an early stage can have cascading, long-term consequences. This is well-documented in the CS education literature, where a rich tradition of research has systematically cataloged persistent student misconceptions in areas ranging from programming fundamentals [13] to algorithm visualization [14]. Early explorations revealed that models like Codex could already solve a significant portion of introductory programming assignments, sparking intense debate about their pedagogical impact [15]. Recent research on CS student attitudes found that while experienced students have rapidly integrated LLMs into their learning workflows, introductory-sequence students remain more cautious, citing concerns about the reliability of LLM responses and the impact on their own learning process [16].

Moreover, recent endeavors to benchmark LLM performance in CS education provide a more nuanced picture. Specifically, studies evaluating LLMs for answering student questions in introductory programming courses, found that models frequently generate complete code solutions rather than guiding students towards discovering the solution themselves. This presents a pattern that can undermine the development of core problem-solving skills [16]. Similarly, benchmarks based on multimodal question-answer data from authentic course forums show that leading LLMs can approach human-level performance in terms of accuracy and clarity. However, they consistently underperform in areas such as personalization and interactive scaffolding, often producing fluent but pedagogically suboptimal or overly direct responses. These findings resonate with the central premise of our work: an LLM response can be technically correct in its final answer while being pedagogically counterproductive or, worse, conceptually misleading in its explanation.

2.3. Bridging the Gap: The Need for SocraticTrap-CS

While existing benchmarks and surveys have illuminated the broad capabilities and limitations of LLMs, a critical gap remains: the systematic analysis of strategic misconceptions within CS education. Previous work has generally focused on detecting overt hallucinations or evaluating general response quality [2,3], while our work addresses the specific, and arguably more insidious case, where an LLM generates a response that is confident, coherent, and structurally sound, yet contains a subtle, domain-specific error that can entrap a learner. As the stochastic parrot metaphor implies, these models are adept at mimicking the form of expert knowledge without capturing its underlying meaning [7]. This makes them particularly effective generators of Socratic traps. This work is, therefore, informed by long-standing theories of misconceptions from the learning sciences, notably Chi’s framework of false beliefs, flawed mental models, and incorrect reasoning processes [11], which we use as a lens to categorize the errors produced by LLMs.

The need for a dedicated benchmark is further motivated by recent advances in LLM reasoning capabilities. The emergence of foundation models represents a paradigm shift in artificial intelligence, with models trained on broad data at scale and adapted to a wide range of downstream tasks [17]. Within this new paradigm, techniques such as chain-of-thought prompting have been shown to elicit step-by-step reasoning from LLMs, dramatically improving their performance on complex multi-step tasks [18]. Subsequent work demonstrated that LLMs can engage in zero-shot reasoning without explicit examples, simply by being prompted with the phrase “Let’s think step by step” [19]. Further improvements have been achieved through self-consistency methods, which sample multiple reasoning paths and select the most consistent answer, enhancing reasoning reliability [20]. These advances culminated in the release of models such as GPT-4, which set new standards for reasoning, instruction-following, and overall capability [21].

However, these advances in reasoning capability present a double-edged sword for education. The very mechanisms that enable chain-of-thought reasoning, zero-shot inference, and self-consistent answer selection also make LLMs more capable of generating coherent, step-by-step explanations that are internally consistent yet conceptually flawed. A model that can reason through multiple steps can also rationalize an incorrect answer in a way that sounds plausible. A model that can generate diverse reasoning paths for self-consistency can also produce multiple convincing but flawed explanations for the same concept. As LLMs become more sophisticated reasoners, the risk of strategic misconceptions may increase rather than decrease. What was once an obvious hallucination may now become a polished, authoritative, and internally coherent explanation that contains subtle conceptual errors, precisely the kind of Socratic trap that our benchmark is designed to detect.

To address this gap, we introduce SocraticTrap-CS, a benchmark specifically designed to elicit, identify, and analyze these strategic misconceptions across a broad spectrum of core CS concepts. Our work contributes a novel annotated corpus and a rigorous methodology for quantifying model propensity for this specific, high-impact class of error. By moving beyond binary notions of right and wrong, and focusing on the pedagogical persuasiveness of an error, our study provides a more relevant assessment of LLM risks in educational practice and establishes a foundation for developing automated detection tools to safeguard learning.

3. Methodology

The methodology of this study is structured into three sequential phases: corpus design and response generation, expert annotation and ground truth establishment and benchmark analysis. Each phase is described in detail in the following subsections.

The proposed SocraticTrap-CS framework consists of four main components operating in sequence: (1) Concept Selection Module: A systematic procedure for identifying core CS concepts with known student misconceptions and verifiable ground truth, using the ACM/IEEE CS2023 curriculum guidelines as a foundation. (2) Prompt Engineering Module: A structured prompt template that elicits three response types (correct, overt hallucination, strategic misconception) while maintaining consistent formatting across all model interactions. (3) Generation and Collection Module: Deployment of seven open-weight LLMs via a local Ollama-based infrastructure, with programmatic parsing of model outputs into individual segments. (4) Evaluation and Annotation Module: A rigorous three-phase annotation process involving calibration, independent annotation by three domain experts, and consensus-based resolution of disagreements, followed by statistical analysis. The framework’s novelty lies in its explicit targeting of strategic misconceptions through structured prompting, its multidimensional evaluation metrics (SMR, persuasiveness, and error type analysis), and its focus on pedagogical trustworthiness rather than factual correctness alone.

Figure 1 provides a high-level overview of the SocraticTrap-CS benchmark construction and evaluation pipeline. The process begins with the selection of 35 core computer science concepts spanning five domains: Algorithms and Data Structures, Programming Languages and Paradigms, Databases, Computer Networks, and Operating Systems. For each concept, a structured prompt is generated, instructing the model to produce three distinct response types: a correct explanation, an overt hallucination, and a strategic misconception. These prompts are submitted to seven open-weight large language models, yielding a total of 735 individual response segments. Each segment is then independently annotated by three domain experts according to the three-class schema (Correct, Overt Hallucination, Strategic Misconception). Finally, statistical analysis is performed on the annotated corpus to benchmark model performance, including one-way ANOVA with Tukey HSD post-hoc correction and chi-square tests for error type distributions.

Figure 1. Overview of the SocraticTrap-CS benchmark construction and evaluation pipeline.

3.1. Research Questions

To guide the empirical investigation, this study addresses the following research questions:

RQ1:

To what extent do contemporary open-weight LLMs generate strategic misconceptions when explicitly prompted to do so?

RQ2:

Are there statistically significant differences between models regarding their strategic misconception rates and persuasiveness?

RQ3:

Are strategic misconceptions distributed uniformly across major computer science domains?

RQ4:

Which error types (factual, logical, or conceptual) dominate within strategically misleading explanations?

RQ5:

What implications do these findings have for the safe integration of LLMs into computer science education?

3.2. Corpus Design and Response Generation

The first phase involved the construction of the SocraticTrap-CS corpus, encompassing the selection of CS domains and concepts, the choice of language models, and the design of structured prompts for response generation.

3.2.1. Domain and Concept Selection

Five core subfields of computer science were selected to ensure broad coverage of the undergraduate CS curriculum, following the ACM/IEEE Computer Science Curricula 2023 guidelines [22]. The selected domains are:

• Algorithms and Data Structures
• Programming Languages and Paradigms
• Databases
• Computer Networks
• Operating Systems

For each domain, seven key concepts were identified through a two-stage selection procedure. An initial candidate list was generated based on their frequency of appearance in undergraduate syllabi from three European universities. This list was subsequently filtered according to two criteria: (a) the existence of documented student misconceptions in the CS education literature [13], and (b) the availability of authoritative, verifiable definitions in peer-reviewed sources. This procedure yielded 35 concepts with clearly defined ground truth boundaries, a necessary prerequisite for reliable expert annotation.

The complete set of 35 concepts is presented in Table 1. For each concept, we verified that authoritative definitions existed in standard CS textbooks or peer-reviewed sources to establish ground truth boundaries for annotation.

Table 1. Complete list of 35 computer science concepts used in the SocraticTrap-CS benchmark, organized by domain.

Domain	Concepts
Algorithms and Data Structures	Dijkstra’s Algorithm, Quicksort, Dynamic Programming,
	Binary Search Trees, Hash Tables,
	Graph Traversal (BFS/DFS), Greedy Algorithms
Programming Languages and Paradigms	Lexical Scope & Closures, Polymorphism,
	Type Inference, Inheritance vs. Composition,
	Functional Programming, Concurrency, Lambda Calculus
Databases	ACID Properties, Normalization (3NF/BCNF),
	Joins (Inner/Outer/Cross), Indexing (B-Tree vs. Hash),
	Transactions & Isolation Levels, SQL Semantics, NoSQL (CAP Theorem)
Computer Networks	TCP vs. UDP, IP Addressing (CIDR/Subnetting),
	HTTP/HTTPS (Request-Response), DNS Resolution,
	Routing Protocols (OSPF/BGP), OSI Model Layers, Congestion Control
Operating Systems	Process Scheduling (SJF/RR), Virtual Memory,
	Deadlock (Conditions/Prevention), File Systems (Inodes/Journaling),
	Semaphores vs. Mutexes, System Calls, Interrupt Handling

3.2.2. Model Selection

The selection of appropriate language models was a critical step in ensuring the validity and generalizability of the benchmark. We selected open-weight models exclusively for three reasons. First, local deployment via Ollama on the Supek supercomputer ensures full reproducibility of our experiments, as all models and generation parameters are fixed and can be rerun by other researchers. Second, open-weight models allow complete control over the generation process without API constraints or cost limitations, enabling exhaustive testing across 35 concepts with consistent parameters. Third, and most critically, while proprietary models such as GPT-4o, Claude, and Gemini represent state-of-the-art capabilities, their closed-source nature, evolving behavior through continuous updates, and API-based access make them unsuitable for the reproducible, controlled benchmarking that is the hallmark of our study. We acknowledge that evaluating these proprietary models would provide additional comparative insights; this constitutes a promising direction for future work.

Seven open-source large language models were selected for local deployment, representing meaningful variation across model families, parameter scales, and architectural approaches, including both dense transformer models and sparse mixture-of-experts (MoE) architectures. All models were deployed locally via Ollama [23] on the Supek supercomputer, the most powerful HPC system in Croatia, operated by the University of Zagreb University Computing Centre (Srce) in partnership with CARNET and other institutions under the Croatian Scientific and Educational Cloud (HR-ZOO) project [24]. Local deployment was chosen to eliminate API costs, ensure full reproducibility, and maintain complete control over the generation process.

The selected models span a broad range of parameter counts, from 7 billion to 235 billion parameters, and originate from five distinct research organizations. Table 2 provides a summary of the evaluated models.

Table 2. Overview of the seven open-weight large language models evaluated in this study.

Model	Organization	Parameters	Architecture
Mistral 7B [25]	Mistral AI	7B	Dense transformer
Ministral 8B [26]	Mistral AI	8B	Dense transformer
Gemma 2 27B [27]	Google DeepMind	27B	Dense transformer
LLaMA 3.3 70B [28]	Meta AI	70B	Dense transformer
DeepSeek-R1 70B [29]	DeepSeek AI	70B	Dense transformer
LLaMA 4 Scout [30]	Meta AI	16 × 17B	Sparse MoE
Qwen3 235B [31]	Alibaba DAMO	235B	Sparse MoE

3.2.3. Prompt Design and Response Generation

A key methodological innovation of this study was the use of a single structured prompt to elicit three distinct response types from each model for each concept. This approach was designed to maximize the efficiency of the generation process while ensuring consistent formatting across all responses, and to minimize prompt-ordering effects that might arise from generating each response type in a separate model interaction.

The prompt template was constructed through an iterative design process. Initial versions were tested on a small subset of concepts ($n=5$) with the Mistral 7B model to evaluate whether the three response types could be reliably elicited. The prompt structure was refined based on the clarity and completeness of the generated responses, ensuring that each response type was sufficiently distinct and that the strategic misconception instruction was understood as requiring a subtle, domain-specific error. The final template was then validated on a held-out set of concepts ($n=3$) by having the authors independently assess whether the generated responses conformed to the intended categories. Prompt templates were not varied between models to ensure comparability, as our goal was to assess intrinsic model behavior under identical conditions.

For each of the 35 concepts, a single structured prompt was submitted to each of the seven models, instructing the model to generate three clearly demarcated responses: a correct explanation, an overt hallucination, and a strategic misconception. Figure 2 visualizes the structure of this prompt.

Figure 2. Prompt template for generating three explanation types per concept.

As illustrated in Figure 2, the prompt template explicitly instructs the model to generate three distinct outputs. The left branch produces a correct explanation, which must be technically accurate and pedagogically appropriate for a first-year CS student. This output serves as the ground truth baseline for evaluating the other two response types. The center branch generates an obvious hallucination, an answer that contains a fundamental, easily detectable error. This type of error is designed to be clearly incorrect, allowing the annotation process to distinguish between trivial mistakes and more subtle misconceptions. The right branch produces the critical output of this study: a strategic misconception. Unlike the obvious hallucination, this output is professionally formulated, internally coherent, and pedagogically plausible, yet contains a subtle, domain-specific inaccuracy. The key distinction lies in the subtlety of the error: while an obvious hallucination would likely be recognized as incorrect by a knowledgeable student, a strategic misconception is designed to be persuasive and potentially misleading, thereby acting as a Socratic trap for the learner.

The model is required to format its response using the delimiters [CORRECT], [OBVIOUS], and [STRATEGIC], which enables reliable programmatic parsing of the generated text. This structured formatting ensures that each of the three explanation types can be extracted automatically before being presented to human annotators.

Each prompt was submitted to all seven models, yielding 245 structured responses (35 concepts × 7 models). Each response was then programmatically parsed into three separate segments using a custom Python 3.13.1 script, yielding a total of 735 individual text segments for expert annotation (245 responses × 3 segments). All generation parameters were kept at model defaults to ensure comparability, and no system prompts, few-shot examples, or chain-of-thought instructions were used, so that each model’s raw capacity for generating the three response types could be assessed in a controlled setting.

One model, DeepSeek-R1, exhibited a specific parsing issue due to its unique output format, which wraps its internal reasoning process within <think> tags. In four instances, these tags interfered with the programmatic segmentation of the response, resulting in unrecoverable text segments that could not be presented to annotators. This resulted in a total of 31 instead of 35 annotation triplets for DeepSeek-R1, a documented limitation of our parsing script’s inability to handle this specific, model-intrinsic output structure.

3.3. Expert Annotation Protocol

The second phase involved the design and implementation of a rigorous annotation protocol to ensure the reliability and validity of the ground truth labels assigned to each generated text segment.

3.3.1. Annotation Schema

To enable systematic classification of model outputs, a three-class annotation schema was developed through an iterative process involving all three annotators. Each text segment was independently annotated according to the following schema:

• Class 0—Correct: Factually accurate, appropriate for a student audience, and free of conceptual errors.
• Class 1—Overt Hallucination: Clearly incorrect or nonsensical; easily identifiable as wrong by a knowledgeable student.
• Class 2—Strategic Misconception: Persuasive, professionally formulated, and internally coherent, but containing a subtle, domain-specific inaccuracy.

Note that while the prompt explicitly instructed models to generate one response of each type, annotators evaluated the actual content of each segment independently, without knowledge of the intended class. A model may therefore produce a response that was prompted as a strategic misconception but that the annotators classify differently, for instance, as an overt hallucination if the error is too obvious, or as correct if the intended misconception was not successfully introduced. This distinction between prompted intent and annotated class is a methodological strength of our approach, as it allows the benchmark to capture the genuine distributional properties of model outputs rather than assuming perfect adherence to the prompt.

For all Class 2 annotations, experts additionally recorded two supplementary dimensions. The first was the error type-factual, logical, or conceptual, following established classifications of misconceptions in the CS education literature, which distinguish between errors grounded in incorrect factual knowledge, flawed reasoning processes, and fundamental misunderstandings of underlying principles [11,13,14].

Factual errors represent incorrect statements about objective, verifiable facts, such as misstating definitions, algorithmic complexities, or protocol properties that can be directly checked against authoritative sources [32]. Factual errors are typically the most straightforward to detect, yet when embedded in otherwise coherent explanations, they can still mislead students who lack the confidence or expertise to independently verify the claims.

Logical errors involve flawed reasoning or invalid inferences drawn from otherwise correct premises [33,34]. Individual statements within the explanation may be factually correct, but the conclusion derived from them is logically invalid. For instance, a model might correctly state that Quicksort has an average-case time complexity of $\mathcal{O}\left(n\log n\right)$ and then conclude that it is therefore suitable for real-time systems. This inference is flawed because real-time systems require guaranteed worst-case performance bounds rather than favorable average-case behavior. Logical errors are particularly dangerous because they often appear as plausible reasoning chains that are difficult to identify without well-developed critical thinking skills.

Conceptual errors represent fundamental misunderstandings of the underlying principles, mechanisms, or relationships governing a domain [35,36,37]. Unlike factual errors, which can often be corrected by simply providing accurate information, conceptual errors reflect deeper inconsistencies in the internal mental model. For example, a model might claim that Dijkstra’s algorithm can handle negative edge weights by first applying the Bellman-Ford algorithm to detect and remove negative cycles, and then proceeding with Dijkstra’s algorithm on the remaining graph. While the explanation may sound technically sophisticated, it reveals a fundamental conceptual misunderstanding: if Bellman-Ford is already being applied, it can solve the single-source shortest path problem directly, rendering the subsequent use of Dijkstra’s algorithm unnecessary and conceptually incoherent. Conceptual errors are arguably the most pedagogically harmful form of strategic misconception because they can persist even when students possess correct factual knowledge and may interfere with the acquisition of more advanced concepts that build upon the misunderstood foundation.

This three-part taxonomy maps directly onto established frameworks in the science of learning. Chi distinguishes between false beliefs (factual), flawed mental models (conceptual), and incorrect reasoning processes (logical), while the CS education literature has repeatedly demonstrated that these distinct error categories require different pedagogical interventions to remediate [11,13]. By annotating each strategic misconception with its error type, the SocraticTrap-CS benchmark provides not only a measure of model risk but also diagnostic information about the nature of that risk, enabling educators to design targeted counter-strategies for each category.

The second supplementary dimension was a persuasiveness rating on a five-point Likert scale (1 = low persuasiveness, 5 = high persuasiveness), adapted from prior work on misinformation credibility [38]. This rating captures the degree to which a strategic misconception could plausibly deceive a student encountering it in an educational context. Importantly, these ratings should be interpreted as expert judgments of potential student susceptibility rather than direct empirical measurements of student deception, belief change, or learning outcomes. The persuasiveness scale operationalizes the core concern of this work: not all Class 2 responses carry the same pedagogical risk, and a model that systematically generates highly persuasive misconceptions represents a qualitatively greater threat than one whose misconceptions are easily dismissed.

3.3.2. Annotator Selection and Calibration

All three annotators are co-authors of this study, each holding a faculty position at the Faculty of Informatics, Juraj Dobrila University of Pula, with demonstrated expertise spanning at least three of the five target domains and a minimum of five years of professional experience in computer science research and teaching. The use of authors as annotators is a recognized practice in specialized annotation tasks where recruiting external expert annotators with the required depth of domain knowledge across multiple CS subfields is logistically infeasible and financially prohibitive [39]. The task demanded not just general NLP annotation skills but deep, active pedagogical and technical expertise to reliably distinguish a subtle conceptual error from a correct explanation, a distinction that would be challenging for non-specialist annotators.

To mitigate potential bias arising from this design, two methodological safeguards were implemented. First, annotation was conducted blind: annotators were presented with text segments only, with all metadata, including the generating model, the concept name, and the intended class from the prompt, removed from the annotation interface. Second, the annotation schema was developed and refined through a calibration phase prior to the main annotation task, as described below.

The three-class schema was finalized through a calibration procedure in which all three annotators independently annotated a shared set of 30 pilot responses drawn from a held-out subset of the corpus (not included in the final benchmark). Following independent annotation, annotators convened to discuss disagreements and align category boundaries, particularly the distinction between Class 1 and Class 2 responses, which represents the most theoretically and practically important boundary in the schema. Pilot responses on which consensus could not be reached were excluded from the final corpus. This calibration procedure is standard practice in high-stakes annotation tasks and has been shown to substantially reduce systematic disagreement arising from divergent interpretations of class definitions [39].

We note that the exceptionally high agreement achieved (see Section 4) is a strong indicator of a clear and robust annotation schema. However, we also acknowledge that when all annotators share the same institutional and research context, a very high Kappa can, paradoxically, also reflect shared interpretive biases rather than purely the quality of the schema itself. We therefore encourage future replication studies with independent annotator pools to further validate the objectivity of the class boundaries.

3.4. Benchmark Analysis

The third phase involved a comprehensive analysis of the annotated corpus to compare model performance, identify domain-specific patterns, and establish the statistical significance of observed differences.

It is important to clarify the object of evaluation in this phase. SocraticTrap-CS does not propose a new model to be benchmarked against competing methods; rather, it is a benchmark framework for evaluating existing, off-the-shelf LLMs on a task they were not originally designed to perform. Consequently, the notion of a “compared method” from conventional machine learning benchmarking does not directly apply. Instead, evaluation proceeds through three complementary, purpose-built metrics, SMR, mean persuasiveness, and error type distribution, each targeting a distinct facet of pedagogical risk, together with the statistical tests described below that establish whether observed differences between models and domains are reliable. This multidimensional design is deliberately broader than conventional accuracy-based or hallucination-detection benchmarks, which typically reduce model quality to a single correctness score; here, a model can simultaneously be evaluated on how often it produces a strategic misconception, how convincing that misconception is, and what kind of conceptual or factual error it reflects.

3.4.1. Model-Level Metrics

To characterize the performance and risk profile of each model, three complementary metrics were defined and computed:

• Strategic misconception rate (SMR): The proportion of responses annotated as Class 2 out of all responses generated by that model ($\mathrm{SMR}=N_{\mathrm{Class} 2}/N_{\mathrm{total}}$). This metric directly quantifies the frequency with which a model produces outputs that were annotated as strategic misconceptions.
• Mean persuasiveness ($\overline{P}$): The average Likert score across all Class 2 responses for a given model. This metric captures the average severity of each model’s strategic misconceptions, complementing the frequency measure provided by SMR.
• Error type distribution: The proportion of factual, logical, and conceptual errors within Class 2 responses for each model. This distribution reveals the structural nature of each model’s error profile and informs targeted pedagogical countermeasures.

Together, these three metrics form a multidimensional profile of model behaviour: SMR quantifies how often a model produces strategic misconceptions; $\overline{P}$ quantifies the average perceived persuasiveness of those misconceptions; and the error type distribution characterizes the nature of the underlying misconception.

3.4.2. Domain-Level Analysis

Beyond model-level comparisons, a domain-level analysis was conducted to examine how strategic misconceptions vary across the five CS subfields. Cross-tabulation of model × domain × error type was performed to identify which models generate the highest rate of strategic misconceptions within each subfield and which error types dominate in each domain. This analysis is motivated by the hypothesis, grounded in the hallucination literature, that LLM error rates are not uniformly distributed across knowledge domains but are shaped by the frequency and consistency of domain-specific content in pretraining corpora [2]. Identifying domain-level variation therefore serves both a scientific purpose, testing this hypothesis in a novel context, and a practical one, enabling educators to prioritize AI literacy interventions in areas of highest risk.

3.4.3. Statistical Methods

A rigorous statistical framework was employed to ensure that observed differences between models and domains were not attributable to chance. One-way analysis of variance (ANOVA) was used as the primary method for comparing strategic misconception rates across models, with the significance threshold set at $\alpha =0.05$. The assumptions of approximate normality and homogeneity of variance were verified using the Shapiro-Wilk and Levene tests, respectively. Where ANOVA yielded a significant omnibus result, Tukey’s Honestly Significant Difference (HSD) post-hoc test was applied to identify pairwise differences between models. Tukey’s HSD was selected over alternatives such as Bonferroni correction because it controls the family-wise error rate while maintaining greater statistical power when the number of comparisons is moderate [9]. Chi-square tests were used to evaluate whether error type distributions differed significantly across models and domains. All statistical analyses were conducted in Python using the SciPy and Statsmodels libraries.

4. Results

The annotation of 722 complete triplets (DeepSeek-R1 contributing 31 instead of 35) yielded a complete labeled dataset for benchmarking. This section presents the inter-annotator agreement analysis, descriptive statistics, comparative model evaluation, error type distributions, and persuasiveness analysis. Throughout this section, we report effect sizes alongside significance tests to provide a complete picture of both statistical and practical significance.

4.1. Inter-Annotator Agreement

Before examining model performance, it is essential to establish the reliability of the ground truth labels against which model outputs are evaluated. Inter-annotator agreement (IAA) was assessed using Fleiss’ $\kappa$ [8] on the full set of 722 complete annotation triplets. The choice of Fleiss’ $\kappa$ over alternative agreement measures was motivated by its suitability for designs involving three or more raters classifying items into nominal categories, as it accounts for chance agreement among all pairs of raters simultaneously. An a priori threshold of $\kappa >0.75$ was established, corresponding to excellent agreement on the Landis-Koch scale [40] and consistent with standards adopted in comparable NLP annotation studies.

The overall agreement reached $\kappa =0.9487$ ($SE=0.0101$, $z=93.55$, $p<0.001$), corresponding to almost perfect agreement and substantially exceeding the established a priori threshold. The observed proportion of agreement ($\overline{P}=0.9658$) was substantially higher than the expected chance agreement (${\overline{P}}_e=0.3334$), confirming that the three-class schema can be applied with high reliability by domain experts. This near-perfect agreement is particularly noteworthy given the inherent difficulty of the annotation task: distinguishing between an overt hallucination (Class 1) and a strategic misconception (Class 2) requires nuanced judgment about the subtlety and persuasiveness of the error, a distinction that might reasonably be expected to produce substantial annotator disagreement.

Figure 3 visualizes the distribution of annotation agreement patterns across all 722 triplets. The donut chart makes immediately apparent the overwhelming dominance of full agreement: in 96.6% of cases, all three annotators independently assigned the same class label. The dominant blue segment represents this high-consensus scenario, while the narrow light blue sliver (3.4%) captures the small fraction of cases where two annotators agreed and one differed. No instances of complete disagreement, where all three annotators assigned different labels, were observed in the entire corpus.

Figure 3. Distribution of inter-annotator agreement patterns across all 722 annotation triplets. The dominant segment (96.6%) represents cases of full agreement; the minority segment (3.4%) represents partial agreement. No instances of complete disagreement were observed.

Pairwise Cohen’s $\kappa$ values further corroborated the consistency of the annotation process. Agreement between Annotator 1 and Annotator 2 was perfect ($\kappa =1.000$, 100% agreement), while Annotator 1 vs. Annotator 3 and Annotator 2 vs. Annotator 3 both yielded $\kappa =0.9231$ (94.88% agreement). The mean pairwise $\kappa$ of 0.9487 indicates strong consensus across all annotator pairs. The slightly lower agreement involving Annotator 3 reflects a small number of edge cases at the Class 1/Class 2 boundary. These disagreements were resolved through a structured consensus discussion following the main annotation phase, yielding the final ground truth labels used in all subsequent analyses. Taken together, these reliability metrics provide strong evidence for the trustworthiness of the ground truth labels.

4.2. Descriptive Statistics

Of the 722 annotated segments, 241 were classification-relevant, specifically, item 3 in each triplet, corresponding to the prompted strategic misconception response. The remaining segments served as context for the annotators but are not the primary focus of the benchmark analysis. The majority-vote class distribution across these 241 classification-relevant segments is presented in Table 3.

Table 3. Distribution of majority-vote annotation labels across 241 classification-relevant segments.

Class	N	Percentage (%)
Class 0 (Correct)	18	7.5
Class 1 (Overt)	2	0.8
Class 2 (Strategic)	221	91.7
Total	241	100.0

The overwhelming majority of prompted strategic misconception responses, 221 out of 241, or 91.7%, were confirmed as Class 2 by the annotators. This high confirmation rate demonstrates that the structured prompt design is effective at eliciting the intended response types across a diverse set of models and concepts. However, the fact that 20 responses (8.3%) were reclassified as either correct or overt hallucinations is methodologically important: it confirms that even under explicit instruction, models do not always produce the intended output, and that expert annotation remains necessary to validate the actual content of generated responses.

The 18 responses reclassified as correct (7.5%) represent cases where the model, despite being prompted to generate a strategic misconception, produced an explanation that the annotators judged to be factually accurate and pedagogically appropriate. The two responses reclassified as overt hallucinations (0.8%) represent the opposite failure mode. Both types of reclassification highlight the value of distinguishing between prompted intent and annotated class.

4.3. Strategic Misconception Rates by Model

The strategic misconception rate (SMR), defined as the proportion of prompted strategic misconception responses confirmed as Class 2 by majority vote, varied considerably across the seven evaluated models. This metric is the primary outcome of the benchmark, as it directly quantifies the frequency with which each model produces responses annotated as strategic misconceptions. Table 4 presents the SMR for each model, sorted in descending order to facilitate comparison.

Table 4. Strategic misconception rates (SMR) by model, sorted in descending order. $N_{\mathrm{Class} 2}$ indicates the number of responses confirmed as strategic misconceptions; $N_{\mathrm{total}}$ indicates the total number of prompted strategic misconception responses.

Model	SMR (%)	NClass 2	${\mathit{N}}_{\mathbf{total}}$
Gemma 2 31B	100.0	35	35
Ministral 8B	100.0	35	35
Qwen3 235B	100.0	35	35
Llama 3.3 70B	97.1	34	35
DeepSeek-R1 70B	96.8	30	31
Llama 4 Scout	91.4	32	35
Mistral 7B	57.1	20	35
Overall	91.7	221	241

For DeepSeek-R1, $N_{\mathrm{total}}=31$ because 4 responses could not be parsed due to the model’s unique output format. This is documented in the Section 3.

The most striking pattern is the sharp bifurcation between Mistral 7B and all other models. Three models, Gemma 2 31B, Ministral 8B, and Qwen3 235B, achieved a perfect SMR of 100%, meaning that every single prompted strategic misconception response they generated was confirmed as Class 2 by the annotators. Two further models, Llama 3.3 70B (97.1%) and DeepSeek-R1 70B (96.8%), were within four percentage points of this ceiling. Even Llama 4 Scout, the second-lowest model, maintained an SMR above 90%. Mistral 7B, by contrast, achieved an SMR of only 57.1% ($n=20$ out of 35), with the remaining 15 responses (42.9%) either failing to introduce the intended error or producing errors too obvious to qualify as strategic.

Figure 4 provides a visual comparison of SMR across models. The bar chart makes immediately apparent the outlier status of Mistral 7B, shown in red, which stands in marked contrast to the uniformly high SMR of the other six models, shown in blue. Values above each bar indicate the exact SMR percentage, allowing precise comparison without consulting the table.

Figure 4. Strategic misconception rates by model. Mistral 7B is the sole model significantly below the near-ceiling performance of the remaining six models. The ANOVA result ($F(6,234)=14.97$, $p<0.001$) is driven entirely by this single outlier.

A one-way ANOVA confirmed that the visual pattern in Figure 4 reflects statistically significant differences across models ($F(6,234)=14.97$, $p<0.001$, ${\eta }^2=0.277$). The effect size, ${\eta }^2=0.277$, is classified as large according to conventional benchmarks, indicating that model identity accounts for approximately 27.7% of the total variance in SMR. However, this large effect must be interpreted with caution, as Levene’s test indicated violation of the homogeneity of variance assumption ($W=14.97$, $p<0.001$). This violation is entirely attributable to the three models with perfect SMR (100%), which by definition have zero variance. The non-parametric Kruskal-Wallis test, which does not assume equal variances, confirmed the same overall pattern ($H=66.57$, $p<0.001$), providing convergent evidence that the observed differences are robust.

To identify the specific source of the significant omnibus effect, Tukey HSD post-hoc comparisons were conducted. The results, presented in Table 5, confirm that Mistral 7B is the sole driver of the significant ANOVA result. Mistral 7B exhibited a significantly lower SMR than all six other models (all $p<0.001$), while no other model pair differed significantly from one another (all $p>0.74$). In other words, from a statistical standpoint, the seven models can be partitioned into exactly two homogeneous subsets: Mistral 7B alone in one subset, and all six other models together in the other.

Table 5. Tukey HSD post-hoc comparisons for strategic misconception rates. Only pairs involving Mistral 7B reached statistical significance at $\alpha =0.05$.

Pairwise Comparison	Difference	q	p
Mistral 7B vs. Gemma 2 31B	0.429	10.65	<0.001
Mistral 7B vs. Ministral 8B	0.429	10.65	<0.001
Mistral 7B vs. Qwen3 235B	0.429	10.65	<0.001
Mistral 7B vs. Llama 3.3 70B	0.400	9.94	<0.001
Mistral 7B vs. DeepSeek-R1 70B	0.396	9.55	<0.001
Mistral 7B vs. Llama 4 Scout	0.343	8.52	<0.001
All other pairwise comparisons: $p>0.05$.

To quantify the magnitude of these pairwise differences beyond statistical significance, Cohen’s d effect sizes were computed. Comparisons involving Mistral 7B yielded d values ranging from 0.84 (vs. Llama 4 Scout) to 1.21 (vs. Gemma 2 31B, Ministral 8B, and Qwen3 235B), all representing large effects according to conventional thresholds ($d>0.8$). These effect sizes indicate that the difference between Mistral 7B and the other models is not merely statistically detectable but substantively meaningful: the typical Mistral 7B response is more than a full standard deviation below the mean SMR of the other models.

4.4. Strategic Misconception Rates by Domain

Having established substantial variation across models, we next examined whether SMR varies systematically across the five CS subfields. This analysis addresses the question of whether certain domains are inherently more susceptible to strategic misconceptions than others, a question with direct implications for curriculum design and AI literacy interventions.

Domain-level SMR values are presented in Table 6. A one-way ANOVA revealed no significant differences across domains ($F(4,236)=0.68$, $p=0.608$, ${\eta }^2=0.011$). The effect size, ${\eta }^2=0.011$, is classified as small, indicating that domain identity accounts for only about 1.1% of the total variance in SMR. Levene’s test confirmed homogeneity of variance across domains ($W=0.68$, $p=0.608$), satisfying the assumptions of the ANOVA.

Table 6. Strategic misconception rates by CS domain. SD = standard deviation; N = number of prompted strategic misconception responses aggregated across all models.

Domain	SMR (%)	SD	N
Algorithms (ALG)	95.8	19.98	48
Computer Networks (NET)	93.8	24.21	48
Operating Systems (OS)	91.8	27.38	49
Programming Languages (PLP)	89.6	30.55	48
Databases (DB)	87.5	33.07	48

We acknowledge that the conclusion of no significant domain-level differences must be interpreted with caution. The small number of concepts per domain (n = 7) means that this analysis has limited statistical power to detect true domain-level effects, particularly if such effects are modest in magnitude. Future work with larger concept sets per domain would be needed to draw stronger conclusions about domain-specific susceptibility to strategic misconceptions.

Figure 5 visualizes these domain-level SMR values as a horizontal bar chart, sorted from lowest to highest SMR. The narrow range of values, from 87.5% (Databases) to 95.8% (Algorithms), is substantively informative. It suggests that no CS subfield appears less susceptible to the generation of strategic misconceptions under the prompting conditions used in this benchmark. Whether the domain involves formal mathematical reasoning (Algorithms), abstract language semantics (Programming Languages), protocol specifications (Computer Networks), system-level abstractions (Operating Systems), or data modeling (Databases), models consistently generate persuasive but flawed explanations at very high rates. Even the lowest observed SMR of 87.5% indicates that strategic misconceptions were generated at high rates across all evaluated domains: nearly nine out of every ten LLM-generated explanations in the Databases domain were judged to contain subtle but consequential inaccuracies.

Figure 5. Strategic misconception rates by CS domain. The narrow range (87.5–95.8%) and non-significant ANOVA ($F(4,236)=0.68$, $p=0.608$) indicate that strategic misconception risk is uniformly high across all five CS subfields.

4.5. Two-Way Interaction: Model × Domain

A two-way ANOVA with model and domain as factors was conducted to examine whether the pattern of model differences in SMR was consistent across CS subfields. This analysis is important because a significant interaction would indicate that model rankings depend on the domain being tested, which would complicate practical recommendations. The analysis confirmed the significant main effect of model ($F(6,206)=14.94$, $p<0.001$, ${\eta }^2=0.277$) and the non-significant main effect of domain ($F(4,206)=0.92$, $p=0.454$, ${\eta }^2=0.011$).

Critically, the model × domain interaction was not significant ($F(24,206)=0.99$, $p=0.475$, ${\eta }^2=0.074$). The non-significant interaction term has an important substantive interpretation: the relative standing of models in terms of SMR was consistent across all five domains. Mistral 7B consistently produced lower SMR regardless of whether the concept belonged to Algorithms, Programming Languages, Databases, Computer Networks, or Operating Systems. Similarly, the six high-SMR models maintained near-ceiling performance across all domains. This consistency across domains strengthens the interpretation that the observed model differences reflect genuine, stable variation in model behavior, such as differences in training data, architecture, or instruction-following capacity, rather than domain-specific artifacts or idiosyncratic responses to particular concept types.

4.6. Error Type Distribution

Beyond the frequency of strategic misconceptions, the nature of the errors has important implications for pedagogical interventions. Each confirmed strategic misconception was classified according to whether it represented a factual error (incorrect statements about verifiable facts), a logical error (flawed reasoning from correct premises), or a conceptual error (fundamental misunderstanding of underlying principles). Across all 221 confirmed strategic misconceptions, conceptual errors dominated, accounting for 66.5% ($n=147$), with factual errors comprising the remaining 33.5% ($n=74$). No logical errors were identified in the corpus.

The complete absence of logical errors is a noteworthy finding that warrants careful interpretation. One possibility is that this reflects a genuine strength of current LLMs: they are highly proficient at maintaining local coherence and constructing valid reasoning chains from given premises, even when those premises are factually incorrect or stem from a flawed conceptual model. In other words, the models are capable of perfectly logical reasoning within an incorrect or incomplete “world model,” which is precisely what makes their errors so insidious from a pedagogical standpoint. An alternative interpretation, however, is that this absence is partly an artifact of our annotation schema and its operationalization. The boundary between a logical error and a conceptual error can be ambiguous; an inference that appears logically invalid may, upon deeper inspection, be the downstream consequence of a more fundamental conceptual misunderstanding. Our annotators may have preferentially categorized such cases as conceptual, effectively absorbing borderline logical errors into the conceptual category. This ambiguity is well-documented in the misconception literature [11] and represents a limitation of any discrete taxonomy of error types. To clarify the distinction between these categories, consider concrete examples from our corpus. A logical error would be: “Quicksort has average-case complexity $O\left(n\log n\right)$, therefore it is suitable for real-time systems”, where the factual premise is correct but the inference is invalid. A conceptual error would be: “Dijkstra’s algorithm can handle negative weights by first applying Bellman-Ford to remove negative cycles”, where the underlying model of how these algorithms relate is fundamentally flawed. We did observe cases of the latter type, but none that were purely the former. Future work could explore finer-grained annotation schemes or use think-aloud protocols to better disambiguate these categories. Despite this potential boundary effect, the practical pedagogical implication remains clear: the majority of strategic misconceptions produced by LLMs are not simple factual slips but reflect deeper issues that would require substantial conceptual intervention to remediate.

4.6.1. Error Types by Model

Table 7 presents the distribution of error types across models. A chi-square test of independence revealed a statistically significant association between model and error type (${\chi }^2\left(6\right)=13.84$, $p=0.031$, Cramér’s $V=0.250$). The effect size, Cramér’s $V=0.250$, is classified as small to medium, indicating that while model identity is reliably associated with error type distribution, the strength of this association is modest.

Table 7. Distribution of error types by model (Class 2 responses only). Row percentages are shown in parentheses.

Model	Conceptual	Factual	Total
Ministral 8B	30 (85.7%)	5 (14.3%)	35
Llama 3.3 70B	25 (73.5%)	9 (26.5%)	34
Llama 4 Scout	21 (65.6%)	11 (34.4%)	32
Gemma 2 31B	21 (60.0%)	14 (40.0%)	35
Qwen3 235B	21 (60.0%)	14 (40.0%)	35
Mistral 7B	15 (75.0%)	5 (25.0%)	20
DeepSeek-R1 70B	14 (46.7%)	16 (53.3%)	30
Total	147 (66.5%)	74 (33.5%)	221

Figure 6 visualizes the distribution of error types across models as a stacked bar chart. The blue portion of each bar represents the proportion of conceptual errors, while the light blue portion represents factual errors. Numbers inside each segment indicate the absolute count, facilitating precise comparison.

Figure 6. Distribution of error types (conceptual vs. factual) by model. Numbers inside bars indicate absolute counts. Conceptual errors dominate across all models except DeepSeek-R1 70B, where factual errors constitute the majority.

The most notable pattern in Figure 6 is the behavior of DeepSeek-R1 70B. This is the only model for which factual errors ($n=16$, 53.3%) outnumber conceptual errors ($n=14$, 46.7%). This stands in marked contrast to Ministral 8B, which exhibited the strongest conceptual bias at 85.7% conceptual ($n=30$) versus 14.3% factual ($n=5$).

4.6.2. Error Types by Domain

Table 8 presents the domain-level error type distributions. A chi-square test revealed significant variation across CS subfields (${\chi }^2\left(4\right)=16.31$, $p=0.003$, Cramér’s $V=0.272$). The effect size, Cramér’s $V=0.272$, is slightly larger than that observed for the model-level analysis, suggesting that domain identity is a somewhat stronger predictor of error type than model identity.

Table 8. Distribution of error types by CS domain (Class 2 responses only). Row percentages are shown in parentheses.

Domain	Conceptual	Factual	Total
Programming Languages (PLP)	36 (83.7%)	7 (16.3%)	43
Databases (DB)	32 (76.2%)	10 (23.8%)	42
Algorithms (ALG)	31 (67.4%)	15 (32.6%)	46
Operating Systems (OS)	27 (60.0%)	18 (40.0%)	45
Computer Networks (NET)	21 (46.7%)	24 (53.3%)	45
Total	147 (66.5%)	74 (33.5%)	221

Figure 7 visualizes these domain-level distributions. A clear gradient is observable across domains. At one extreme, Programming Languages and Paradigms shows the strongest conceptual bias (83.7% conceptual, 16.3% factual), suggesting that LLM errors in this domain predominantly reflect fundamental misunderstandings of language semantics, type systems, and paradigm principles. At the other extreme, Computer Networks shows the most balanced distribution (46.7% conceptual, 53.3% factual), indicating that errors in this domain are approximately equally likely to arise from incorrect factual knowledge as from conceptual misunderstandings.

Figure 7. Distribution of error types (conceptual vs. factual) by CS domain. Domains are ordered from strongest conceptual bias (Programming Languages, 83.7%) to most balanced distribution (Computer Networks, 46.7% conceptual).

This domain-level pattern is interpretable in terms of the epistemic structure of the disciplines. Programming language theory is characterized by formal, axiomatic knowledge structures where concepts are defined relationally, for example, the relationship between lexical scope and closures, or between inheritance and the Liskov Substitution Principle. Errors in such domains naturally manifest as conceptual misunderstandings of these relationships. Computer networking, by contrast, involves a substantial body of specification-driven factual knowledge, TCP header fields, CIDR notation formats, HTTP status codes, where errors are more likely to take the form of incorrect factual claims. The practical implication is that AI literacy interventions may need to be tailored to the dominant error type of each domain: instructors in programming-heavy courses should emphasize conceptual verification strategies, while those teaching networking should emphasize factual cross-referencing against authoritative protocol specifications.

4.7. Persuasiveness Scores

While SMR quantifies the frequency of strategic misconceptions, it does not capture their severity. A model that generates many strategic misconceptions of low persuasiveness may pose less practical risk than a model that generates fewer but highly persuasive ones. To address this dimension, persuasiveness was rated on a 5-point Likert scale for all 221 confirmed strategic misconceptions. Each annotator independently estimated how convincing each Class 2 response might appear to a first-year CS student and the three ratings were averaged to produce a composite persuasiveness score for each segment.

The overall mean persuasiveness was $M=3.71$ ($SD=0.64$, $Mdn=3.67$, $IQR=[3.33,4.33]$, range $=[2.3,5.0]$). This mean, approaching the upper end of the scale (where 4 corresponds to “very persuasive”), indicates that the typical strategic misconception in the SocraticTrap-CS corpus was rated as moderately to highly persuasive. The standard deviation of 0.64 indicates meaningful variability around this mean. The interquartile range of $[3.33,4.33]$ shows that the middle 50% of persuasiveness ratings fell between “moderately persuasive” and “very persuasive,” with a substantial right tail extending to the maximum value of 5.0 (“highly persuasive”). Even the minimum observed persuasiveness of 2.3 exceeded the scale midpoint (2.5), suggesting that models rarely produce strategic misconceptions that are easily dismissed as suspicious.

Table 9 presents the mean persuasiveness scores by model, sorted in descending order. The seven models span nearly the full range of the persuasiveness scale, from 2.95 (Mistral 7B, approaching “moderately persuasive”) to 4.17 (Qwen3 235B, exceeding “very persuasive”).

Table 9. Mean persuasiveness scores by model, sorted in descending order. SD = standard deviation; N = Class 2 responses with valid persuasiveness ratings.

Model	Mean	SD	N
Qwen3 235B	4.17	0.45	35
Ministral 8B	4.11	0.54	35
Llama 3.3 70B	3.90	0.50	34
Gemma 2 31B	3.82	0.44	35
DeepSeek-R1 70B	3.40	0.55	30
Llama 4 Scout	3.18	0.44	32
Mistral 7B	2.95	0.49	20

Figure 8 visualizes these persuasiveness scores as a bar chart with error bars representing $\pm 1$ standard deviation. The dashed horizontal line at $M=3.71$ indicates the grand mean, facilitating visual identification of models above and below the overall average.

Figure 8. Mean persuasiveness scores by model (Class 2 responses only). Error bars represent $\pm 1$ SD. The dashed horizontal line indicates the grand mean ($M=3.71$).

A one-way ANOVA on persuasiveness scores revealed a highly significant effect of model ($F(6,214)=26.90$, $p<0.001$, ${\eta }^2=0.430$). This is the largest effect size observed in the study: model identity accounts for 43% of the total variance in persuasiveness scores, substantially exceeding the variance explained by model identity in SMR (${\eta }^2=0.277$). This indicates that persuasiveness is more strongly model-dependent than the mere propensity to generate strategic misconceptions.

Figure 8 reveals a clear three-tier stratification. The top tier consists of Qwen3 235B ($M=4.17$), Ministral 8B ($M=4.11$), and Llama 3.3 70B ($M=3.90$), which are statistically indistinguishable from one another but significantly more persuasive than all models in the bottom tier. The middle tier comprises Gemma 2 31B ($M=3.82$) and DeepSeek-R1 70B ($M=3.40$). The bottom tier consists of Llama 4 Scout ($M=3.18$) and Mistral 7B ($M=2.95$).

Tukey HSD post-hoc comparisons, presented in Table 10, identified 11 significant pairwise differences out of 21 possible comparisons. The pattern is one of clear separation between the extremes: the three most persuasive models consistently differ from the two least persuasive models, while adjacent models within each tier are generally not significantly different from one another.

Table 10. Significant Tukey HSD post-hoc comparisons for persuasiveness scores. Only comparisons reaching statistical significance at $\alpha =0.05$ are shown.

Pairwise Comparison	Difference	q	p
Qwen3 235B vs. Mistral 7B	1.221	12.64	<0.001
Ministral 8B vs. Mistral 7B	1.164	12.05	<0.001
Qwen3 235B vs. Llama 4 Scout	0.994	11.79	<0.001
Llama 3.3 70B vs. Mistral 7B	0.952	9.80	<0.001
Ministral 8B vs. Llama 4 Scout	0.937	11.12	<0.001
Gemma 2 31B vs. Mistral 7B	0.869	8.99	<0.001
Qwen3 235B vs. DeepSeek-R1 70B	0.771	8.99	<0.001
Ministral 8B vs. DeepSeek-R1 70B	0.714	8.33	<0.001
Llama 3.3 70B vs. Llama 4 Scout	0.725	8.54	<0.001
Gemma 2 31B vs. Llama 4 Scout	0.642	7.61	<0.001
DeepSeek-R1 70B vs. Mistral 7B	0.450	4.52	0.026

Only significant comparisons are shown; all other pairs: $p>0.05$.

The joint analysis of SMR and persuasiveness yields what is perhaps the most practically important finding of this benchmark. Qwen3 235B and Ministral 8B represent the worst-case scenario for educational deployment: both models achieved perfect SMR (100%), meaning they generated strategic misconceptions for every single prompted concept, and their misconceptions were rated as the most persuasive ($M=4.17$ and $M=4.11$, respectively). A student interacting with these models would encounter strategic misconceptions at maximal frequency and maximal persuasiveness, a combination that represents the highest possible pedagogical risk.

At the opposite extreme, Mistral 7B represents the safest profile: it generated strategic misconceptions at the lowest rate (57.1%) and these misconceptions were rated as the least persuasive ($M=2.95$). This dual advantage is noteworthy because it rules out a potential trade-off interpretation: Mistral 7B is not simply sacrificing persuasiveness to achieve a lower SMR, nor is it trading off SMR for lower persuasiveness. Rather, it outperforms the other models on both dimensions simultaneously, making it the most educationally suitable model among those evaluated.

The dissociation between SMR and persuasiveness observed for Llama 4 Scout further demonstrates the value of multidimensional risk profiling. Llama 4 Scout exhibited a high SMR (91.4%) but the second-lowest persuasiveness ($M=3.18$), indicating that while this model frequently generates strategic misconceptions, those misconceptions are less polished and less likely to deceive students than those produced by models in the top persuasiveness tier. This dissociation would be invisible in a single-metric evaluation focusing solely on SMR, underscoring the importance of the multidimensional approach adopted in this benchmark.

4.8. Summary of Key Findings

The benchmark analysis of 722 annotated text segments across seven open-weight LLMs and five CS domains yielded five principal findings:

• Inter-annotator agreement was almost perfect ($\kappa =0.949$), validating the three-class annotation schema and confirming that the distinction between correct responses, overt hallucinations, and strategic misconceptions can be drawn reliably by domain experts. This establishes the trustworthiness of the ground truth labels used throughout the benchmark.
• Strategic misconception rates were uniformly high across models, with the striking exception of Mistral 7B (57.1%), which was significantly lower than all other models ($p<0.001$ for all pairwise comparisons, Cohen’s d ranging from 0.84 to 1.21). The remaining six models exhibited near-ceiling SMR (91.4–100%), with three models achieving perfect scores.
• Conceptual errors dominated over factual errors across all models (66.5% vs. 33.5%), and no logical errors were identified. Error type distributions varied significantly across both models ($p=0.031$) and domains ($p=0.003$). Programming Languages showed the strongest conceptual bias (83.7%); Computer Networks showed the most balanced profile.
• Domain-level SMR did not differ significantly ($p=0.608$, ${\eta }^2=0.011$), indicating that strategic misconception risk is pervasive across all five CS subfields. The non-significant model × domain interaction ($p=0.475$) confirms that model differences are stable across domains.
• Persuasiveness was strongly model-dependent (${\eta }^2=0.430$). Qwen3 235B and Ministral 8B were identified as the riskiest models, combining perfect SMR with the highest persuasiveness. Mistral 7B emerged as the safest model on both dimensions simultaneously.

5. Discussion and Limitations

The findings of this study provide strong evidence that strategic misconceptions represent a distinct and pervasive risk in the educational deployment of contemporary LLMs. However, a critical methodological consideration in interpreting these high SMRs is the explicit nature of the prompting strategy. Because models were directly instructed to generate strategic misconceptions, the SMR primarily reflects their capacity for adversarial instruction-following rather than their spontaneous tendency to produce such errors in unprompted, naturalistic educational dialogues. We acknowledge this as a significant limitation of the current benchmark design. Nonetheless, this instruction-following capacity is itself pedagogically relevant: it shows that, when prompted to do so, most evaluated models can produce conceptually flawed explanations that experts judge to be moderately to highly persuasive. What the benchmark measures is therefore a latent capability rather than an observed disposition, but a capability that, once present, can be activated under the right conditions. In a real-world educational setting, a well-intentioned but poorly phrased student query (e.g., “Explain why Dijkstra’s algorithm works with negative weights”) could inadvertently serve as a similarly adversarial prompt, triggering the same capability. Therefore, while our SMR values should not be interpreted as estimates of the base rate of such errors in naturalistic use, they serve as a conservative upper bound on model capability and highlight a latent risk that can be activated under specific, and not entirely unrealistic, conditions. Future work employing naturalistic study designs, where students interact with LLMs without explicit misconception prompts, is a necessary next step to estimate real-world prevalence.

We acknowledge that this study lacks direct empirical validation with students. Our persuasiveness ratings reflect expert judgments of how convincing a strategic misconception might appear to a first-year CS student, not actual measurements of student deception, belief change, or learning outcomes. This is an important limitation, as expert assessments may not perfectly align with student vulnerability, particularly for novice learners who may lack the domain knowledge to detect subtle conceptual errors. Future work should complement our expert-based evaluation with empirical studies measuring actual student responses to LLM-generated strategic misconceptions, potentially through controlled experiments where students rate the credibility of explanations or through pre/post-test designs measuring conceptual change after exposure. Such studies would provide crucial evidence about the real-world pedagogical impact of the strategic misconceptions identified in our benchmark.

Another significant limitation concerns model selection. The benchmark focuses exclusively on open-weight models that can be deployed locally and evaluated under fully reproducible conditions. Consequently, proprietary systems such as GPT-4o, Claude, and Gemini were not included. While our rationale for this choice was methodological, ensuring reproducibility and controlling for the rapid, undocumented updates that characterize commercial models, it limits the generalizability of our findings to the broader landscape of LLMs used in educational settings. Proprietary models may exhibit different patterns of strategic misconception generation, potentially higher or lower SMR, and different error type distributions. We encourage future work to extend the SocraticTrap-CS methodology to proprietary models through carefully controlled API-based experiments, while noting the challenges of reproducibility this approach entails.

The exceptionally low SMR of Mistral 7B (57.1%) compared to all other models invites careful interpretation. The most straightforward conclusion is that Mistral 7B is the “safest” model among those evaluated, a finding supported by its lower persuasiveness scores as well. However, a compelling alternative explanation must be considered: that Mistral 7B’s lower SMR reflects a more general deficit in instruction-following capability rather than a genuinely lower propensity for generating misconceptions. As the smallest model in the benchmark by parameter count (7B), its reduced capacity may have made it less adept at the complex task of constructing a subtle, intentional error while maintaining overall coherence, causing it to default to simpler, more obviously wrong outputs or to correct explanations. Our current experimental design cannot definitively disentangle these two interpretations. Both are, however, educationally relevant: a model that fails to follow instructions is, in practice, less likely to produce a well-disguised Socratic trap, whether because it is “safe” or simply less capable of the sophisticated generation required. Future work could include a dedicated instruction-following baseline task to partial out this variance.

The patterns of strategic misconceptions we observed do not exist in a vacuum; they mirror and potentially amplify well-documented student misconceptions in CS education. For example, the prevalence of conceptual errors in our benchmark aligns with Qian and Lehman’s findings that students often hold robust, theory-like misconceptions about programming constructs rather than simple factual errors [13]. Our finding that algorithms and programming languages are highly susceptible to conceptual errors echoes Sorva’s work on the difficulty students face in building accurate mental models of program execution [14]. This congruence suggests that the “Socratic traps” generated by LLMs are not random fabrications but exploit the same conceptual fault lines that make these topics challenging for human learners. The danger, then, is that an LLM does not just introduce a new error; it can validate and reinforce a pre-existing, documented misconception that a student may already be developing. This ecological link to the established CS education literature strengthens the real-world relevance of our benchmark and underscores its value for anticipating and mitigating AI-induced learning obstacles.

A particularly important finding of this work is the uniformly high prevalence of strategic misconceptions across models. With the exception of Mistral 7B, all evaluated models produced strategic misconceptions for over 90% of prompted concepts. This suggests that modern instruction-tuned LLMs are highly capable of producing explanations that satisfy the formal characteristics of pedagogically plausible pedagogical discourse despite containing subtle conceptual inaccuracies. Rather than producing random or incoherent failures, the evaluated models consistently generated responses characterized by plausibility, coherence, and subtle conceptual deviation. From an educational perspective, this is arguably more concerning than overt hallucination because an obviously incorrect answer can often be rejected by students or instructors with minimal effort, whereas a persuasive but subtly flawed explanation may reinforce incorrect mental models while appearing pedagogically valid. On this note, the persuasiveness analysis further strengthens this interpretation. The highest-risk models, particularly Qwen3 235B and Ministral 8B, not only generated strategic misconceptions at the highest rates, but also produced the most persuasive responses overall. This combination suggests that advances in fluency, coherence, and instruction alignment may unintentionally amplify pedagogical danger rather than reduce it. Consequently, the same qualities that make LLMs attractive educational tools may also render their misconceptions more difficult to identify critically. The observed separation between models with high misconception frequency and those with lower persuasiveness additionally highlights the importance of evaluating educational LLMs using multidimensional risk profiles rather than relying solely on binary correctness metrics.

An especially noteworthy result is the dominance of conceptual errors over factual ones. Across the corpus, most strategic misconceptions reflected deeper misunderstandings of relationships, mechanisms, or governing principles rather than isolated factual mistakes. This is particularly relevant in computer science education, where learning often depends on constructing internally coherent mental models of abstract systems. Conceptual misunderstandings may therefore have longer-lasting educational consequences than factual inaccuracies, as they can influence how students interpret future concepts that build upon the same foundation. The strong conceptual bias observed in Programming Languages and Paradigms further supports this interpretation, given the relational and highly abstract nature of concepts such as scope, inheritance, polymorphism and type systems. At the same time, the variation in error type distributions across domains suggests that the epistemic structure of a discipline shapes how misconceptions manifest. Domains such as Computer Networks, which rely heavily on standardized technical specifications and protocol-level knowledge, showed a more balanced distribution between conceptual and factual errors. In contrast, domains grounded in formal abstractions and conceptual relationships showed substantially stronger conceptual bias. These findings imply that strategies for mitigating LLM-related educational risks may need to be domain-sensitive rather than universal. For example, networking courses may benefit more from factual verification and source cross-referencing, whereas programming-focused courses may require stronger emphasis on conceptual reasoning and mental model validation.

Furthermore, the near absence of logical errors within the annotated corpus may indicate that contemporary LLMs are generally effective at maintaining surface-level logical coherence, even when the conceptual premises underlying the explanation are flawed. This observation aligns with emerging evidence that LLMs can generate internally consistent reasoning chains without necessarily possessing stable conceptual representations of the domains they describe. Pedagogically, such errors may be particularly difficult for novice learners to detect because students frequently evaluate explanations according to coherence and fluency rather than through formal conceptual verification. Also, lower rates of strategic misconceptions were generally associated with lower persuasiveness scores, whereas models exhibiting near-ceiling misconception rates tended to produce more convincing and authoritative responses. Although such differences may partially reflect variation in overall expressive capacity and instruction-following behavior, this suggests that highly polished and fluent explanations are not necessarily safer in educational settings. On the contrary, increased linguistic coherence and authority may enhance the perceived credibility of subtly incorrect explanations. This raises the assumption that improvements in general model capability automatically translate into greater educational suitability and highlights the need for evaluation frameworks specifically designed to capture pedagogical risk rather than relying solely on conventional performance metrics.

Several methodological strengths of this study support the robustness of these conclusions. The annotation process achieved almost perfect inter-annotator agreement, demonstrating that strategic misconceptions constitute a sufficiently coherent and operationalizable category of model behavior to be identified reliably by domain experts. In addition, the distinction between prompted intent and annotated class proved methodologically valuable, as it allowed the benchmark to capture genuine model behavior rather than assuming perfect compliance with prompt instructions. The multidimensional evaluation framework, combining misconception frequency, persuasiveness, and error type analysis, also provides a more comprehensive assessment of educational risk than conventional accuracy-based benchmarks.

Nevertheless, the study has several limitations that should be considered when interpreting the results. First, as noted above, the use of explicit prompting to elicit strategic misconceptions differs from naturalistic educational interactions, where such errors may emerge more implicitly. The SMR values reported here should therefore be interpreted as measures of model capability under adversarial conditions rather than estimates of real-world prevalence. Second, the benchmark is restricted to computer science education and therefore may not generalize directly to other academic disciplines. Fields with different epistemic structures may exhibit different misconception dynamics. Third, although expert annotation ensured high-quality labels, all annotators originated from the same institutional context, which may introduce subtle interpretive biases despite the high agreement scores obtained. Fourth, the parsing failures encountered with DeepSeek-R1’s self-referential output format highlight a practical challenge in benchmarking frontier models whose behaviors deviate from expected interaction norms.

These limitations point toward several promising directions for future research. One important avenue involves extending the benchmark to additional STEM and non-STEM disciplines in order to examine whether similar misconception patterns emerge across domains with different conceptual structures. Another direction concerns the development of automated detection systems capable of identifying strategic misconceptions in real time, potentially through hybrid approaches combining transformer-based classifiers with retrieval or symbolic verification mechanisms. Further work is also needed to investigate how learners with different levels of expertise respond to strategic misconceptions, particularly whether novice students are disproportionately vulnerable to persuasive conceptual errors. Finally, longitudinal educational studies could help determine the extent to which repeated exposure to such misconceptions affects conceptual development, retention, and problem-solving ability over time.

Overall, the results of this study suggest that the primary challenge of integrating LLMs into education may not lie in overt factual unreliability alone, but in the generation of explanations that appear pedagogically sound while embedding subtle conceptual distortions. As LLMs become increasingly integrated into educational workflows, the ability to identify, evaluate, and mitigate these strategic misconceptions will become essential for the development of reliable and pedagogically responsible AI-supported learning environments.

6. Conclusions

This paper introduced SocraticTrap-CS, a novel publicly available benchmark designed to identify and analyze strategic misconceptions generated by large language models in computer science education. Unlike overt hallucinations, strategic misconceptions represent a subtler and potentially more harmful category of error: responses that are coherent, persuasive, and pedagogically plausible, yet conceptually flawed. Through the construction of a human-annotated corpus spanning five core computer science domains and seven contemporary open-weight LLMs, this study provides one of the first systematic empirical investigations of this phenomenon within educational contexts.

The results demonstrate that, when explicitly prompted, current-generation LLMs can reliably produce strategic misconceptions and that experts judge these responses to be moderately to highly persuasive. Under these adversarial conditions, nearly all evaluated models complied at very high rates, while the majority of these errors reflected conceptual misunderstandings rather than isolated factual inaccuracies. These figures characterize model capability under explicit elicitation rather than the prevalence of such errors in naturalistic use. Persuasiveness varies substantially across models, indicating that pedagogical risk cannot be adequately captured through binary correctness metrics alone. Together, these results suggest that improvements in fluency, coherence, and instruction-following do not necessarily correspond to improvements in educational reliability.

Beyond the benchmark itself, this work contributes a multidimensional framework for evaluating pedagogical risk in AI-generated educational content. By combining misconception frequency, persuasiveness, and error type analysis, the proposed methodology enables a more nuanced assessment of LLM behavior than conventional evaluation approaches focused solely on accuracy or general hallucination detection. The near-perfect inter-annotator agreement achieved in this study further demonstrates that strategic misconceptions constitute a coherent and operationalizable category of model failure that can be identified reliably by domain experts.

Additionally, the findings of this work have important implications for the future integration of LLMs into educational practice. As these systems become increasingly embedded in tutoring platforms, programming assistants, and self-directed learning workflows, the ability to detect and mitigate strategically misleading explanations will become essential for maintaining conceptual integrity in learning environments. In particular, the prevalence of persuasive conceptual errors highlights the importance of developing AI literacy skills that extend beyond simple fact-checking toward deeper conceptual verification and critical evaluation of generated explanations.

Ultimately, the central contribution of SocraticTrap-CS lies in reframing hallucination in educational AI not merely as a problem of factual incorrectness but as a problem of pedagogical trustworthiness. By providing both a benchmark and an analytical framework for studying these failures, this work establishes a foundation for future research aimed at developing safer, more reliable, and pedagogically responsible AI-supported learning systems. Future research could extend the benchmark to additional academic disciplines, investigate the impact of strategic misconceptions on real student learning outcomes, and explore automated detection approaches capable of identifying such errors in real time. More broadly, the presented results suggest that evaluating educational AI systems requires not only measuring what models know but also understanding how they fail, how persuasive those failures are, and how those failures may shape the learner’s mental model.