A Problem-Level Cognitive Framework for Analyzing Learning Performance: An Empirical Application in AI-Supported and Teacher-Mediated Statistics Instruction

Abstract

Students with similar overall test scores may nevertheless succeed or struggle with very different kinds of problems. Aggregate performance measures can conceal these differences when assessment problems require different forms of reasoning or different levels of structural complexity. Existing cognitive theories and educational taxonomies describe important aspects of knowledge, reasoning, and complexity, but were not designed to classify individual problems simultaneously according to these two characteristics. This study therefore develops a purpose-built Problem-Level Cognitive Framework that represents assessment problems along two analytically independent dimensions: the dominant reasoning required for a conceptually adequate solution—Formal, Procedural, Theoretical, or Critical—and the structural complexity of the minimal valid solution path, expressed through seven ordinal difficulty levels (D1–D7). The framework was empirically examined using semester-long data from 121 undergraduates enrolled in one of two instructional conditions: AI-supported or teacher-mediated introductory statistics instruction. Binary outcomes for each student–problem pair were analyzed using generalized linear mixed-effects models with crossed random effects for students and problems. Successful problem solving varied substantially across reasoning dimensions, difficulty levels, and assessment timings. In contrast, instructional condition made little explanatory contribution, and neither the condition-by-reasoning nor the condition-by-difficulty interaction was significant. The framework thus revealed a stable cognitive organization of performance across two substantially different instructional environments, while also identifying pronounced variation among cognitively differentiated problems. By separating the form of reasoning governing a valid solution from the structural complexity of the required solution process, the framework transforms heterogeneous assessment problems into cognitively comparable analytical units. It provides an operational basis for investigating problem-level performance structures and for examining their stability across learners, assessments, instructional environments, and educational domains.

1. Introduction

Artificial intelligence (AI) systems capable of generating explanations, solving problems, producing worked examples, and providing immediate feedback are becoming integral components of higher education (Deriba et al., 2026; Su et al., 2026). In statistics education, these systems can participate directly in method selection, procedural execution, interpretation, and evaluation, potentially altering the processes through which students arrive at successful solutions (J. Zhang et al., 2026; Liang et al., 2026; Farrelly & Baker, 2023; Yang et al., 2025; Lindell & Stöhr, 2026).

Despite increasing research attention, findings on AI-supported learning remain inconsistent, with positive, negligible, negative, and context-dependent effects reported across educational settings (Wu, 2026; Chiu et al., 2026; T. Zhang et al., 2026; Lehtinen et al., 2026). These inconsistencies may reflect genuine differences among learners, disciplines, instructional designs, and forms of AI use. They may also arise, however, from representing learning performance through outcomes aggregated across cognitively heterogeneous assessment problems.

Aggregate comparisons can establish whether instructional conditions differ in overall performance, but they cannot show whether similar outcomes arise from similar patterns of success across different cognitive demands (Kalogeratos et al., 2026; Zawacki-Richter et al., 2019; Mollick & Mollick, 2023).

Addressing this limitation requires analysis at the level of individual assessment problems. The present study responds by developing and empirically applying a Problem-Level Cognitive Framework in a semester-long comparison of AI-supported and teacher-mediated introductory statistics instruction.

1.1. The Problem of Interpreting AI Effects in Education

This drawback of aggregate assessment is particularly consequential in statistics, where superficially similar problems may require different forms of reasoning. Successful performance may depend primarily on procedural execution, formal representation, theoretical understanding, or critical evaluation, and these demands cannot be reduced to a single general competence (Chance, 2002; Garfield & Ben-Zvi, 2009; Pfannkuch et al., 2010). A student may correctly execute a statistical procedure while being unable to justify its selection, interpret its output, or evaluate the validity of the resulting conclusion.

Differences among problems also cannot be represented adequately by the difficulty observed alone. Empirical difficulty reflects not only the structure of a problem but also the characteristics of the learners, the instructional environment, and the assessment context. Moreover, problems with similar success rates may require qualitatively different reasoning, whereas problems governed by the same form of reasoning may differ substantially in the complexity of their solution processes.

These distinctions are especially relevant to AI-supported learning because AI may support particular cognitive demands more effectively than others. Aggregate comparisons cannot determine whether AI changes performance broadly or selectively across different problem characteristics. A cognitively informative analysis must therefore distinguish two properties of individual assessment problems: the form of reasoning that governs a valid solution and the structural complexity of the solution required.

1.2. Toward a Problem-Level Cognitive Framework

Established cognitive theories and educational taxonomies provide important foundations for understanding learning and problem solving. The distinction between procedural and conceptual knowledge differentiates the ability to execute procedures from understanding the principles and relationships that give those procedures meaning (Rittle-Johnson & Alibali, 1999). Educational taxonomies characterize cognitive demand through progressively more complex forms of activity, from recognition and application to analysis and evaluation (Anderson & Krathwohl, 2001). Research on mathematical representation emphasizes that successful problem solving also depends on interpreting and coordinating symbolic, graphical, and other representational forms (Ainsworth, 2006; Duval, 2026), while statistics education research highlights the distinctive roles of statistical reasoning, inferential understanding, and critical evaluation (Garfield & Ben-Zvi, 2009; delMas et al., 2007; Wild & Pfannkuch, 1999). These traditions describe complementary aspects of cognition, but none were designed to classify individual assessment problems simultaneously according to the dominant form of reasoning required for a valid solution and the structural complexity of the solution process.

This unresolved analytical requirement motivated the development of the Problem-Level Cognitive Framework. The framework is not proposed as a replacement for established cognitive theories or as a general theory of cognition. Rather, it is a purpose-built analytical framework that draws on existing theoretical distinctions and reorganizes them into an operational structure for examining performance across cognitively heterogeneous assessment problems.

The framework represents each problem along two complementary but analytically independent dimensions. The first identifies the dominant form of reasoning required for a conceptually adequate solution: Formal, Procedural, Theoretical, or Critical. Because an individual problem may involve several forms of reasoning, classification is based on necessity rather than mere presence. A problem is assigned to the reasoning dimension whose absence would make a conceptually adequate solution impossible. This necessity-based dominance principle identifies the reasoning that governs the validity of the solution without implying that other cognitive processes are absent.

The second dimension represents ordinal cognitive difficulty through the structural complexity of the minimal valid solution path. Difficulty is therefore not defined by observed success rates or perceived effort, both of which depend partly on learners and instructional context. Instead, it reflects the number and organization of logically dependent reasoning operations required for the shortest conceptually adequate solution. The seven levels (D1–D7) range from direct recognition and routine execution to coordinated method selection, inferential evaluation, and justified statistical judgment.

The distinction between the two dimensions is fundamental. The reasoning dimension identifies what kind of cognitive activity governs a valid solution, whereas ordinal difficulty identifies how structurally complex that solution is. Problems governed by different forms of reasoning may occupy the same difficulty level, while problems within the same reasoning dimension may require solution paths of substantially different complexity. Figure 1 summarizes this two-dimensional structure and shows how each assessment problem is located independently according to its dominant reasoning requirement and ordinal cognitive difficulty.

Figure 1. Two-dimensional structure of the Problem-Level Cognitive Framework. Each assessment problem is classified independently according to the dominant form of reasoning required for a conceptually adequate solution and the ordinal cognitive difficulty represented by the structural complexity of its minimal valid solution path. The matrix defines 28 theoretically possible problem classes; it does not imply that assessment problems must be distributed evenly across the cells.

Two examples illustrate how the two classifications operate independently. Consider first a problem asking students to compute a 95% confidence interval when the z-based procedure is explicitly specified. The problem is classified as Procedural (P) because successful completion depends primarily on correctly executing a predetermined statistical procedure; no method selection, representational interpretation, or evaluation of inferential validity is required. Its minimal valid solution path contains two logically dependent reasoning operations: recognizing the specified procedure and applying it correctly. It is therefore classified as D2.

Now consider a problem in which a researcher uses a pooled two-sample t-test despite substantial differences in group variances and students must evaluate whether the resulting conclusion is methodologically justified. This problem is classified as Critical (C) because recognizing the violated equal-variance assumption is not sufficient: successful completion requires judging how that violation affects the validity of the inference and whether the conclusion remains defensible. Its minimal valid solution path contains six logically dependent reasoning operations: (1) identifying the equal-variance assumption of the pooled procedure; (2) recognizing that the observed variance difference is substantial; (3) determining that this constitutes a violation of the assumption; (4) evaluating how the violation affects the validity of the pooled test; (5) considering whether an alternative procedure, such as Welch’s t-test, would be more appropriate; and (6) formulating a justified conclusion about the defensibility of the original inference. Because six reasoning operations are required, the problem is classified as D6. Thus, the C classification identifies the type of reasoning that governs a valid solution, whereas D6 represents the structural complexity of the minimal valid solution path. Full classification procedures and five worked examples are provided in Supplementary Material S1.3.

The principal contribution of the present study is therefore not another aggregate comparison of AI-supported and teacher-mediated instruction, but a problem-level analytical framework that separates the type of reasoning required for a valid solution from the structural complexity of the solution process. This distinction makes it possible to determine whether similar overall achievement conceals systematic differences in the kinds of statistical problems students solve successfully.

By locating individual assessment problems within this two-dimensional structure, the framework converts cognitively heterogeneous problems into comparable analytical units. It enables learning performance and instructional effects to be examined separately across forms of reasoning and levels of structural complexity, thereby supporting a more differentiated analysis of successful problem solving across diverse educational contexts, including comparisons between contrasting instructional environments.

2. Operationalization and Empirical Calibration of the Framework

2.1. Classification by the Minimal Valid Solution Path

The ordinal cognitive difficulty of an assessment problem is determined from its minimal valid solution path: the shortest sequence of logically dependent reasoning operations sufficient to produce a conceptually adequate solution. The path excludes optional calculations, redundant intermediate steps, unnecessarily extended derivations, and alternative procedures that are not required for validity.

A reasoning operation is counted only when it produces an indispensable intermediate result, decision, interpretation, or judgment on which a subsequent part of the solution depends. Elementary actions belonging to the same conceptual step—such as entering values into a formula or completing routine arithmetic—do not constitute separate operations unless they require distinct decisions. Conversely, operations cannot be combined when the output of one is logically required before the next can be performed.

The number of indispensable operations in the minimal valid solution path determines the ordinal difficulty level defined in Table 1.

Table 1. Operational definition of ordinal cognitive difficulty.

Level	Minimal Valid Solution Path	Operational Definition
D1	1 operation	Direct recognition, identification, retrieval, or execution requiring no dependent intermediate step.
D2	2 operations	One initial recognition or decision followed by one dependent operation.
D3	3 operations	A short sequence in which two intermediate operations are required before the valid response can be produced.
D4	4 operations	A coordinated solution requiring four indispensable operations linked through a continuous dependency structure.
D5	5 operations	An extended solution path involving five dependent operations, typically including an intermediate selection, comparison, or interpretation.
D6	6 operations	A complex solution path requiring six dependent operations and the coordination of several intermediate results or decisions.
D7	7 operations	The longest solution structure represented in the framework, requiring seven indispensable operations and culminating in an integrated inference, evaluation, or justified judgment.

The levels are ordinal rather than metric. A D6 problem requires a longer minimal valid solution path than a D5 problem, but the difference between D5 and D6 cannot be interpreted as quantitatively equivalent to the difference between D1 and D2. The classification identifies increasing structural complexity, not equal increments in psychological or empirical difficulty.

Adjacent levels are separated by a single criterion: whether an additional reasoning operation is indispensable to a conceptually adequate solution. If removing an operation leaves the solution valid, that operation is not counted. If its removal breaks the logical dependency chain or makes the conclusion unjustified, it remains part of the minimal valid solution path. This rule prevents difficulty levels from being inflated by lengthy calculations, verbose explanations, or inefficient solution strategies.

The classification is independent of observed success rates, estimated learner effort, instructional familiarity, topic, response format, and surface length. A long computational problem may therefore receive a relatively low classification when its solution follows a short, predetermined reasoning sequence, whereas a briefly worded problem may receive a high classification when it requires several dependent decisions before a justified conclusion can be reached.

Ordinal difficulty describes the length and dependency structure of the solution path, but not the kind of reasoning that governs its validity. The latter is classified separately through the dominant reasoning requirement described in Section 2.2. Concrete applications of the D1–D7 classification are presented in Section 2.3 using reconstructed solution paths from an internationally established and widely used undergraduate statistics textbook corpus.

The complete ordinal difficulty framework, including its conceptual rationale and the detailed operational criteria used to reconstruct the minimal valid solution path and assign the corresponding D1–D7 level, is provided in Supplementary Material S1.1.

2.2. Classification by Dominant Reasoning Requirement

Whereas ordinal cognitive difficulty is determined by the structure of the minimal valid solution path, the reasoning dimension identifies the type of reasoning that governs the validity of the solution. Classification therefore concerns neither the topic of the problem nor every cognitive operation involved, but the reasoning requirement that ultimately makes the solution conceptually adequate. The four reasoning dimensions and their operational classification criteria are summarized in Table 2.

Table 2. Operational definitions of the dominant reasoning dimensions.

Reasoning Dimension	Governing Requirement	Operational Classification Criterion
Formal (F)	Construction, interpretation, or transformation of symbolic, mathematical, tabular, or graphical representations	The solution is valid only if the relevant relationships between representations are correctly established or transformed.
Procedural (P)	Correct execution of a statistical operation, algorithm, or predetermined sequence	The required method is specified or directly identifiable, and validity depends primarily on its correct execution.
Theoretical (T)	Application of statistical principles to method selection, inference, or interpretation	Correct execution alone is insufficient; the solution must use the theoretical conditions or principles that justify the method or conclusion.
Critical (C)	Evaluation of assumptions, evidence, conclusions, limitations, or the appropriateness of a statistical claim	The solution requires a justified evaluative judgment for which no predetermined procedure alone can establish the answer.

Because a problem may involve several forms of reasoning, classification follows a necessity-based dominance principle. The relevant question is not which activity is most visible or occupies the greatest number of steps, but which reasoning requirement cannot be removed without making the solution conceptually inadequate. Operations belonging to other dimensions may be indispensable components of the solution path yet remain supporting operations if they do not govern the validity of the final response.

For example, a problem may require a calculation before the adequacy of a statistical conclusion can be evaluated. The calculation is procedurally necessary, but the problem is classified as Critical when the validity of the response ultimately depends on a justified evaluation of the conclusion. Conversely, a problem remains Procedural when the requested interpretation follows directly from a specified computation and requires neither theoretical justification nor independent evaluative judgment.

When two dimensions initially appear equally indispensable, the complete minimal valid solution path is examined to distinguish supporting operations from the reasoning requirement governing the final conclusion. If validity depends on correctly representing a relationship, the problem is classified as Formal; if it depends on executing an identifiable procedure, it is classified as Procedural; if it depends on applying a statistical principle, it is classified as Theoretical; and if it depends on evaluating the warrant, adequacy, or limitations of a conclusion, it is classified as Critical.

Figure 2 summarizes the complete classification procedure. The minimal valid solution path provides the common analytical basis: its indispensable and logically dependent operations determine ordinal cognitive difficulty, whereas the necessity-based dominance principle identifies the reasoning dimension governing solution validity. The two classifications are made independently and jointly locate each problem within the framework.

Figure 2. Classification procedure within the Problem-Level Cognitive Framework. Classification begins with reconstruction of the minimal valid solution path. Its structurally dependent reasoning operations determine ordinal cognitive difficulty (D1–D7), whereas the necessity-based dominance principle identifies the reasoning dimension whose absence would make a conceptually adequate solution impossible. The two classifications jointly locate each assessment problem within the framework.

Illustrative examples of the application of the reasoning-dimension classification procedure are provided in Supplementary Material S1.2.

2.3. Development and Calibration Using an Internationally Established Textbook Item Bank

The framework was developed and calibrated using the publisher-provided test bank accompanying Statistics for Business and Economics (McClave et al., 2008), hereafter referred to as the textbook item bank. This widely used and internationally established textbook supports undergraduate statistics education, and its accompanying item bank comprises a broad and systematically organized collection of assessment problems covering the principal content areas of an introductory statistics curriculum. It therefore provided a heterogeneous reference base for developing the classification rules and examining whether they could be applied consistently across different statistical topics and problem formats.

The item bank served as a developmental and calibration resource rather than as an empirical dataset of student performance. For each problem, the minimal valid solution path was reconstructed, and only those reasoning operations that were indispensable to a conceptually adequate solution were retained. The number and dependency structure of these operations determined the ordinal cognitive difficulty classification (D1–D7). After the difficulty level had been established, the necessity-based dominance principle was applied to identify the Formal, Procedural, Theoretical, or Critical reasoning requirement governing solution validity.

Given the size of the textbook item bank, the initial classification was conducted with AI assistance. Each problem was provided to the AI together with the operational definitions, decision rules, and classification criteria of the framework. The AI then generated a preliminary classification of the problem’s ordinal cognitive difficulty level and dominant reasoning dimension. Repeated comparisons across the item bank were used to refine the boundaries between adjacent difficulty levels and to distinguish validity-governing reasoning from supporting operations. Particular attention was given to cases in which surface characteristics could produce misleading classifications. A problem involving extensive computation, for example, does not necessarily occupy a high difficulty level if the calculation follows a short and predetermined reasoning structure. Conversely, a briefly worded problem may receive a higher classification when a valid response requires several logically dependent decisions, interpretations, or judgments. Similarly, the presence of calculation does not make a problem Procedural when the validity of the final response is governed by theoretical justification or critical evaluation.

These preliminary classifications were subsequently subjected to human validation. Each problem was independently reviewed by two instructors experienced in statistics education, who evaluated the AI-generated classifications using the same operational criteria. Agreement was nearly complete for the dominant reasoning dimensions and reached 94% for the ordinal difficulty levels. Cases in which either reviewer disagreed with the preliminary classification, or in which the two reviewers differed from one another, were discussed by the six-member author panel until consensus was reached. Thus, AI was used to conduct the initial large-scale classification, whereas every final classification was determined through human review and, where necessary, panel consensus.

Five detailed worked examples of the classification process are provided in Supplementary Material S1.3. These examples reconstruct the minimal valid solution paths of selected item-bank problems and demonstrate how ordinal cognitive difficulty is classified before the dominant reasoning dimension is identified. They also show that problems addressing the same statistical topic may differ in both classifications, whereas problems from different topics may share the same two-dimensional location within the framework.

The item bank was organized into 11 sections corresponding to the principal content areas of the course. Because probability distributions were taught and analyzed separately for discrete and continuous distributions, STAT-04 was divided into two sections. Table 3 defines the abbreviated section codes used in the subsequent tables and figures and reports the number of classified problems in each section.

Table 3. Statistical content areas represented by the item-bank section codes.

Item-Bank Section Code	Statistical Content Area	Course Week	Number of Problems
STAT-01	Introduction to Statistics	1	82
STAT-02	Descriptive Statistics	1–2	190
STAT-03	Probability	3	237
STAT-04-1	Probability Distributions: Discrete Distributions	4	134
STAT-04-2	Probability Distributions: Continuous Distributions	5	138
STAT-05	Statistical Sampling	6	52
STAT-06	Estimation Theory	8	135
STAT-07	One-Sample Hypothesis Testing	9	144
STAT-08	Two-Sample Inference	10	102
STAT-09	Categorical Data Analysis	11	60
STAT-10	Nonparametric Statistical Methods	12	49
Total			1323

Note: Course weeks indicate when the corresponding content areas were taught. Weeks 7 and 13 were reserved for the first and second mid-term examinations, respectively. STAT-04 was divided into discrete and continuous probability distributions because these areas were taught in separate weeks and analyzed separately in the item bank.

The distribution of ordinal cognitive difficulty levels across the item-bank sections is reported in Table 4. All sections contained problems at more than one difficulty level, and several content areas extended across a broad portion of the D1–D7 scale. The distributions nevertheless differed across sections, reflecting differences in the number and dependency structure of the indispensable reasoning operations represented in each content area. The highest level, D7, occurred only rarely in the item bank, whereas D1–D3 accounted for the majority of problems. Importantly, the variation observed within individual sections shows that ordinal cognitive difficulty does not function merely as a proxy for statistical topic.

Table 4. Distribution of ordinal cognitive difficulty levels across the textbook item bank.

Item-Bank Section	D1	D2	D3	D4	D5	D6	D7	Total
STAT-01	40	26	16	0	0	0	0	82
STAT-02	100	16	59	2	3	5	5	190
STAT-03	56	92	34	7	47	1	0	237
STAT-04-1	8	104	14	0	1	6	1	134
STAT-04-2	12	34	86	6	0	0	0	138
STAT-05	23	9	14	5	0	0	1	52
STAT-06	28	4	66	6	20	11	0	135
STAT-07	5	41	18	58	15	7	0	144
STAT-08	22	3	32	13	22	10	0	102
STAT-09	12	16	17	1	12	2	0	60
STAT-10	3	3	17	12	10	4	0	49
Total	309	348	373	110	130	46	7	1323

Note: Counts represent the number of problems assigned to each ordinal cognitive difficulty level (D1–D7) according to the proposed structural classification framework. The item-bank section codes and their corresponding statistical content areas are defined in Table 3.

Figure 3 visualizes the distributions reported in Table 4 using row percentages. Within each item-bank section, the percentages show how the classified problems are distributed across the seven ordinal cognitive difficulty levels. The heatmap demonstrates both the concentration of particular difficulty levels within some sections and the presence of multiple difficulty levels within the same statistical content area. This supports the distinction between the topic addressed by a problem and the structural complexity of its minimal valid solution path.

Figure 3. Row-percentage heatmap of ordinal cognitive difficulty distributions across the textbook item-bank sections identified in Table 3. Within each row, cell values represent the percentage of problems in the corresponding item-bank section assigned to each ordinal cognitive difficulty level (D1–D7); percentages therefore sum to 100% across each row.

After the ordinal cognitive difficulty classifications had been established, each problem was assigned to the reasoning dimension governing the validity of its solution. The distribution of the four dominant reasoning dimensions across the item-bank sections is reported in Table 5. Procedural and Theoretical problems were the most frequent overall, but all four dimensions occurred across multiple content areas. The distributions also differed substantially among sections. For example, Theoretical problems predominated in Introduction to Statistics and Descriptive Statistics, Procedural problems predominated in Probability and Probability Distributions, and Formal problems were particularly prominent in One-Sample Hypothesis Testing. Critical problems were less frequent but occurred in most content areas. These patterns indicate that the dominant reasoning dimension does not merely reproduce the statistical topic of a problem.

Table 5. Distribution of dominant reasoning dimensions across the textbook item bank.

Item-Bank Section	Formal (F)	Procedural (P)	Theoretical (T)	Critical (C)	Total
STAT-01	12	0	66	4	82
STAT-02	4	76	105	5	190
STAT-03	23	178	34	2	237
STAT-04-1	0	89	39	6	134
STAT-04-2	0	86	52	0	138
STAT-05	6	18	27	1	52
STAT-06	3	90	29	13	135
STAT-07	46	17	74	7	144
STAT-08	25	50	14	13	102
STAT-09	12	35	6	7	60
STAT-10	13	26	5	5	49
Total	144	665	451	63	1323

Note: Counts represent the number of problems assigned to each dominant reasoning dimension within the item bank accompanying Statistics for Business and Economics (McClave et al., 2008). The item-bank section codes and their corresponding statistical content areas are defined in Table 3.

Figure 4 visualizes the distributions reported in Table 5 using row percentages. Within each item-bank section, the percentages show how the classified problems are distributed across the four dominant reasoning dimensions. The heatmap makes the within- and between-section variation more readily visible and shows that the reasoning dimensions are distributed across the curriculum rather than confined to particular statistical content areas.

Figure 4. Row-percentage heatmap of dominant reasoning-dimension distributions across the textbook item-bank sections identified in Table 3. Within each row, cell values represent the percentage of problems assigned to each dominant reasoning dimension; percentages therefore sum to 100% across each row. F = Formal; P = Procedural; T = Theoretical; C = Critical.

Together, the worked examples, numerical distributions, and heatmaps demonstrate the operational coverage and internal differentiation of the framework across a heterogeneous collection of authentic undergraduate statistics assessment problems. The analysis shows that neither ordinal cognitive difficulty nor dominant reasoning dimension is reducible to statistical topic and that the two classifications capture distinct structural properties of problems. This calibration does not, by itself, establish psychometric validity or generalizability beyond statistics education. Rather, it establishes a transparent and consistently applicable classification system that can subsequently be examined using independent assessment problems and observed student-performance data.

2.4. Research Objectives and Research Questions

The primary objective of this study was to empirically examine the proposed Problem-Level Cognitive Framework using observed student-performance data. Applied in the context of AI-supported and traditional teacher-mediated introductory statistics instruction, the framework was used to investigate the simultaneous and mutually adjusted associations of ordinal cognitive difficulty and dominant reasoning dimension with the probability of successful problem solving, and to determine whether the association between instructional condition and performance varied across either dimension. Accordingly, the study adopted an integrated modeling strategy in which both dimensions of the cognitive framework—ordinal cognitive difficulty and dominant reasoning dimension—were considered simultaneously with instructional condition, assessment timing, and instructor. The following research questions were addressed:

• RQ1. To what extent does instructional condition (AI-supported versus traditional teacher-mediated instruction) influence the probability of successful problem solving when both dimensions of the proposed cognitive framework are considered simultaneously?
• RQ2. How do ordinal cognitive difficulty levels (D1–D7) influence the probability of successful problem solving after accounting for dominant reasoning dimension and instructional condition?
• RQ3. How do dominant reasoning dimensions (F, P, T, and C) influence the probability of successful problem solving after accounting for ordinal cognitive difficulty and instructional condition?
• RQ4. Do the effects of instructional condition vary systematically across ordinal cognitive difficulty levels?
• RQ5. Do the effects of instructional condition vary systematically across dominant reasoning dimensions?

These questions were investigated using problem-level performance data collected through Immediate and Consolidated assessments and analyzed using the inferential models described in the following sections.

3. Methodology

3.1. Overall Experimental Design

The study employed a quasi-experimental controlled multifactor instructional design to investigate successful problem solving under AI-supported and traditional teacher-mediated introductory statistics instruction. The design combined a between-group comparison of instructional conditions with repeated assessment timing and problem-level cognitive classification. This structure enabled instructional effects to be examined across ordinal cognitive difficulty levels and dominant reasoning dimensions while distinguishing performance observed immediately after instruction from performance demonstrated in subsequent cumulative examinations.

The study was conducted at the University of Dunaújváros, Hungary, during the second semester of the 2025–2026 academic year. A total of 121 second-year undergraduate students participated, of whom 74 were assigned to the AI-supported instructional condition and 47 to the traditional teacher-mediated condition. None had previously completed a university-level statistics course. To assess the academic comparability of the two instructional groups before the intervention, students’ first-year mathematics results were examined. The comparison indicated relative homogeneity in prior mathematical performance, supporting the suitability of the groups for controlled comparison within the institutional course setting.

Students were assigned to instructional groups through the university’s standard administrative procedures before the study began. Group membership was therefore determined independently of the research intervention and did not involve student self-selection into instructional conditions. Although students were not individually randomized, the institutional assignment procedure preserved the authentic organization of the course and supported the ecological validity of the comparison.

The intervention was embedded within a regular semester-long introductory statistics course. Three instructors each taught two parallel groups during the same semester: one traditional teacher-mediated group and one AI-supported group. The parallel groups followed the same syllabus, textbook, sequence of statistical topics, assessment schedule, grading rules, instructional duration, and Moodle-based learning environment. Students in both conditions also had unrestricted access to the same instructional materials outside scheduled classroom sessions, and independent AI use during examination preparation was not restricted. Accordingly, the study did not compare access to AI with the absence of AI access. Instead, it compared two substantially different forms of classroom instructional mediation operating within a common institutional learning environment.

Learning outcomes were measured using two complementary assessment formats. Immediate assessments were administered directly after the completion of individual instructional topics and were intended to capture short-term problem-solving performance. Two Consolidated examinations, administered in Weeks 7 and 13, assessed retained and integrated knowledge across multiple previously covered topics. This repeated assessment structure made it possible to distinguish performance demonstrated immediately after classroom learning from more cumulative performance emerging after subsequent study and integration. The weekly topic sequence and assessment schedule are summarized in Table 6.

Table 6. Weekly topics and assessment schedule.

Week	Topic	Assessment Type	Assessment Label	Maximum Points
1	STAT-01 Introduction to Statistics; STAT-02 Descriptive Statistics (Part 1: qualitative data only)	Formative Immediate (ungraded)	Formative Immediate Test	–
2	STAT-02 Descriptive Statistics (Part 2: remaining content)	Immediate	Immediate Test 1	2
3	STAT-03 Probability	Immediate	Immediate Test 2	2
4	STAT-04-1 Probability Distributions: Discrete Distributions	Immediate	Immediate Test 3	2
5	STAT-04-2 Probability Distributions: Continuous Distributions	Immediate	Immediate Test 4	2
6	STAT-05 Statistical Sampling	Immediate	Immediate Test 5	2
7	First Mid-Term Examination (ZH1)	Consolidated	Consolidated Examination 1	50
8	STAT-06 Estimation Theory	Immediate	Immediate Test 6	2
9	STAT-07 One-Sample Hypothesis Testing	Immediate	Immediate Test 7	2
10	STAT-08 Two-Sample Inference	Immediate	Immediate Test 8	2
11	STAT-09 Categorical Data Analysis	Immediate	Immediate Test 9	2
12	STAT-10 Nonparametric Statistical Methods	Immediate	Immediate Test 10	2
13	Second Mid-Term Examination (ZH2)	Consolidated	Consolidated Examination 2	50

Note: The formative assessment administered in Week 1 was ungraded and was not included in the inferential analyses. Immediate assessments measured performance directly following individual instructional topics, whereas Consolidated Examinations 1 and 2 assessed knowledge accumulated across the preceding sections of the course.

Every assessment problem included in the analytical dataset was subsequently classified according to the Problem-Level Cognitive Framework. Classification first established the ordinal cognitive difficulty level (D1–D7) and then identified the dominant reasoning dimension (F/P/T/C). The classification procedure, its independent review, and the resulting analytical dataset are described in Section 3.3.

3.2. Instructional Conditions

Both instructional conditions operated within the same course structure, with an identical content sequence, textbook, assessment schedule, grading rules, and standardized 135-min weekly sessions. Despite this equivalence, they differed fundamentally in how classroom learning was mediated. The design deliberately maximized the contrast in instructional mediation while maintaining the same curricular content, assessment conditions, and overall course organization across the two instructional environments. The structure of the instructional sessions is summarized in Table 7.

Table 7. Structure of the 135-min instructional sessions across instructional conditions.

Phase	Duration	Purpose	Traditional Teacher-Mediated Condition	AI-Supported Condition
I. Organizational overview (non-content)	5–10 min	Clarifying workflow and task order	Procedural overview only; no conceptual content	Same procedural overview; no conceptual content
II. Learning phase	80–90 min	Core knowledge acquisition	Instructor-led conceptual teaching, including explanations, worked examples, derivations, guided practice, and direct problem-solving support	AI-mediated learning, with AI serving as the primary source of explanations, examples, procedural guidance, and problem-solving assistance; instructor involvement restricted to organizational, technical, and workflow support
III. Immediate assessment	25–30 min	Measuring same-session learning	Completed individually without AI or external assistance	Completed individually without AI or external assistance
IV. Post-assessment clarification (non-intervention)	10–15 min	Clarifying misunderstandings revealed by the assessment	Brief clarification of misunderstandings; no new content introduced	Same clarification; no new content introduced and no preview of the next topic

Each instructional session began with a common organizational overview lasting approximately 5–10 min. This introductory phase was intentionally restricted to procedural and administrative information provided identically in both instructional conditions, including the organization of the session, the sequence of classroom activities, notation conventions, assessment logistics, and access to learning resources. No conceptual teaching, worked statistical examples, inferential explanation, or guided problem solving occurred during this common phase.

Following the organizational overview, the two instructional conditions diverged substantially. In the traditional teacher-mediated condition, classroom learning followed a conventional instructor-led format. Instructors provided direct conceptual explanations, demonstrated worked statistical examples, guided students through problem-solving procedures, clarified misunderstandings, and offered continuous support throughout the learning phase. Statistical methods, inferential reasoning, and interpretation strategies were explicitly discussed, and students could request direct conceptual assistance at any point during the session.

In the AI-supported condition, classroom learning relied primarily on AI systems for explanations, worked examples, procedural guidance, and problem-solving assistance. Although instructors remained continuously present, they intentionally withheld direct conceptual teaching, inferential explanations, worked statistical solutions, and guided problem-solving support. Their involvement was restricted to organizational coordination, technical assistance, workflow clarification, and support related to the operational use of AI tools. Thus, the contrast concerned the primary mediator of classroom learning rather than the curricular content, time allocation, learning resources, or assessment conditions.

Before the experimental phase, students participated in an orientation session focused on the practical use of AI systems for learning statistics. The first instructional week also provided an opportunity to become familiar with AI-supported learning workflows, prompting strategies, and the use of AI-generated explanations during problem solving. Students were permitted to use any AI system; however, Microsoft 365 Copilot was used almost exclusively because institutional access was provided to all participants.

At the end of each instructional session, students completed an Immediate assessment directly following the learning phase. These assessments were completed individually under controlled classroom conditions, without access to AI systems or other external assistance. Consequently, although students in the AI-supported condition relied extensively on AI during classroom learning, students in both conditions were evaluated under identical independent problem-solving conditions.

Each session concluded with a brief post-assessment clarification phase. In both conditions, instructors could address misunderstandings revealed by the Immediate assessment, but no new curricular content was introduced, and the next topic was not previewed. Because this phase occurred after the assessment used to measure same-session performance, it did not influence the corresponding Immediate assessment outcome.

Accordingly, the study compared two deliberately contrasting forms of classroom instructional mediation while holding the principal curricular, temporal, and assessment conditions constant. Subsequent successful problem-solving performance could therefore be examined across ordinal cognitive difficulty levels and dominant reasoning dimensions using the Problem-Level Cognitive Framework introduced in Section 2.

3.3. Problem Classification and Analytical Dataset

Each problem administered in the Immediate and Consolidated assessments was classified according to the Problem-Level Cognitive Framework introduced and calibrated in Section 2. Classification first established the problem’s ordinal cognitive difficulty level (D1–D7) and then identified its dominant reasoning dimension (F/P/T/C).

The McClave reference corpus was used exclusively for the development and calibration of the framework. None of its problems was used during instruction or included in the Immediate or Consolidated assessments. The instructional and assessment problems addressed comparable statistical content but were developed independently and were never identical to those in the McClave corpus. Consequently, no problem from the reference corpus formed part of the empirical student-performance dataset analyzed in this study.

The analytical dataset consisted exclusively of problem-level records representing each student’s response to each problem included in the Immediate and Consolidated assessments administered during the instructional intervention. The response was coded as 1 if the problem was solved successfully and 0 if it was not. Each row also recorded the student’s instructional condition, the timing of the assessment, and the problem’s ordinal cognitive difficulty level and dominant reasoning dimension. Because each student responded to multiple problems, the dataset contained repeated binary responses from the same students. Student, instructional-group, and instructor identifiers were therefore retained to account for the resulting dependency structure in the statistical analyses.

For students who participated in a given assessment, all assigned problems were included in the analytical dataset; therefore, problems left unanswered or not successfully solved were coded as 0, irrespective of whether a solution attempt was recorded. Assessments missed because of student absence were not included, and no problem-level observations were generated for those occasions. Thus, the dataset comprised all assigned problems from assessments actually completed by participating students, rather than only those problems that were explicitly attempted. No observations meeting these inclusion criteria were excluded, and there were no missing values in the resulting binary outcome variable.

The final analytical dataset contained 13,136 individual problem responses: 7656 from Immediate assessments, 3120 from Consolidated Examination 1, and 2360 from Consolidated Examination 2. Table 8 reports the distribution of these responses across assessment timings, ordinal cognitive difficulty levels, and dominant reasoning dimensions.

Table 8. Distribution of student–problem observations in the analytical dataset.

Analytical Variable	Category	Number of Observations
Assessment timing	Immediate	7656
	Consolidated Examination 1	3120
	Consolidated Examination 2	2360
Ordinal cognitive difficulty	D1	1034
	D2	1493
	D3	1451
	D4	2001
	D5	2136
	D6	3678
	D7	1343
Dominant reasoning dimension	Formal (F)	826
	Procedural (P)	3839
	Theoretical (T)	3493
	Critical (C)	4978
	Total	13,136

Whereas Table 8 describes the distribution of student–problem observations, the structural composition of the assessment problem set itself is documented separately in Supplementary Material S2.1–S2.4.

The specification of the unit of analysis, the resulting dependency structure, and the statistical models used to analyze these observations are presented in Section 3.4.

3.4. Statistical Analysis

3.4.1. Unit of Analysis and Dependency Structure

Following the descriptive characterization of the problem ecosystem, inferential analyses were designed to examine how successful problem solving varied across the proposed cognitive framework and whether these patterns differed between the AI-supported and traditional instructional conditions. Each observation represented the response of one student to one specific problem, with the outcome coded as 1 for a successful solution and 0 for an unsuccessful solution. Each observation therefore represented a binary outcome that was modeled as a Bernoulli variable.

Treating each student’s response to an individual assessment problem as a separate observation does not imply that all observations are statistically independent. Each student attempted multiple problems, meaning that multiple observations originated from the same student. At the same time, each specific problem was attempted by multiple students, meaning that multiple observations also shared the same problem. Consequently, the dataset contains two distinct sources of dependence: observations may be related because they come from the same student or because they involve the same problem. Student and Problem therefore form crossed classification factors, creating a dependency structure that may affect statistical inference if the resulting correlations are sufficiently strong.

3.4.2. Evaluation of Alternative Modeling Strategies

The choice of the inferential model was determined by two fundamental characteristics of the data. First, the outcome variable was binary rather than continuous, requiring the use of a generalized linear model (GLM) rather than a classical general linear model (LM). Unlike the classical linear model, which assumes a continuous response with normally distributed errors, the generalized linear model allows response variables to follow distributions from the exponential family.

Under the assumption of conditionally independent observations, standard binary logistic regression would represent the natural baseline model for the inferential analyses. In practice, logistic regression often remains reasonably robust even when observations exhibit a modest degree of correlation. The present dataset, however, contains two potential sources of dependence whose magnitude cannot be assumed to be negligible.

Importantly, this dependency structure does not correspond to a conventional repeated-measures design in which repeated observations represent multiple measurements of the same experimental unit under identical observational conditions. Instead, each observation corresponds to a different assessment problem, while each problem is simultaneously solved by multiple students. Consequently, the data form a crossed rather than a nested dependency structure, making generalized linear mixed-effects models with crossed random effects more appropriate than standard repeated-measures approaches.

Furthermore, the same assessment problems were administered to students across both instructional conditions and across instructors. Thus, individual problems were not uniquely associated with a particular instructional condition, student group, or instructor. Instead, each problem was answered by students taught under different instructional conditions and by different instructors, creating a crossed rather than nested data structure. Treating problems as nested within instructional groups or instructors would therefore incorrectly imply that each problem occurred in only one group or was administered exclusively by one instructor. To reflect the actual study design, problems were modeled as a separate crossed random effect, allowing the analysis to account for systematic differences in difficulty among problems while estimating instructional-condition and instructor effects from responses to a shared set of assessment problems.

Several statistical approaches have been proposed for analyzing correlated binary data. Among these, generalized estimating equations (GEE) and generalized linear mixed-effects models (GLMMs) are the two most widely used extensions of standard logistic regression. GEE produces population-averaged estimates while accounting for within-cluster correlation through a working correlation structure. In contrast, a GLMM explicitly models the sources of dependency by introducing random effects associated with the clustering variables. As a result, a GLMM produces conditional estimates while simultaneously partitioning the unexplained variability attributable to the different clustering factors.

The objectives of the present study favored the latter approach. The primary aim was not only to estimate the effects of instructional condition, assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, and instructor, but also to obtain valid statistical inference while explicitly accounting for the two crossed sources of heterogeneity arising from students and problems. Because individual students differ in their overall probability of solving statistical problems successfully, and individual problems differ in their intrinsic difficulty beyond the measured explanatory variables, these two sources of variability should be incorporated directly into the statistical model rather than treated merely as nuisance correlations. Generalized linear mixed-effects models provide a principled framework for accomplishing this by estimating the fixed effects of the explanatory variables while simultaneously modeling unexplained variation through random effects.

Based on these considerations, generalized linear mixed-effects modeling was selected as the primary inferential framework. Instructional condition, assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, and instructor were specified as fixed effects because they constituted the explanatory variables of substantive interest. Student and Problem were included as crossed random effects to account for the two sources of dependency arising from repeated observations from the same students and repeated use of the same assessment problems. This specification allowed the fixed effects to be estimated while appropriately accounting for unexplained heterogeneity associated with both clustering factors.

The following section presents the model specification, including the selected fixed and random effects and the evaluation of the prespecified interaction terms.

3.4.3. Model Specification

Based on the methodological considerations presented in the previous sections, the primary inferential analyses were performed using a generalized linear mixed-effects model (GLMM) for binary response data.

Let $Y_{ij}$ denote the outcome of student $i$ on problem $j$, where

$$Y_{ij}=\left\{\begin{array}{cc}1,& \mathrm{if}\mathrm{the}\mathrm{problem}\mathrm{was}\mathrm{solved}\mathrm{successfully},\\ 0,& \mathrm{otherwise}.\end{array}\right.$$

Accordingly,

$$Y_{ij}\sim \mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left(p_{ij}\right),$$

where

$$p_{ij}=P\left(Y_{ij}=1\right)$$

is the probability that student $i$ correctly solves problem $j$.

The generalized linear mixed-effects model relates this probability to the explanatory variables through the logit link function,

$${\eta }_{ij}=\log \left({\displaystyle \frac{p_{ij}}{1-p_{ij}}}\right),$$

where ${\eta }_{ij}$ is the linear predictor, representing the logarithm of the odds that student $i$ correctly solves problem $j$. Equivalently, the probability of a correct solution is given by the inverse logit transformation,

$$p_{ij}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\eta }_{ij}\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\eta }_{ij}\right)}},$$

showing that the linear predictor is mapped onto the probability of a correct solution, which is necessarily bounded between 0 and 1.

The linear predictor was specified as

$$\begin{array}{c}{\eta }_{ij}={\beta }_0+{\beta }_1{\mathrm{M}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{o}\mathrm{d}}_{ij}+{\beta }_2{\mathrm{A}\mathrm{s}\mathrm{s}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}}_{ij}+{\beta }_3{\mathrm{D}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{y}}_{ij}+{\beta }_4{\mathrm{D}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_{ij}+{\beta }_5{\mathrm{I}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}}_{ij}\\ +{\beta }_6{\left(\mathrm{M}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{o}\mathrm{d}\times \mathrm{D}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{y}\right)}_{ij}+{\beta }_7{\left(\mathrm{M}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{o}\mathrm{d}\times \mathrm{D}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\right)}_{ij}+u_i+v_j,\end{array}$$

where $i=1,\dots ,N$ indexes students and $j=1,\dots ,M$ indexes problems. The parameter ${\beta }_0$ denotes the overall intercept. The terms ${\beta }_1,\dots ,{\beta }_7$ denote the sets of regression coefficients associated with the explanatory variables and the two prespecified interaction terms. The random intercept $u_i$ represents the student-specific effect, whereas the random intercept $v_j$ represents the problem-specific effect.

Note. Assessment, Difficulty, Dimension, and Instructor are categorical variables with multiple levels. Consequently, each of these factors is represented in the fitted model by multiple regression coefficients according to the selected contrast coding. For clarity of presentation, each factor is expressed as a single symbolic term in the model equation.

Model estimation was performed using maximum likelihood procedures for generalized linear mixed-effects models with a logit link function. During model development, two theoretically prespecified interaction terms—Instructional Condition × Ordinal Cognitive Difficulty and Instructional Condition × Dominant Reasoning Dimension—were evaluated in alternative model specifications. Neither interaction significantly improved model fit, and both were therefore excluded from the final model. The overall contributions of the fixed effects were evaluated using Type III Wald chi-square tests. The prespecified interaction models were compared with the main-effects model using likelihood-ratio tests and Akaike’s Information Criterion (AIC); Bayesian Information Criterion (BIC) was additionally reported for the final model. These comparisons were used to identify the most parsimonious specification that adequately represented the data while remaining consistent with the theoretical framework of the study.

Model adequacy was assessed using standard diagnostic procedures for generalized linear mixed-effects models. These included evaluation of model convergence, examination of the estimated random-effect variances, assessment of potential overdispersion, and inspection of residual diagnostics.

All analyses were conducted using Minitab^® Statistical Software, version 22.5.1 (64-bit), and RStudio version 2026.07.0+139 “Pacific Dogwood” for Windows, with Quarto version 1.9.38. The R-based analyses used the readxl, dplyr, lme4, car, emmeans, DHARMa, performance, broom.mixed, and writexl packages for data import and processing, fitting and evaluating the generalized linear mixed-effects models, estimating marginal means and model-adjusted probabilities, conducting residual diagnostics, calculating model-performance indices, organizing model outputs, and exporting the results.

4. Results

This section presents the inferential findings obtained from the generalized linear mixed-effects models described in Section 3.4. The analyses examine the relative contributions of instructional condition, assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, and instructor to successful problem solving while accounting for the cross-classified dependency structure arising from repeated observations within both students and problems. The results are presented in three stages: first, the overall effects of the explanatory variables are examined; second, the magnitude and direction of the main effects are summarized; finally, interaction analyses evaluate whether the relationship between instructional condition and problem-solving success varies across ordinal cognitive difficulty levels or dominant reasoning dimensions.

4.1. Overall Factors Associated with Problem-Solving Success

As summarized in Table 9, the generalized linear mixed-effects model revealed a highly asymmetric pattern of effects. The Type III Wald tests (testing whether each fixed effect contributed significantly to the model after adjustment for all remaining predictors) showed statistically significant effects of ordinal cognitive difficulty, dominant reasoning dimension, assessment timing, and instructor. Thus, each of these predictors provided statistically significant explanatory information regarding the probability of successful problem solving beyond that contributed by the remaining predictors. In contrast, instructional condition showed no statistically significant overall effect after adjustment for the remaining explanatory variables (χ² = 0.03, p = 0.856).

Table 9. Overall Generalized Linear Mixed-Effects Model Predicting Problem-Solving Success. Panel A. Type III Wald Tests of Fixed Effects. Panel B. Random Effects. Panel C. Model Performance.

Panel A
Predictor	Wald χ2	df	p
Instructional condition	0.03	1	0.856
Assessment timing	75.73	2	<0.001
Ordinal cognitive difficulty	154.98	6	<0.001
Dominant reasoning dimension	92.79	3	<0.001
Instructor	106.26	2	<0.001
Panel B
Random effect	Variance	SD
Student	0.312	0.558
Assessment problem	0.158	0.397
Panel C
Statistic		Value
AIC		14,600.8
BIC		14,727.9
Marginal R2		0.129
Conditional R2		0.238
Intraclass correlation coefficient (ICC)		0.125

Panel A presents Type III Wald χ² tests evaluating the overall contribution of each fixed effect after adjustment for all remaining predictors. Panel B reports the estimated variance components for the crossed random intercepts representing students and assessment problems. Panel C summarizes the overall performance of the final generalized linear mixed-effects model. Model diagnostics indicated satisfactory fit (DHARMa uniformity test: p = 0.464; dispersion test: p = 0.984; outlier test: p = 0.433).

The estimated random effects demonstrated substantial variability among both students and assessment problems, confirming the importance of accounting for the cross-classified dependency structure. The model also showed satisfactory overall performance, with a marginal R² of 0.129 and a conditional R² of 0.238, indicating that incorporating both fixed effects and crossed random effects substantially improved the explanation of variation in problem-solving success.

Overall, the global mixed-effects model indicates that successful problem solving was more strongly associated with characteristics of the learning and assessment process—including assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, and instructor—than with instructional condition itself.

The following section examines the magnitude and direction of these effects.

4.2. Magnitude and Direction of the Main Effects

Estimated marginal probabilities of successful problem solving were calculated for each level of the fixed effects, adjusted for the remaining predictors included in the model. These adjusted probabilities quantify the magnitude and direction of the effects identified by the GLMM while expressing the results on the probability scale. The complete set of model-estimated probabilities and their corresponding 95% confidence intervals is presented in Table 10, while the principal findings are illustrated in Figure 5 and Figure 6.

Table 10. Model-Estimated Probabilities of Successful Problem Solving. Panel A. Instructional Condition. Panel B. Assessment Timing. Panel C. Ordinal Cognitive Difficulty. Panel D. Dominant Reasoning Dimension. Panel E. Instructor.

Panel A
Instructional Condition	Estimated Probability of Success	95% CI
Traditional	0.731	0.684–0.774
AI-supported	0.735	0.693–0.773
Panel B
Assessment	Estimated probability of success	95% CI
Immediate	0.729	0.664–0.786
Consolidated Examination 1	0.632	0.580–0.681
Consolidated Examination 2	0.818	0.778–0.852
Panel C
Difficulty level	Estimated probability of success	95% CI
D1	0.852	0.808–0.887
D2	0.857	0.823–0.886
D3	0.780	0.737–0.818
D4	0.688	0.639–0.733
D5	0.653	0.601–0.701
D6	0.598	0.546–0.647
D7	0.612	0.549–0.671
Panel D
Dominant reasoning dimension	Estimated probability of success	95% CI
Formal (F)	0.702	0.638–0.760
Procedural (P)	0.668	0.621–0.712
Theoretical (T)	0.744	0.704–0.782
Critical (C)	0.805	0.772–0.834
Panel E
Instructor	Estimated probability of success	95% CI
HP	0.625	0.556–0.689
KK	0.671	0.621–0.718
TA	0.859	0.829–0.885

Figure 5. Adjusted probabilities of successful problem solving by instructional condition and assessment timing. (Points represent estimated marginal probabilities obtained from the generalized linear mixed-effects model, and horizontal lines denote 95% confidence intervals. Estimates are adjusted for all fixed effects and the crossed random effects of students and assessment problems.).

Figure 6. Adjusted probabilities of successful problem solving by ordinal cognitive difficulty, dominant reasoning dimension, and instructor. (Points represent estimated marginal probabilities obtained from the generalized linear mixed-effects model, and horizontal lines denote 95% confidence intervals. Estimates are adjusted for all fixed effects and the crossed random effects of students and assessment problems.).

The estimated probability of successful problem solving was virtually identical under the two instructional conditions (Table 10, Panel A). As illustrated in Figure 5A, the adjusted probabilities were 0.731 for the Traditional condition and 0.735 for the AI-supported condition, with almost completely overlapping 95% confidence intervals. This graphical pattern is fully consistent with the non-significant main effect of instructional condition identified by the GLMM (OR = 1.02, 95% CI: 0.822–1.266, p = 0.856), indicating that no statistically detectable overall difference in problem-solving success was associated with instructional condition.

Assessment timing, in contrast, produced substantial differences in performance (Table 10, Panel B). As shown in Figure 5B, the adjusted probability of success declined from 0.729 during the Immediate assessments to 0.632 in the first Consolidated Examination before increasing markedly to 0.818 in the second Consolidated Examination. The confidence intervals demonstrate that the second consolidated assessment was associated with a considerably higher probability of successful problem solving than the preceding assessment occasions, consistent with the significant main effect of assessment timing identified by the GLMM.

Ordinal cognitive difficulty exhibited a pronounced and systematic performance gradient (Table 10, Panel C). As illustrated in Figure 6A, the adjusted probability of successful problem solving remained high for the two lowest difficulty levels (D1 = 0.852 and D2 = 0.857), declined to 0.780 at D3, and then decreased progressively through D4 (0.688), D5 (0.653), and D6 (0.598). Although D7 showed a slight numerical increase (0.612), its confidence interval overlapped substantially with that of D6, indicating that performance remained comparatively low at D6 and D7, with substantially overlapping confidence intervals. Overall, the adjusted probabilities reveal a clear inverse relationship between ordinal cognitive difficulty and the likelihood of successful problem solving.

Substantial differences were also observed across the four dominant reasoning dimensions (Table 10, Panel D). Figure 6B shows that problems classified as requiring Critical reasoning were associated with the highest adjusted probability of success (0.805), followed by Theoretical (0.744), Formal (0.702), and Procedural reasoning (0.668). These results demonstrate that dominant reasoning dimension remained significantly associated with performance after adjustment for all other variables included in the model. The ordering of these probabilities does not constitute an ordinal hierarchy among the reasoning dimensions, which represent qualitatively different forms of reasoning that can occur across multiple levels of cognitive difficulty.

Instructor-related differences remained evident after statistical adjustment (Table 10, Panel E). As presented in Figure 6C, the adjusted probability of successful problem solving ranged from 0.625 for instructor HP to 0.671 for instructor KK and reached 0.859 for instructor TA. These differences persisted after simultaneously controlling instructional condition, assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, and the crossed random effects of students and assessment problems, indicating that instructor-related variability represented an independent source of variation in problem-solving performance. Instructor was included primarily as a control variable to account for systematic differences among the three participating teachers, thereby allowing the effects of instructional condition and the cognitive characteristics of the assessment problems to be estimated more accurately, rather than to compare the effectiveness of the individual instructors. Because inference was restricted to these three specific instructors rather than to a broader population of teachers, instructor was modeled as a fixed effect rather than as a random effect.

Taken together, the model-estimated probabilities reinforce the conclusions of the generalized linear mixed-effects analysis. Whereas instructional condition was associated with virtually identical probabilities of successful problem solving across the two instructional conditions, ordinal cognitive difficulty, dominant reasoning dimension, assessment timing, and instructor all exhibited substantial and practically meaningful differences. The graphical presentation of the estimated marginal probabilities complements the inferential results by illustrating both the magnitude and precision of the fixed effects while accounting for the multilevel structure of the data. This representation facilitates interpretation of the final GLMM and highlights the more pronounced contribution of problem-related characteristics than of instructional condition in explaining successful problem solving.

4.3. Interaction Effects Between Instructional Condition and Problem Characteristics

Although the generalized linear mixed-effects model revealed no significant overall effect of instructional condition, it remained possible that AI-supported and traditional instruction differed within specific regions of the cognitive framework. For example, AI support might have been particularly effective for problems at lower difficulty levels while becoming less beneficial as cognitive complexity increased, or its effectiveness might have varied across different dominant reasoning dimensions. To examine these possibilities, three interaction models were compared with the main-effects GLMM.

Table 11 summarizes these model comparisons. Each interaction model was evaluated against the main-effects model using a likelihood-ratio (LR) test, which determines whether the additional interaction parameters produce a statistically significant improvement in model fit. Model selection was further supported by the Akaike Information Criterion (AIC), which evaluates the trade-off between goodness of fit and model complexity. Lower AIC values indicate preferable models. The difference between the AIC of an interaction model and that of the reference model (ΔAIC) therefore indicates whether the additional complexity is justified. Positive ΔAIC values imply that the interaction model fits the data less efficiently than the simpler main-effects model.

Table 11. Comparison of Nested Interaction Models with the Main-Effects GLMM.

Model	Compared with	LR Test (χ2)	df	p	AIC	ΔAIC
Main-effects GLMM	—	—	—	—	14,600.77	0.00
+Instructional Condition × Ordinal Cognitive Difficulty	Main-effects GLMM	3.06	6	0.801	14,605.47	+4.70
+Instructional Condition × Dominant Reasoning Dimension	Main-effects GLMM	1.62	3	0.655	14,610.03	+9.26
+Both interaction terms	Main-effects GLMM	4.36	9	0.886	14,614.42	+13.64

Adding the Instructional Condition × Ordinal Cognitive Difficulty interaction did not significantly improve the model (LR χ² = 3.06, df = 6, p = 0.801). The increase of 4.70 AIC units further indicated that introducing six additional interaction parameters did not provide sufficient improvement in model fit to justify the increased model complexity. Consequently, the relationship between ordinal cognitive difficulty and the probability of successful problem solving remained statistically comparable in the AI-supported and traditional instructional conditions. The gradual decline in performance across the seven difficulty levels therefore reflected a common performance pattern across the two conditions rather than a statistically detectable condition-specific pattern. Descriptive, unadjusted performance profiles across ordinal cognitive difficulty levels, assessment timings, and instructional conditions are provided in Supplementary Material S2.5 (Figure S2.5.1).

The Instructional Condition × Dominant Reasoning Dimension interaction yielded a similar result. The likelihood-ratio test was not statistically significant (LR χ² = 1.62, df = 3, p = 0.655), while the AIC increased by 9.26 units relative to the reference model. Thus, allowing the association between instructional condition and performance to differ across the Formal, Procedural, Theoretical, and Critical reasoning dimensions produced a more complex model without improving its explanatory performance. Although these reasoning dimensions differed substantially in their overall probabilities of successful problem solving (Section 4.2), those differences remained essentially unchanged across the two instructional conditions. Descriptive, unadjusted performance profiles across dominant reasoning dimensions, assessment timings, and instructional conditions are provided in Supplementary Material S2.6 (Figure S2.6.1).

Finally, the model containing both interaction structures simultaneously also failed to outperform the simpler main-effects model (LR χ² = 4.36, df = 9, p = 0.886). Despite introducing nine additional interaction parameters, the model exhibited the largest AIC increase (ΔAIC = +13.64), indicating the poorest balance between model fit and complexity among the models considered. Taken together, both the likelihood-ratio tests and the information-criterion comparisons consistently supported retaining the simpler main-effects GLMM as the final analytical model.

4.4. Summary of the Main Findings

The generalized linear mixed-effects analyses consistently indicated that successful problem solving was more strongly associated with problem-related characteristics than with instructional condition. Ordinal cognitive difficulty, dominant reasoning dimension, assessment timing, and instructor all exhibited significant associations with performance, whereas the overall effect of instructional condition remained non-significant after accounting for these factors and the crossed random effects of students and assessment problems.

The estimated marginal probabilities demonstrated systematic performance differences across difficulty levels, reasoning dimensions, assessment occasions, and instructors. In contrast, the probabilities associated with the AI-supported and traditional instructional conditions were nearly identical. Furthermore, none of the interaction analyses indicated that the association between instructional condition and performance varied across ordinal cognitive difficulty levels or dominant reasoning dimensions. The observable problem-solving patterns identified within the proposed framework therefore remained statistically comparable across both instructional environments.

These findings provide the empirical basis for the interpretation presented in the following Discussion section, where the implications of the proposed problem-level cognitive framework and the limited contribution of instructional condition are considered in the broader context of AI-supported statistics education.

5. Discussion

The principal contribution of the present study is the development and initial empirical demonstration of a two-dimensional Problem-Level Cognitive Framework that treats ordinal cognitive difficulty and dominant reasoning demand as analytically distinct characteristics of individual assessment problems. The framework does not propose that difficulty or different forms of reasoning are previously unrecognized educational phenomena. Rather, its contribution lies in formalizing these characteristics within a joint problem-level analytical structure and making it possible to examine how successful performance is distributed across both dimensions simultaneously.

The framework was applied in a semester-long introductory statistics course involving substantially different AI-supported and traditional instructional conditions. This setting provided a particularly demanding first empirical application because statistical problem solving involves heterogeneous forms of reasoning, multiple levels of cognitive complexity, repeated student–problem interactions, and potentially substantial variation associated with students, assessment problems, instructors, and assessment occasions. Within this environment, the framework produced interpretable, theoretically coherent, and statistically distinguishable performance patterns. Ordinal cognitive difficulty and dominant reasoning dimension were both significantly associated with successful problem solving, while their relationships with performance remained statistically comparable across the two instructional conditions. The following sections discuss the contribution of the framework, the cognitive structure revealed through its empirical application, the absence of the expected instructional asymmetry, and the implications for AI-supported statistics education.

5.1. Contribution of the Problem-Level Cognitive Framework

Existing educational theories and taxonomies have long distinguished among levels of cognitive demand and among different forms of knowledge or reasoning. However, these distinctions are frequently used as instructional objectives, learner-related constructs, or hierarchical classifications in which qualitatively different cognitive processes are positioned along a single progression. The framework proposed in the present study adopts a different analytical perspective. It locates both dimensions at the level of the individual assessment problem and separates the level of cognitive complexity from the dominant form of reasoning required for successful solution.

This separation extends, rather than replaces, established approaches to cognitive classification. The procedural–conceptual distinction differentiates execution from understanding (Rittle-Johnson & Alibali, 1999), while educational taxonomies distinguish levels of cognitive demand (Anderson & Krathwohl, 2001). Research on mathematical representation and statistical reasoning further demonstrates that successful performance may depend on qualitatively different representational, inferential, and evaluative processes (Ainsworth, 2006; Duval, 2026; Garfield & Ben-Zvi, 2009; Wild & Pfannkuch, 1999). The present framework brings these complementary distinctions to the level of individual assessment problems by representing reasoning demand and structural complexity separately within the same analytical system.

This distinction is important because reasoning type and cognitive difficulty do not represent interchangeable properties. A problem dominated by Procedural reasoning is not necessarily cognitively simple, just as a problem dominated by Critical reasoning is not necessarily highly complex. Each reasoning dimension may occur across multiple levels of difficulty, while problems at a similar level of cognitive difficulty may require fundamentally different forms of reasoning. Treating ordinal cognitive difficulty and dominant reasoning demand as complementary but analytically independent dimensions therefore avoids collapsing qualitatively different problem characteristics into a single hierarchy.

The framework also changes the unit at which learning performance is examined. Overall test scores, course averages, and group-level comparisons summarize achievement, but they provide limited information about the particular cognitive demands under which successful performance occurs. By classifying individual problems along both dimensions and analyzing student–problem interactions, the proposed framework makes it possible to identify where performance changes across the cognitive landscape of an assessment ecosystem. It therefore does not replace established measures of achievement; instead, it provides a structured means of decomposing those outcomes according to the cognitive characteristics of the problems from which they are constructed.

The empirical application supports the analytical usefulness of this distinction. Ordinal cognitive difficulty and dominant reasoning dimension both remained significantly associated with successful problem solving after simultaneous adjustment for instructional condition, assessment timing, instructor, and the crossed random effects of students and assessment problems. The framework therefore revealed differentiated performance patterns that could not have been obtained from a single comparison of overall achievement between the instructional conditions. Importantly, this does not constitute comprehensive validation of the proposed framework. The present study represents its first empirical demonstration within one disciplinary and institutional setting. Nevertheless, the emergence of interpretable and statistically distinguishable patterns provides initial evidence that the framework captures meaningful aspects of the cognitive structure of assessment problems.

The methodological contribution is reinforced by the use of crossed random-effects modeling. Students responded to multiple problems, and the same assessment problems were completed by multiple students. Accounting for variation associated with both students and problems allowed the fixed associations of the two framework dimensions to be examined without treating repeated student–problem observations as independent. In this respect, the combination of explicit problem classification and an appropriate multilevel analytical model provides a general strategy for investigating learning performance as a structured set of interactions between learners and cognitively differentiated problems.

The value of the framework therefore lies not in rediscovering that some problems are more difficult than others or that different problems require different forms of reasoning. Its contribution is to formalize these two properties as independent problem-level dimensions, operationalize them within a reproducible classification procedure, and incorporate them jointly into an empirical model of successful problem solving. Rather than treating learning performance as a unitary outcome, the framework makes its underlying problem-level cognitive organization analytically visible.

5.2. Cognitive Structure of Successful Problem Solving

The first empirical application of the framework revealed a pronounced relationship between ordinal cognitive difficulty and successful problem solving. Adjusted probabilities remained high at D1 and D2, declined at D3, and decreased more substantially from D4 onward. Performance remained comparatively low at D6 and D7, although the small numerical increase from D6 to D7 and the substantial overlap between their confidence intervals indicate that these two levels should not be interpreted as empirically distinct performance regimes in the present dataset.

This overall pattern provides initial empirical support for the proposed difficulty structure as a representation of meaningful differences in the cognitive demands of statistical problems. The results are consistent with cognitive and developmental perspectives according to which increasing problem complexity involves changes in the organization and coordination of cognitive operations rather than merely greater effort or a larger quantity of knowledge (Fischer & Bidell, 2007). In the present framework, progression from D1 to D7 represents increasingly demanding combinations of recognition, procedural execution, method selection, inferential reasoning, interpretation, and evaluative judgment. The observed decline in successful performance suggests that this ordinal structure captured practically relevant differences among the assessment problems.

At the same time, the findings do not establish that the seven levels constitute a universally applicable psychological scale. The categories were applied to a particular set of problems in introductory statistics, and their empirical differentiation inevitably depends partly on the composition of that problem set and the learner population examined. The present evidence is therefore best interpreted as an initial demonstration that the proposed ordering produces a theoretically coherent performance gradient. Replication across other assessment ecosystems is required to determine whether the complete seven-level structure remains stable and sufficiently differentiated in other contexts.

Dominant reasoning dimension provided an additional and analytically distinct organization of performance. Even after adjustment for ordinal cognitive difficulty and the other variables in the model, the probability of successful problem solving differed significantly among Formal, Procedural, Theoretical, and Critical problems. This result is important because it demonstrates that performance was not organized by cognitive difficulty alone. Problems classified at comparable levels of complexity may still produce different success patterns depending on the dominant form of reasoning they require.

The comparatively high adjusted probability associated with Critical problems deserves particular attention. This result should not be interpreted as showing that Critical reasoning is inherently easier than the other forms of reasoning or that it occupies the highest position in an ordinal hierarchy. The four reasoning dimensions represent qualitatively different cognitive demands and are analytically independent of the D1–D7 difficulty scale. Critical problems can occur at several difficulty levels, just as Formal, Procedural, and Theoretical problems can.

The observed differences may partly reflect the composition and formulation of the problems assigned to each dimension. Some Critical problems in the present assessment ecosystem may have permitted successful solution through interpretation, evaluation, or strategic recognition without requiring the same degree of sequential precision as certain Formal or Procedural problems. Formal and Procedural problems may be particularly sensitive to executional errors when success depends on symbolic consistency, correct method implementation, or the accurate completion of several connected operations. These explanations remain provisional because the framework classified the dominant demand embedded in each problem but did not directly observe the cognitive processes used by individual students. The findings therefore establish differences among the classified problem sets, not an inherent hierarchy of difficulty among the reasoning dimensions.

Considered together, the two dimensions provided complementary information. Ordinal cognitive difficulty captured the progression of cognitive complexity, whereas dominant reasoning dimension differentiated the principal form of cognitive activity required by a problem. Their simultaneous contribution to the model supports the central conceptual premise of the framework: cognitive complexity and reasoning demand should be examined together without reducing one to the other. This two-dimensional representation offers a more differentiated view of successful problem solving than either a single difficulty scale or a categorical reasoning taxonomy could provide alone.

5.3. The Missing Instructional Asymmetry

The AI-supported and traditional conditions were intentionally designed to create substantially different classroom experiences. The traditional condition relied on teacher-mediated conceptual explanation and guided instruction, whereas the AI-supported condition placed considerably greater responsibility on students to work with AI-based support and the available learning resources. Under such differentiated conditions, a marked divergence in successful problem solving might reasonably have been expected, particularly for problems involving greater cognitive complexity (Kirschner et al., 2006).

The generalized linear mixed-effects analyses did not support this expectation. Instructional condition showed no statistically significant overall association with successful problem solving after adjustment for assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, instructor, and the crossed random effects of students and assessment problems. The adjusted probabilities for the two conditions were nearly identical. Furthermore, neither the Instructional Condition × Ordinal Cognitive Difficulty interaction nor the Instructional Condition × Dominant Reasoning Dimension interaction improved model fit. No statistically detectable condition-specific divergence therefore emerged across either dimension of the proposed framework.

This absence of instructional asymmetry is particularly informative when interpreted through the framework. A conventional comparison of overall achievement would have shown only that the two instructional conditions produced similar aggregate performance. The problem-level analysis provides a more differentiated conclusion: the relationship between successful problem solving and both cognitive complexity and dominant reasoning demand remained statistically comparable across the two instructional environments. Thus, the framework showed not merely that overall performance was similar, but that the observable cognitive organization of performance did not differ detectably between the conditions.

The absence of a statistically significant overall instructional-condition effect is consistent with the heterogeneous findings reported in the broader literature on AI-supported learning, where positive, negligible, negative, and context-dependent effects have all been observed (Wu, 2026; Chiu et al., 2026; T. Zhang et al., 2026; Lehtinen et al., 2026). It also accords with research on instructional mediation showing that the effectiveness of instructional support depends on the interaction among learner characteristics, task demands, and the form of guidance provided rather than on the presence or absence of a particular instructional medium alone (VanLehn, 2011; Ma et al., 2014). The present results add a problem-level perspective to this literature by showing that the absence of an overall condition effect was accompanied by statistically comparable performance patterns across both cognitive difficulty and reasoning dimensions.

The interpretation of this finding nevertheless requires caution. The consolidated examinations reflected the complete learning process occurring both within and outside classroom sessions. Both groups had access to the same Moodle-based materials, and students were not prohibited from independently using AI tools while preparing for the examinations. The two conditions should therefore not be understood as completely isolated “AI” and “non-AI” environments. Similarities during the consolidated assessments may partly reflect overlapping resources and self-regulated learning outside the formal classroom conditions.

These contextual factors cannot fully explain the observed pattern, however, because the absence of a substantial instructional difference also appeared in the Immediate assessments administered directly after classroom instruction. These assessments were completed before extended independent preparation could occur and therefore provide the most direct available comparison of the differentiated classroom environments. Even under these more immediate conditions, the expected large performance divergence did not emerge.

The absence of a significant instructional-condition effect should not be interpreted as proof that AI-supported and traditional instruction are equivalent. The study was not designed as an equivalence or non-inferiority trial, and the quasi-experimental implementation within authentic university instruction does not support such a conclusion. Nor does the finding imply that the two environments produced identical learning processes. Students may have used different strategies, forms of support, or cognitive pathways while reaching statistically comparable outcomes. The results establish only that no overall or framework-specific difference in successful problem solving was detected under the conditions examined.

The theoretically important finding is therefore not simply the absence of a group difference. It is that the newly proposed framework revealed a coherent problem-level cognitive structure that remained visible across two substantially different instructional environments. The AI-supported and traditional conditions served as a demanding empirical context in which to examine whether the framework could identify stable patterns beyond a single form of instruction. The results provide initial evidence that it can.

5.4. Implications for AI-Supported Statistics Education

The findings have implications for both the analysis and design of AI-supported statistics education. Most importantly, they demonstrate the limitations of evaluating AI-supported learning solely through comparisons of total scores or mean performance. Similar overall achievement can conceal important differences in where students succeed or struggle across levels of cognitive complexity and forms of reasoning. A problem-level cognitive analysis can reveal whether an instructional approach is associated with localized advantages, weaknesses, or changes that disappear when performance is aggregated.

In the present study, no such condition-specific pattern was detected. The association between successful problem solving and both framework dimensions remained statistically comparable across AI-supported and traditional instruction. This indicates that access to AI-supported learning was not accompanied by the disappearance of performance differences associated with the cognitive characteristics of the assessment problems. Students in both environments continued to encounter greater difficulty in the higher-complexity regions of the framework, and both groups exhibited differentiated success patterns across the four reasoning dimensions.

These results suggest that AI support should not be assumed to eliminate or fundamentally simplify the cognitive demands embedded in statistical problems. AI may change how learners obtain explanations, receive feedback, explore solution strategies, or complete procedural operations, but the structural demands of method selection, inference, interpretation, and evaluation remain relevant to observable performance. The proposed framework offers a way to identify the regions of an assessment ecosystem in which these demands become most consequential.

The findings also raise the possibility that AI-supported learning may redistribute some instructional functions. Procedural demonstration, routine practice, and immediate feedback could potentially be supported through AI-assisted independent work, allowing teacher-mediated instruction to concentrate more strongly on conceptual understanding, interpretation of statistical evidence, evaluation of assumptions, and critical judgment. This interpretation is compatible with the absence of substantially weaker performance in the AI-supported condition, including during the Immediate assessments. However, the study did not directly manipulate the allocation of instructional functions or examine the mechanisms through which students used AI. The possible redistribution of instructional roles should therefore be treated as a hypothesis for future educational research rather than as an effect demonstrated by the present analysis.

This interpretation is compatible with previous work distinguishing the educational functions of procedural guidance from those requiring conceptual explanation, interpretation, and higher-order judgment (Garfield & Ben-Zvi, 2009; VanLehn, 2011; Ma et al., 2014). From this perspective, AI support need not be understood simply as a substitute for teacher mediation. Its educational relevance may instead depend on which components of statistical problem solving can be supported effectively through automated guidance and which continue to require explicit conceptual or evaluative mediation.

The framework may be particularly useful for such future investigations because it provides a structured basis for connecting instructional design to the cognitive characteristics of problems. Instead of asking only whether AI-supported instruction improves overall performance, researchers can examine whether its association with performance varies across specific difficulty regions or reasoning demands. This can support more precise questions about where AI assistance is beneficial, where teacher mediation remains especially important, and where apparently successful performance may depend on different underlying cognitive processes.

Outcome data alone cannot determine whether students in the two conditions reached similar answers through similar forms of reasoning. Process-oriented evidence—including AI interaction logs, solution traces, think-aloud protocols, cognitive interviews, and analyses of error patterns—would be needed to determine whether statistically comparable outcomes conceal different strategies, levels of understanding, or forms of dependence on external support. Combining these sources with the Problem-Level Cognitive Framework could extend the present outcome-oriented analysis into a more comprehensive account of how cognitive demands are encountered and managed within AI-supported learning.

5.5. Limitations and Future Research

Several limitations should be considered when interpreting the present findings and evaluating the proposed framework.

First, the framework was designed to classify the cognitive characteristics of assessment problems and analyze observable problem-solving performance. It does not directly measure learners’ internal cognitive representations, solution strategies, conceptual structures, or the processes through which successful answers were produced. The empirical patterns should therefore be interpreted as relationships between classified problem characteristics and observable success rather than as direct evidence of the complete cognitive architecture underlying statistical reasoning.

Second, the framework has so far been applied within a single introductory statistics course at one institution. Although the semester-long authentic setting strengthens ecological validity, it limits generalizability. The observed structure may depend partly on the statistical content, assessment design, prior preparation of the students, and institutional context. Applications across other statistical topics, levels of study, educational systems, and learner populations are necessary to evaluate the robustness and transferability of the framework.

Third, the classification of problems necessarily depends on the operational definitions adopted for ordinal cognitive difficulty and dominant reasoning dimension. The use of independent expert evaluations and consensus-based resolution of disagreements strengthened the classification process, but alternative taxonomies or operational criteria might produce different assignments. Future research should examine inter-rater reliability across less closely connected evaluator groups and compare the proposed classifications with alternative cognitive frameworks.

Fourth, the number and content of problems were not evenly distributed across all combinations of difficulty levels and reasoning dimensions. The adjusted differences among categories may therefore partly reflect the composition of the particular assessment ecosystem. Further applications should include more systematically balanced problem sets where feasible, allowing the two dimensions and their possible interrelationships to be examined across a wider and more evenly represented range of cognitive demands.

Fifth, the instructional conditions were not completely isolated. Both groups had access to common Moodle-based materials, and independent AI use during preparation for the consolidated examinations was not prohibited. Consequently, the later assessments represented the combined influence of classroom instruction and broader self-regulated learning. Although the Immediate assessments reduced this concern, they did not eliminate all possible overlap between the learning experiences of the two groups.

Sixth, the comparison was quasi-experimental and implemented within authentic instructional practice. Although the analysis adjusted for instructional condition, assessment timing, ordinal cognitive difficulty, dominant reasoning dimension, instructor, and the crossed effects of students and assessment problems, unmeasured differences among students or course sections may still have influenced performance. The non-significant instructional-condition effect therefore cannot establish equivalence or non-inferiority between AI-supported and traditional instruction.

Seventh, instructor remained significantly associated with successful problem solving after adjustment for the other modeled variables. Instructor was included as a fixed control variable, and the study was neither designed nor powered to compare teacher effectiveness. With only three participating instructors, these estimates may capture multiple unmeasured contextual differences and should not be interpreted as rankings or evaluations of the individual teachers.

Despite these limitations, the study provides a substantive first empirical demonstration of the Problem-Level Cognitive Framework. Its application produced a theoretically coherent difficulty gradient, differentiated patterns across dominant reasoning dimensions, and a more detailed understanding of performance than could have been obtained from aggregate comparisons alone. Future research should test whether these patterns can be replicated across independently constructed problem sets, disciplines, institutions, and instructional environments. Particular attention should be given to the stability of the D1–D7 structure, the empirical independence of difficulty and reasoning dimensions, and the extent to which the framework can identify localized instructional effects that remain hidden in overall achievement measures.

Combining the framework with process-oriented evidence would be especially valuable. Direct observations of reasoning, solution pathways, AI interactions, and error structures could clarify whether the problem classifications correspond to the cognitive processes actually employed by learners. Such research would move beyond the initial analytical demonstration provided here and support a more comprehensive evaluation of the framework’s validity, robustness, and educational usefulness.

6. Conclusions

The principal contribution of this study is the development and initial empirical demonstration of a two-dimensional Problem-Level Cognitive Framework that separates ordinal cognitive difficulty from dominant reasoning demand. Rather than treating successful problem solving as a single overall outcome, the framework classifies individual assessment problems along two complementary cognitive dimensions and makes it possible to examine how performance is distributed across them. Its novelty lies not in proposing that problems differ in difficulty or reasoning requirements, but in formalizing these properties as analytically independent problem-level characteristics and incorporating them jointly into an empirical model of learning performance.

In practical terms, the framework allows instructors and assessment designers to distinguish whether lower performance is associated primarily with increasing solution complexity or with the particular type of reasoning required by a problem.

The framework was applied in a semester-long introductory statistics course involving substantially different AI-supported and traditional instructional conditions. Within this demanding first application, both ordinal cognitive difficulty and dominant reasoning dimension were significantly associated with successful problem solving after adjustment for assessment timing, instructor, and the crossed random effects of students and assessment problems. The resulting patterns were interpretable and theoretically coherent: performance was generally lower toward the higher-complexity regions of the framework, while the four reasoning dimensions exhibited distinct adjusted probabilities of success without forming an ordinal hierarchy.

The empirical application also demonstrated that the framework can move beyond a simple comparison of overall achievement. Although instructional condition showed no statistically significant overall association with successful problem solving, the problem-level analysis further established that no condition-specific variation was detected across either ordinal cognitive difficulty or dominant reasoning dimension. Thus, the observable cognitive organization of successful performance remained statistically comparable across the AI-supported and traditional environments. This finding does not establish the equivalence of the two instructional approaches, but it demonstrates the framework’s capacity to identify structural regularities in performance under substantially different instructional conditions.

From an educational perspective, the results indicate that AI-supported learning was not accompanied by the disappearance of the cognitive demands embedded in statistical problems. AI may potentially redistribute some instructional functions by supporting procedural demonstration, routine practice, and immediate feedback, while teacher-mediated instruction concentrates more strongly on conceptual understanding, interpretation, evaluation of assumptions, and critical judgment. Determining whether such redistribution occurs—and whether similar outcomes are achieved through different cognitive processes—will require experimental and process-oriented evidence beyond the outcome data analyzed here.

Overall, the study provides initial evidence that the proposed framework can serve as a systematic analytical tool for identifying where and in what forms cognitive demands are associated with successful problem solving. Further applications across independently constructed problem sets, disciplines, educational settings, and learner populations are required to evaluate its robustness and transferability. Nevertheless, the present study demonstrates the framework’s central analytical promise: rather than reducing learning performance to a single aggregate result, it makes the underlying problem-level cognitive structure of successful performance empirically visible.