MCD-Temporal: Constructing a New Time-Entropy Enhanced Dynamic Weighted Heterogeneous Ensemble for Cognitive Level Classification

Abstract

Accurate classification of cognitive levels in instructional dialogues is essential for personalized education and intelligent teaching systems. However, most existing methods predominantly rely on static textual features and a shallow semantic analysis. They often overlook dynamic temporal interactions and struggle with class imbalance. To address these limitations, this study proposes a novel framework for cognitive-level classification. This framework integrates time entropy-enhanced dynamics with a dynamically weighted, heterogeneous ensemble strategy. Specifically, we reconstruct the original Multi-turn Classroom Dialogue (MCD) dataset by introducing time entropy to quantify teacher–student speaking balance and semantic richness features based on Term Frequency-Inverse Document Frequency (TF-IDF), resulting in an enhanced MCD-temporal dataset. We then design a Dynamic Weighted Heterogeneous Ensemble (DWHE), which adjusts weights based on the class distribution. Our framework achieves a state-of-the-art macro-F1 score of 0.6236. This study validates the effectiveness of incorporating temporal dynamics and adaptive ensemble learning for robust cognitive level assessment, offering a more powerful tool for educational AI applications.

1. Introduction

Accurately assessing students’ cognitive levels is a critical component of achieving personalized teaching and remains a core challenge in the field of educational artificial intelligence [1]. This study tackles a key research question: How can we effectively improve cognitive level classification in instructional dialogues? The goal is to enhance both classification accuracy and robustness. This will be achieved by integrating temporal dynamic features, such as time entropy and semantic richness. Additionally, a dynamically weighted heterogeneous ensemble strategy will be used. The approach specifically addresses the common challenge of class imbalance.

Nowadays, main research methods suffer from significant limitations. Current assessment systems overly rely on static textual features [2] and a shallow semantic analysis [3], and fail to adequately consider the systematic impact of two key factors in the teaching process: dynamic temporal behavioral features and differences in teaching styles [4] on learning effectiveness. Furthermore, models struggle with the prevalent class imbalance [5], which leads to poor performance on minority classes. Although Chen et al. (2024) released the MCD dataset [6], providing an essential foundational resource for dialogue understanding research, but it lacks quantitative representation of temporal features (e.g., time entropy calculation [7]) and fails to establish a classification framework for teaching styles, severely limiting the optimization potential of adaptive teaching strategies in educational practice.

As illustrated in Figure 1a, the research motivation diagram, traditional methods rely on static features and fractional timestamps [8], making it difficult to comprehensively characterize the essence of instructional dialogues, which hinders the enhancement of student cognitive levels and teaching quality.

To address these limitations, this study proposes a novel framework (Figure 1b) that shifts the paradigm from static feature analysis to modeling instructional dynamics. Building on established uses of entropy to quantify patterns in educational settings [9], we introduce time entropy, a metric that operationalizes this shift by quantifying the balance of teacher–student speaking turns. Combined with semantic richness [10] and other dynamic features, this measure enables the direct assessment of teaching quality and student engagement as interactions unfold.

Moreover, this study starts with comprehensive data preprocessing and feature extraction, followed by teaching style based on the time entropy calculation, a semantic analysis, identification of golden speeches via a weighted scoring mechanism, and finally an ensemble modeling strategy utilizing four machine learning algorithms. The algorithms include Extreme Gradient Boosting (XGBoost [11]), Light Gradient Boosting Machine (LightGBM [12]) combined with stratified k-Fold cross-validation [13], and Synthetic Minority Over-Sampling Technique (SMOTE [14]) oversampling techniques to address the class imbalance problem. The comparison of the deficiencies in the previous work and our contributions are shown in Appendix A.

Our contributions in this work are summarized as follows:

1.

We propose time entropy and semantic richness features to dynamically quantify teaching style and dialogue quality, moving beyond static textual analysis.

2.

We design a DWHE that adaptively fuses XGBoost [11] and LightGBM [12], innovating in handling class imbalance with a data-driven weighting strategy.

3.

We construct the MCD-temporal dataset, enhancing the original MCD dataset with nine dynamic features to support temporal-semantic research in education.

4.

We demonstrate superior and robust performance, achieving a macro-F1 score of 0.6236 and showing significant gains on the challenging minority class.

2. Related Work

2.1. Current State and Challenges of Cognitive Assessment in Instructional Dialogues

Current research on cognitive level classification in instructional dialogues predominantly relies on static textual features (e.g., word frequency [15], syntactic structures [16]), and a shallow semantic analysis, and fails to model the dynamic interactions inherent in teaching processes. Most studies employ traditional machine learning methods [17] or single deep learning models and overlook the temporal structure of dialogues and the moderating effect of teaching style. Moreover, the prevalent issue of class imbalance further exacerbates model training, leading to poor performance on minority classes. Although existing resources like the MCD dataset provide a foundation for multi-turn dialogue understanding, they lack dynamic features such as time entropy and quantitative teaching style representations, which limits the adaptability and interpretability of models in real educational settings [18]. Our work addresses these gaps by reconstructing the MCD dataset into MCD-temporal dataset, incorporating temporal dynamics and semantic richness features and proposing a dynamic ensemble framework specifically designed for imbalanced educational data.

2.2. Application and Limitations of Temporal Dynamic Features in the Educational Analysis

Temporal dynamic features (e.g., speech rhythm, response latency) have recently been introduced into the educational analysis to capture temporal patterns in teaching interactions. Time entropy, as a metric for measuring the balance of teacher–student speaking time, has shown preliminary correlations with teaching style and cognitive engagement. However, existing studies often rely on theoretical thresholds for style categorization, lack data-driven adaptive classification methods, and fail to deeply integrate temporal features with semantic information. Furthermore, traditional approaches overlook the synergy between temporal features and semantic richness. This oversight prevents them from building a unified representation for fine-grained cognitive level assessment. In contrast, our methodology introduces a data-driven percentile-based approach for teaching style classification and jointly models temporal entropy with latent semantic density to create a more holistic feature set for cognitive level prediction.

2.3. Advances in Ensemble Learning for Imbalanced Educational Data

To address class imbalance in educational data, ensemble learning methods [19] such as XGBoost [11] and LightGBM [12] are widely adopted, which leverage weighted loss functions or sampling strategies [20] to improve minority class recognition. However, current methods mostly employ homogeneous ensembles or static weighting schemes and fail to fully exploit the complementary strengths of heterogeneous models. Particularly in instructional dialogues, significant variations in data distribution across teaching style necessitate dynamic weight adjustment mechanisms. The proposed DWHE framework innovatively combines the cost-sensitive learning [21] of XGBoost with the automatic class balancing of LightGBM [12] and employs a data-driven weighting strategy to achieve more accurate and robust classification across cognitive levels.

3. Dataset

3.1. Dataset Reconstruction and Enhancement

This study reconstructs and enhances the MCD dataset to form the MCD-temporal dataset. The MCD dataset contains 5226 instructional dialogue samples, each characterized by 25 static features, with labels for three cognitive levels (mastery, understanding, and apprentice). While retaining all original features, the new dataset adds nine dynamic features. The specific process is illustrated in Figure 2.

The schematic diagram of the MCD-temporal dataset reconstruction process illustrates a three-step reconstruction process. The process involves: (1) filtering out invalid dialogue segments and separating utterances by speaker type; (2) calculating the four main types of numerical features: temporal features, semantic features, linguistic features, and interaction features; and (3) merging all new features with the original feature table through ID connection, handling missing values, and completing dataset reconstruction. This process makes the new features systematic and reproducible.

A comparison between the original and enhanced datasets is presented in

Table 1. Comparison of feature categories between the MCD dataset and the proposed MCD-temporal dataset.

Feature Category	MCD	MCD-Temporal (Ours)
Basic Statistical Features	Dialogue turn num, etc.	Remain unchanged.
Temporal Features	Elapsed time std, etc.	Time Entropy
		Teaching Style
Semantic Features	N-gram similarity, etc.	Semantic richness
Student Features	Student x num, etc.	Lexical Diversity
		Explanation Ratio
		Student Mastery Score
Teacher Features	Key question words, etc.	Question Ratio
		Explanation Ratio
		Guidance Ratio
		Teacher Guidance Score
Interaction Features	×	Student Response Depth
		Teacher Question Quality
		Dialogue Coherence
Comprehensive Evaluation Features	×	Golden Speech Ratio
		Interaction Quality Score

.

Table 1 systematically compares the feature systems of the two datasets. The left side lists the static features of the original MCD dataset, while the right side presents nine new dynamic features that cover multiple dimensions such as temporal interaction, semantic depth, linguistic expression, and interaction quality [22]. The new features focus on dynamic interaction patterns and linguistic cognitive characteristics during the teaching process [23], and provide richer temporal and semantic information for cognitive level identification.

The MCD-temporal dataset introduces the following new features:

(1)

Temporal Features: Time entropy quantifies the teacher–student speaking balance to categorize instructional styles, ranging from lecture-heavy (low entropy) to dialogic and student-active (high entropy). Teaching style classifies patterns, enabling tailored analysis of instructional efficacy.

(2)

Semantic Features: Semantic richness evaluates information density, signaling dialogue substance and cognitive demand.

(3)

Student Features: Lexical diversity reflects vocabulary breadth, pointing to language development needs. Explanation ratio measures logical elaboration, tracing the growth of structured reasoning. Student mastery score evaluates proficiency, informing personalized learning paths.

(4)

Teacher Features: Question ratio quantifies questioning, assessing the use of inquiry. Explanation ratio assesses exposition, measuring direct knowledge transfer. Guidance ratio evaluates prompting, gauging scaffolding effectiveness. Teacher guidance score integrates impact, benchmarking instructional quality.

(5)

Interaction Features: Student response depth measures relevance, capturing moment-to-moment engagement quality. Teacher question quality identifies open-ended prompts, highlighting catalysts for critical thinking. Dialogue coherence evaluates continuity, ensuring the conversation builds knowledge logically.

(6)

Comprehensive Evaluation Features: Golden speech ratio detects pivotal responses, spotlighting transformative learning moments. Interaction quality score rates effectiveness, quantifying the overall pedagogical value of a dialogue.

3.2. Detailed Feature Computation

This section provides explicit computational formulas for all new dynamic features in the MCD-temporal dataset.

(1) Time Entropy: For each individual dialogue, let $T_{\mathrm{teacher}}$ be the total teacher speaking time and $T_{\mathrm{student}}$ be the total student speaking time. The teacher speaking proportion is computed as the following:

$$p_t={\displaystyle \frac{T_{\mathrm{teacher}}}{T_{\mathrm{teacher}}+T_{\mathrm{student}}}}.$$

(1)

applying Laplace smoothing ($ϵ=1\times {10}^{-10}$), the time entropy is calculated as:

$$H\left(T\right)=-\left[(p_t+ϵ)·{log}_2(p_t+ϵ)+(1-p_t+ϵ)·{log}_2(1-p_t+ϵ)\right].$$

(2)

(2) Teaching style is assigned per dialogue and classified as spoon-feeding, balanced, and inquiry-based using the 25th ($Q_{25}$) and 75th ($Q_{75}$) percentiles of all $H\left(T\right)$ values.

(3) Semantic richness evaluates the informational density through TF-IDF vectorization [24]:

$$\mathrm{Semantic}\mathrm{Richness}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left({\displaystyle \frac{1}{M_i}}\sum _{j=1}^{M_i}\mathrm{TF}-\mathrm{IDF}\left({\mathrm{text}}_{ij}\right)\right),$$

(3)

where N is the number of dialogue samples and $M_i$ is the number of valid text segments in the i-th sample.

(4) Lexical diversity measures vocabulary variation in student responses:

$$\mathrm{Lexical}\mathrm{Diversity}={\displaystyle \frac{|\mathrm{unique}\_\mathrm{words}|}{\mathrm{total}\_\mathrm{words}}},$$

(4)

where $\mathrm{unique}\_\mathrm{words}$ is the set of distinct words and $\mathrm{total}\_\mathrm{words}$ is the total word count.

(5) Explanation ratio for students quantifies explanatory language usage:

$$\mathrm{Explanation}\mathrm{Ratio}={\displaystyle \frac{\mathrm{count}\_\mathrm{explanatory}}{\mathrm{total}\_\mathrm{student}\_\mathrm{utterances}}},$$

(5)

counting utterances with explanatory keywords (e.g., “because”, “so”, “therefore”).

(6) Student mastery score integrates Lexical diversity and explanation ratio:

$$\mathrm{Student}\mathrm{Mastery}\mathrm{Score}={(\mathrm{Lexical}\mathrm{Diversity}·\mathrm{Explanation}\mathrm{Ratio})}^{1/2}.$$

(6)

components are ensured to be at least 0.001.

(7) Question ratio assesses teacher questioning frequency:

$$\mathrm{Question}\mathrm{Ratio}={\displaystyle \frac{\mathrm{count}\_\mathrm{questions}}{\mathrm{total}\_\mathrm{teacher}\_\mathrm{utterances}}},$$

(7)

identifying utterances with question keywords.

(8) Explanation ratio for teachers uses the same calculation as students:

$$\mathrm{Explanation}\mathrm{Ratio}={\displaystyle \frac{\mathrm{count}\_\mathrm{explanatory}}{\mathrm{total}\_\mathrm{teacher}\_\mathrm{utterances}}}.$$

(8)

(9) Guidance ratio evaluates teacher’s guiding language:

$$\mathrm{Guidance}\mathrm{Ratio}={\displaystyle \frac{\mathrm{count}\_\mathrm{guidance}}{\mathrm{total}\_\mathrm{teacher}\_\mathrm{utterances}}},$$

(9)

based on guidance keywords (e.g., “think”, “analyze”).

(10) Teacher guidance score synthesizes three ratios:

$$\mathrm{Teacher}\mathrm{Guidance}\mathrm{Score}={(\mathrm{Question}\mathrm{Ratio}·\mathrm{Explanation}\mathrm{Ratio}·\mathrm{Guidance}\mathrm{Ratio})}^{1/3}.$$

(10)

(11) Student response depth measures answer relevance using the Jaccard similarity:

$$\mathrm{Jaccard}(T_i,S_i)={\displaystyle \frac{|T_i\cap S_i|}{|T_i\cup S_i|}},$$

(11)

where $T_i$ and $S_i$ represent word sets. The final score averages similarity across adjacent pairs.

(12) Teacher question quality identifies open-ended questions:

$$\mathrm{Teacher}\mathrm{Question}\mathrm{Quality}={\displaystyle \frac{\mathrm{count}\_\mathrm{open}\_\mathrm{ended}}{\mathrm{total}\_\mathrm{teacher}\_\mathrm{utterances}}},$$

(12)

counting open-ended questions while excluding closed-ended markers.

(13) Dialogue coherence assesses semantic continuity:

$$\mathrm{Jaccard}(U_{i-1},U_i)={\displaystyle \frac{|U_{i-1}\cap U_i|}{|U_{i-1}\cup U_i|}},$$

(13)

where $U_{i-1}$ and $U_i$ represent word sets. The final score averages similarity across consecutive pairs.

(14) Golden speech ratio identifies high-quality student responses:

$$\mathrm{Golden}\mathrm{Speech}\mathrm{Ratio}={\displaystyle \frac{\mathrm{count}\_\mathrm{golden}}{\mathrm{total}\_\mathrm{student}\_\mathrm{utterances}}}.$$

(14)

a golden utterance must meet thresholds in length, complexity, and explanation scores.

(15) Interaction quality score provides a comprehensive evaluation by averaging four core metrics:

$$\begin{array}{cc}\hfill \mathrm{Interaction}\mathrm{Quality}\mathrm{Score}={\displaystyle \frac{1}{4}}(& {\displaystyle \frac{\mathrm{Time}\_\mathrm{Entropy}}{1.0}}\hfill \\ \hfill & +\mathrm{Golden}\_\mathrm{Speech}\_\mathrm{Ratio}\hfill \\ \hfill & +\mathrm{Student}\_\mathrm{Mastery}\_\mathrm{Score}\hfill \\ \hfill & +\mathrm{Teacher}\_\mathrm{Guidance}\_\mathrm{Score}).\hfill \end{array}$$

(15)

the code for calculating all the aforementioned dynamic features is publicly available in our Gitee repository, ensuring full reproducibility.

4. Methodology

The overall framework of the proposed temporal-semantic adaptive ensemble framework is shown in Figure 3, mainly comprising three core modules: (a) time style classification, (b) latent semantic density extraction, and (c) dynamic weighted heterogeneous ensemble. This framework systematically extracts dynamic features from raw dialogue data and employs an adaptive ensemble strategy for precise classification.

4.1. Time Entropy Style Categorization

This module (Figure 3a) aims to quantitatively assess the balance of teacher–student speaking time and classify the teaching style into three types: spoon-feeding, balanced, and inquiry-based. The process is as follows:

Our approach uses the time entropy and teaching style features defined in Section 3.2 to quantify teacher–student interaction patterns. We use a data-driven percentile-based method rather than fixed theoretical thresholds to establish classification boundaries. The division details of teaching style are shown in Algorithm 1.

Algorithm 1: Teaching style classification

    Spoon-feeding teaching style: The teacher dominates the exposition, which limits students’ verbal contributions. It is typically associated with lower time entropy, indicating a discourse dominated by the teacher. This style aligns with traditional, transmission-based instructional models [25], which often prioritize knowledge transfer over the development of critical thinking skills, potentially limiting students’ cognitive engagement.

Balanced teaching style: This style facilitates scaffolded interactions where the teacher provides guidance while actively encouraging student participation. It is captured by moderate time entropy, reflecting a dynamic balance in the dialogue. This interaction pattern embodies the principle of scaffolding within a constructivist framework [26], where the teacher supports students in taking on increasing responsibility for their learning, thereby fostering a collaborative knowledge-building process.

Inquiry-based teaching style: This pedagogy encourages student exploration and the verbalization of their reasoning. It is associated with higher time entropy, signifying a more student-centered dialogue. This is a direct manifestation of inquiry-based learning [27] and social constructivism [28], where learning is driven by students’ questions and peer interaction. The high time entropy quantitatively captures the active cognitive engagement and metacognitive processes (e.g., verbalizing reasoning) that are central to deep learning in these theories.

This empirical percentile-based approach adapts to the inherent distribution of the data [29]. It replaces fixed theoretical bounds with robust, context-aware thresholds. These thresholds reflect actual teaching patterns observed in the classroom.

4.2. Latent Semantic Density Extraction

This pattern is responsible for capturing the latent semantic information density within the dialogue, going beyond surface-level word statistics. Quantifies the semantic density and informational richness of the dialogue, serving as an auxiliary feature to assess both student comprehension levels and the quality of instructional guidance provided by teachers. Its process is shown in Figure 3b.

4.2.1. TF-IDF Vectorization with SVD for Extraction

To capture the latent semantic information in instructional dialogues, we transform the textual content into a numerical representation using TF-IDF vectorization, which effectively weights words by their importance. We then apply truncated singular value decomposition (SVD [30]) to reduce dimensionality and extract salient semantic features. This process results in a compact latent semantic vector for each dialogue, which serves as the basis for semantic richness estimation and enhances the generalization capability of our feature set.

4.2.2. Role of Semantic Richness in the Ensemble

The semantic richness feature, as formally defined by Equation (3), quantifies the informational density of each dialogue. Within our ensemble framework, it provides a crucial semantic signal that complements the temporal dynamics captured by time entropy. This integration is designed to provide a more nuanced representation of dialogues, with the aim of improving discriminative capacity for cognitive level assessment beyond what is possible with lexical or turn-based features alone. The inclusion of this semantic dimension is essential for robust classification, as it directly reflects the complexity and depth of both instructional content and student responses.

4.3. Dynamic Weighted Heterogeneous Ensemble

This module represents the core contribution of our work, with its architecture detailed in Figure 3c. We designed a dynamically weighted heterogeneous ensemble framework that effectively integrates the strengths of two powerful base learners, XGBoost [11] and LightGBM [12] and introduces an adaptive weighting strategy to address the pronounced class imbalance inherent in educational dialogue data. The detailed hyperparameter optimization process and optimal configurations are provided in Appendix B.

4.3.1. Dynamic Weight Calculation Strategy

The dynamic weight calculation strategy adaptively fuses predictions from two heterogeneous base learners—XGBoost and LightGBM—by dynamically adjusting their contribution weights based on multiple factors: the class distribution of the training data (especially the proportion of the minority understanding class), class overlap characteristics, and the relative performance of each base learner across different cognitive levels. This multi-faceted approach effectively addresses the complex class imbalance issues inherent in educational dialogue data.

The detailed procedure is described in Algorithm 2, which operates as follows: First, a dynamic threshold $r_{\mathrm{threshold}}$ = 0.1826 was empirically determined through five-fold cross-validation analysis, representing the 15th percentile of understanding class distribution with 8% conservative adjustment to enhance robustness against class imbalance.

Algorithm 2: Dynamic weight calculation strategy

$$r_{\mathrm{threshold}}=max\left(0.12,Q_{\alpha }\left({\left[r_i\right]}_{i=1}^{n_{\mathrm{folds}}}\right)\times 0.92\right)\mathrm{where}\alpha =0.15,n_{\mathrm{folds}}=5$$

(16)

For complete reproducibility, including detailed fold-wise threshold calculations, parameter selection rationale, and statistical validation of the optimized threshold $r_{\mathrm{threshold}}=0.1826$, refer to Appendix C.

Second, the degree of class overlap is quantified via a k-nearest neighbors approach (k = 5), measuring the average proportion of same-class neighbors within the five nearest neighbors for each sample.

Third, for each base learner, F1 scores are evaluated through three-fold cross-validation within the training set, with a combined score calculated as 60% weight on understanding class performance and 40% weight on overall performance. The performance-based weights serve as the foundation and are computed as:

$$w_{\mathrm{xgb}}={\displaystyle \frac{F{1}_{\mathrm{xgb}}}{F{1}_{\mathrm{xgb}}+F{1}_{\mathrm{lgb}}}},w_{\mathrm{lgb}}={\displaystyle \frac{F{1}_{\mathrm{lgb}}}{F{1}_{\mathrm{xgb}}+F{1}_{\mathrm{lgb}}}}$$

(17)

where $F{1}_{\mathrm{xgb}}$ and $F{1}_{\mathrm{lgb}}$ are the average F1 scores for the understanding class achieved by XGBoost and LightGBM, respectively.

This strategy offers three key advantages: (1) Adaptability: The weighting scheme dynamically adjusts based on data distribution. This design aims to enhance model robustness across different dialogue contexts and teaching styles. (2) Interpretability: The weighting logic is transparent and grounded in imbalanced learning theory, providing clear insight into the model’s decision-making process. (3) Performance: The ensemble leverages the complementary strengths of XGBoost and LightGBM. It is engineered to achieve superior generalization, particularly for recognizing minority classes.

Overall, this comprehensive strategy provides a principled, data-driven approach to handling class imbalance in educational AI applications.

4.3.2. Weighted Probability Fusion Mechanism

For an input sample X, the final prediction probability Pensemble of the ensemble model is calculated by:

$$\begin{array}{c}\hfill P_{\mathrm{ensemble}}(Y |=c\left|X\right)=w_{\mathrm{xgb}}·P_{\mathrm{xgb}}(Y=c|X)+w_{\mathrm{lgb}}·P_{\mathrm{lgb}}(Y=c|X)\end{array}$$

(18)

where c is the class label, $w_{xgb}$ and $w_{lgb}$ are the dynamic weights, and $w_{xgb}+w_{lgb}=1$. The final classification result is the class with the highest probability.

4.3.3. Feature Importance Analysis

The ensemble framework provides a unified and robust view into the model’s decision-making process through a weighted-average feature importance mechanism. This approach synthesizes the feature importance scores from both base learners, XGBoost [11] and LightGBM [12], into a single, authoritative ranking. The core of this synthesis is a weighted average, where the dynamic ensemble weights—determined by the class distribution of the training data—are applied to the importance vectors from each model. This calculation is formally represented by the equation:

$$\begin{array}{c}\hfill I_{\mathrm{ensemble}}=w_{\mathrm{xgb}}·I_{\mathrm{xgb}}+w_{\mathrm{lgb}}·I_{\mathrm{lgb}}\end{array}$$

(19)

where $I_{\mathrm{xgb}}$ and $I_{\mathrm{lgb}}$ denote the feature importance vectors extracted from the XGBoost and LightGBM models, respectively. The weights $w_{\mathrm{xgb}}$ and $w_{\mathrm{lgb}}$ are the same dynamic ensemble weights used for probability fusion, satisfying $w_{\mathrm{xgb}}+w_{\mathrm{lgb}}=1$. This ensures that the final importance score for each feature reflects its contribution to the ensemble’s final prediction, weighted by the confidence in each base model.

This method offers a significant advantage by adapting to the data distribution. In scenarios of severe class imbalance, where XGBoost [11] is assigned a higher weight, features it deems critical for identifying minority classes will be proportionally emphasized in the unified importance score. Consequently, this mechanism moves beyond providing a mere black-box prediction; it delivers actionable insights by clearly identifying the most influential drivers of the model’s decisions, such as time entropy or semantic richness, thereby offering educators a interpretable understanding of the key factors affecting cognitive level classification.

4.3.4. Comprehensive Evaluation System

The framework employs two cross-validation strategies: teacher-aware group cross-validation to prevent data leakage across teachers, and stratified cross-validation for ablation studies. The group-based approach ensures that all dialogues from the same teacher remain within the same fold, effectively preventing teacher-specific data leakage. It particularly guarantees sufficient representation of minority classes in every validation round, leading to more reliable performance estimation.

The primary evaluation metric is macro-F1, as it treats all classes equally and comprehensively reflects the model’s overall performance on both majority and minority classes—making it especially suitable for imbalanced classification scenarios like the one in this task. Additionally, accuracy, recall, precision, and receiver operating characteristic-area under curve (ROC-AUC [31]) values are used as supplementary metrics for a multi-dimensional comprehensive assessment, enabling a thorough evaluation of the model’s classification capability and generalization performance. Furthermore, the area under the precision–recall curve, known as average precision (AP), was used to evaluate performance for each class, particularly the challenging minority class.

The innovativeness of this ensemble framework lies in the following: (1) Weights dynamically adapt to the data distribution, rather than being statically set. (2) It organically combines the different advantages of XGBoost (cost-sensitive learning) and LightGBM (automatic balancing) in addressing the class imbalance problem. (3) The probability fusion mechanism enhances the reliability and robustness of the final predictions by integrating the base models’ predictive probabilities through weighted combination, thereby preserving their uncertainty in classification.

4.4. Decision Flow for Practical Implementation

To facilitate practical adoption of the proposed framework, Figure 4 presents a concise decision flowchart that illustrates the complete pipeline from temporal dynamics analysis to final ensemble prediction. This visual guide demonstrates how time entropy quantifies classroom interaction patterns, how these patterns inform teaching style classification, and ultimately how the identified style guides the dynamic weight calculation for heterogeneous ensemble fusion.

As shown in the figure, the decision-making process begins with time entropy feature extraction, and the teaching dialogue is classified into three teaching styles through a data-driven percentile method. The classification result, together with the semantic richness feature, jointly affects the selection of the dynamic weight calculation strategy. Specifically, when the proportion of understanding class samples is low and the degree of category overlap is high, the framework tends to adopt a weight distribution dominated by XGBoost. In cases where there is significant category overlap, LightGBM should be given priority. Ultimately, the prediction results of the two base learners are integrated through a weighted probability fusion mechanism, and the final cognitive level classification is output after post-processing optimization.

This streamlined workflow highlights the core innovation of this framework: organically integrating the dynamic characteristics of teaching interaction (time entropy, teaching style) and the characteristics of data distribution (category imbalance, overlap degree) into the integrated weight decision, achieving context-based adaptive classification.

5. Experiments

This section aims to answer the study’s core questions through a series of experiments. First, we evaluate the overall classification performance of the proposed framework to verify its effectiveness. Subsequently, the contributions of the dynamic weighting mechanism and heterogeneous integration were deeply analyzed through the ablation study. We further evaluated the reliability and robustness of the results through statistical significance tests and model discriminant ability analysis. Finally, through fine-grained error analysis, model bottlenecks are diagnosed and directions for future improvements are pointed out. All experiments were carried out around the core research questions, aiming to verify the value of dynamic feature integration and adaptive ensemble learning in solving the problems of cognitive level classification and category imbalance.

5.1. Experimental Setup

Teaching style was assigned per dialogue based on time entropy thresholds computed separately within each training fold. To prevent teacher-specific data leakage, we employed teacher-aware cross-validation where all dialogues from the same teacher were kept within the same fold.

We fitted all preprocessing steps—including SMOTE oversampling (k = 3), feature standardization, one-hot encoding, and time entropy percentile computation for style assignment—exclusively on training folds. This ensured that no test set information influenced feature engineering or model training.

5.1.1. Experimental Dataset

Experiments in this study were conducted on the reconstructed MCD-temporal dataset. This dataset contains 5226 instructional dialogue samples, with cognitive level labels divided into three classes: mastery, understanding, and apprentice. The dataset exhibits significant class imbalance ($ImbalanceRatio=2.53$). The specific distribution is shown in

Table 2. Student cognitive level classification: class distribution and sample statistics.

Class	Sample Count	Percentage
Mastery	2619	50.1%
Understanding	1037	19.8%
Apprentice	1570	30.1%

.

5.1.2. Evaluation Metrics and Experimental Design

To comprehensively evaluate model performance, this study uses macro-F1 as the core evaluation metric, supplemented by accuracy, recall, precision, and ROC-AUC [31] values. The ROC-AUC was computed using the one-vs.-rest approach, which evaluated each class against the rest to provide a macro-average ROC-AUC score. Experiments employed stratified k-fold cross-validation, ensuring the class distribution in each fold was consistent with the whole—particularly verifying that the proportion of the understanding class remained stable across all folds (approximately 19.8% of samples)—with the random seed set to 42. All experiments were conducted under the same hardware environment (Intel Core i9-13900H, 32GB RAM) and software environment (Python 3.9, scikit-learn 1.3).

5.1.3. Comparison Methods

To validate the effectiveness of the proposed DWHE, we compare the following six methods:

Logistic regression [32] (with class weight adjustment), random forest [33], balanced random forest [34], XGBoost [11], LightGBM [12], DWHE (our Method).

The selection of these baseline models is motivated by their representativeness in tackling classification tasks, particularly those with imbalanced data. Logistic regression serves as a simple yet strong linear benchmark. random forest represents traditional bagging ensemble methods, while balanced random forest is its variant specifically designed for imbalanced datasets through class-weighted bootstrapping. XGBoost and LightGBM are state-of-the-art gradient boosting frameworks that are highly effective across diverse domains and are commonly used as strong baselines for tabular data classification. Comparing against these established and powerful models allows for a comprehensive and convincing evaluation of the proposed DWHE framework’s advantages.

Furthermore, to verify the effectiveness of the ensemble strategy and dynamic weighting mechanism, the following ablation studies are designed: XGBoost_Only: using only the XGBoost model; LightGBM_Only: using only the LightGBM model; and static_Weight_Ensemble: fixed weight ensemble (0.5 for XGBoost and 0.5 for LightGBM).

5.2. Results and Analysis

5.2.1. Dataset Statistical Analysis

To empirically validate the impact of temporal dynamics and pedagogical features on cognitive level classification, this study first conducts a comprehensive exploratory analysis of the MCD-temporal dataset. The following figures reveal key relationships among time entropy, teaching style, interaction quality, and cognitive levels, providing an important data foundation and theoretical basis for subsequent modeling.

Figure 5 illustrates the distribution of cognitive levels across eight equally divided time entropy groups. As time entropy increases (from left to right), indicating a more balanced distribution of teacher–student speaking time, the proportion of mastery-class students gradually decreases, while apprentice-class students increase significantly. For example, in the highest entropy group, apprentice and understanding students account for over 55%, suggesting that a more balanced dialogue structure may facilitate active participation by beginners. Understanding-class students show some fluctuation but generally account for between 8% and 25% across all groups. While not perfectly stable, their proportion is consistently lower than that of the mastery and apprentice classes in most bins.

This trend not only verifies a significant correlation between temporal dynamics (quantified by time entropy) and cognitive levels but also interprets its pedagogical value. The increase in apprentice-class students within high time entropy dialogues suggests that balanced turn-taking may enhance participation among beginners. Consequently, time entropy transcends its mathematical formulation to serve as a diagnostic indicator of classroom interaction quality, guiding educators toward more inclusive teaching practices.

This bar chart with error bars (Figure 6) displays the average interaction quality score and standard deviations for each teaching style. The inquiry-based teaching style has the highest interaction quality (0.373 ± 0.046), followed by the balanced style (0.344 ± 0.051), and the spoon-feeding style has the lowest (0.272 ± 0.082). The results show that teaching style with higher time entropy (i.e., more balanced teacher–student dialogue) are often accompanied by better interaction quality, validating the effectiveness of time entropy as a feature for capturing teaching dynamics.

As shown in Figure 7, apprentice-class students have the highest golden speech ratio ($0.439\pm 0.257$), exceeding both mastery $(0.351\pm 0.195)$ and understanding $(0.334\pm 0.164)$ levels. This suggests that beginners may produce high-quality responses more frequently, such as by asking questions or providing detailed explanations. This counterintuitive result reveals the complexity of cognitive engagement and implies that traditional assumptions about student performance need to be re-examined.

In summary, the above figures show a consistent pattern: teaching style with higher time entropy (e.g., inquiry-based) are often associated with higher participation of apprentice students, better interaction quality, and higher-quality golden speeches from beginners. This indicates that a balanced dialogue structure not only promotes participation among students at different cognitive levels but may also encourage students with weaker knowledge foundations to contribute higher-quality dialogues. These findings further confirm the importance of incorporating dynamic features such as time entropy and teaching style into cognitive level classification models, as they capture key teaching interaction mechanisms that may be overlooked by pure semantic analysis.

5.2.2. Overall Performance Comparison

Following the comprehensive statistical analysis of the MCD-temporal dataset and its relationship with cognitive levels, we evaluated the overall classification performance of the proposed DWHE framework. We compare it against several baseline methods to validate its superiority in terms of key metrics such as macro-F1.

Table 3. Macro-F1 performance comparison of various methods.

Method	Macro-F1 ↑	Std. Dev. ↓
Random Forest [33]	0.6156	0.0149
Logistic Regression [35]	0.6233	0.0081
Balanced Random Forest [34]	0.6195	0.0130
XGBoost [11]	0.5937	0.0111
LightGBM [12]	0.6035	0.0157
DWHE (ours)	0.6236	0.0110
The arrows indicate the desired direction of performance: ↑ higher values are better; ↓ lower values are better.

shows the macro-F1 performance of each method. The proposed framework demonstrates state-of-the-art performance ($macro\text{-}F1=0.6236\pm 0.0110$), achieving the highest performance among all compared methods. It indicates that simple category balance strategies are insufficient to fully capture the dynamic semantics and temporal patterns in instructional dialogues.

5.2.3. Ablation Study Analysis

Having established the overall superiority of the DWHE framework, we now conduct a systematic ablation study to dissect its internal mechanisms and validate the effectiveness of both the dynamic weighting strategy and the heterogeneous ensemble design.

The numerical results in

Table 4. Ablation study of the proposed ensemble framework.

Model Variant	XGBoost	LightGBM	Dynamic Weighting	Macro-F1 ↑	Std. Dev. ↓	Understanding F1 ↑
XGBoost_Only	✓	×	×	0.6205	0.0128	0.2468
LightGBM_Only	×	✓	×	0.6152	0.0176	0.3695
Static Weight Ensemble	✓	✓	×	0.6182	0.0137	0.3196
DWHE (ours)	✓	✓	✓	0.6236	0.0110	0.4044

provide a systematic ablation study validating our proposed framework: the heterogeneous ensemble strategy itself (static_Weight_Ensemble, $macro\text{-}F1=0.6182$) significantly outperforms individual models (XGBoost_Only: 0.6205, LightGBM_Only: 0.6152), while the dynamic weighting mechanism (DWHE, 0.6236) further enhances the ensemble performance. This conclusively demonstrates the critical role of the dynamic weight adjustment strategy in handling class imbalance, as its data-driven adaptability proves superior to a static weighting scheme.

Most significantly, our DWHE framework demonstrates remarkable effectiveness in minority class recognition. DWHE achieved the highest understanding F1 (0.4044), representing a 61.8% improvement over XGBoost_Only and a 24.9% improvement over Static Weight Ensemble. This performance pattern confirms that our dynamic weighting strategy successfully reallocates model attention to the challenging minority class.

This ablation study conclusively demonstrates the critical role of the dynamic weight adjustment strategy in handling class imbalance, as its data-driven adaptability proves superior to static weighting schemes for educational dialogue classification tasks.

5.2.4. Statistical Significance Test

We used paired t-tests (results from five-fold CV) to perform statistical significance analysis ($\alpha =0.05$) comparing DWHE with other methods. The results indicate that our method achieves statistical significance ($p<0.05$) in comparisons against XGBoost ($p=0.012$), while differences with other methods were not statistically significant (logistic_Regression: p = 0.891, balanced_RF: p = 0.419). The difference with LightGBM ($p=0.062$) approached but did not reach the threshold for statistical significance, suggesting a trend that warrants further investigation.

For the ablation study, the paired t-tests yielded p-values of 0.576, 0.211, and 0.241 when comparing DWHE against XGBoost_Only, LightGBM_Only, and Static_Weight _Ensemble, respectively.

These statistical results, combined with its top-ranked macro-F1 score, suggest that the DWHE framework provides a consistent and practically valuable improvement for cognitive level classification, even though the gains over some strong baselines do not always reach the threshold of statistical significance in this evaluation setup.

5.2.5. ROC and Model Discriminative Ability

We evaluated the framework’s discriminative capability using ROC analysis with the One-vs.-Rest approach. The ensemble achieved a macro-average ROC-AUC of 0.8071, demonstrating strong overall performance across all cognitive levels. This macro-average, which summarizes the model’s overall discriminative ability, is the arithmetic mean of the class-specific AUCs: mastery (0.86), understanding (0.69), and apprentice (0.87). All values are significantly above the random baseline of 0.500.

As shown in Figure 8, the ROC curves indicate excellent separation for mastery and apprentice classes, while the understanding class presented the greatest challenge, yet still achieved a respectable ROC-AUC of 0.69.

Error analysis revealed that the understanding class had the highest error rate (48.56%), with primary confusion between mastery and understanding classes: about 24.43% of mastery samples were misclassified as understanding, and 25.00% of understanding samples were misclassified as mastery. This phenomenon may indicate either fundamental feature overlap or potential inconsistencies in the annotation criteria for intermediate cognitive states.

The maintained discriminative capability for the challenging understanding class validates the effectiveness of the dynamic weighting strategy and heterogeneous ensemble approach. The model’s balanced performance makes it suitable for real-world educational applications requiring reliable identification of all cognitive levels.

5.2.6. Fine-Grained Error Analysis and Feature Refinement Directions

To address the critical challenge of classifying the understanding class, we conducted a detailed error analysis. This investigation provides essential insights into the model’s limitations and establishes a clear path for future feature enhancement. The analysis leverages both the normalized confusion matrix and per-class precision–recall curves to dissect the model’s performance at a granular level.

The normalized confusion matrix, presented in Figure 9, reveals the most significant classification challenge: a pronounced bidirectional confusion between the mastery and understanding classes. We observe that 23.28% of true mastery dialogues are misclassified as understanding. Conversely, 28.37% of true understanding dialogues are incorrectly predicted as mastery. This pattern indicates a substantial feature space overlap between these two cognitive levels. In particular, the high confusion between mastery and understanding directly suggests inherent ambiguity in labeling the transitional understanding class, as its distinction from stable mastery can be subtle and context-dependent in dialogue. The current feature set appears insufficient to capture the subtle distinctions that separate a student who has fully mastered a concept from one who is in a transitional state of understanding. This overlap likely stems from similar linguistic patterns and interaction dynamics exhibited by students at these adjacent cognitive stages.

Further evidence of this difficulty is provided by the per-class precision–recall curves in Figure 10. The understanding class achieves an AP of only 0.33. This value is substantially lower than the AP for mastery (0.85) and apprentice (0.73). The low AP score quantitatively confirms the model’s struggle to reliably identify the understanding class. The shape of the precision–recall curve for understanding shows rapid performance decay, meaning the model cannot maintain high precision as it attempts to recall more instances of this class. This behavior is characteristic of a minority class that is not well-separated from others, aligning perfectly with the confusion matrix results.

Beyond the quantitative analysis, we performed a qualitative examination of misclassified samples. This review identified recurring scenarios that contribute to errors. A dominant pattern is teacher dominance, where extended teacher monologues overshadow student contributions, thereby suppressing the linguistic cues necessary for accurate cognitive level assessment. Another common issue is Topic Shift, where an abrupt change in the dialogue’s subject disrupts the contextual flow, making it difficult to evaluate the depth of student understanding coherently. These observations, combined with the quantitative data, suggest that the existing features, while effective for the mastery and apprentice classes, lack the sensitivity needed to isolate the understanding class.

The insights from this analysis directly inform our strategy for feature refinement. To better disentangle the mastery and understanding classes, we will develop more nuanced features. Promising directions include creating a feature that quantifies the ratio of exploratory to declarative language in student responses, which may better capture the transitional nature of understanding. Additionally, implementing a turn-level analysis of time entropy could mitigate the effects of teacher dominance by highlighting local balances of interaction. Finally, a dedicated dialogue coherence metric, tracking semantic similarity across consecutive utterances, could help the model remain robust against topic shifts. These targeted enhancements are expected to improve the model’s discriminative power, particularly for the most challenging understanding class, leading to more robust and accurate cognitive level assessment.

6. Discussion

6.1. Experimental Validation and Core Advantages of the DWHE Framework

This study systematically validates the effectiveness of the proposed DWHE framework for cognitive level classification in instructional dialogues through experiments. Its main advantages are:

Effectiveness of dynamic features: The introduction of dynamic features such as time entropy and semantic richness significantly enhances the model’s perception of teaching style.

Innovativeness of the ensemble strategy: The dynamic weight adjustment mechanism effectively integrates the advantages of XGBoost and LightGBM, balancing class imbalance [36] and model diversity.

Strong generalization ability: The framework demonstrated stable performance in five-fold cross-validation, as evidenced by the low standard deviation of the macro-F1 score ($\pm 0.0110$), and maintained effective discrimination across all classes, achieving a macro-average ROC-AUC of 0.8071 in a multi-class imbalanced scenario.

Implications for Teaching Practice. The findings of this study offer valuable insights for both educators and intelligent tutoring system designers. The strong correlation between time entropy, teaching style, and cognitive outcomes provides empirical support for the educational theory of cognitive engagement, which posits that active student participation is crucial for deep learning. This suggests that teachers can be trained to monitor and adjust their speaking time distribution consciously. For instance, reducing teacher-dominated talk (low entropy) in favor of more balanced or inquiry-based styles (high entropy) could promote deeper cognitive engagement, especially among apprentice-class students. Furthermore, the proposed framework can be integrated into real-time teaching assistants, providing immediate feedback to teachers on their interaction style and predicted student comprehension, thereby enabling just-in-time pedagogical adjustments. The dynamic weighting mechanism also illustrates a path toward adaptive educational AI [37] that tailors its reasoning process based on the observed classroom dynamics, moving beyond static one-model-fits-all solutions.

6.2. Implications and Implementation Paths of Teaching Practice

The implementation of this framework in a real educational environment can be achieved through the following paths:

Teacher teaching feedback tool: Integrate the framework into teaching software to provide teachers with reports after class that include time entropy, teaching models, and predicted cognitive level distribution. For instance, when the report consistently shows that the teaching mode leans towards the “indoctrination style” and is associated with a lower cognitive level, it can prompt teachers to adjust the structure of the dialogue between teachers and students, increase heuristic questioning and opportunities for students to express themselves.

Adaptive learning system intervention module: In the online learning platform, the framework is used to make real-time or quasi-real-time inferences about students’ cognitive levels. For instance, the system can dynamically adjust subsequent dialogue strategies or the difficulty of learning content based on the cognitive level that students are continuously at. For example, it can provide more explanatory materials for students at the apprentice level or pose challenging questions to those approaching the mastery level.

Educational quality assessment indicators: The educational administrative department can anonymously summarize and analyze the interaction quality score, teaching models and other indicators of multiple classrooms. For instance, by comparing the distribution of teaching models and the quality of interaction among different schools or regions, effective teaching practices can be identified, and empirical evidence can be provided for the design of teacher professional development programs.

7. Conclusions

This study addresses the critical challenge of cognitive level classification in instructional dialogues by proposing a novel framework that integrates dynamic temporal features and a dynamically weighted heterogeneous ensemble learning method. The enhanced MCD-temporal dataset, with its new time entropy and semantic richness features, provides a more comprehensive representation of teaching style. The proposed ensemble learner effectively combines the strengths of XGBoost and LightGBM, incorporating an adaptive weighting strategy that responds to class imbalance conditions.

Experimental results validate the superiority of the proposed DWHE framework ($macro-F1=0.6236$), demonstrating statistically significant improvements over key baselines. The method demonstrates superior performance on the minority understanding class (F1 score of 0.4044), while maintaining competitive performance on the majority mastery and apprentice classes (ROC-AUC > 0.86).

Despite the promising results, this study has several limitations that point to future research directions. First, the model was trained and validated primarily on dialogues from a specific academic subject (mathematics), which may limit its immediate generalizability to other disciplines with distinct pedagogical patterns. Future work will involve validating the framework on interdisciplinary dialogue datasets to enhance its robustness. Second, the significant confusion between mastery and understanding classes highlights the challenge of annotation ambiguity for intermediate cognitive states. To address this, a key future direction is to develop a fine-grained annotation rubric with concrete examples (e.g., defining specific linguistic markers that distinguish ’proficient application’ from ’initial comprehension’) to guide and standardize the labeling process. Third, the current feature set is derived solely from textual and temporal metadata. Incorporating multimodal data, such as audio prosody and visual cues of engagement, could provide a more holistic understanding of cognitive states. Finally, exploring the framework’s adaptability for real-time cognitive state inference in live classroom settings represents a critical step towards practical application.