ANT-KT: Adaptive NAS Transformers for Knowledge Tracing

Abstract

Knowledge Tracing aims to assess students’ mastery of knowledge concepts in real time, playing a crucial role in providing personalized learning services in intelligent tutoring systems. In recent years, researchers have attempted to introduce Neural Architecture Search (NAS) into knowledge tracing tasks to automatically design more efficient network structures. However, existing NAS-based methods for Knowledge Tracing suffer from excessively large search spaces and slow search efficiency, which significantly constrain their practical applications. To address these limitations, this paper proposes an Adaptive Neural Architecture Search framework based on Transformers for KT, called ANT-KT. Specifically, we design an enhanced encoder that combines convolution operations with state vectors to capture both local and global dependencies in students’ learning sequences. Moreover, an optimized decoder with a linear attention mechanism is introduced to improve the efficiency of modeling long-term student knowledge state evolution. We further propose an evolutionary NAS algorithm that incorporates a model optimization efficiency objective and a dynamic search space reduction strategy, enabling the discovery of high-performing yet computationally efficient architectures. Experimental results on two large-scale real-world datasets, EdNet and RAIEd2020, demonstrate that ANT-KT significantly reduces time costs across all stages of NAS while achieving performance improvements on multiple evaluation metrics, validating the efficiency and practicality of the proposed method.

1. Introduction

Knowledge Tracing (KT) [1] aims to assess the mastery of knowledge points by the student in real time and is one of the key links in the realization of personalized adaptive learning [2]. With the rapid development of online education platforms [3,4], the huge amount of data from learning records provides a brand new opportunity to build Knowledge Tracing models. By modeling student exercise-solving sequences, KT can be used to dynamically predict students’ knowledge states, providing an important basis for downstream tasks such as the learning path planning and recommendation of learning material. In recent years, deep learning has witnessed substantial advancements in the domain of KT. Nevertheless, prevailing models continue to exhibit limitations in capturing students’ forgetting behavior, extracting structural information between exercises and knowledge concepts, and other pertinent aspects. Consequently, the exploration of more effective KT models is imperative to improve the intelligence of adaptive learning systems.

Research on Knowledge Tracing dates back to the 1990s with the introduction of the Bayesian Knowledge Tracing (BKT) model [5]. Subsequently, various probabilistic graphical models were proposed, including dynamic Bayesian networks [6] and hidden Markov models [7]. While these models describe the state transitions of student knowledge using predefined rules, they have limitations in representation and generalization. Deep learning has since revolutionized Knowledge Tracing. The Deep Knowledge Tracing (DKT) model [8] pioneered the use of RNNs to model student interactions as sequences. Dynamic Key-Value Memory Network (DKVMN) [9] further has improved this by introducing memory mechanisms to capture the dynamic changes of knowledge state. The Self-Attentive Knowledge Tracing (SAKT) model, as outlined in the seminal work of Pandey et al. [10], employs a sophisticated form of self-attention to model inter-question relationships. Meanwhile, the Separated Self-Attentive Neural Knowledge Tracing (SAINT) model [11] marked a significant leap. It applied Transformers to knowledge tracing, achieving substantial performance improvements through complex exercise interaction modeling. Despite these advancements in characterizing learning processes and knowledge point relationships, most methods relied on fixed network structures, overlooking the potential of architecture optimization. Recent work [12] has explored applying Neural Architecture Search (NAS) [13] to Transformer-based knowledge tracing models, aiming for automatic optimal architecture discovery. However, the computational demands and time requirements of NAS methods present challenges for efficient and practical application in knowledge tracing.

To address the limitations of applying neural architecture search (NAS) to KT tasks, this paper proposes an Adaptive Neural Architecture Search framework based on Transformers (ANT-KT). Traditional Transformer encoders primarily rely on self-attention mechanisms to model sequence dependencies. However, self-attention has limitations in capturing local features. To overcome this issue, an enhanced encoder is designed that combines convolution operations [14] with state vectors [15], enabling the simultaneous modeling of both local and global dependencies in sequences. This allows the encoder to better capture short-term changes in students’ knowledge states and incremental progress during the learning process. Furthermore, to improve efficiency when handling long sequences, an optimized decoder is proposed that introduces a linear attention mechanism. Unlike traditional decoders that compute self-attention through pairwise comparisons across all sequence positions, the linear attention transforms the attention computation into matrix multiplication via linear mappings. This significantly reduces time complexity while maintaining performance, enhancing the model’s ability to capture long-term dependencies and more accurately predict knowledge state changes over extended learning periods. Moreover, to enhance search efficiency, an improved evolutionary algorithm is introduced. Existing evolutionary algorithms typically optimize model performance in isolation, often ignoring the balance between model performance and computational efficiency [16]. To address this, a model optimization efficiency objective is designed within the evolutionary algorithm, establishing a balanced optimization function considering both performance and computational efficiency. Additionally, a dynamic search space reduction strategy is adopted to adjust the search space during the search process, enabling more efficient discovery of high-performing and computationally efficient architectures. The main contributions of this paper are summarized as follows:

• Proposes ANT-KT, an adaptive neural architecture search architecture based on Transformer for Knowledge Tracing tasks. ANT-KT integrates an innovative Transformer variant and an improved evolutionary algorithm to enhance model performance and architecture search efficiency.
• Designs an enhanced encoder–decoder architecture that strengthens the modeling of local and global dependencies. The encoder introduces state vectors combined with convolution operations to capture short-term knowledge state transitions, while the decoder employs a novel linear attention mechanism for efficient fusion of long-term global information, enabling more accurate modeling of students’ learning dynamics over time.
• Designs an improved evolutionary algorithm to further enhance search efficiency. An optimization efficiency objective in the model is introduced to establish a balanced optimization objective function between performance and computational efficiency. Additionally, a dynamic reduction strategy is adopted to dynamically adjust the search space during the search process, enabling a more efficient discovery of neural network architectures that are both high-performance and computationally efficient.
• Conducts extensive experiments on two large-scale real-world datasets, EdNet and RAIEd2020, and compares with multiple existing methods. The results demonstrate that the proposed method achieves improvements across multiple evaluation metrics while also enhancing search efficiency, confirming the framework’s advantages in both performance and efficiency.

2. Related Work

Knowledge Tracing: Knowledge Tracing has evolved from Bayesian parameter estimation [5] to sophisticated deep learning paradigms that capture temporal dynamics and structural dependencies. Early deep approaches like [8] pioneered RNN/LSTM-based sequential modeling, later extended with temporal features [17,18] and enhanced loss functions [19,20]. Concurrently, structural modeling emerged through graph networks [21,22] encoding concept–question relationships and memory-augmented architectures [9,23] storing long-term dependencies. The advent of Transformer-based models (SAKT [10], AKT [24]) introduced attention-driven interaction weighting, further advanced by hybrid approaches like [25] dynamically fusing temporal trajectories with graph structures and [26] integrating language models for inductive knowledge inference. And recently, Large Language Models (LLMs) have been explored for knowledge tracing tasks [26,27]. For instance, SINKT [26] integrates LLMs for inductive knowledge inference by leveraging pre-trained language understanding, while [27] investigates zero-shot KT using GPT-based models. Although these approaches show promise in handling cold-start scenarios and cross-domain generalization, they face challenges in computational overhead and interpretability. Despite these advances, existing KT models still struggle to balance prediction accuracy and efficiency, highlighting the need for adaptive architectures that can better capture students’ evolving knowledge states.

However, existing work is mainly limited to the original Transformer architecture, ignoring the latest progress in structural optimization.

NAS in Knowledge Tracing: Although deep Knowledge Tracing models have achieved remarkable performance gains in predicting students’ learning states, their architectures are still predominantly designed manually, which makes it difficult to ensure a globally optimal structure. To address this limitation and further improve model performance while reducing the burden of manual design, researchers have increasingly turned to Neural Architecture Search (NAS) [13] techniques in the context of Knowledge Tracing. In particular, Ding et al. pioneered this line of research by introducing the NAS-Cell method [28], which employs reinforcement learning to search for optimal LSTM cell structures. Their approach yielded architectures that surpass manually designed counterparts, leading to significant improvements across multiple Knowledge Tracing tasks. Building on this foundation, Yang et al. further advanced the integration of NAS into Knowledge Tracing by proposing ENAS-KT [12], which not only adopts a more comprehensive and systematic search strategy but also constructs a task-specific search space tailored to the characteristics of Knowledge Tracing. Collectively, these studies highlight both the feasibility and the promising potential of NAS in driving the next stage of development in Knowledge Tracing research.

Moreover, existing NAS-based KT approaches primarily focus on improving predictive accuracy, while neglecting the efficiency of the search process and the resulting model. This imbalance hinders their scalability to large-scale educational datasets and real-time applications. To address these limitations, this paper proposes ANT-KT, an adaptive NAS framework based on Transformer architectures. By introducing an enhanced encoder–decoder structure and an efficiency-aware evolutionary search strategy, ANT-KT achieves a better balance between model performance and computational efficiency, as demonstrated through extensive experiments on large-scale datasets.

3. Methodology

3.1. Problem Definition

Let $S=\{s_1,s_2,\dots ,s_n\}$ be a student’s interaction sequence, where $s_i=(e_i,t_i,r_i)$ for $i=1,2,\dots ,n$. Here, i represents the index of the interaction step in the sequence, $e_i$ denotes the exercise attempted, $t_i$ indicates the response time, and $r_i\in \{0,1\}$ represents the response correctness ($r_i=1$ for correct, $r_i=0$ for incorrect). Given an interaction sequence S, the goal of Knowledge Tracing is to correctly predict the probability $P(r_{n+1}=1|S,e_{n+1})$ of the answer for the next exercise $e_{n+1}$. Obviously, how to accurately model the evolving knowledge state of students based on their historical interaction data S is a key factor affecting the prediction accuracy.

3.2. Proposed Method: ANT-KT Framework

Since the students’ historical interaction datasets are collected based on their mastery of knowledge during the learning process, they inherently possess time-series characteristics. Furthermore, drawing inspiration from the excellent performance of Transformer in language sequence generation tasks within the field of natural language processing, this paper proposes an Adaptive Neural Architecture Search framework based on the Transformer (ANT-KT) for Knowledge Tracing tasks. As shown in Figure 1, the ANT-KT framework consists of three components: (1) a selective hierarchical input module, which is designed to extract and select feature inputs of different granularities and types; (2) an encoder, in which state vectors and convolutional operations are combined to capture both local and global dependencies within the exercise sequence; and (3) a decoder, where the information obtained from the encoder is leveraged to predict students’ knowledge states by introducing linear attention and a selective output mechanism.

3.3. Adaptive Embedding Selection Module

The adaptive embedding selection module is designed to adaptively select the most relevant embedding subset from the candidate embedding set ${\mathcal{X}}_{embed}=\{{\mathbf{embed}}_i\in {\mathbb{R}}^{n\times D}\mid 1\le i\le Num\}$ according to a given student’s interaction sequence S, for subsequent processing by the encoder and decoder. Herein, n denotes the length of the student’s interaction sequence S, D represents the dimension of each embedding and $Num$ is the total number of candidate embeddings. Meanwhile, the candidate embedding set ${\mathcal{X}}_{embed}$ is derived by performing embedding mapping on discrete features (such as exercise $e_i$) in the student interaction sequence S and normalizing continuous features (such as response time $t_i$).

To achieve adaptive selection, the module introduces two binary vectors ${\mathbf{b}}_{En}\in {\{0,1\}}^{1\times Num}$ and ${\mathbf{b}}_{De}\in {\{0,1\}}^{1\times Num}$ as the embedding selectors for the encoder and decoder, respectively. Each element $b_i\in \{0,1\}$ indicates whether the i-th candidate embedding ${\mathbf{embed}}_i$ is selected. In other words, $b_i=1$ indicates that the candidate embedding vector ${\mathbf{embed}}_i$ is selected; otherwise, it is not selected. To ensure non-empty inputs for the encoder and decoder, the selectors need to satisfy the following constraints:

$$\begin{array}{cc}1\le & \sum _{i=1}^{Num}{b}_i,b_i\in {\mathbf{b}}_{En},\\ 1\le & \sum _{i=1}^{Num}{b}_i,b_i\in {\mathbf{b}}_{De}.\end{array}$$

(1)

Based on selectors ${\mathbf{b}}_{En}$ and ${\mathbf{b}}_{De}$, two embedding subsets ${\mathcal{X}}_{En}$ and ${\mathcal{X}}_{De}$ can be obtained from the candidate embedding set ${\mathcal{X}}_{embed}$:

$$\begin{array}{cc}{\mathcal{X}}_{En}& =\{{\mathbf{embed}}_i\mid {\mathbf{embed}}_i\in {\mathcal{X}}_{embed},b_i=1,b_i\in {\mathbf{b}}_{En}\},\\ {\mathcal{X}}_{De}& =\{{\mathbf{embed}}_i\mid {\mathbf{embed}}_i\in {\mathcal{X}}_{embed},b_i=1,b_i\in {\mathbf{b}}_{De}\}.\end{array}$$

(2)

This candidate embedding sets are employed the hierarchical embedding fusion strategy $\mathrm{HierFuse}(·)$ proposed by [12]. In this strategy, the selected embeddings are first pairwise fused and then are fed into a linear transformation to map the fused features into a tensor of dimension $n\times D$, thereby generating the inputs ${\mathbf{H}}_{\mathrm{En}}^0$ and ${\mathbf{H}}_{\mathrm{De}}^0$ for the encoder and decoder:

$$\begin{array}{cc}{\mathbf{H}}_{En}^0& =\mathrm{HierFuse}\left({\mathcal{X}}_{En}\right),\\ {\mathbf{H}}_{De}^0& =\mathrm{HierFuse}\left({\mathcal{X}}_{De}\right).\end{array}$$

(3)

In the module, the $\mathrm{HierFuse}(·)$ can be described as follows:

$$\mathrm{HierFuse}\left(\mathbf{X}\right):\mathbf{output}=\mathrm{Linear}\left(\mathrm{Concat}\right(\mathbf{temp}\left)\right),$$

(4)

and

$$\mathbf{temp}=\bigcup _{{\mathbf{x}}_i,{\mathbf{x}}_j\in \mathbf{X},i\ne j}\mathrm{Tanh}\left(\mathrm{Linear}\left([{\mathbf{x}}_i,{\mathbf{x}}_j]\right)\right).$$

(5)

where $[{\mathbf{x}}_i,{\mathbf{x}}_j]$ represents the concatenation of embeddings ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$, $\mathrm{Tanh}(·)$ is the hyperbolic tangent activation function, $\mathrm{Linear}(·)$ denotes a linear transformation, $\mathbf{temp}$ is a temporary variable storing the pairwise fused embeddings, and $\mathbf{output}$ is the final output tensor of dimension $n\times D$ generated by the $\mathrm{HierFuse}(·)$ function.

Through the adaptive embedding selection module, ANT-KT can flexibly select the most relevant subset of embeddings according to the requirements of the Knowledge Tracing task and integrate them into the optimal input representations for the encoder and decoder using the hierarchical embedding fusion strategy.

3.3.1. Enhanced Encoder

The enhanced encoder extends the Transformer encoder by introducing mechanisms to better capture both global and local dependencies in student interaction sequences. While self-attention effectively models long-range relations, it overlooks short-term patterns. To address this, we integrate convolutions and state vectors, enabling the encoder to jointly capture fine-grained learning fluctuations and overall knowledge evolution, thus providing richer representations for subsequent prediction. This module consists of $N_E$ stacked encoder units, which progressively model local and global dependencies in the student interaction sequence.

Corresponding to the input definition of the encoder in Formula (3), let ${\mathbf{H}}_{\mathrm{En}}^i$ denote the output of the i-th encoder unit, where ${\mathbf{H}}_{\mathrm{En}}^0$ represents the input feature embedding generated by the selective input module. Hence, the calculation process of the encoder unit is as follows:

$${\mathbf{Z}}_{\mathrm{En}}^i=\mathrm{Mamba}\left({\mathbf{H}}_{\mathrm{En}}^{i-1}\right),$$

(6)

where ${\mathbf{Z}}_{\mathrm{En}}^i$ denotes the intermediate global feature representation and are extracted by the Mamba, which processes the output from the previous encoder unit. By using selective state space selection, the Mamba module can effectively capture the global features to better model the student state. It employs a global state dimension $d_{\mathrm{state}}=16$ and a convolutional channel size $d_{\mathrm{conv}}=4$, enabling efficient modeling of long-range dependencies in the interaction sequence.

Next, local features of the sequence are captured by a one-dimensional convolution operation:

$${\mathbf{C}}_{\mathrm{En}}^i={\mathbf{H}}_{\mathrm{En}}^{i-1}+\mathrm{Conv}1\mathrm{d}\left({\mathbf{H}}_{\mathrm{En}}^{i-1}\right),$$

(7)

where a kernel size 4 is applied by the Conv1d to model short-term dependencies. And the residual connection is introduced to preserving local features for subsequent computations.

After local feature extraction, the hybrid attention mechanism ${\mathbf{M}}_{\mathrm{En}}^i$ integrates multiple types of feature interactions. The hybrid mechanism dynamically combines different candidate operations (e.g., self-attention, gated convolution) as follows:

$${\mathbf{M}}_{\mathrm{En}}^i\left(\mathbf{x}\right)=\sum _{k=1}^K{\alpha }_k^i{o}_k\left(\mathbf{x}\right),$$

(8)

where $o_k\left(\mathbf{x}\right)$ represents the k-th candidate operation, and ${\alpha }_k^i$ is a learnable weight normalized using the softmax function, ensuring ${\sum }_{k=1}^K{\alpha }_k^i=1$. This mechanism allows the encoder to adaptively adjust to varying sequence patterns, effectively integrating global and local dependencies.

The output of the hybrid attention mechanism is processed using residual connections and layer normalization to stabilize training:

$${\mathbf{H}}_{\mathrm{En}}^i=\mathrm{LN}({\mathbf{C}}_{\mathrm{En}}^i+{\mathbf{M}}_{\mathrm{En}}^i\left({\mathbf{C}}_{\mathrm{En}}^i\right)),$$

(9)

where LN denotes layer normalization, which mitigates internal covariate shifts and ensures smoother optimization.

Finally, the encoder unit refines its output through a feed-forward network (FFN), which expands the feature space with a hidden layer dimension of $d_{\mathrm{ffn}}=4\times D$, before projecting back to the original dimension D:

$${\mathbf{H}}_{\mathrm{En}}^i={\mathbf{H}}_{\mathrm{En}}^i+{\mathbf{M}}_{\mathrm{En}}^i\left(\mathrm{FFN}\left({\mathbf{H}}_{\mathrm{En}}^i\right)\right).$$

(10)

By stacking $N_E$ such encoder units, the enhanced encoder progressively extracts contextual features from the interaction sequence, resulting in the final representation:

$${\mathbf{H}}_{\mathrm{En}}^{N_E}=\mathrm{Encoder}\left({\mathbf{X}}_{\mathrm{En}}\right),$$

(11)

where ${\mathbf{H}}_{\mathrm{En}}^{N_E}$ can capture local and global dependencies in sequences while effectively utilizing input embeddings selected by the adaptive embedding selection module to achieve more accurate modeling of students’ cognitive states.

3.3.2. Optimized Decoder

The decoder aims to predict the student’s response based on the encoder’s output $H_{En}^{N_E}$. It consists of $N_D$ stacked units, where each unit includes self-attention, linear attention, and a feed-forward network (FFN). The decoder maintains a consistent embedding dimension d across all layers, with the initial input $H_{De}^0$, where ${input}_{\mathrm{En}}$ is generated from the selective input module.

Each decoder unit first processes its input through a normalization layer to ensure numerical stability. Self-attention is then applied to capture local dependencies within the sequence. Through residual connections, the updated representation $Z_{De}^j$ retains the original feature information while enhancing contextual modeling. The self-attention mechanism is formulated as follows:

$$Z_{De}^j=H_{De}^j+\mathrm{SelfAttn}\left(H_{De}^j\right),$$

(12)

where $H_{De}^j$ is the input of the j-th decoder unit, and $Z_{De}^j$ represents the intermediate representation.

To efficiently model global dependencies while maintaining computational feasibility, this paper designs a linear attention mechanism. The mechanism begins by generating query (Q), key (K), and value (V) matrices. Specifically, the decoder’s intermediate representation $Z_{De}^j$ and the encoder’s output $H_{En}^{N_E}$ are used for these computations. The query is obtained by applying a linear transformation to the concatenated encoder and decoder outputs:

$$Q=W_a[H_{En}^{N_E};Z_{De}^j],$$

(13)

where $W_a\in {\mathbb{R}}^{2D\times D}$ is a learnable transformation matrix, and $[;]$ denotes concatenation along the feature dimension. The keys and values are derived solely from the encoder’s output as follows:

$$K=W_b{H}_{En}^{N_E},$$

(14)

$$V=W_c{H}_{En}^{N_E}.$$

(15)

Here, $W_b,W_c\in {\mathbb{R}}^{D\times D}$ are learnable transformation matrices, and $H_{En}^{N_E}\in {\mathbb{R}}^{L\times D}$ represents the encoder’s output sequence. This ensures that the keys and values fully capture the global dependencies encoded by the encoder.

To compute the attention distribution, the scaled dot-product mechanism is applied between the queries and keys:

$$E=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right),$$

(16)

where $E\in {\mathbb{R}}^{L\times L}$ represents the attention weights. This matrix encodes the relevance of each element in the encoder’s output to the decoder’s intermediate representation, allowing for selective focus on the most relevant global features.

Using the attention weights, the global context representation is obtained by weighting the value matrix V:

$$F=EV,$$

(17)

where $F\in {\mathbb{R}}^{L\times D}$ aggregates the most relevant information from the encoder outputs. This context representation encapsulates global dependencies and is critical for refining the decoder’s features.

To integrate the global context F into the decoder’s representation, a nonlinear transformation is applied:

$${\tilde{Z}}_{De}^j=Z_{De}^j+\mathrm{ReLU}(W_{f}F+Z_{De}^j),$$

(18)

where $W_f\in {\mathbb{R}}^{D\times D}$ is a learnable transformation matrix. The addition operation ensures that the local features in $Z_{De}^j$ are preserved, while the global context F enhances the representation with long-range dependencies. The use of the ReLU activation function introduces nonlinearity, improving the model’s capacity to capture complex interactions.

This mechanism effectively balances computational efficiency with representational power, reducing the quadratic complexity of traditional self-attention mechanisms while maintaining the ability to model global dependencies.

To further refine the features, local convolutional operations and hybrid modules are applied. The convolutional layer captures short-term dependencies, while the hybrid operation $M_{De}^j$ dynamically selects kernel sizes and specific operations. The refined features are updated as follows:

$$H_{De}^j={\tilde{Z}}_{De}^j+M_{De}^j\left(\mathrm{Conv}1\mathrm{d}\left({\tilde{Z}}_{De}^j\right)\right).$$

(19)

Finally, the feed-forward network expands the feature space with a hidden layer dimension $d_{ffn}=4\times D$. The output of the FFN is combined with the features from the convolutional operation through residual connections:

$$H_{De}^j=H_{De}^j+M_{De}^j\left(\mathrm{FFN}\left(H_{De}^j\right)\right).$$

(20)

The final decoder output $H_{De}^{N_D}$ is passed through a linear transformation and a Sigmoid function to predict the probability of the student’s response being correct:

$$R=\sigma (w_r{H}_{De}^{N_D}+b_r),$$

(21)

where $w_r\in {\mathbb{R}}^D$ and $b_r$ are the weights and biases of the fully connected layer. The model is trained by minimizing the binary cross-entropy loss:

$$\mathcal{L}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_{i}log{R}_i+(1-y_i)log(1-R_i)\right],$$

(22)

where $y_i\in \{0,1\}$ represents the ground truth label, and $R_i$ is the predicted probability for the i-th sample.

3.4. Neural Architecture Search

To find the optimal network configuration, define the above model as a flexible search space, as shown in Figure 2. And apply NAS for optimization. The search space contains the following three groups of key parameters:

Input feature selection vectors: $b_{\mathrm{En}},b_{\mathrm{De}}\in {\{0,1\}}^{1\times \mathrm{Num}}$, where $Num$ is the total number of candidate features, controlling which input features will be used for the encoder and decoder.

Hybrid operation parameters: The hybrid operation parameters of the encoder and decoder are ${\alpha }_{En}\in {\mathbb{R}}^{N_E\times K}$ and ${\alpha }_{De}\in {\mathbb{R}}^{N_D\times K}$, where K is the number of candidate operations. Different operations (such as convolution, feed-forward network, etc.) in each hybrid operation are weighted and combined through weights $\alpha$, defining the final structure of the model.

Number of layer selection: The number of encoder layers $N_E$ and the number of decoder layers $N_D$.

3.4.1. Objective Function

Let $\theta$ be all the weight parameters of the Transformer Transformer variant model and $\mathcal{A}$ represent the search space, then the goal of joint learning can be expressed as follows:

$$\begin{array}{c}\hfill \underset{a\in A}{min}{\mathcal{L}}_{val}(w^{\ast }\left(\alpha \right),\alpha ),\end{array}$$

(23)

where $\mathcal{L}train$ and $\mathcal{L}val$ represent the loss functions of the model on the training set and validation set, respectively, $\alpha$ represents the architecture parameters in the search space, and $w^{\ast }\left(\alpha \right)$ represents the optimal weights obtained by training with the fixed architecture $\alpha$.

3.4.2. Search Strategies

The search process employs two key strategies:

• Weighted Combination of Operations: Each operation $o^i$ is assigned an importance weight ${\alpha }_i$:

  $$o\left(x\right)=\sum _{i=1}^K{\displaystyle \frac{exp\left({\alpha }_i\right)}{{\sum }_{j=1}^{K}exp\left({\alpha }_i\right)}}{o}^i\left(x\right).$$

  (24)
  This allows flexible combination of different structural elements, improving generalization.
• Weight Sharing: Certain parameters are shared across different architectural configurations, significantly reducing computational overhead.

These strategies work in tandem to explore a wide range of model structures while maintaining computational feasibility.

3.4.3. Improved Evolutionary Algorithm

To further improve the efficiency of architecture search, this paper proposes improvements to the traditional evolutionary algorithm, shown in Figure 3. Existing evolutionary algorithms typically focus solely on optimizing model performance while neglecting the balance between model performance and computational efficiency. To address this issue, we introduce a model optimization efficiency objective. Specifically, we design an objective function that comprehensively considers both model performance and computational efficiency:

$$f\left(a\right)={\displaystyle \frac{ACC\left(a\right)}{T{\left(a\right)}^{\beta }}},a\in A$$

(25)

where $ACC\left(a\right)$ denotes the accuracy of the model using architecture a on the validation set, $T\left(a\right)$ represents the inference time of the model, and $\beta$ is a balancing factor. By maximizing this objective function, the evolutionary algorithm can discover architectures that are both high-performing and computationally efficient.

Additionally, to further accelerate the search process, we employ a dynamic reduction strategy to adjust the size of the search space. Specifically, during the evolutionary process, the number of candidate operations is dynamically adjusted based on the fitness distribution of individuals in the population. As the evolution progresses, candidates with lower fitness are gradually eliminated, leading to a progressively smaller search space, thereby significantly reducing the time cost of the search.

3.4.4. Alternating Optimization

To efficiently optimize the architecture parameters $\alpha$ and weight parameters w, we adopt the gradient-based optimization method DARTS. The core idea of DARTS is to relax the discrete architecture selection into a continuous parameter mixture, so that gradient information can be used to optimize the architecture parameters and weight parameters simultaneously. In each training step, data $D_{train}$ is first sampled from the training set, and the gradient is calculated based on the current architecture parameters ${\alpha }_t$ to update the weight parameters:

$$w_{t+1}=w_t-{\eta }_w{\nabla }_w\mathcal{L}train(w_t,{\alpha }_t,Dtrain).$$

(26)

Then, data $D_{val}$ is sampled from the validation set, and the gradient is calculated based on the updated weight parameters $w_{t+1}$ to update the architecture parameters:

$${\alpha }_{t+1}={\alpha }_t-{\eta }_{\alpha }{\nabla }_{\alpha }\mathcal{L}val(wt+1,{\alpha }_t,D_{val}),$$

(27)

where ${\eta }_w$ and ${\eta }_{\alpha }$ represent the learning rates of weights and architecture, respectively. Through alternating optimization, the model can continuously update the architecture to adapt to the current task during the training process. After the search is over, for each position that requires a decision, we select the operation with the largest $\alpha$ value as the final architecture:

$$o^{\ast }=\underset{{\alpha }_i}{argmax}{o}^{\left(i\right)}.$$

(28)

The operation with the largest weight ${\alpha }_i$ is selected as the optimal architecture ${\alpha }^{\ast }$. After fixing the optimal architecture ${\alpha }^{\ast }$, the model weights are retrained on the complete dataset to obtain the final optimized model. Through the above steps, the design space of the Transformer architecture can be explored efficiently and comprehensively, and the optimal architecture suitable for the Knowledge Tracing task can be automatically discovered.

4. Experiments

In this section, the experimental setup, result analysis, and further ablation studies are introduced to comprehensively evaluate the effectiveness of our proposed model, ANT-KT, in the knowledge-tracing task in detail. Hence, we validated the performance of the model on two large-scale Knowledge Tracing datasets and compared it with multiple strong baselines.

4.1. Datasets

During the experiment, we opted for the two most mainstream real-word educational datasets, EdNet [29] and RAIEd2020 [30], to rigorously evaluate the model’s performance. The key characteristics of these datasets are summarized in

Table 1. Statistics of the two largest education datasets, EdNet and RAIEd2020.

Datasets	# of Interactions	# of Students	# of Exercises	# of Skills
	(# of Tags)	(# of Tag-Sets)	(# of Bundles)	(# of Explanations)
EdNet	95,293,926 (302)	84,309 (1792)	13,169 (9534)	7 (-)
RAIEd2020	99,271,300 (189)	393,656 (1520)	13,523 (-)	7 (2)

.

The datasets include feature embeddings for exercises, skills, tags, tag-sets, and bundles/explanations, as shown in Table 1.

In addition to these five embeddings, the candidate input embedding $X_{embed}$ incorporates seven additional feature embeddings, for a total of 12 embeddings: Answer embedding, Continuous elapsed time embedding, Categorical elapsed time embedding in seconds, Continuous lag time embedding, Categorical lag time embeddings in seconds, minutes, and days.

By leveraging these large-scale, real-world datasets with rich feature embeddings, the experiments provide a convincing validation of the proposed methodology in authentic educational contexts.

4.2. Baseline Models

We compared our model with the following Knowledge Tracing models:

• DKT [8]: The DKT model is the first KT method leveraging deep neural networks, commonly used as a baseline model. It focuses only on concept labels, ignoring exercise-level information.
• HawkesKT [31]: This model uses Hawkes processes to model the temporal dynamics in student interactions, enabling more accurate predictions of knowledge states by capturing the time-sensitive effects of prior responses.
• CT-NCM [32]: The CT-NCM method incorporates a continuous-time neural cognitive model that captures knowledge forgetting and learning progress through time-aware neural architectures.
• SAKT [10]: A self-attentive Knowledge Tracing model that utilizes attention mechanisms to model relationships among exercises, capturing dependencies without relying on sequence order.
• AKT [24]: The AKT method employs context-aware representations and Rasch embeddings to dynamically link students’ historical responses with their future interactions.
• SAINT [11]: This model adopts a Transformer-based encoder–decoder structure to integrate both student and exercise interactions for more comprehensive Knowledge Tracing.
• SAINT+ [33]: Building on SAINT, SAINT+ incorporates additional temporal features such as elapsed time and lag time, enhancing its ability to model temporal dependencies in student learning.
• NAS-Cell [28]: This method applies reinforcement learning to optimize RNN cell structures for Knowledge Tracing tasks, achieving architectures superior to manually designed ones.
• DisKT [34]: The DisKT model addresses cognitive bias in Knowledge Tracing by disentangling students’ familiar and unfamiliar abilities through causal intervention, using a contradiction attention mechanism to suppress guessing/mistaking effects and integrating an Item Response Theory variant for interpretability.
• ENAS-KT [12]: An evolutionary neural architecture search method tailored for KT, which designs a search space specialized for capturing Knowledge Tracing dynamics effectively.
• AAKT [35]: The AAKT model reframes Knowledge Tracing as a generative autoregressive process, which alternatively encodes question–response sequences to directly model pre- and post-response knowledge states, and incorporates auxiliary skill prediction and extra exercise features by enhancing sequences through state-of-the-art Natural Language Generation (NLG) techniques.

4.3. Implementation Details

During the experiment, all students were randomly partitioned into three subsets with respective proportions of 70%, 10%, and 20%, serving as the training set, validation set, and test set. Meanwhile, the maximum length of the input sequences was set to 100; for sequences of student learning interaction that exceeded this length, truncation was performed to generate multiple subsequences. And a 5-fold cross-validation strategy was adopted to ensure the robustness of the experimental results. For the evolutionary search algorithm, we set the population size to 50 and the maximum number of generations to 100. These parameters follow established practices in evolutionary neural architecture search [36,37]. The balance factor $\beta$ = 0.5 in Equation (25) is determined based on the Pareto optimization principle [38], which suggests equal weighting when objectives are normalized to comparable scales. This configuration ensures that neither prediction accuracy nor computational efficiency dominates the search process. In terms of model configuration, the number of blocks (N) was set to 4, the embedding dimension (D) was 128, and the hidden dimension of the feed-forward network (FFN) was configured as 128, too. These settings balance representational capacity and computational efficiency, following the principle of compact architectures widely adopted in resource-constrained sequential prediction tasks [37,39,40]. For the training process of the model, the hyperparameters were specified as follows: the number of training epochs was 60, the initial learning rate was 1 × 10⁻³, the dropout rate was 0.1, and the batch size was 128. And all experimental trials were conducted on a single server with an RTX3090 graphics processing unit.

4.4. Main Results

Table 2. Comparsion of various models on EdNet and RAIEd2020 datasets. Best results are highlighted in Bold, and the second best results are underlined.

Model	EdNet			RAIEd2020
	RMSE	ACC	AUC	RMSE	ACC	AUC
DKT	0.4653	0.6537	0.6952	0.4632	0.6622	0.7108
HawkesKT	0.4475	0.6888	0.7487	0.4453	0.6928	0.7525
CT-NCM	0.4364	0.7063	0.7743	0.4355	0.7079	0.7771
SAKT	0.4405	0.6998	0.7680	0.4381	0.7035	0.7693
AKT	0.4399	0.7016	0.7686	0.4368	0.7076	0.7752
SAINT	0.4322	0.7132	0.7825	0.4310	0.7143	0.7862
SAINT+	0.4285	0.7188	0.7916	0.4272	0.7192	0.7934
NAS-Cell	0.4345	0.7143	0.7796	0.4309	0.7167	0.7839
DisKT	0.4592	0.6863	0.7384	-	-	-
ENAS-KT	0.4209	0.7295	0.8062	0.4196	0.7313	0.8089
AAKT	0.4064	0.7554	0.7827	-	-	-
ANT-KT	0.4062	0.7553	0.8387	0.4122	0.7438	0.8239
Improve	−1.47%	0%	3.25%	−0.74%	1.25%	1.50%

presents the performance evaluation and comparison results between the ANT-KT model and current mainstream Knowledge Tracing models. Specifically, all data from the comparative models are derived from the results reported in their original papers, and the evaluation is conducted based on the tree metrics: RMSE (Root Mean Square Error), ACC (Accuracy), and AUC (Area Under the ROC Curve). Among these metrics, a smaller RMSE indicates better performance, while large values of ACC and AUC correspond to superior model performance.

As shown in Table 2, the ANT-KT model exhibits excellent performance on both the EdNET and RAIEd2020 datasets. Notably, it outperforms the baseline model ENAS-KT across all three evaluation metrics. In terms of Root Mean Square Error (RMSE), ANT-KT reduces the error to 0.4062—a further improvement compared to AAKT—while also achieving a 1.47% reduction relative to ENAS-KT. Beyond RMSE, ANT-KT also delivers impressive results in the AUC metric: it boosts the score by 3.25% when compared to ENAS-KT, which is recognized as the state-of-the-art method in this field. Consistent with its performance on EdNET, ANT-KT maintains significant advantages over the baseline ENAS-KT across all evaluation metrics when tested on the RAIEd2020 dataset as well.

4.5. Ablation Studies

As shown in

Table 3. Performance of ablation studies.

Dataset	Metric	Baseline	Encoder	Decoder	ANT-KT
EdNet	RMSE	$0.4209$	$0.4246$	$0.4200$	$\mathbf{0}.\mathbf{4062}$
	ACC	$0.7295$	$0.7243$	$0.7424$	$\mathbf{0}.\mathbf{7553}$
	AUC	$0.8062$	$0.7993$	$0.8208$	$\mathbf{0}.\mathbf{8387}$
RAIEd2020	RMSE	$0.4196$	$0.4275$	$0.4147$	$\mathbf{0}.\mathbf{4122}$
	ACC	$0.7313$	$0.7198$	$0.7408$	$\mathbf{0}.\mathbf{7438}$
	AUC	$0.8089$	$0.7939$	$0.8201$	$\mathbf{0}.\mathbf{8239}$

, we performed ablation experiments by individually adjusting the key modules of the enhanced encoder and decoder to analyze the impact of different components of ANT-KT on the model performance.

The results show that the optimized decoder contributes most significantly to the improvement of model performance. For example, on the EdNet dataset, compared with the baseline model (which adopts standard MHSA in the decoder), the introduction of our optimized decoder (with linear attention) improves AUC from 0.8062 to 0.8208. This direct comparison confirms that linear attention can more effectively capture long-range dependencies within student interaction sequences. This allows the model to better utilize historical information during inference, leading to more accurate predictions. This finding aligns with the current research trend in the Knowledge Tracing field, which emphasizes the impact of students’ long-term learning processes on knowledge states.

In contrast, the contribution of the enhanced encoder is relatively limited and, in some cases, even results in slight performance degradation. The observed performance degradation when employing the enhanced encoder in isolation (AUC decreasing from 0.8062 to 0.7993 on EdNet) can be explained by two factors. First, the Conv1d operations, while effective at capturing short-term patterns, may disproportionately emphasize local features and thereby disrupt the modeling of long-range semantic dependencies that are crucial in Knowledge Tracing tasks. Second, in the absence of the optimized decoder’s linear attention mechanism, which serves to re-integrate global contextual information, the encoder’s locally extracted representations remain insufficiently refined.

However, when the enhanced encoder and the optimized decoder are used together, the model achieves further performance improvements. For example, on the EdNet dataset, ANT-KT achieves an AUC of 0.8387, which is higher than the 0.8208 obtained using only the optimized decoder. This suggests that, although the enhanced encoder alone may introduce interference, it can generate a synergistic effect when combined with the optimized decoder. The optimized decoder may mitigate the local-global imbalance introduced by the enhanced encoder by strengthening global information fusion, allowing the model to consider both local features and global semantics.

Overall, the ablation experiments reveal the mechanisms and interactions of different components in ANT-KT. The optimized decoder plays a critical role in improving performance by capturing long-range dependencies, while the enhanced encoder complements the model by extracting richer local information. Although the enhanced encoder may introduce interference when used alone, it produces a complementary effect when working synergistically with the optimized decoder, jointly driving the improvement of model performance.

4.6. Analysis of Search Optimization Strategies and Training Efficiency Comparison

To verify the performance advantages of ANT-KT in training efficiency, we selected NAS-based method ENAS-KT, the classic Transformer-based model Saint+, and several variants of ANT-KT for comparative experiments. We evaluated the time consumption of three key stages: supernet training, evolutionary search, and final architecture training.

Table 4. Comparison of training time and performance across different models and ANT-KT variants.

Model	Supernet Training Time (h)	Evolutionary Search Time (h)	Final Architecture Training Time (h)	AUC	ACC
Saint+	NAN	NAN	2.381	0.7934	0.7192
ENAS-KT	41.863	12.8	8.784	0.8089	0.7313
ANT-KT (Encoder)	41.438	8.65	6.066	0.7993	0.7243
ANT-KT (Decoder)	39.767	8.921	17.85	0.8208	0.7424
ANT-KT	39.288	8.81	5.39	0.8387	0.7553
ANT-KT (Search optimization)	39.288	7.42	4.357	0.8228	0.7439

shows the training time of different models in these three stages. As shown in Table 4, compared with ENAS-KT, in the supernet training stage, ANT-KT requires 39.28 h, which is 6.2% less than ENAS-KT’s 41.86 h.

During the evolutionary search phase, the dynamic search space reduction mechanism in ANT-KT is guided by a three-dimensional fitness vector that includes AUC, ACC, and latency (Lines 3–7 of Algorithm A1 in Appendix A). Specifically, this three-dimensional fitness vector can effectively guide the search by analyzing the elite individuals on the Pareto frontier (Lines 15–19 of Algorithm A1 in Appendix A) and adaptively adjusting the search boundaries (Lines 20–21). As shown in Table 4, this mechanism effectively reduces the time of the model framework in the evolutionary search phase, indicating that it can balance model performance and computational efficiency simultaneously.

In the evolutionary search phase, the unique advantages of ANT-KT’s search strategies are evident when comparing the optimal architecture discovered by ANT-KT as shown in Figure 4a, to that discovered by the original evolutionary algorithm as shown in Figure 4b. The original evolutionary algorithm tends to identify complex architectures dominated by MHSA modules, leading to high computational overhead. In contrast, ANT-KT’s optimal architecture incorporates multiple FFN and convolution modules in the encoder and balances the use of MHSA and FFN in the decoder. This design, which balances local feature extraction and global information fusion, improves model performance while effectively controlling computational complexity. As shown in Table 4, ANT-KT reduces search time from ENAS-KT’s 12.8 h to 7.42 h, a reduction of 42.0%. Additionally, compared to ANT-KT without the efficiency-aware search strategy, the search time is further reduced by 15.7%, demonstrating the effectiveness of the algorithm. Despite the introduction of efficiency considerations in the search optimization strategy, performance is not sacrificed. ANT-KT achieves an AUC of 0.8228 and an ACC of 0.7439, outperforming most baseline models. This highlights ANT-KT’s excellent balance between efficiency and performance, validating the effectiveness and robustness of its search optimization strategy.

5. Conclusions

This paper presents ANT-KT, a Transformer-based adaptive neural architecture search (NAS) framework designed to address the limitations of existing methods in Knowledge Tracing tasks, particularly in search efficiency and modeling accuracy. By integrating an enhanced encoder, an optimized decoder, and a refined evolutionary algorithm, ANT-KT achieves a significant balance between computational efficiency and predictive performance. Experimental results on two large-scale datasets demonstrate that ANT-KT consistently outperforms state-of-the-art baselines across multiple metrics, highlighting its ability to capture both short-term and long-term dependencies in student interaction data. Ablation studies further confirm the effectiveness of each proposed component, showcasing their contributions to the framework’s overall performance.

The findings of this study underline the potential of NAS-driven approaches in adaptive Knowledge Tracing and personalized learning systems. Future research could extend ANT-KT to a broader range of educational tasks and datasets, as well as explore further optimizations in the search process to enhance scalability and generalizability. Furthermore, given the still relatively lengthy training times, future work will focus on enhancing efficiency through progressive NAS and distillation-based search. This work offers valuable insights for advancing intelligent education systems and adaptive learning technologies.