Pre-Training Multi-Concept Question Embeddings for Knowledge Tracing

Abstract

With the rise of online education platforms, the vast number of accumulated data conceal patterns regarding the evolution of students’ knowledge states. Tracking a student’s knowledge state through their historical interaction records is known as knowledge tracing, a critical task in intelligent educational systems. Current research predominantly focuses on designing high-performance networks to enhance knowledge state tracking capabilities, often employing simplistic methods for question embedding, such as one-hot encoding or graph-based representations. This paper proposes a pre-training model for multi-concept question embedding (Pre-training Multi-concept Question Embedding, PMQE), aimed at providing a robust upstream tool for knowledge tracing and intelligent education fields. We first leverage the textual information in the dataset, representing concepts using their names or descriptive text embeddings. We then utilize the structural information from the question–concept graph, applying graph convolutional networks to derive question embeddings that integrate both textual semantics and structural information. Additionally, auxiliary information (such as question difficulty) is utilized for joint model optimization. The model outputs include question embeddings, question–concept weight matrices, and question difficulty, which can be used for knowledge tracing and other downstream tasks. Experiments conducted on two real-world datasets and multiple models validate the effectiveness of our approach.

1. Introduction

In intelligent educational systems, knowledge tracing is a crucial task aimed at modeling students’ proficiency levels based on their historical answering records and assessing their current knowledge state [1,2]. The task is typically evaluated by the ability to predict whether a student will answer the next question correctly. To address this problem, numerous methods have been developed. A key element in the definition of the knowledge tracing task is the embedding of the questions themselves during student interactions. Most existing methods do not focus on the embedding of questions but instead jointly train the representation of questions within the framework of a predictive model using supervised methods. However, due to the potential insufficiency of interaction data for some questions or concepts, the issue of data sparsity arises, limiting the expressiveness of embeddings obtained through supervised methods. As illustrated in Figure 1, knowledge tracing involves analyzing students’ mastery of different concepts at each time step based on recorded interactions with a sequence of questions, which include information about the questions, the concepts covered, and the correctness of the responses. Typically, answering correctly indicates an improvement in the student’s mastery of the corresponding concepts, as shown with $q_4$, where the student’s proficiency in concept $c_2$ increases. Conversely, an incorrect response, such as with $q_3$, suggests a decline in proficiency for concepts $c_1$ and $c_3$. The objective of this study is to construct question embeddings from a multi-concept perspective, allowing for a more accurate reflection of the diverse concepts covered by a question.

Current pre-training methods for question embeddings do not consider the textual information of concepts or questions, nor do they account for the possibility of a question encompassing multiple concepts.

Based on this analysis and consideration, we propose a pre-training model that integrates textual embeddings and structural information to uniformly model the relationships between questions and concepts, providing high-quality embeddings for knowledge tracing and its downstream tasks. Specifically, we initially employ large language models to obtain textual embeddings of concepts. Then, we construct a question–concept graph from the interaction history and use graph convolutional methods to update the embeddings of questions and concepts, subsequently reconstructing the graph and optimizing embeddings through graph loss calculation. The experimental results on two real-world datasets demonstrate that compared to traditional supervised question embedding methods in knowledge tracing, our pre-trained embeddings can enhance model performance and improve interpretability.

Our contributions are as follows:

• We developed a method to obtain both question embeddings and question–concept weight matrices, facilitating downstream tasks such as knowledge tracing and question recommendation. This approach transitions from traditional single-concept to multi concept modeling.
• By leveraging the relatively fixed nature of concept texts, we utilized methods from other text embedding models to represent concepts, enriching the structured information of the question–knowledge bipartite graph with additional textual semantic information.
• We designed a difficulty analyzer that assesses question difficulty based on its embeddings and auxiliary information, which can be utilized for question evaluation and incorporated into downstream tasks [3,4,5].

2. Related Work

The related work is categorized into the following aspects: the knowledge tracing task itself, representations of question embeddings, and other methods utilized in this paper.

2.1. Knowledge Tracing

The digitalization of education has generated a vast amount of learning interaction records [1,2,5]. Knowledge tracing aims to estimate students’ current knowledge states from these records, making it a fundamental task in the realm of intelligent education. Below, we briefly review related work on knowledge tracing, with a focus on developments before and after the introduction of deep learning.

2.1.1. Knowledge Tracing Before Deep Learning

The task of knowledge tracing was originally proposed in the last century. Bayesian Knowledge Tracing (BKT) [6] assumes that a student’s knowledge state is a set of binary variables, using a Hidden Markov Model to update the probability of mastering concepts based on learner performance. During the subsequent period, factor analysis models were primarily used to accomplish the knowledge tracing task. This theory played a significant role in educational assessment, with models such as Learning Factors Analysis (LFA) [7] and Knowledge Tracing Machines (KTM) [8] representing logical knowledge tracing methods. These models excelled in both interpretability and performance compared to Bayesian Knowledge Tracing.

2.1.2. Deep Learning-Based Knowledge Tracing

In 2015, Deep Knowledge Tracing (DKT) [9] was the first to successfully apply LSTM based [10] deep neural networks to the knowledge tracing task, achieving commendable results. The Dynamic Key-Value Memory Network (DKVMN) [11] utilized dynamic key value matrices to store relationships between concepts and questions, as well as students’ knowledge states. Attentive Knowledge Tracing (AKT) [12] employed a monotonic attention mechanism to simulate student forgetting and enhanced model interpretability using the Rasch model. Graph-based Knowledge Tracing (GKT) [13] introduced graph neural networks to the knowledge tracing task for the first time and proposed various graph construction methods. These efforts have fully integrated knowledge tracing into the era of deep learning.

2.2. Question Embedding Representation

Question embedding is often a crucial component of the knowledge tracing task. Most knowledge tracing models treat it as an internal task, synchronously constructing question representations during the training process or using simple methods to differentiate questions. Some research, however, focuses on modeling question embeddings, even treating it as a standalone task and training a separate model [14].

2.2.1. Traditional Question Embedding

In traditional deep learning approaches to knowledge tracing, such as those represented by DKT, questions are typically embedded using simple one-hot encoding. To mitigate the issue of data sparsity, some studies simplify datasets from the question level to the concept level, using concepts to represent entire questions. This approach alleviates data sparsity to some extent but overlooks the nuances between different questions. Other works incorporate question embeddings into the joint learning process within knowledge tracing models. For instance, AKT [12] introduces the Rasch model, which captures variations in questions under the same concept. Some studies attempt to construct question embeddings using textual information, such as EKT [15] and EERNN [16], which employ a private dataset to embed entire question texts using word2vec [17]. Additional research extracts information from knowledge structure graphs, as seen in SKT [18], JKT [19], and TSKT [20].

2.2.2. Pre-Trained Question Embedding

Recent studies have isolated question embedding as a standalone task, independent from the knowledge tracing task [21,22]. This unified approach to constructing question embeddings, which is not task-specific, can be utilized across multiple knowledge tracing models and other downstream tasks like question recommendation and learning path planning. This allows downstream tasks to focus on their core objectives.

PEBG [23] represents the question–concept relationship as a bipartite graph, using a simple dot product layer to learn low-dimensional embeddings for questions, and demonstrates its effectiveness in question representation through visualization techniques. PERM [24] employs a two-layer attention mechanism to further uncover semantic information in student–question and student–concept interactions from the global student–question–concept relationships.SEEP [25] constructs a heterogeneous graph of student–question–concept from the dataset, decomposing it into two semantic perspectives and using node-level attention mechanisms to obtain embeddings for questions and concepts.BiCo [26] chooses two similar yet distinct perspectives (subjective and objective) as semantic sources for question embedding, designing a pre-training model for question embeddings that achieves state-of-the-art performance on two real-world datasets, proving its robust expressive capabilities.

Overall, PEBG pioneered the use of pre-training for constructing question embeddings, establishing a fundamental structure for obtaining embeddings from question–concept bipartite graphs and difficulty constraints, thereby laying the groundwork for subsequent studies. However, none of these four models account for the possibility that a single question may involve multiple concepts, nor do they utilize the semantic information readily available from the textual embeddings of concepts. Moreover, by explicitly modeling the multi-concept matrix, we can better serve downstream applications.

2.3. Graph Convolutional Networks

Graph Convolutional Networks (GCNs) [27] are neural network architectures designed to process graph-structured data, and they have been widely applied across various domains, including social network analysis, recommendation systems, and bioinformatics. GCNs perform convolution operations on graph structures to learn node representations, effectively capturing dependencies between nodes. LightGCN [28] aims to simplify the design of traditional GCNs by removing unnecessary components, such as non-linear activation functions and complex aggregation functions. Instead, it updates the central node’s features by directly aggregating the features of neighboring nodes. These characteristics enable us to efficiently and cost-effectively extract knowledge structure information.

3. Problem Formulation

For a student $s\in S$, we define their historical learning interaction sequence as $s={\left\{e_j\right\}}_{j=1}^n$, where $e_j$ represents the j-th interaction, and S is a set of all students’ sequences. Each interaction is a tuple $e=\langle q,r\rangle$, where q denotes the question involved in this interaction, and $r\in \{0,1\}$ is a binary value indicating whether the student’s response was correct. Each question q includes a set of concepts and auxiliary information, represented as $q=\langle \left\{c\right\},a\rangle$. A question may encompass one or more concepts. The set of all concepts is denoted as $\mathrm{C}=\left\{c\right\}$, and the set of all questions is denoted as $\mathrm{Q}=\left\{q\right\}$.

The auxiliary information for a question is derived from statistical data within the dataset. Different datasets provide varying types of auxiliary information. For instance, in the Assist09 dataset, we can derive statistics such as the question error rate d, average number of hints $h_{avg}$, maximum number of hints $h_{max}$, average first response time $t_{avgfirst}$, average response time $t_{avgresp}$, and average number of attempts $t_{attempt}$.

The set of concepts associated with a question indicates which concepts are relevant to that question, while the auxiliary information reflects aspects related to the question’s difficulty.

The knowledge tracing task aims to determine a student’s knowledge state given their past interaction sequence s. Due to the nature of real-world data collection, explicit knowledge states cannot be directly used as targets for the task. Therefore, the primary objective of the knowledge tracing task is to predict the probability that the student will answer the next question correctly. Specifically, given the current sequence $s=\{e_1,e_2,\dots ,e_j\}$, the task is to predict the probability of correctness for the next interaction $e_{j+1}$, denoted as $P_{j+1}$.

3.1. Definition 1: Question–Concept Bipartite Graph

In most datasets, a significant portion of questions correspond to more than one concept. We construct a graph $G=(Q,C,E)$ to analyze the relationships between questions and concepts. Here, Q denotes the set of question nodes, C represents the set of concept nodes, and E signifies the set of edges between question nodes and concept nodes. The values of the edge set E are determined by the co-occurrence relationships between questions and concepts. Specifically, $E=\left[A_{ij}\right]\in {\{0,1\}}^{Q\times C}$, with:

$$A_{ij}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{question}{q}_i\mathrm{includes}\mathrm{concept}{c}_j\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(1)

3.2. Definition 2: Question Difficulty

Following previous research [29], we consider question difficulty to be a crucial attribute that not only aids the knowledge tracing task but also plays an important role in other tasks within intelligent educational systems. The difficulty of a question $d_i$ is statistically derived from the dataset as follows:

$$d_i={\displaystyle \frac{{\sum }_{s\in S}{\sum }_{e_j\in s}\mathbb{I}(q_{e_j}=q_i\wedge r_{e_j}=0)}{{\sum }_{s\in S}{\sum }_{e_j\in s}\mathbb{I}(q_{e_j}=q_i)}}$$

(2)

where $\mathbb{I}(·)$ is the indicator function that equals 1 if the condition is satisfied, and 0 otherwise. The exercise and the response corresponding to the interaction record $e_j$ are represented as $q_{e_j}$ and $r_{e_j}$, respectively.

4. Method

This chapter introduces our proposed PMQE model. As shown in Figure 2, we first obtain text embeddings of concepts using a large language model, which serves as their initial embeddings. These embeddings are then multiplied by a trainable weight matrix to generate the initial embeddings of questions. A concept-question bipartite graph is constructed based on their co-occurrence relationships in the dataset, and LightGCN is applied to capture structural information, producing refined embeddings for both concepts and questions. The similarity between these embeddings is then used to construct a new graph. Graph embedding loss is computed using both the original bipartite graph and the reconstructed graph. After obtaining updated question embeddings, they are processed through a difficulty analysis module, incorporating auxiliary information, to derive difficulty representations. These representations are then compared with statistical difficulty data from the dataset to compute the difficulty loss. The final optimization objective considers both the graph embedding loss and the difficulty contrastive loss.

4.1. Concept Text Embedding

Given that concepts do not directly interact with students, we utilize existing text embedding models for their representation. Recently, large language models (LLMs) [30] have rapidly advanced, and we employ these models’ text embedding methods to embed concept texts, effectively leveraging the semantic information learned from vast corpora. The textual information corresponding to concepts is relatively straightforward, and we denote the resultant text embeddings as $C\in {\mathbb{R}}^{\left|\mathrm{C}\right|\times d}$, where $\left|\mathrm{C}\right|$ is the number of concepts and d is the embedding dimension. For each concept $c_i$, its text embedding is represented as the i-th row of the matrix C.

Considering the different lengths of text embeddings obtained from various models and the special techniques employed by OpenAI, we limit the length of text embeddings from OpenAI to match those from other text embedding models. For the OpenAI text-embedding-3-large model, we followed OpenAI’s recommended methods for shortening text embeddings, extracting different lengths as needed to evaluate the impact of varying embedding lengths.

We start by computing the L2 norm (Euclidean norm) ${\parallel \mathbf{x}\parallel }_2=\sqrt{{\sum }_{i=1}^n{x}_i^2}$ of a text embedding vector $\mathbf{x}=(x_1,x_2,\dots ,x_n)$, where $x_i$ represents the i-th component of the embedding vector and n is the number of target dimensions. Then, we normalize the vector $\mathbf{x}$:

$${\mathbf{x}}_{\mathrm{normalized}}=\left\{\begin{array}{cc}\mathbf{x},\hfill & {\mathrm{if} \parallel \mathbf{x}\parallel }_2=0\hfill \\ {\displaystyle \frac{\mathbf{x}}{{\parallel \mathbf{x}\parallel }_2}},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

4.2. Question Embedding

To achieve the embedding representation of questions, we multiply the concept embeddings by a weight matrix $W_{C2Q}\in {\mathbb{R}}^{\left|\mathrm{C}\right|\times \left|\mathrm{Q}\right|}$ to obtain the matrix $Q\in {\mathbb{R}}^{\left|\mathrm{Q}\right|\times d}$. This matrix explicitly represents the relational weights between questions and concepts, thereby fully reflecting the influence of concepts in the question embeddings. Here, $\left|\mathrm{C}\right|$ denotes the number of concepts, and $\left|\mathrm{Q}\right|$ denotes the number of questions. For each question $q_i$, its text embedding is represented as the i-th row of the matrix Q.

4.3. Embedding Update Based on LightGCN

Building on this explicit modeling, we further optimize the embeddings by employing the message-passing mechanism of LightGCN to obtain updated embeddings $C^{\prime }$ and $Q^{\prime }$ from the question–concept bipartite graph G. Through message passing on the bipartite graph, our question embeddings capture more structural information on top of the semantic information from text embeddings. This shifts the semantic space of the embeddings from pure text to the knowledge tracing domain, enhancing their expressive power in this context.

4.4. Graph Constraint

After generating the new embeddings, we use these embeddings to reconstruct the bipartite graph $\widehat{G}$ and compare it with the original question–concept bipartite graph G. We calculate the loss $L_G$ to optimize the embeddings. The reconstruction of the bipartite graph is based on the similarity between concepts and questions, constructing an edge between two nodes if their similarity exceeds a certain threshold. In the reconstructed bipartite graph $\widehat{G}$, the relationship is defined as:

$${\widehat{r}}_{ij}=\sigma \left({\mathit{q}}_i^T{\mathit{c}}_j\right),i\in \left[1,\dots ,\left|Q\right|\right],j\in \left[1,\dots ,\left|C\right|\right]$$

(4)

where $\sigma$ is the sigmoid function to ensure the output is between 0 and 1, and $\widehat{q}$ and $\widehat{c}$ represent the updated embeddings of questions and knowledge points.

We use the cross-entropy loss function to optimize the reconstructed bipartite graph against the original question–concept graph built from the dataset:

$${\mathcal{L}}_G(\mathit{Q},\mathit{C})=\sum _{i=1}^{\left|Q\right|}\sum _{j=1}^{\left|C\right|}-(r_{ij}log{\widehat{r}}_{ij}+(1-r_{ij})log(1-{\widehat{r}}_{ij}))$$

(5)

This loss function encourages the reconstructed graph to closely resemble the original graph by penalizing discrepancies between them.

4.5. Auxiliary Information Constraint

In addition, the model integrates the question embeddings $Q^{\prime }$ obtained from graph convolution with auxiliary information to estimate the question difficulty $\widehat{D}$ and calculates the loss against the difficulty D derived from dataset statistics. By combining the reconstruction graph loss $L_G$ and the difficulty loss $L_D$, the model optimizes the embeddings. This approach not only refines the question embeddings but also enhances their accuracy based on difficulty analysis.

We treat each question individually, collecting continuous variables from the dataset as inputs for the question difficulty analysis module. For the Assistment-2009 dataset, we use the following data as inputs: total number of hints, average number of hints, first response time, average response time, and number of attempts. We compute the average for each variable across the dataset and normalize these averages across all questions, feeding these normalized values into a multi-layer perceptron (MLP) to obtain $D_A$. Simultaneously, the question embeddings are fed into another MLP to obtain $D_Q$, and both results are combined through a fully connected (FC) layer to produce the final difficulty representation $\widehat{D}$:

$$\widehat{D}=\mathrm{FC}\left(\left[{\mathrm{MLP}}_A\left({\displaystyle \frac{\mathbf{a}-\mu }{\sigma }}\right),{\mathrm{MLP}}_Q\left(\mathbf{q}\right)\right]\right)$$

(6)

where $\mathbf{a}$ is the vector of auxiliary information for the questions, consisting of continuous variables. $\mu$ and $\sigma$ are the mean and standard deviation of the auxiliary information across all questions in the dataset. ${\mathrm{MLP}}_A$ and ${\mathrm{MLP}}_Q$ are two different multi-layer perceptrons. FC is a fully connected layer. $\widehat{D}$ is the final difficulty representation.

For optimizing difficulty with auxiliary information constraints, we use the error rate, statistically derived from the dataset, as the label:

$${\mathcal{L}}_D\left(\mathit{Q}\right)=\sum _{i=1}^{\left|Q\right|}-(D_{i}log{\widehat{D}}_i+(1-D_i)log(1-{\widehat{D}}_i))$$

(7)

This loss function helps align the model’s difficulty prediction with observed error rates, thereby refining the embeddings through both structural and difficulty constraints.

4.6. Joint Optimization

Our model performs joint optimization by combining both graph constraints and auxiliary information constraints:

$$\mathcal{L}=\lambda {\mathcal{L}}_G+(1-\lambda ){\mathcal{L}}_D$$

(8)

where $\lambda \in (0,1)$ is a tunable hyperparameter that controls the weighting between the graph constraint and the auxiliary information constraint. This balanced approach allows the model to effectively leverage both structural and difficulty information to optimize the question embeddings.

5. Experiments

5.1. Datasets

In selecting datasets, we considered that some datasets do not have recorded textual fields for concepts. We chose the following two real-world datasets for our experiments:

Assistment-2009: This dataset originates from the Assistment platform and includes 346,860 recorded interactions from 4217 students, covering 110 different concepts. It is a fundamental dataset widely used in knowledge tracing tasks.

Junyi: This is a learning platform constructed based on open-source code released by Khan Academy.

We performed uniform preprocessing on the datasets. Considering the richness of real-world data, we used the following methods to ensure low data sparsity: interactions without knowledge point information were removed, student data with too few learning records (fewer than 5) were excluded, and interactions involving knowledge points with too few corresponding questions (fewer than 10) were also removed. Some users with too few interactions were removed, which may result in the insufficient sampling of short term or intermittent learners, potentially reducing the predictive performance for this user group. Our model aims to construct more expressive exercise embeddings. However, in achieving the goal of constructing more expressive exercise embeddings, this approach can improve data quality, reduce noise, optimize computational resources, and enhance the stability and generalization ability of the model.

Table 1. Dataset statistics.

Dataset	Assist09	Junyi
Students	2932	10,000
Concepts	105	835
Questions	16,811	835
Interactions	268,271	353,835

presents the statistics of the datasets after preprocessing.

5.2. Compared Models

In this study, we aim to validate the effectiveness of our proposed model. To this end, we integrate the pre-trained model into classic deep knowledge tracing models and carefully evaluate its performance enhancements. Additionally, to further substantiate the conclusions of our study, we compare our approach with several existing pre-training methods. These comparative experiments not only demonstrate the superiority of our model but also highlight its potential advantages in the relevant field.

• Knowledge Tracing Models:

  (a)

  DKT: The first model to apply deep learning to the field of knowledge tracing, utilizing an RNN with sigmoid activation. It simplifies questions by representing them with single concepts to alleviate data sparsity issues.

  (b)

  DKVMN: This model uses Memory-Augmented Neural Networks to construct the main framework of the knowledge tracing model. Specifically, it employs a static matrix to store knowledge concepts and a dynamic matrix to store and update the mastery level of response concepts.

• Pre-Trained Models:

  (a)

  PEBG: To our knowledge, the earliest method to treat question representation as a standalone task separated from knowledge tracing. It represents the question–concept relationship as a bipartite graph to optimize question embeddings.

  (b)

  BiCo: Utilizes self-supervised dual graph contrastive learning with objective and subjective semantics to pre-train question embeddings.

5.3. Experimental Setup and Hyperparameters

The experiments in this paper were conducted on an NVIDIA RTX 4090 24G (NVIDIA Corporation, Santa Clara, CA, USA), and the models and related code were implemented using PyTorch 2.1.2. For each dataset, 70% of the data was used for training, and 30% was used for testing. The learning rate was set to 0.001, with LightGCN configured to have three layers, and the number of epochs was set to 1000. The hyperparameters that needed adjustment in PMQE were $\lambda$ and the embedding dimension d. Based on our experiments, we selected $\lambda =0.73$ and $d=128$. In the knowledge tracing models, the learning rate was set to 0.006, and the number of epochs was set to 300.

5.4. Performance

We selected several classic models in the knowledge tracing task to validate the effectiveness of our question embeddings, including the classic deep knowledge tracing model DKT and the Dynamic Key-Value Memory Network (DKVMN), replacing their question representation components with the outputs of our pre-trained model. We also compared our method with previous pre-training approaches like PEBG and BiCo. Through these comparisons across different models, we aim to comprehensively assess the performance improvement brought by our designed pre-trained question-embedding module in the knowledge tracing task.

Students’ response sequences are recorded as either correct (1) or incorrect (0). Therefore, we formulate the knowledge tracing task as a binary classification problem. Based on this, we utilize Accuracy (ACC) and the Area Under the Receiver Operating Characteristic Curve (AUC) as our evaluation metrics. ACC measures the proportion of correctly predicted responses among all responses, providing a straightforward assessment of overall model performance. AUC, on the other hand, evaluates the model’s ability to distinguish between correct and incorrect responses by computing the area under the ROC curve, which plots the true positive rate against the false positive rate. A higher AUC value indicates the better discriminative capability of the model.

The experimental results, as shown in

Table 2. Performance comparison of PMQE.

	Assist09		Junyi
Model	ACC	AUC	ACC	AUC
DKT	71.42	73.73	69.42	70.3
DKVMN	72.18	74.61	70.6	71.8
PEBG+DKT	74.7	76.46	71.7	73.86
PEBG+DKVMN	75.83	77.87	72.23	74.75
BiCo+DKT	75.41	76.2	71.89	73.44
BiCo+DKVMN	77.19	78.31	73.15	75.17
PMQE+DKT	75.95	77.18	71.67	73.73
PMQE+DKVMN	77.74	78.86	73.52	75.81

, demonstrate that the models augmented with PMQE exhibit superior performance across all metrics. On the Assist09 dataset, the PMQE+DKVMN model achieved the highest accuracy (ACC) of 77.74% and an AUC value of 78.86%, representing improvements of 5.56% and 4.25% over the baseline DKVMN model, respectively. Similarly, on the Junyi dataset, the PMQE+DKVMN model outperformed other combinations, achieving an accuracy of 73.52% and an AUC of 75.81%, which are 2.92% and 4.01% higher than the DKVMN baseline, respectively. Furthermore, models utilizing PMQE outperformed previous pre-training approaches.

The improvements in various performance metrics indicate that PMQE is an effective method for question representation, significantly enhancing predictive capabilities in knowledge tracing tasks and providing new insights and tools for educational data mining.

5.5. Question–Concept Matrix

We visualized the change in concepts for a particular student (as shown in Figure 3). The figure is divided into three parts.

The top black-and-white matrix represents the coverage of concepts in the questions during interactions. The darker the color, the greater the weight of that concept. Most questions involve a single concept, while a few involve two or three. The middle section uses solid or hollow circles to indicate whether the response was correct during the interaction, with solid circles representing correct answers and hollow circles representing incorrect ones. The bottom color matrix displays the predicted knowledge state of the student for each concept, with each column representing the state after each interaction. Colors closer to yellow indicate better mastery of that concept as predicted by the model.

Our observations are as follows.

For single concept interactions, as seen in the third and fourth columns, both questions pertain to concept $c_3$. The third question was answered correctly, while the fourth was answered incorrectly, suggesting that the corresponding knowledge state should decrease and then increase. This matches the changes shown in the knowledge state matrix.

Multiple concept interactions, as observed in the tenth column, involve concepts $c_5$ and $c_6$, and the question was answered correctly. Therefore, we predict an increase in knowledge state for $c_5$ and $c_6$, which is consistent with the changes shown in the figure.

This indicates that our weight matrix can accurately describe the relationship between questions and concepts.

5.6. Ablation Study

To validate the effectiveness of each module in our model, we designed the following ablation experiments:

• PMQE: The initial model.
• no-text-emd: Removed the text embedding of concepts, using one-hot encoding for concept representation as in most methods.
• no-c2q-weight: Removed the question-to-concept weight matrix, decoupling question embeddings from concepts and training with independent weights.
• no-side-info: Removed additional auxiliary information used for difficulty analysis, relying solely on question embeddings to compute difficulty, i.e.,

  $$\widehat{D}={\mathrm{MLP}}_Q\left(\mathbf{q}\right)$$

  (9)
• no-difficulty: Removed the entire difficulty analysis module, effectively removing the difficulty constraint, and used only the question–concept bipartite graph to guide model optimization. Without the difficulty analysis module, the model’s loss degenerates to:

  $$\mathcal{L}(\mathit{Q},\mathit{C})={\mathcal{L}}_G$$

  (10)

For the ablation study conducted using the Assist09 dataset with the DKT model, the results are presented in Figure 4. By systematically removing key components of the PMQE model, we observed that each module significantly impacts knowledge tracing performance. Specifically, removing the concept text embedding (no-text-emb) resulted in the greatest performance decline (a decrease of 1.06% in ACC and 0.68% in AUC), underscoring the importance of semantic representations provided by pre-trained language models for modeling concepts. The absence of the question-to-concept weight matrix (no-c2q-weight) led to reductions of 0.71% in ACC and 0.21% in AUC, indicating that explicitly modeling the association weights between questions and concepts effectively captures complex relational structures. When auxiliary information for questions was removed (no-side-info), the performance drop was relatively minor (0.32% in ACC and 0.15% in AUC), suggesting that the fundamental embeddings already capture core features, though auxiliary information still provides supplementary discriminative features for difficulty evaluation. Notably, eliminating the difficulty analysis module (no-difficulty) resulted in ACC and AUC decreases of 0.79% and 0.70%, respectively, confirming the critical role of a dynamic, multi-source difficulty assessment mechanism in accurately estimating student abilities. Overall, the experimental results demonstrate that the model integrates semantic embeddings, structural constraints, and cognitive diagnostic theories to achieve multidimensional optimization in the knowledge tracing task.

6. Conclusions and Future Work

In this paper, we proposed a pre-training method to obtain high-quality question embeddings. Our approach first utilizes large language models to derive text representations of concepts, which are then used to obtain initial question representations through a coefficient matrix. These representations are further refined using graph convolution methods to produce updated embeddings. The model optimizes its weights by reconstructing the graph structure and comparing it against the original question–concept graph. Additionally, the model incorporates auxiliary information such as question difficulty to jointly optimize the model’s weights. Comparative experiments across multiple datasets demonstrate that our method offers superior performance, while ablation studies confirm the effectiveness of each designed module. The model outputs include question embeddings and a weight relationship matrix between questions and concepts, which can be applied to knowledge tracing and its downstream tasks to enhance performance and interpretability.

The prerequisite relationships between concepts were not considered, but we believe these relationships are an important structural aspect rather than relying solely on our question–concept bipartite graph. Secondly, the difficulty analysis module heavily depends on the statistical information of exercises, leading to a cold-start problem for newly introduced exercises. Additionally, due to dataset limitations, we only used the textual information of knowledge point names, which prevents us from fully leveraging the rich semantics of complete exercises.

Future work directions include: (1) considering prerequisite relationships between concepts; and (2) the current model is trained on specific datasets and relies on specific concept and question IDs. We aim to train on multiple datasets to develop a model that works from question text, enabling it to function independently of question or concept IDs for improved generalizability.