Detecting Student Engagement in an Online Learning Environment Using a Machine Learning Algorithm

Abstract

This paper examines online learner engagement, a complex concept encompassing several dimensions (behavioral, emotional, and cognitive) and recognized as a key indicator of learning effectiveness. Engagement involves participation, motivation, persistence, and reflection, facilitating content understanding. Predicting engagement, particularly behavioral engagement, encourages interaction and aids teachers in adjusting their methods. The aim is to develop a predictive model to classify learners based on their engagement, using indicators such as academic outcomes to identify signs of difficulty. This study demonstrates that engagement in quizzes and exams predicts engagement in lessons, promoting personalized learning. We utilized supervised machine learning algorithms to forecast engagement at three levels: quizzes, exams, and lessons, drawing from a Kaggle database. Quiz and exam scores were employed to create predictive models for lessons. The performance of the models was evaluated using classic metrics such as precision, recall, and F1-score. The Decision Tree model emerged as the best performer among those evaluated, achieving 97% and 98.49% accuracy in predicting quiz and exam engagement, respectively. The K-Nearest Neighbors (KNN) and Gradient Boosting models also showed commendable performance, albeit slightly less effective than the Decision Tree. The results indicate a strong correlation between engagement predictions across the three levels. This suggests that engagement in quizzes and exams, known as assessments, is a pertinent indicator of overall engagement. Active learners tend to perform better in these assessments. Early identification of at-risk learners allows for targeted interventions, optimizing their engagement.

1. Introduction

Adaptive e-learning environments have revitalized e-learning worldwide by making learning resources affordable and available. With this transition, many challenges have emerged [1]. The learning process can be attributed to a decline in learner motivation and interest due to a lack of interaction with the instructor [2], leading to course dropout or even abandonment of the learning path. Learner engagement in a learning environment is considerably crucial, as it is linked to learning rates [3]. Learner engagement can be defined as a combination of the cognitive, behavioral, and emotional aspects of the learner to accomplish a task [4,5] by matching academic success, perseverance, satisfaction, and a sense of belonging. It can also be defined as a unit for measuring the learner’s degree of interaction and effort [6]. A significant level of engagement requires adaptive and flexible environments able to match different levels of engagement and learners’ preferences to have a thorough and holistic understanding of their behaviors in a learning environment. According to [7], learner engagement can be divided into four domains, namely: emotional, behavioral, cognitive, and agentic engagement. The learner’s cognitive aspect refers to their psychological engagement in educational activities, including motivation to learn, understanding, and willingness to master skills [8,9]. The behavioral aspect refers to his participation, effort, and perseverance in activities.

Our work is based on the conception of engagement proposed by [10], who defines it according to three dimensions: behavioral, emotional, and cognitive. Additionally, it is inspired by the theory of self-determination [11], according to which the satisfaction of fundamental needs such as autonomy, the feeling of competence, and social relations plays an important role in maintaining long-term engagement.

Predicting learner engagement, particularly behavioral, in e-learning environments is crucial, as many studies have shown. This enhances learning dynamics by giving students a sense of ownership over their education and empowering instructors to modify the way they teach based on their interactions and behaviors. Ultimately, this leads to the development of recommender systems. The current work focuses on identifying the most effective machine learning techniques for predicting engagement, as explored by [12,13,14,15].

Therefore, our paper aims to propose an intelligent predictive model to classify learners according to their engagement in the virtual learning environment (VLE). The input data represent students, exams, quizzes, and lessons, and the algorithm must predict whether the learner is engaged. In addition, the paper also presents the methods used in a large body of work on learner modeling and adaptive learning.

The rest of the paper is divided into five sections. The second section explains related work. Section 3 and Section 4 describe the dataset and methodology, respectively. Section 5 and Section 6 present the results and conclusion, respectively.

2. Related Work

Learner performance is the measure of progress in a learning path. It is generally influenced by several elements, including engagement, which is important in e-learning. Many studies have examined the measurement of this performance by exploring learner engagement, particularly in e-learning environments. Most studies have affirmed the positive impact of information technologies in these environments [16,17].

Certainly, these technologies facilitate access to content and enhance learner interaction. They enable learners to acquire not only knowledge, but also behavioral skills such as problem solving, analytical, critical, and synthesizing thinking, and collaborative work, which is an essential element of engagement [18]. Reasons why learners feel a sense of belonging to their learning environments [19]. Consequently, it is worth focusing on learner engagement to ensure effectiveness [20]. Dixson has proposed that skills, participation, performance, and emotion represent measurement indicators and components of engagement [21]. This highlights a holistic approach to understanding how these factors interact. Competence manifests itself in learning style, while participation appears in the interactivity of different peers through discussions and sending messages that make the learning environment dynamic.

The performance component is promoted by results and grades from past exams and quizzes. In addition, the majority of studies use surveys and observations [17,22], while others use approaches based on machine learning [20,23]. However, other studies adopt a more quantitative approach, collecting information from learners in real time to assess their engagement. This section focuses specifically on research into learner engagement in e-learning environments. To this end, various methods and input data were deployed to examine the correlation between learner information and engagement.

The authors of [24] surveyed 325 students to examine their goal of continued use of asynchronous online courses based on four key factors (intrinsic motivation, extrinsic motivation, perception of multiple sources, and cognitive engagement). They revealed that cognitive engagement was the only factor linked to the continued use of online courses. This observation underlines the importance of cognitive engagement in developing learners’ perseverance. Other researchers have looked at analyzing learners’ emotions and facial expressions to predict their engagement by using facial recognition techniques [25,26,27]. For example, Bhardwaj used computer vision techniques to measure learner engagement, highlighting the importance of emotions. Moreover, learner engagement was predicted based on images collected in real time by webcams. They combined both deep learning-based methods for engagement detection and facial emotion recognition [7].

In another study, machine learning methods were used to predict early performance and therefore engagement of learners based on specific characteristics (such as gender, age, etc.) and engagement indicators (such as course satisfaction, frequency of logging in, accessing, and downloading courses) and assessment scores. This design underlines the importance of combining demographic data for better engagement accuracy [12]. In another study [28], learner engagement was the basis for analyzing the test results of five convolutional neural network (CNN) models.

Different predictive models with different algorithms (Naïve Bayes, Random Forest, K-Nearest Neighbors (k-NN), Support Vector Machine (SVM), and Artificial Neural Networks (ANN) Data Mining and machine learning algorithms) have been used [29] to predict learner performance based on the analysis of demographic, engagement, and performance data from the Open University dataset taken from Kaggle. This multitude of algorithms underlines the importance of choosing the right model based on the characteristics of the data. The results showed that k-NN and ANN performed best.

Several works have used the Open University Learning Analytics Dataset (OULAD) [30], which includes information on 22 courses and 32,593 students. The specificity of this dataset lies in the assembly of demographic data as well as learner interaction data in the virtual learning environment (VLE). To investigate the impact of different activities on learner engagement, ref. [31] exploited cross-lag modeling techniques and concluded that course design and student perception variables have a large impact compared to individual characteristics on engagement. No less than multitasking harms engagement. That said, the quality of content conception has an important role to play in improving engagement.

A study by [32] intended to examine the correlation between learning material and learners’ final grades using a statistical approach. Ref. [33] showed that machine learning algorithms are effective in predicting learner engagement. According to [34], the various works that correspond to learner engagement used machine learning or deep learning models (XGboost, LightGBM, random forest, neural networks, CATBoost, and stacking methods).

Lima, Costa, Araújo, and Dorça explored the relationship between learner engagement and academic performance, taking into account learners’ self-regulation profiles. Clustering algorithms such as K-means, EM, and Agglomerative were applied to the OULAD database. The results showed a positive correlation: the more learners interact with learning resources, the better their academic performance. This finding reinforces the idea that interaction is a lever for academic success [35].

Alruwais and M. Zakariah aimed to predict learner engagement based on data taken from an online learning platform (VLE). The data were prepared using data preparation techniques and then evaluated by various machine learning algorithms. The CATBoost model showed the best performance, with an accuracy of around 92.23%, a precision of 94.40%, and a recall of 100% [6].

Similarly, ref. [36] used deep learning to build a CNN model from data consisting of videos and images of learners at Ibn Tofaïl University in Kenitra, Morocco. The study aimed to measure learners’ levels of engagement and emotions during the learning process. The results show that it is possible to quantify learners’ participation and identify their emotions, to make decisions to improve interaction and learning. In another work [37], the same authors examined the increase in drop out rates linked to learners’ emotional and health status. For this purpose, they adopted an approach to analyzing learners’ video sequences, which enables both their emotions to be analyzed and their level of engagement to be measured. This approach combines advanced algorithms and machine learning methods. The results provide valuable insights into learners’ experiences, influencing the teaching methodologies adopted by teachers.

Recently, several works have revealed advanced approaches for detecting learner engagement by exploring artificial intelligence, as well as emotional and cognitive data. For example, the automated analysis of collaborative online discussions, in synchronous and non-synchronous contexts, through a probabilistic clustering approach, enables the identification of learners’ roles and transition categories to better understand the complexity of their engagement [38].

Other studies have adopted an automated configurational approach, combining bidirectional encoder representation from transformers (BERT) and fuzzy set qualitative comparative analysis (fsQCA). The aim is to understand the multidimensional characteristics of learner engagement in MOOCs and to identify specificities across disciplines [39].

Although our study focuses on behavioral engagement, due to the limitations of the available data, we recognize the importance and significant contribution of recent approaches that lead to the detection of the emotional and cognitive dimensions of engagement. Exploiting multimodal data offer a promising avenue for enriching engagement analysis, which we plan to exploit in future work.

Our study follows the continuity of research on the analysis of learner engagement in e-learning environments such as those already mentioned. However, in contrast to previous work, which focused mainly on behavioral or emotional aspects, our approach is distinguished by the integration of concrete data from learners’ interactions with quizzes and exams, enabling more accurate prediction of engagement in lessons.

This work makes an innovative contribution to the field of e-learning by focusing on the analysis of engagement through quizzes, exams, and lessons. Based on measurable indicators such as academic performance and engagement, it aims to personalize the learning experience. Using machine learning (ML) techniques, we offer an effective solution to improve pedagogical personalization.

3. Materials and Methods

3.1. Dataset

The goal of this study was to propose an intelligent predictive model for classifying students according to their participation in the virtual learning environment (VLE). The algorithm should be able to predict whether the learner is engaged or not based on input data, which includes student information, exams, quizzes, and lessons.

For this study, the Kaggle Student Engagement dataset that is open to the public was used [40]. This dataset includes the details about learner participation in exams, quizzes, and lessons, such as the course title, level, date the course was scored, exam category, exam length, and correctness of the answer, as well as the number of quizzes, exams, and lessons that the learner completed, as well as the date of engagement and the exam result, and more information.

The dataset contains 11 CSV files, summarized in

Table 1. Dataset content.

Course info	The file gives information about the available courses.
course_id		[2–57]
course_title		46 unique values
Course ratings	The file features student course rating information.
course_id		[2–57]
student_id		[259 k–295 k]
course_rating		[1–5]
date_rated		[2022-06-16–2022-10-20]
Exam info	The file includes details of exams passed by students on the Table platform.
Exam Id		[118–834]
exam_category		[1–4]
exam_duration		[5–130]
Quiz info	The file features details of quizzes completed by students.
quiz_id		[1–580]
question_id		[1–1258]
answer_id		[1–4757]
answer_correct		True\False
Student engagement	The file includes information on student engagement using the Table platform.
engagement_id		[1–2.65 m]
student_id		[259 k–296 k]
engagement_quizzes		1\0
engagement_exams		1\0
engagement_lessons		1\0
date_engaged		[2022-01-01–2022-10-20]
Student exams	The file includes information on student participation in the Table.
exam_attempt_id		[173 k–240 k]
student_id		[259 k–295 k]
exam_id		[97–833]
exam_result		Categorical [0–100]
exam_completion		Numerical [0.07–198]
date_exam_completed		Date [2022-01-01–2022-10-20]
Hub Questions	The file features data on questions asked by students.
hub_question_id		[7619–10.4 k]
student_id		[259 k–294 k]
date_question_asked		Date [2022-01-03–2022-10-20]
Student info	The file includes information on students who have registered for the Table platform.
student_id		[259 k–294 k]
student_country		Categorical
date_registered		[2022-01-01–2022-10-20]
Student learning	The file includes information on student use of the Tableau platform, such as the number of minutes viewed, the date of use, and whether the student found the experience successful or not.
student_id		[259 k–296 k]
course_id		[2–67]
minutes_watched		Numeric [0–1.71 k]
date_watched		Date [2022-01-01–2022-10-20]
Student purchases	This dataset provides information on the number of times a student has used the Table platform, the duration of use, and the commitment date.
purchase_id		[15.8 k–23.2 k]
student_id		[259 k–296 k]
purchase_type		Annual\Monthly\Other
date_purchased		[2022-01-01–2022-10-20]
Student quizzes	The file includes details of student quiz performance, such as quiz ID, question ID, answer ID, and student ID.
student_id		[259 k–296 k]
quiz_id		[21–580]
question_id		[22–1258]
answer_id

, covering the different learning data: course info, course ratings, exam info, quiz info, student engagement, student exams, hub questions, student info, student learning, student purchases, and student quizzes. Each CSV file contains data related to its topic. Table 1 shows a general overview of the files, facilitating understanding of the data used.

The steps followed to develop the most appropriate predictive model of student engagement are briefly illustrated in Figure 1.

The first step was to select and extract the data in an organized and well-prepared way so that our model is not at risk of being biased. It also involved cleaning the dataset of poorly annotated data, missing data, and duplicate data. We then proceeded to the most characteristic phase of our methodology, the modeling step, which consisted of training the algorithms and tuning the hyperparameters. Finally, after the model testing step, we evaluated and compared the performance of each model to identify the most effective one.

3.2. Data Pre-Processing

Due to their format, the raw datasets were typically inappropriate for analysis. Thus, preparing the data before performing advanced analytical exploration was crucial. Our initial task was to combine these files. For this, we used the similar features of each file. Due to the size of our data, we excluded two optional files—Student Purchases and Student Information—from the combined dataset.

Initially, the pre-final dataset contained 22 features (namely: engagement_id, student_id, engagement_quizzes, engagement_exams, engagement_lessons, date_engaged, quiz_id, question_id, answer_id, answer_correct, course_id, course_rating, date_rated, minutes_watched, date_watched, exam_attempt_id, exam_id, exam_result, exam_completion, date_exam_completed, exam_category, exam_duration). Secondly, we developed two features called “Difference_rated_watched” and “Difference_examcompleted_engaged” that depict the differences between the features “date_rated” and “date_watched” and the differences between “date_exam_completed” and “date_engaged” in order. In addition, the ‘engagement_id’ feature was eliminated.

The “answer_correct” feature in the native dataset was set to “y” if the learner’s answer was correct and to “n” if, it was incorrect. The feature values in our dataset were successively assigned the values “1” and “0”. The value of the “exam_duration” feature was also adjusted in minutes. The prediction targets had assigned values of 1 for engaged students and 0 for unengaged learners. The two variables, Difference_rated_watched and Difference_examcompleted, were selected from the list below. More precisely,

• Difference_rated_watched = date_rated − date_watched;
• Difference_examcompleted = date_exam_completed − date_engaged.

These dates were converted into the standard YYYY-MM-DD format and then transformed into duration through direct subtraction of datetime-type objects. In this way, we were able to express the time differences between the student’s engagement and certain actions.

Finally, the final dataset is a CSV file with 19 features. Details of the features are as follows (

Table 2. The model’s training features.

Name of the Feature	Description	Range of Values
student_id	The student identification code.	[258 k–259 k]
engagement_quizzes	Student engagement in the quizzes.	1\0
engagement_exams	Student engagement in the exams.	1\0
engagement_lessons	Student engagement in the exams.	1\0
quiz_id	The quiz identification code.	[43–550]
question_id	The question identification code.	[72–1203]
answer_id	The answer identification code.	[294–4545]
answer_correct	If the student has correctly or incorrectly answered the question.	1 \0
course_id	The course identification code.	[2–54]
course_rating	The mark assigned to the course by the student.	[2–5]
minutes_watched	Number of minutes watched per student.	[0.1–322]
exam_attempt_id	The identification of the student’s examination attempt.	[189 k–240 k]
exam_id	The exam identification code.	[118–830]
exam_result	The exam outcomes.	[0–100]
exam_completion_time	The duration of time the student needs to pass the exam.	[0.23–130.0]
exam_category	Category of exam.	[1–4]
exam_duration	Examination duration.	[5–130]
Difference_rated_watched	The duration between the course evaluation date and the student’s engagement date.	[−108–205]
Difference_examcompleted	The time between the exam completion date and the student’s engagement date.	[−66–252]

).

Data pre-processing, which involved cleaning, transforming, and structuring raw data to make it more appropriate for analysis and processing, was a vital and integral component of data science. This significance affected the reliability, accuracy, and quality of the results.

Numerous factors, such as noise, missing numbers, inaccurate data entry, issues with data quality, technical issues, and more, could have led to data loss. To prevent biased or inaccurate outcomes, this required proper processing.

3.2.1. Missing Values

Missing values or data meant that data did not exist in a dataset. Missing values were often present in the dataset and might have been due to errors in data collection or non-recording. False conclusions, biased findings, or loss of information could have been the result of missing values, which could have affected the results and the model’s quality.

For our study, we applied a reasoned treatment based on the nature of the data distribution. In some cases, the mean was used when missing values were limited. In other situations, where this seemed more appropriate, the next available value was used.

As the majority of values were treated during the preparation of our final dataset, we had no missing values. The average of the numerical characteristics was used to process them.

3.2.2. Normalization and Encoding

Normalization was an equal step, generally applied in data pre-processing, which aimed to ensure that all variables are on the same scale. It contributed to the learning process of machine learning models by preventing certain variables from dominating or biasing the model due to their range of values, i.e., variables with high values were of more importance than those with low ones. For many, if not all, machine learning algorithms, this step was essential.

There were three main techniques for normalizing data: Min–Max Scaling, which kept all values in a range between 0 and 1; Standardization, which reduced variable values to a mean of 0 and a standard deviation of 1; and Robust Scaling, which was less sensitive to aberrant values than the other two techniques and employed the median and quartile. Categorical data coding was a manner of converting categorical (non-numerical) data into numerical data because the bulk of machine learning algorithms only take numerical data.

Regarding categorical variables, the majority of machine learning algorithms could only process them in numerical form. It was, therefore, necessary to transform them into numerical values. This could have been done in various ways: for example, ordinal variables were directly encoded with integers representing an order, while nominal variables were transformed by techniques such as One-Hot Encoding, which creates a binary column for each category.

For our dataset, we applied a StandardScaler to standardize numerical variables, ensuring that all variables were on the same scale. Regarding the categorical variables, they were appropriately transformed into numerical values, depending on their type (ordinal or nominal), to ensure their compatibility with the machine learning models used.

3.3. Modeling

For the prediction of learner engagement, we trained several models while optimally adjusting their hyperparameters. This section discusses the different algorithms used to find the best predictive model: SVM, Naive Bayes, Decision Tree, Random Forest, k-NN, Logistic Regression, Gradient Boosting, Ada Boost, and Multi-Layer Perceptron (MLP).

• Support Vector Machine (SVM)

SVM is one of the most popular and robust machine learning algorithms for classification and regression. Vladimir Vapnik first proposed them in 1960 [41]. The main objective of the SVM algorithm is to find a hyperplane able to separate and maximize the margin between different classes in a multidimensional space. According to the training data, this hyperplane is a maximum margin classifier and is therefore considered optimal [42]. The main property of SVMs is their performance and ability to solve binary classification problems, i.e., efficiently separating data into two distinct categories.

• Naive Bayes

Naive Bayes is also a popular machine learning technique based on Bayes’ theorem with considerable conditional independence between variables [43], hence the name Naive [44]. It can predict the class of a sample using the conditional probabilities of the variables given. It is also known as Idiot Bayes, Simple Bayes, and Independence Bayes. Despite the simplicity of its hypothesis, which may reduce its performance in the presence of highly correlated variables, Naive Bayes remains an efficient algorithm, fast and simple to implement.

• Decision Tree

The Decision Tree algorithm is one of the most powerful and commonly used algorithms in machine learning and data mining [45] due to its simplicity of analysis and accuracy on multiple forms of data [46]. This algorithm is represented as an architecture tree composed of nodes, branches, and leaves. Each internal node shows a feature, and each branch represents a decision based on that feature, while each leaf represents the final decision [47,48]. To entrain the best feature at each decision node, the Decision Tree uses various techniques such as entropy, gini index, or information gain. The Decision Tree is developed iteratively, choosing partitions between classes at each stage. Once the tree has been built, it can be used to divide new data according to the decision rules defined in the tree [47].

• Random Forest

Random Forest is a machine learning technique based on a set of Decision Trees that are independent of each other. It combines the prediction results of each Decision Tree. The average of the predictions of each Decision Tree is the final prediction result of the Random Forest.

• KNN

KNN is the most basic classification technique in the field of machine learning. It is founded on the approximation principle, i.e., those similar samples are those that are in proximity to each other [49]. The KNN algorithm relies on calculating the distance between data points, usually using Euclidean distance, although other distance measures can be adopted. Despite its simplicity, KNN can be vulnerable to the value of K (the number of neighbors) and the characteristics of the dataset.

• Logistic regression

Logistic regression is also a classification technique commonly used in machine learning due to its simplicity, interoperability, and ability to provide class probability. Despite what its name may suggest, logistic regression is mainly for binary classification. Its objective is to model the probability that a given observation belongs to one of two possible classes based on its characteristics.

The idea of this algorithm resides in the use of the sigmoid or logistic function to transform the number given by the hyper-right into a probability. In other words, turn a linear combination of characteristics into a probability.

• Gradient Boosting

Gradient Boosting is a machine learning ensemble technique known for its power and accuracy, as it combines several simpler machine learning models to build a more robust model.

This algorithm is based on operating iteratively on simple or weakly weighted models (usually shallow Decision Trees) while adjusting each to correct errors, i.e., minimizing the loss function that estimates the model’s ability to make predictions with the data provided. This is achieved by using optimization techniques.

• Ada Boost

Ada Boost, also known as adaptive boosting, belongs to the family of machine-learning ensemble techniques. It is commonly used in classification problems. As it is an ensemble technique, the final model is based on the combination of several weak models (such as Decision Trees). Its characteristic is that this classifier focuses more on poorly classified examples, adjusting their weights to give more value to the difficult cases when training the additional classifiers created during the iterative process, thus increasing the overall accuracy of the final model.

• Multi-Layer Perceptron

Multi-Layer Perceptron, also known as a multi-layer neural network, is one of the deep neural networks widely used for complex classification, regression, and data modeling problems. It is composed of several interconnected layers of neurons: an input layer, one or more hidden layers, and an output layer. Each layer is based on the output of the previous layer.

A multi-layer neural network relies primarily on the backpropagation algorithm, which iteratively adjusts the weights in the network to minimize the loss function.

In our work, we used a hierarchical neural network architecture as a multi-layer perceptron comprising four distinct layers: an input layer with 16 units representing input features, two similar hidden layers composed of 50 units, and a ReLU activation function. In addition, dropout regularization is used to avoid overfitting by randomly deleting neurons during the training process. Finally, an output layer is composed of a single unit using a sigmoid activation function for binary classification.

To briefly and clearly illustrate the various steps involved in our work, Figure 2 presents a schematic overview of the process that was followed. It includes the main stages: data pre-processing (missing value processing, normalization), feature selection, classification model training, and performance evaluation. This representation provides an overall, structured understanding of the methodology followed.

We had three targets: engagement quizzes, engagement exams, and engagement lessons. For each target, we selected features that would be used. However, for the third ‘Engagement lessons’, we used the results of the metrics evaluating engagement with quizzes and exams as features for this model. This approach seemed logical and coherent for assessing students’ overall engagement in learning for the following reasons:

• Lesson assimilation is assessed using quizzes and exams focusing on lesson content;
• Engagement in quizzes and exams is an indicator of the learner’s overall engagement behavior if the learner actively and fully engages in quizzes and exams and demonstrates engagement behavior in the learning process;
• The fact that learners participate effectively and do well in quizzes and exams testifies to their behavior and involvement in the content taught in the corresponding lessons.

4. Results

This study seeks to build a classification model most adapted to predict learner engagement.

The Student Engagement Dataset from Kaggle and the features were selected to feed our model. The dataset, consisting of 19 features, was created by extracting and combining various columns, and a data pre-processing step was carried out.

Before presenting the results of our analysis, it is essential to understand the process and flow of our work. As mentioned in the previous section, we set three targets: engagement quizzes, engagement exams, and engagement lessons. To predict engagement lessons, we used the results of the engagement quizzes and engagement exams evaluation metrics.

For the study of each model, we involved several key steps: feature selection, testing on several algorithms (namely: SVM, Naive Bayes, Decision Tree, Random Forest, k-NN, Logistic Regression, Gradient Boosting, Ada Boost, and MLP), tuning the parameters of each algorithm to optimize performance. We then divided our dataset into training and test sets to train and evaluate the model. Techniques such as cross-validation were used to ensure model robustness. In addition, evaluation metrics like accuracy, recall, F1-score, and precision. Accuracy presents the number of correct predictions divided by the total number of predictions made. Recall measures the proportion of positive examples correctly classified among all positive ones. The F1-score combines both precision (minimizing false positives) and recall (correctly identifying positive examples) in a single measure. Precision measures the proportion of true positives among all the positive predictions in the model. We then carried out an in-depth comparison of each algorithm’s performance. These tests enabled us to determine which algorithm would perform best.

Before presenting the results, it is important to justify the choice of models used. These models were chosen carefully, depending on their performance, interpretability, and adaptability to the e-learning system:

• Decision Tree is highly recognized for its readability and transparency, and the extracted rules are easy for technical experts to interpret. In an educational context, it is important, even crucial, to understand why a model predicts a certain behavior for a learner.
• KNN was selected for its simplicity and effectiveness in capturing similarities, making it particularly relevant to apply in personalized or anticipatory recommendations
• Gradient Boosting was chosen for its high performance in structured data and its ability to capture complex interactions while limiting over-learning thanks to regularization mechanisms.
• Concerning the other models not retained,
• SVM, despite its power, presents significant computational complexity for nonlinear kernels, as well as its visibility in large-scale online systems;
• Logistic Regression does not exploit nonlinear interactions between variables at the level of learner behavior;
• Random Forest is conceptually similar to Gradient Boosting, but offers better performance and overfitting control.

As a next step, we examined

Table 3. Evaluation metrics of the used models.

Model	Engagement Quizzes				Engagement Exams				Engagement Lessons
	Accuracy	Recall	F1-Score	Precision	Accuracy	Recall	F1-Score	Precision	Accuracy	Recall	F1-Score	Precision
SVM	0.773	1.00	0.87	0.77	0.616	0.280	0.39	0.63	0.987	1.0	0.993	0.987
Naive Bayes	0.771	0.983	0.87	0.78	0.647	0.418	0.51	0.65	0.662	0.658	0.794	1.0
Decision Tree	0.952	0.964	0.97	0.97	1.0	1.0	1.00	1.00	0.984	0.984	0.992	0.984
Random Forest	0.795	0.999	0.88	0.79	0.808	0.669	0.75	0.86	0.984	1.0	0.992	0.984
k-Nearest Neighbors	0.944	0.967	0.96	0.96	0.991	0.987	0.99	0.99	0.984	0.988	0.992	0.995
Logistic Regression	0.767	1.0	0.868	0.767	0.597	0.300	0.397	0.586	0.987	1.0	0.993	0.987
Gradient Boosting	0.949	0.984	0.97	0.95	0.935	0.868	0.92	0.98	0.984	1.0	0.992	0.984
Ada Boost	0.854	0.854	0.91	0.85	0.824	0.824	0.78	0.86	0.984	0.984	0.992	0.984
Multi-Layer Perceptron(MLP)	0.767	0.999	0.87	0.77	0.718	0.619	0.66	0.70	0.984	1.0	0.992	0.984

, which compares the evaluation metrics of the algorithms used to predict each target. The analysis of these metrics helped us identify the model that best met our prediction objective.

According to the table, our main concern, which is also the core of our work, lies in identifying the best performing for predicting learner engagement. As distinguished, our model takes into account a variety of data sources, encompassing the learner’s multiple interactions with their learning environment. Our ultimate goal was to predict the level of engagement of these learners within their learning paths.

In this context, the following figure (Figure 3) represents a detailed comparison of the performance of three algorithms (Decision Tree, KNN, and Gradient Boosting) in predicting the three target engagement quizzes, engagement exams, and engagement lessons. Each algorithm is assessed using four evaluation metrics: accuracy, recall, F1-score, and precision. The metrics are represented on the y-axis, while the colored bars represent the performance of each. By examining the graph attentively, we can quickly identify which algorithm performs best for each engagement target.

The three models, Decision Tree, KNN, and Gradient Boosting, were carefully parameterized and evaluated to determine their ability to predict learner engagement on specific targets. The hyperparameters were adjusted using the test-and-error method. Although this empirical approach provides a fast view of performance, the integration of rigorous cross-validation combined with systematic hyperparameter optimization would enhance the robustness and generalizability of the results, particularly in a large-scale deployment. The details of each model are as follows:

• Decision Tree
  • The model was trained using entropy as the criterion for building the Decision Tree;
  • Training and test data were standardized using StandardScaler;
  • Model performance was evaluated in predicting engagement on the three targets.
• KNN
  • The model was trained with n_neighbors = 3 to make decisions;
  • Training and test data were standardized using StandardScaler;
  • Model performance was evaluated in predicting engagement on the three targets.
• Gradient boosting
  • Optimal parameters were chosen, including n_estimators = 100, learning_rate = 0.1 and random_state = 42 to ensure reproducibility of results;
  • Training and test data were standardized using StandardScaler;
  • Model performance was evaluated in predicting engagement on the three targets.

Each of these models has its strengths and weaknesses, and evaluating their performance gives us a better understanding of their suitability for engagement prediction tasks.

Several algorithms were tested to predict engagement from extracted features. Of these, Decision Tree, KNN, and Gradient Boosting performed best. In addition to the performance obtained, it is worth noting some of the limitations of these algorithms, as identified in the literature. Naive Bayes, for example, assumes independence between data, which can lead to an underestimation of performance when variables are highly correlated. SVM, while effective for well-separable data, can be sensitive to the choice of kernel and parameters. Finally, Random Forest, Logistic Regression, and Ada Boost can be affected by parameter overfitting, particularly in the presence of noise. This passage allows us to contextualize the performances obtained and shows that, despite theoretical limitations, the algorithms chosen responded well to the objectives of modeling learners’ behavioral engagement.

Figure 4 shows the precision–recall curve for evaluating the performance of the three algorithms. The precision–recall curve gives a perspective for evaluating the performance of classification models. All three models show well-determined precision and recall for the three targets, represented by the classification of many examples as positive. However, the Decision Tree curve represents the highest precision and recall compared to the others. This model was trained using entropy as a criterion for building the Decision Tree. Training and test data were standardized using StandardScaler.

5. Discussion

Machine learning is arguably an essential method and concept for predicting results from training data. It is known for its performance and the results it delivers. Indeed, machine-learning techniques, such as classification, have become ubiquitous in complex data analysis. These approaches make it possible to build predictive models from a dataset while developing significant knowledge in various domains.

This research aims to contribute to the existing literature on e-learning engagement by proposing a predictive model to classify learners according to their engagement in an e-learning environment. This model could detect learners who may be having trouble or exhibiting signs of disengagement, allowing them to provide targeted support and assistance. We used supervised machine learning algorithms to predict learner engagement. We trained these models on a significant dataset and assessed their performance using several evaluation metrics, including precision, recall, accuracy, and F1-score. The creation of precision-recall curves offers a comprehensive overview of each model. These tools generate essential information for anticipating learner behavior, paving the way for more precise and effective educational interventions.

Highly engaged learners tend to participate more effectively in different activities and may show indications of engagement that are not as evident in less engaged learners. Even so, this suggests that the approaches employed may be valid for detecting early signs of difficulties or academic failure in learners. Relying not only on learning outcomes, this requires the intervention of tutors or a personalized referral system to identify strengths and develop gaps, along with goal-setting to improve each learner’s learning throughout the learning journey [50].

The analysis indicated that the Decision Tree, KNN, and Gradient Boosting models demonstrated relatively higher performance in data categorization than the other algorithms. Each algorithm has features that make it suitable for the task of predicting engagement in an online learning environment. For example, the Decision Tree is valued for its simplicity of interpretation and its ability to handle complex data with little pre-processing, which is crucial in diverse e-learning environments. KNN, on the other hand, is well suited to capturing similarities between learners, while Gradient Boosting offers solid predictive performance, albeit at a higher computational cost. These specific advantages explain why we have selected these models to predict learner engagement.

These results suggest a linear relationship between inputs and learner engagement. Indeed, these classification algorithms produced better results, indicating an increased ability to predict learner behaviors based on their activities, an indicator of engagement.

The prediction model for the Decision Tree algorithm for target engagement_quizzes had an accuracy of 95.27%, a recall of 96.44%, an F1-score of 97%, and a precision of 97%. For engagement_exams, accuracy, and recall, the F1-score is 100%. Moreover, for engagement_lessons, accuracy is 98.49%, recall is 98.49%, F1-score is 99.24%, and precision is 98.49%. For target engagement_quizzes, accuracy, recall, f1-score, and KNN prediction model precision were 94.47%, 96.76%, 96%, and 96%, respectively. For the engagement_exams target, accuracy, recall, f1-score, and KNN prediction model precision were 99.18%, 98.79%, 99%, and 99%, respectively. For engagement_lessons, accuracy, recall, f1-score, and KNN prediction model precision were 98.43%, 98.86%, 99.2%, and 99.5%, respectively. The Gradient Boosting model, for target engagement_quizzes, accuracy, recall, f1-score, and precision were 94.97%, 98.46%, 97%, and 95%, respectively. For the target engagement_exams, the accuracy, recall, F1-score, and precision of the KNN prediction model were 93.54%, 86.87%, 92%, and 98%, respectively. Finally, for engagement_lessons, accuracy, recall, f1-score, and precision were 98.49%, 100%, 99.24%, and 98.49%, respectively.

The results obtained from the models reveal a strong correlation between predictions of engagement in quizzes, exams, and lessons. These results confirm that these items, namely quiz and exam engagement scores, referred to as assessment engagement, can be considered relevant indicators of lesson engagement. As mentioned above, engagement in these assessments directly correlates with understanding the concepts taught. Thus, learners who actively engage in lessons are more likely to excel and engage in quizzes and exams. This reinforces the validity of our approach, which considers quiz and exam engagement as robust indicators of lesson engagement.

Indeed, engagement in quizzes and exams reflects the effort made by learners to understand the concepts covered in the lessons. Although these two targets are distinct, they are all interconnected, providing a comprehensive overview of learner engagement.

The precision–recall curve is a commonly used evaluation tool in machine learning to measure the performance of classification models. For all three algorithms—namely, Decision Tree, KNN, and Gradient Boosting—three models were trained to predict learner engagement. In these models, the curves (Figure 4) showed a good balance between precision and recall, illustrated by their typical decreasing shape. This means that an increase in recall generally leads to a decrease in precision.

In terms of lesson engagement, the curve showed a small decline in accuracy compared to quizzes and exams, but recall remained high. Analysis of the results also revealed similar performance for quiz and exam engagement, indicating consistency in learners’ assessment behaviors. We can deduce that learners engaged in quizzes are also engaged in exams, confirming the validity and relevance of these indicators.

As shown in Figure 3, the Decision Tree model presented the best evaluation metrics, while the neural network model obtained the lowest results compared to the other algorithms. The model evaluation presents an in-depth assessment of the model’s performance on a dataset to measure its ability to predict future data. It also enabled us to measure its ability to predict learner engagement in e learning about their behaviors.

To corroborate our results, we also compared actual labels with those predicted by the best-performing algorithms to obtain a more nuanced view of their effectiveness and to assess the accuracy of the models on real data:

• Decision Tree—Engagement quizzes;
• Decision Tree—Engagement exams;
• Decision Tree—Engagement lessons (results of quizzes and exams will not be presented, we will only present other features);
• KNN—Engagement quizzes;
• KNN—Engagement exams;
• KNN—Engagement lessons (results of quizzes and exams will not be presented, we will only present other features);
• Gradient Boosting—Engagement quizzes;
• Gradient Boosting—Engagement exams;
• Gradient Boosting—Engagement lessons (results of quizzes and exams will not be presented, we will only present other features).

The results in the tables highlight a correspondence between the real labels and the predictions of all three algorithms, showing the robustness of our models for classifying learner engagement. The Decision Tree model performed best, with an almost perfect correspondence between real and predicted labels. However, some differences were observed in KNN predictions, possibly revealing a sensitivity to certain types of data. Regarding the Gradient Boosting model, although it performed well, it showed more errors, probably due to the complexity of the data.

For the Decision Tree algorithm,

Table 4. Comparison of real and predicted labels for quiz engagement—Decision Tree model.

Index	Feature	Feature Value	Prediction for Engagement_Quizzes	True Label for Engagement_Quizzes
0	student_id	277,353	0	0
	quiz_id	539
	question_id	72
	answer_id	294
	answer_correct	1
5	student_id	281,295	1	1
	quiz_id	43
	question_id	72
	answer_id	294
	answer_correct	1
10	student_id	279,508	0	0
	quiz_id	539
	question_id	789
	answer_id	3109
	answer_correct	1

,

Table 5. Comparison of real and predicted labels for exam engagement—Decision Tree model.

Index	Feature	Feature Value	Prediction for Engagement_Exams	True Label for Engagement_Exams
0	student_id	277,353	0	0
	exam_id	758
	exam_category	1
	exam_duration	30
	exam_attempt_id	225,667
	exam_result	100
	exam_completion_time	4.45
	Difference_examcompleted_engaged	70
5	student_id	281,295	1	1
	exam_id	755
	exam_category	2
	exam_duration	10
	exam_attempt_id	217,024
	exam_result	0
	exam_completion_time	3.92
	Difference_examcompleted_engaged	0
10	student_id	279,508	0	0
	exam_id	758
	exam_category	1
	exam_duration	30
	exam_attempt_id	212,786
	exam_result	58
	exam_completion_time	11.8
	Difference_examcompleted_engaged	20

and

Table 6. Comparison of real and predicted labels for lesson engagement—Decision Tree model.

Index	Feature	Feature Value	Prediction for Engagement_Lessons	True Label for Engagement_Lessons
0	student_id	272,842	1	1
	course_id	37
	course_rating	5
	minutes_watched	14.8
	Difference_rated_watched	4
5	student_id	286,529	1	1
	course_id	21
	course_rating	5
	minutes_watched	14.0
	Difference_rated_watched	45.7
10	student_id	277,322	1	1
	course_id	42
	course_rating	5
	minutes_watched	45.7
	Difference_rated_watched	−3

show that the Decision Tree algorithm exhibits excellent correspondence between actual labels and predictions for all three targets (quiz, exam, and lesson engagement) on randomly drawn indexes, indicating good model performance. Then, for the KNN algorithm, the results in

Table 7. Comparison of real and predicted labels for quiz engagement—KNN model.

Index	Feature	Feature Value	Prediction for Engagement_Quizzes	True Label for Engagement_Quizzes
2174	student_id	289,270	1	0
	quiz_id	539
	question_id	789
	answer_id	3109
	answer_correct	1
20	student_id	268,011	0	0
	quiz_id	539
	question_id	789
	answer_id	3109
	answer_correct	1

,

Table 8. Comparison of real and predicted labels for exam engagement—KNN model.

Index	Feature	Feature Value	Prediction for Engagement_Exams	True Label for Engagement_Exams
2174	student_id	268,011	0	0
	exam_id	707
	exam_category	2
	exam_duration	20
	exam_attempt_id	232,336
	exam_result	100
	exam_completion_time	6.38
	Difference_examcompleted_engaged	20
10	student_id	268,011	0	0
	exam_id	403
	exam_category	1
	exam_duration	25
	exam_attempt_id	206,211
	exam_result	75
	exam_completion_time	25
	Difference_examcompleted_engaged	16

and

Table 9. Comparison of real and predicted labels for lesson engagement—KNN model.

Index	Feature	Feature Value	Prediction for Engagement_Lessons	True Label for Engagement_Lessons
4588	student_id	279,508	1	1
	course_id	14
	course_rating	5
	minutes_watched	9.6
	Difference_rated_watched	−16
20	student_id	268,011	1	1
	course_id	3
	course_rating	5
	minutes_watched	107.4
	Difference_rated_watched	58

confirm that the predictions are globally consistent, but a notable error was observed for quiz engagement, particularly for index 2174 in Table 7, suggesting a sensitivity of the model to certain types of data. For the Gradient Boosting algorithm,

Table 10. Comparison of real and predicted labels for quiz engagement—Gradient Boosting model.

Index	Feature	Feature Value	Prediction for Engagement_Quizzes	True Label for Engagement_Quizzes
2174	student_id	289,270	0	0
	quiz_id	539
	question_id	789
	answer_id	3109
	answer_correct	1
20	student_id	268,011	0	0
	quiz_id	539
	question_id	789
	answer_id	3109
	answer_correct	1

,

Table 11. Comparison of real and predicted labels for exam engagement—Gradient Boosting model.

Index	Feature	Feature Value	Prediction for Engagement_Exams	True Label for Engagement_Exams
10	student_id	268,011	0	0
	exam_id	403
	exam_category	1
	exam_duration	25
	exam_attempt_id	206,211
	exam_result	75
	exam_completion_time	25
	Difference_examcompleted_engaged	16
2174	student_id	289,270	0	0
	exam_id	707
	exam_category	2
	exam_duration	20
	exam_attempt_id	232,336
	exam_result	100
	exam_completion_time	6.38
	Difference_examcompleted_engaged	20

and

Table 12. Comparison of real and predicted labels for lesson engagement—Gradient Boosting model.

Index	Feature	Feature Value	Prediction for Engagement_Lessons	True Label for Engagement_Lessons
2174	student_id	289,270	1	1
	course_id	2
	course_rating	4
	minutes_watched	32.3
	Difference_rated_watched	2
20	student_id	268,011	1	1
	course_id	3
	course_rating	5
	minutes_watched	107.4
	Difference_rated_watched	58

also show a good correspondence between actual labels and predictions for the three targets, similar to that of the Decision Tree, demonstrating that the Gradient Boosting model is also robust in predicting learner engagement.

These remarks lead us to recommend the use of the Decision Tree model in environments where high accuracy is crucial for predicting engagement. However, the KNN model proves more suitable in contexts where similarity between learners plays a crucial and determining role.

Overall, these results offer promising prospects for the implementation of pedagogical interventions based on engagement prediction.

We also noted that some characteristics unquestionably influence learner engagement; however, others have a weaker impact. This indicates that although some learners are globally engaged in their learning, they may display different levels of involvement depending on the attributes taken into account. These variations underline the value of tailoring the learning process to the specific needs of each learner.

In terms of practical applications, these results can be used to develop personalized learning systems that quickly identify learners at risk of dropping out. Based on these predictions, recommendations for appropriate pedagogical interventions can be proposed, enabling systems to anticipate learners’ difficulties and adjust their learning processes accordingly. This opens up interesting prospects for improving the quality of e-learning.

Computational efficiency is a key factor in the deployment of e-learning platforms. Decision Tree is more adapted to environments requiring regular maintenance and continuous recalibration, given its fast training and inference speed, even on moderately-sized datasets. However, KNN, while offering significantly better performance with easier learning, requires a higher inference time, especially at large scales, as it has to traverse the entire set of examples for each prediction. Gradient reinforcement requires more training time owing to the sequential nature of the boosted trees, but presents good robustness and reasonable inference time once training is complete.

However, in an e-learning context, the choice of model depends not only on its performance, but also on its ability to adapt to operational constraints such as real-time response time, scalability of updates, and transparency.

Although this study focuses on behavioral engagement, it is essential to highlight the impact of cultural and institutional biases on learner engagement. For example, student-country differences can significantly influence student engagement. Although these biases were not taken into account in the models of this study, it would be relevant to incorporate them in future work. This could include analyzing learner subgroups and adapting models to cultural and institutional specificities, to better personalize learning. By taking these dimensions into account, learning systems could offer a more inclusive and equitable experience, fostering stronger engagement and better academic success for a diverse population of learners.

Furthermore, demographic factors such as socioeconomic status or educational history can affect the prediction or even distort it, risking misclassifications that can eliminate some learners from personalized support. This risk can be attenuated by incorporating algorithmic regularization techniques, model auditing with equity indicators (subgroup evaluation), diversification of training models, or human intervention before an automated action is triggered.

These approaches simultaneously reduce implicit discrimination and guarantee automation to strengthen decision-making and the ethics of AI systems.

6. Conclusions

The lack of interaction in learning environments is a serious challenge. Predicting learner engagement is of key importance, enabling us to understand the interaction and behavior of each learner. It opens the way to early detection of difficulties, personalization of learning paths, immediate feedback, interactive communications, and the creation of a sense of belonging. Our work is in line with previous work that focused mainly on behavioral or emotional dimensions and stands out for its integration of concrete data, measurable indicators derived from learner interactions (quizzes, exams), and the application of machine learning techniques. In this context, we tested nine machine learning algorithms on a Kaggle database. The database was subjected to various pre-processing techniques. The performance of these algorithms was evaluated using measures such as accuracy, recall, F1 score, and precision. The Decision Tree model proved to be the best-performing of those evaluated.

In the future, we plan to explore other dimensions of online learner engagement, including emotional and cognitive aspects. Specifically, this will include factors such as learning style, abundance, and learning materials, as well as other relevant parameters such as frequency of access to courses, participation in discussion forums, and collaboration with other learners. There are, however, several limitations. Our study relies in part on self-reported data, which may introduce a degree of subjectivity on the part of learners. Furthermore, the absence of real-time interventions limits the possibilities for immediate personalization and reactive adaptation as part of the learning process. These aspects represent future research prospects aimed at further enriching our approach. By considering these multiple dimensions, we intend to gain a holistic view of student engagement, enabling a more precise adaptation of pedagogical strategies and ongoing optimization of the online learning experience.