Predicting At-Risk Students in an Online Flipped Anatomy Course Using Learning Analytics

Abstract

When using the flipped classroom method, students are required to come to the lesson after having prepared the basic concepts. Thus, the effectiveness of the lecture depends on the students’ preparation. With the ongoing COVID-19 pandemic, it has become difficult to examine student preparations and to predict student course failures with limiting variables. Learning analytics can overcome this limitation. In this study, we aimed to develop a predictive model for at-risk students who are at risk of failing their final exam in an introductory anatomy course. In a five-week online flipped anatomy course, students’ weekly interaction metrics, quiz scores, and pretest scores were used to design a predictive model. We also compared the performances of different machine learning algorithms. According to the results, the Naïve Bayes algorithm showed the best performance for predicting student grades with an overall classification accuracy of 68% and with at-risk prediction accuracy of 71%. These results can be used as a traffic light project wherein the “at-risk” group will receive the red light, and thus, will require more effort to engage with the content and they might need to solve the quiz tests after an individual study period.

1. Introduction

Flipped classrooms (FC) are a teaching method that flips traditional teaching methods and makes it possible to spare time in class activities [1,2,3]. In traditional classrooms, students usually participate in classroom activities with no preparation. The teacher provides the basic concepts related to the lesson and the expectation is that students will understand and comprehend these concepts. Since the in-class activity time is limited, they devote most of the lesson time to these activities [4]. At the end of the lesson, the teacher assigns students homework that includes higher-level cognitive tasks like application, analysis, and synthesis [5,6,7]. However, since the students are out of the classroom, these tasks are very difficult to complete without a guide. The FC approach removes these limitations of traditional classrooms [8,9]. Teaching the basic concepts related to learning goals before the lesson by presenting different resources not only ensures that different student characteristics are considered [10] but requires students to reach these goals before the lesson [11]. At the beginning of the lesson, a short quiz is applied in which the lesson preparations of the learners are determined [12,13]. The quiz allows for the students’ deficiencies to be determined as well as for feedback to be given to the student and tutor. In the remaining class time, activities that activate higher-level cognitive processes, such as classroom discussions, problem-solving, and case-based learning processes are implemented [14,15,16]. Within this paradigm, the teacher is a guide and student-centered learning processes are applied [17,18].

FC practices increase students’ interactions with their content and enable more in-class activities [8], increase student motivation [19,20], enable the retention of the information [21,22], improve problem-solving skills [23,24], and increase learner satisfaction. With the COVID-19 period, in particular, online flipped classroom applications have become widespread [25,26]. The synchronous participation of students in their courses may be limited in distance education, and this limited period of availability must be supported by application activities. Learning management system (LMS) interactions with content that includes course resources such as educational videos, online discussions, documents, and simulations have grown increasingly relevant and also support FC applications. While students who participate in a lecture following preparation can benefit more from online education, students who cannot make these preparations may not learn effectively and may fail the course. The decrease in student–instructor interactions and tracking, the difficulty of directly observing learners who are at risk of failing, as well as the difficulty of intervening before failure occurs, are among the most important limitations of flipped classroom practices. However, learning analytics can overcome these limitations.

Learning analytics consists of four elements: collecting data on learners and the learning environment, pre-processing the data collected, applying machine learning algorithms and other statistical methods to determine the patterns of the students, and interventions to increase success and prevent failure [27,28,29]. These components also enable the creation of educational early warning systems. An early warning system aims to predict academic performance or course achievement at an earlier stage [27]. Identifying students at risk of academic failure and providing them with the help they need may help these students succeed. “At-risk” refers to students at risk of failing either their final exam or failing to progress [30]. Identifying students who are at risk of failing or not completing courses can benefit higher education institutions [31]. For this purpose, researchers or instructors can apply learning analytics to create predictive models to enable the early detection of students with failure risk and provide them with the appropriate feedback and intervention. One of the most widely used techniques in early warning systems is supervised learning. These techniques are efficient in helping to solve non-linear real-time problems as classification plays a vital role in machine-based learning algorithms [32]. Supervised machine learning is a technique in which machines are trained using data that is labeled with attributes, and based on this data, the output is predicted in the previous experience. Classification is a supervised machine learning concept, where a class label is predicted for a sample of input data. In previous studies, predictive models were created based on classification methods; researchers usually compared five classifiers to forecast accuracy and identify the best performing classification algorithm. These classifiers are k-nearest neighbors (kNN) [33], decision trees (DT) [34], naïve Bayes (NB) [35], random forest (RF) [36], and support vector machines (SVM) [37]. In summary, to create an early warning system in an online course that is flipped: (1) students’ LMS interactions must be transformed into a weekly interaction metric and merged with their quiz scores for each week; (2) data pre-processing procedures must be applied for an appropriate predictive model; (3) a model must be created to predict the students’ final grades with two-class (pass/fail or safe/at-risk) classification and the best performing algorithm must be determined based on the model performance metrics; (4) the model must be used as an early warning system for at-risk students before the final exam. This “warning message” represents the most important stage of learning analytics, “interventions”.

Hung, Shelton [38] applied machine learning techniques to develop a general model for higher education and K-12. They used several variables including gender, grade, LMS interactions, content views, participation in online discussions, number of replies to comments, and student grade checks. For classification predictions, they applied three commonly used machine learning techniques: SVM, deep neural network (DNN), and RF. According to the results, DNN and RF showed accuracy rates in predicting at-risk students of between 82.6% and 85% for the K-12 data and between 95% and 97% for higher education data. Hung, Wang [39] applied time-series clustering to early identify at-risk online students. Their study aimed to deliver more accurate predictions than those of traditional analysis and to identify the at-risk patterns of the students based on the model, which comprise interaction and personal student data. Students’ LMS interaction data, including the number of course materials accessed, number of forums read, number of discussions posted, as well as the number of replies from these environments, were collected. In addition, other variables such as the final grade (which was also used as a supervised variable), gender, ethnicity, and admission status were included in the analysis. The results showed that a time-series approach was able to identify at-risk students during the semester for early intervention. In addition, the decision tree was the best performing and most stable model. Akçapınar, Altun [27] employed learning analytics to design and develop an early warning system for at-risk students. Their study aimed to compare the classification accuracy of machine learning algorithms in identifying at-risk students and to investigate the effect of different pre-process applications on prediction performance. In their study, they used LMS interaction features including several question ratings, unique session days, unique posting days, the total number of posts, the number of total sessions, the number of tags created, session duration, and the number of responses to the discussion forums. They included student grades in the analyses as a supervised learning variable for different classification techniques, such as classification tree, CN2 rules, NB, neural network, k-NN, RF, and SVM. The k-NN algorithm accurately predicted at-risk students at the end of the term with a rate of 89%. In addition, students who were at-risk at the end of term were predicted to a rate of 74% in 3 weeks. Saqr, Fors [29] applied learning analytics to a medical course for the early detection of under-achieving students in a blended learning environment. In their predictive study, they analyzed the LMS interactions of 133 medical students in order to develop a model for final grades. The collected data included Moodle LMS login counts, number of course views, number of forum views, total time spent in LMS, and formative assessment scores. According to their results, the predictive model showed 63.5% accuracy and 53.9% of at-risk student prediction. They also analyzed the predictive importance of the linear modeling and found login counts and views of course information were the most important measures for the predictive model.

Purdue University used the “course signals” project as an early warning system in their LMS [40]. Students are unaware of how they are progressing within their courses, for example, if they only take a few assessments during their courses and do not see their final grades until the end of the term. Signals remedy this problem. The components, including students’ performance, effort, prior academic achievements, and student characteristics, are weighted and fed into an algorithm. This creates a traffic signal based on the probability of success or failure [40,41]. The red light shows a high likelihood of failure, yellow shows potential problems, and green shows a high likelihood of success. Motivational feedback typically includes remarks like “Well done” or “Fantastic”, whereas informative feedback provides details on the learner’s progress, and how it compares to where they were previously or to others. A green traffic signal represents positive feedback, which the instructor can reinforce with a message of encouragement [42]. An example of this kind of intervention is a study by Foster and Siddle [30] who used an LMS that generated “no-engagement” alerts if students did not engage with any resources for 14 consecutive days. They found that learning analytics-based interventions had a positive effect on student engagement and participation.

Studies in the literature frequently used the flipped classroom method in anatomy lessons [21,43,44,45,46]. This is because the basic concepts in anatomy are suitable for the level of knowledge and comprehension and can be given to students as homework. In-class activities may be administered via the means of problem-based scenarios, case-based discussions, and question–answer techniques. Students may not use higher-level cognitive activities in out-of-class activities. A well-designed flipped anatomy course can provide a deeper learning process via in-class activities. However, the prediction of course success based on student interactions and quiz scores in flipped classrooms is a subject that requires further investigation as it has the potential to aid in the prevention of student failure. Thus, this study aims to create a predictive model to identify at-risk students in an online flipped anatomy course. The study aims to answer two research questions: (1) Which supervised learning technique can predict at-risk students in an online flipped anatomy course? (2) What is the classification accuracy of the best algorithm for predicting at-risk students in an online flipped anatomy course?

2. Materials and Methods

2.1. Participants

This study aimed to predict at-risk students in a flipped classroom. Thus, the study included pharmacy students enrolled in an anatomy course during the 2020–2021 academic year; 69 out of 75 first-year pharmacy students took part in the study. Every participant in this study volunteered to participate and their log data were collected in the Moodle LMS. The researchers then combined exam scores with the pre-processed log data.

2.2. Flipped Anatomy Course

We implemented this study as a part of a compulsory anatomy course (PHM101) for undergraduate pharmacy students who are studying at a public university in Asia. Before the application, an exam was applied to determine the students’ prior knowledge within the scope of the course. This exam aimed to measure the basic anatomy concepts given at the beginning of the course as well as the knowledge regarding related subjects before their enrollment in the course. The data collection procedure was completed within a six-week period of the course. Details of the course schedule and topics have been provided in the table.

The course was delivered fully online via the flipped classroom method during the COVID-19 lockdown period. The instructor shared educational videos and resources related to course topics online on LMS. Students were expected to prepare for the relevant resources during the period between the sharing date of the resources and the date of the lesson. There are time intervals in the “student preparation” column in

Table 1. Data collection weeks, exams, in-class activities, and variable names.

Weeks	Course Topics	Student Preparation	In-Class Exam	In-Class Activities	Variables Names
-	Prior Knowledge Exam: 28 November 2020				Prior
1	Urinary System (US)	December 1–7	Quiz US	Case Studies and Discussions	w1_eng & exam1
2	Reproductive System (RS)	December 8–14	Quiz RS	Case Studies and Discussions	w2_eng & exam2
3	Nervous System (NS)	December 15–22	Quiz NS	Case Studies and Discussions	w3_eng & exam3
4	Spinal Cord and Spinal Plexuses (SCSP)	December 23–29	Quiz SCSP	Case Studies andDiscussions	w4_eng & exam4
5	Cranial Nerves and Autonomic Nervous System (CNAN)	December 30– January 9	Quiz CNAN	Case Studies andDiscussions	w5_eng & exam5
6	Final Exam: 10 January 2021				final (target variable)

. At the beginning of the lesson, quizzes were given to measure student preparation. Students’ misconceptions and errors in the quizzes were discussed; discussions and question–answer techniques were used during the in-class (synchronous virtual classroom lessons) activities.

2.3. Variables

In this study, we exported Moodle raw data as an Excel file comprising 119,517 lines with 9 columns: time, username, affected user, event context, component, event name, description, origin, and IP address. We used the MoodleMiner tool (http://bote2.hacettepe.edu.tr/moodleminer/; accessed on 25 January 2022) to transform the raw dataset into a transactional format in which the students were placed as unique rows and their interaction variables were placed as columns to be analyzed [47]. The raw data comprised 34,277 lines, including every students’ interactions with the course contents. After pre-processing, seven variables were exported; these are given in

Table 2. Pre-processed Moodle interaction variables and their descriptions.

No	Interaction Variable	Description
1	n_session:	The number of sessions by the student
2	n_ShortSession	The number of short sessions by the student
3	d_Time:	The total time the student has spent on the Moodle LMS
4	n_UniqueDay	The number of unique days logged in by the student
5	n_TotalAction	The number of total activities
6	n_CourseView	The number of course (Anatomy) views
7	n_ResourceView	The number of course resource views

.

MoodleMiner calculates the “engagement score” by using these seven variables. As a pre-process, the tool calculates percentile rank by averaging seven variables. Thus, the calculated value is called the “engagement score”, and it also provides percentile values. For instance, if a student’s engagement score is 90, this also means that their activity is higher than the 90% percent of other students.

In this study, we calculated the “engagement score” for each week of data collection and merged them with students’ preparations (in-class quiz scores) for each week (Table 1, variables column). We aimed to create a predictive model for the final score (target variable/dependent variable) by using eleven independent variables (prior, exam1, exam2, exam3, exam4, exam5, w1_eng, w2_eng, w3_eng, w4_eng, and w5_eng).

2.4. Data Analysis

To answer the research questions, we applied the knowledge discovery in databases process (KDD). The KDD is the development of good measures of interesting and discovered patterns [48]. According to Reference [49], data mining, defined as the extraction of interesting (non-trivial, implicit, previously unknown, and potentially useful) patterns or knowledge, plays an important role in the KDD process. Al Shalabi, Shaaban [50] emphasized that data mining, which seeks to discover unrecognized associations between data items in an existing database, is extracting valid, previously unseen or unknown, and comprehensible information from large databases. KDD consists of five steps: data selection, data pre-processing, data mining, pattern evaluation, and knowledge discovery [49].

We used the Orange data mining (DM) tool for each step of the KDD process (Figure 1). The tool is useful for visual programming and explorative data analysis and can be written in Python. Orange has multiple components known as widgets [51] which are dragged–dropped and connected to each other for the KDD procedures. In our predictive model, we applied the steps of the KDD process. These steps are explained in the following section.

2.4.1. Data Selection

We only combined students’ Moodle interaction and exam scores. The log data usernames field was used to collect the interaction features of the students. This information was merged with students’ exam scores, final scores, prior knowledge scores, and in-class quiz scores.

2.4.2. Data Preprocessing

The discretization process is one of the most important data preprocessing tasks in data mining. Presently, many discretization methods, including Boolean reasoning, equal frequency binning, and entropy, are used [52]. All our continuous variables were transformed into two categories, which were divided based on cut-off scores. The cut-off scores for the engagement scores of each (w1_eng—w5_eng) week were decided at 50 points. This is because calculating these values between 0 and 100 based on the percentile rank and the median value (50) of this score can classify students with high engagement (1) and low engagement (0). Cut-off scores for exam results (prior and exam1–exam5) were decided as 60 points, which is also a passing grade for the courses at our university. For the exam, scores above 60 points were passed (1), while all other scores and the students who did not take the exams were coded as failed (0). The cut-off score for the final examination was also determined to be 60 points. Therefore, at-risk students were coded as 0 (below 60) and we coded the others as safe (1). Discretization makes it possible to apply supervised learning techniques to detect at-risk students via classification techniques [27,53].

2.4.3. Data Mining

This study aimed to create a predictive model for pharmacy students’ final scores based on their Moodle interaction data and flipped classroom preparations. For this purpose, we applied five commonly used classification algorithms: k-nearest neighbors (kNN), decision trees (DT), naïve Bayes (NB), random forest (RF), and support vector machines (SVM).

The k-nearest neighbors (kNN) algorithm is also called the lazy learning algorithm and has been successfully applied in real applications with big data [33]. The algorithm uses several distance metrics, such as Euclidian and Manhattan distances, to solve the classification problem of a categorical dependent variable.

Decision trees (DT) are also used for classification problems for categorical dependent variables. Researchers use DT-based learning algorithms in prediction or rule induction solutions for educational problems. Investigating the effect of demographic characteristics such as regional, socio-economic, and educational levels, age, gender, and disability status on academic outcomes in the online learning environment [54] is an example of the use of decision trees in education.

Naïve Bayes (NB) classification aims to determine the class of dependent variable and is calculated by the probabilities of the classes [55]. Predicting students’ graduation rates via their various achievement indicators [35] is an example of the usage of the NB algorithm in an educational setting.

The random forest (RF) algorithm is a statistical framework with a very high generalization accuracy and quick training times for classification tasks [56]. RF uses tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees, which comprise randomly selected dependent and independent variables in the forest [57]. The prediction of university students’ academic success via course examination results [36] is an example of random forest classification approach.

Support vector machines (SVM) aim to identify a hyperplane which can meet classification requirements [58]. SVM can identify students’ literacy skills (low, high) based on eye movement metrics during text reading [59] or classify learners in multi-class (high, average, or low) categories.

2.4.4. Performance Evaluation

Since our study aimed to classify students as either at-risk or safe and compare machine learning classification methods based on the accuracy of prediction models, we used performance metrics and confusion matrices for model evaluation. The confusion matrix is provided in

Table 3. Structure of confusion matrix.

		Predicted Values
		At-Risk	Safe
Actual Values	At-Risk	TP	FP
	Safe	FN	TN

, divided into four sections representing at-risk and safe situations in a binary classification problem.

TP (True Positive): The number of unsuccessful students that were classified as “at-risk”.

TN (True Negative): The number of successful students that were classified as “safe”.

FP (False Positive): The number of unsuccessful students that were classified as “safe”.

FN (False Negative): The number of successful students that were classified “at-risk”.

Classification Accuracy (CA): The ratio of the total number of correct predictions and the total number of predictions (1).

Recall: The percentage of total results correctly classified by the algorithm (2).

Precision: The ratio between the True Positives and all the Positives (3).

F-Measure: The Harmonic mean of the Precision and Recall (4).

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(1)

$$Recall=\frac{TP}{TP+FN}$$

(2)

$$Precision=\frac{TP}{TP+FP}$$

(3)

$$F-measure=2\ast \frac{Precision \ast Recall}{Precision+Recall }$$

(4)

As the purpose of this study was to predict the students who would fail the final examination (at-risk), TP (True Positive) classification values will be considered. Thus, for the model evaluation, we focused on the models with a higher TP value. We also used 10-fold cross-validation during testing and scoring of the models created.

3. Results

3.1. Research Question 1: Which Supervised Learning Technique Can Predict At-Risk Students in an Online Flipped Anatomy Course?

We used independent variables to predict the target variable (final score). Model performance metrics have been provided in

Table 4. Evaluation results of the algorithms.

Model	AUC	CA	F	Precision	Recall
RF	0.795	0.696	0.533	0.571	0.500
DT	0.794	0.696	0.588	0.556	0.625
NB	0.703	0.681	0.607	0.531	0.708
SVM	0.690	0.681	0.476	0.556	0.417
kNN	0.689	0.667	0.489	0.524	0.458

.

The target variable (final score) was classified into two categories with a cut-off score of 60. According to the classification algorithms model performances, the naïve Bayes algorithm performed the best when predicting the final scores in a flipped anatomy classroom model with a classification accuracy of 68% and a recall value of 71%.

3.2. What Is the Classification Accuracy of the Best Algorithm for Predicting At-Risk Students in an Online Flipped Anatomy Course?

In our model, the naïve Bayes algorithm performed well when detecting at-risk students. The confusion matrix of the algorithm is given in

Table 5. Confusion matrix of naïve Bayes algorithm.

		Predicted
		At-Risk	Safe	Total
	At-Risk	17	7	24
Actual	Safe	15	30	45
	Total	32	37	69

.

According to Table 5, the naïve Bayes algorithm correctly classified 17 out of 24 (71%) unsuccessful students (TP value) before the final examination in a flipped anatomy course. The algorithm’s interpretation of 7 students led to mistakes, as the model classified some successful students as “at-risk” by mistake (i.e., registering as a False Positive—FP).

4. Discussion

In flipped classrooms, students’ pre-class activities, content–student interactions, and in-class quiz phases deliver more effective lectures. Tracking these activities via learning analytics enables the early prediction of at-risk students. This study aimed to create a predictive algorithm for final scores based on their interaction and preparation for a flipped anatomy course.

The results showed that the created model achieved at-risk prediction accuracy of 71%. This finding suggests that major determinants of student performance in flipped classrooms are students’ prior knowledge scores, weekly engagement scores, and weekly quiz scores. We also found that the NB algorithm performed well when detecting at-risk students. In order to attain the best performance, it is often necessary to combine different data sources like user profiles and sequentially dynamic data from sources like interaction logs [60]. In our study, we only merged weekly engagement scores with weekly quiz scores of the students, which might explain why different studies resulted in better scores of different machine learning algorithms. On the other hand, in our study, the RF model also showed a high accuracy for the general prediction model. Another study also found RF models with high prediction accuracy for both K-12 education and higher education [38]. Hung, Shelton [38] found that DNN and RF can accurately predict at-risk students for both K-12 and higher education in the range of 82.6–97%. As seen in the literature, combining many datasets can boost the prediction model’s accuracy [27,38,61]. The RF technique works based on the decision trees [57]. Therefore, DT models can also show high performance for the prediction of at-risk students. Similarly, our study also showed high accuracies with DT models. Akçapınar, Altun [27] investigated the impact of various pre-processing applications on prediction performance by adjusting the cut-off scores. This is another important issue for prediction studies. In this study, we selected the cut-off scores based on the decision of pass grade for university, however, changing the cut-off scores can also affect the model’s performance. Selecting norm-based criteria can help researchers generalize their models. They also found that the k-NN algorithm accurately predicted at-risk students at the end of the term with a rate of 89%. In addition, students who were at-risk at the end of term were predicted to a rate of 74% in 3 weeks. In our study, k-NN, which is also called “lazy learning”, showed sub-standard performance. In the literature, k-NN is usually used with continuous independent variables. That might explain why k-NN did not show high accuracy for prediction performance. Our study can predict “at-risk” students just before the final exam and this is the limitation of our study. Application of equal-frequency discretization can be the reason for higher accuracies in their predictive model for identifying at-risk students. The balanced distribution of the dependent variable might be a reason for the success of these methods. In addition, their study also included several metrics including participation in forums, writing posts, and creating tags. These actions might lead to higher classification performance. Setting a cut-off score with the passing grade of the university may change the performance of the algorithms. Previous studies mostly used median values or equal-frequency discretization for categorizing the continuous variables. That may lead to a differentiation in performing different machine learning algorithms. Saqr, Fors [29] used conventional statistical approaches to a medical course for the early detection of under-achieving students. The predictive model showed 63.5% accuracy and 53.9% of at-risk student prediction. According to our results, machine learning algorithms performed better when compared to statistical models such as logistic regression.

The sudden outbreak of the COVID-19 pandemic forced many institutions throughout the world to transition to emergency remote teaching [62,63]. Management of this crisis has depended on the participation of students and instructors in online learning processes and the effective use of online instructional materials and activities [64,65]. The biggest problem that emerged in these processes was the necessity of tracking students’ interactions with these instructional materials, tasks, assignments, and their active participation by the instructors [66,67]. Although educators can directly observe the general condition and progress of the students through formative assessments and in-class activities, it becomes difficult to follow up, give feedback, and prevent student failure in online learning environments. After the COVID-19 pandemic, educational institutions carried out hybrid (blended) learning guided by the curriculum, and the predictive model performance can increase by combining different data sets in face-to-face and online learning environments to prevent student failure by identifying students at risk. We can also use this model as an early warning system in similar crises and lockdown situations in the future.

There are some contradictions in the literature in terms of combining students’ demographics with interaction data. For example, Hung, Wang [39] stated that none of the students’ demographics were selected as important predictors for identifying at-risk patterns. On the other hand, there is also significant correlation between online learners’ performance and their demographic characteristics [54]. Combination of different datasets can increase the model’s performance; however, we could not collect multiple data sources because of the lockdown process. Another limitation of this study is prevention of downloading the online course materials. We used scripts to prevent the students who are trying to download the video lecture assignments and online contents. However, some students may have downloaded the videos or files via browser extensions or other plug-ins, resulting in poor model performance. According to Wilson, Watson [68], it is still unclear if learning analytics are useful for measuring, predicting, and improving student performance. Because learning analytics examines interaction data based on clickstream behavior, which assumes that online navigation is directly linked to learning processes, however, this assumption ignores some situations like disorientation or searching. Therefore, we received verbal feedback from the participants for the convenience of the instructional methods and materials, but it is still another limitation of our study. When using learning analytics in the field of health education, students’ behavioral, emotional, and cognitive engagement should also be considered.

5. Conclusions

The flipped classroom approach is effective in terms of student participation in the classroom and providing higher-order thinking skills [8,12,16]. However, the most important limitation of the method is the difficulty in ensuring that students are prepared for class activities. A predictive method, which also has a higher prediction performance on summative exam scores, can be used for the early warning system. For this purpose, learning analytics can benefit instructors and researchers in identifying at-risk students based on their interactions with the learning management systems. Applying several pre-processing techniques and comparing the effect of different machine learning algorithms on classification performance can benefit the models created.

For further studies, the results of this study can be used as a traffic light project [69]. Misclassifications (FN and FP) can be represented with a yellow light, so the students need to be careful about the final examination. The green light can be shown to the students in the TN group, and motivational feedback can be provided. The TP group needs to receive a red light and must put in the effort to engage with the content, and they might also need to solve the quiz test after an individual study period. In addition, further studies also can combine “no-engagement” alerts during the online anatomy course applications. These kinds of interventions may also prevent the failure of the students.