AI-Boosted Affective Real-Time Educational Software Adaptation

Abstract

Nowadays, educational software across all learning levels is increasingly enhanced with Artificial Intelligence (AI), primarily through content generation or post-session learning analytics. However, most existing systems remain weakly connected to learners’ real-time affective states and rarely exploit emotional information as a direct control signal for instructional adaptation. In this work, we propose a proof-of-concept closed-loop affect-aware educational adaptation framework that integrates real-time facial emotion recognition into a dynamic learning control system. The proposed approach is built upon a dual-model ensemble architecture, combining a transformer-based model (CAGE) and a CNN-based model (DDAMFN++) trained on large-scale in-the-wild datasets. To bridge heterogeneous emotion representations, we introduce a probabilistic fusion strategy that aligns continuous valence–arousal predictions with discrete emotion classification via a Gaussian Mixture Model (GMM), enabling unified emotion inference in real time. Based on the fused emotional state, a temporal aggregation mechanism is applied to capture sustained affective trends rather than transient expressions. These aggregated signals are then mapped to instructional decisions through an emotion-driven adaptive control policy, which adjusts activity difficulty using an Average Emotion Score (AES). This establishes a fully automated closed-loop adaptation cycle, where detected learner affect directly influences the learning environment without requiring explicit user input or post-session questionnaires. The framework is integrated into an open-source educational platform (eduActiv8) to demonstrate feasibility and system-level behavior. Results from alpha-level validation show that the system can continuously monitor learner affect, generate interpretable emotional analytics, and dynamically adjust task difficulty in real time, while reducing user interaction overhead. This study contributes a modular architecture for affect-aware educational systems by combining real-time ensemble emotion recognition, probabilistic fusion of heterogeneous outputs, and closed-loop instructional adaptation. The proposed framework provides a foundation for future research in scalable, emotion-driven intelligent tutoring and adaptive learning environments.

1. Introduction

Educational software at all levels of learning has been evolving since the early 1960s, consistently driven by advances in Information and Communication Technologies (ICT). This is largely due to the fact that teaching and training were among the earliest and most natural applications of computing. As a result, a wide range of software paradigms emerged, including computer-assisted instruction (CAI) [1], constructionist learning environments [2], simulation and modeling systems [3], and edutainment software [4].

The pace of development in educational technology has significantly accelerated with the advent of Artificial Intelligence (AI). Early work by Polson et al. [5] laid the foundations for Intelligent Tutoring Systems (ITS), which simulate human tutoring by incorporating a student model, a domain model, and a pedagogical model. While initial ITS implementations relied on rule-based logic as early as the 1960s, more advanced capabilities—such as Natural Language Processing (NLP) for learning analytics—emerged in the 2010s. In the current decade, Generative AI (GenAI) has further transformed the field by enabling personalized explanations, conversational tutoring, and adaptive content generation.

A rapidly growing research direction within this domain is the integration of affective computing and emotion-aware adaptation in educational systems. In particular, facial emotion recognition (FER) [6,7,8] has gained prominence as a non-intrusive means of capturing learners’ affective states. Modern FER systems leverage deep learning architectures, including Convolutional Neural Networks (CNNs), Long Short-Term Memory networks (LSTMs), and Vision Transformers [9,10,11,12], to estimate either discrete emotional categories [13] or continuous valence–arousal representations [14].

While physiological-signal-based affective computing approaches (e.g., electrodermal activity, heart rate variability, or EEG) can provide valuable and potentially more direct indicators of emotional states, their deployment in real educational environments is often constrained by practicality considerations. In particular, such methods require learners to wear dedicated sensors or specialized hardware, which can be intrusive, uncomfortable, and disruptive to natural classroom behavior. This significantly limits their scalability and suitability for routine use in educational settings. In contrast, FER relies on standard, low-cost, and widely available cameras (e.g., webcams already integrated into personal computers and online learning platforms), making it a far more practical and non-intrusive solution for real-time educational applications where ease of deployment and minimal user burden are essential.

Over the past decade, research on educational software based on emotional perception has progressed through several distinct stages. Early approaches primarily focused on emotion detection and recognition, aiming to accurately classify learners’ affective states using facial expressions or physiological signals. Subsequent work extended this direction toward engagement and affect monitoring, where emotional information was used to provide feedback to instructors or to support post-hoc learning analytics. More recent studies have explored multimodal frameworks that combine facial, vocal, and behavioral cues to improve robustness and capture the complexity of learner states. This evolution reflects a gradual shift from isolated emotion recognition toward more integrated and context-aware learning support mechanisms.

Recent studies have extended FER and affective computing toward educational applications, often through multimodal approaches. For instance, Li [15] proposes a multimodal framework for children’s online learning that combines emotion monitoring with pedagogically grounded intervention strategies, while Villegas et al. [16] integrate a multimodal emotion detection system into a real-world virtual learning platform (Microsoft Teams) to enhance student engagement through instructor feedback. Similarly, Saleem and Aslam [17], Zhou et al. [18], and Khediri et al. [19] employ multimodal deep learning techniques to analyze engagement and emotional states, whereas Salloum et al. [20] and Aly [21] focus on improving emotion or engagement recognition accuracy using AI-driven models. Garcia-Garcia et al. [22] further propose hierarchical multimodal frameworks for emotion detection.

Despite this significant progress, a clear gap remains between emotion perception and its direct pedagogical exploitation. Most existing approaches primarily emphasize emotion recognition, engagement estimation, or instructor-centered feedback, rather than fully automated, real-time adaptation of the learning process itself. In many cases, systems either rely on multimodal complexity, predefined intervention strategies, or explicit user input, and often operate in a semi-passive or post-hoc analytical manner. Consequently, there remains a significant gap between emotion detection capabilities and their seamless integration into adaptive educational software that dynamically modifies content in real time without user intervention.

In the current paper, we focus on exploiting affective computing, and especially FER technology, as an integral component of educational software, aiming to bridge this gap. The proposed approach emphasizes real-time, closed-loop adaptation, where emotional state estimation directly influences the learning experience.

The novelties of our approach are summarized as follows:

• The content presented to the user is automatically adapted based on their current emotional state, inferred from facial expressions, enabling a seamless and fully automated learning experience without requiring explicit user input.
• A closed-loop mechanism is introduced, where emotion recognition is directly linked to dynamic content adaptation, moving beyond passive monitoring or instructor feedback paradigms found in prior work.
• Self-assessment and third-party (tutor) evaluation are supported through the combined analysis of conventional performance metrics and emotion-based statistics collected during each session.
• The need for post-hoc questionnaires is eliminated, reducing user burden and mitigating biases associated with retrospective self-reporting.

The remainder of the paper is structured as follows. Section 2 reviews relevant background work on the integration of artificial intelligence in educational software, highlighting the research gap addressed in this study. Section 3 presents the proposed architecture and describes its components in a step-by-step manner. Section 4 reports the results obtained from integrating the proposed framework into an existing educational activities platform, followed by a discussion of the findings, their implications, and the relation of the derived system metrics to commonly used learning effectiveness indicators. Section 5 discusses the current limitations of the proposed approach. Finally, Section 6 concludes the paper and outlines directions for future research.

2. Related Work

A plethora of free and open-source tools aiming at different age groups and learning aims has emerged since the 1980s, when the FOSS movement first appeared [23]. One can distinguish different software categories based on their specific purpose. LMSs (Learning Management Systems) such as Moodle [24] are software applications that are used for creating, deploying and tracking educational courses and related material. Assessment software, such as TAO Community Edition [25], enables the user to prepare, deliver and report on assessments in order to evaluate an individual’s knowledge and skills. Another category of educational software is intelligent tutoring systems, such as OATutor [26], which simulate human tutors and try to provide automatic feedback or instruction to learners, customized to their needs. Lastly, learning activity platforms, like GCompris [27], are used to deploy educational activities that fall into different subjects such as math, language and science, usually in a gamified way.

The last category is the one we chose to focus on, since our architecture can be integrated in such applications in a quite straightforward manner. In

Table 1. Free and open-source educational activities software and AI integration.

Software	Category	Technologies	Target Age/Domain	AI Integration
eduActiv8 [28]	Activity-based learning platform	Python, Pygame	Early education (3–10)	No
GCompris [27]	Educational activity suite	C++, Qt	Early education (2–10)	No
TuxMath [29]	Math learning game	C, SDL	Elementary math	No
TuxTyping [30]	Typing tutor game	C, SDL	Children/beginners	No
Childsplay [31]	Educational game suite	Python	Preschool/early school	No
KHangMan [32]	Vocabulary learning game	C++, Qt	Language learning	No
KTurtle [33]	Programming learning environment	C++, Qt	Programming education	No
Scratch [34]	Visual programming platform	JavaScript	Programming/creativity	Limited
Blockly Games [35]	Programming activity suite	JavaScript	Programming education	No
KBruch [36]	Math learning tool	C++, Qt	Elementary math	No
Kalzium [37]	Chemistry learning environment	C++, Qt	Science education	No
GeoGebra [38]	Interactive mathematics platform	Java, JavaScript	Math education	Experimental
TurtleBlocks [39]	Visual coding environment	JavaScript, Python	Programming education	No
OpenBoard [40]	Interactive whiteboard platform	C++, Qt	Classroom teaching	No
H5P [41]	Interactive content framework	JavaScript	General education	Experimental

we present an extensive list of free and open-source educational activities software. Most of them aim at primary education students and are implemented in a variety of modern programming languages (C, C++, Javascript or Python). TuxMath, Kbruch and GeoGebra focus on math education, TuxTyping, KHangMan on language skills, KTurtle, Scratch, Blockly Games and TurtleBlocks are visual programming platforms, Kalzium concerns chemistry learning concepts, OpenBoard is an interactive whiteboard, H5P is a web content creator, and the rest (eduActiv8, GCompris and Childsplay) are complete educational activity suites.

As one can easily notice, most of these applications are not integrated with AI at all. Scratch supports extensions that allow blocks to be used in order to convert text to speech, translate text between languages, or detect motion via a webcam. GeoGebra has some AI features, including equation solving assistance, parsing of math equations input, and recognizing scanned math problems. H5P has limited AI integration, focusing on generating quiz questions from text and suggesting summaries or fill-in-the-blank exercises.

In all cases, AI is primarily actualized in the form of chatbot-based interaction or content generation mechanisms, with little to no consideration of the learner’s affective state or real-time adaptive behavior.

While Table 1 highlights the limited integration of AI in existing educational activity platforms—primarily in the form of chatbot-based or content-generation functionalities—it is important to distinguish these approaches from emotion-aware adaptive systems. Most current AI-enabled educational tools focus on user-initiated interaction, where adaptation is driven by explicit input (e.g., text prompts) or predefined logic. In contrast, emotion-assisted systems aim to infer the learner’s internal state and adjust the learning experience accordingly.

As shown in

Table 2. Comparison between the proposed framework and existing emotion-aware educational systems.

Approach	Emotion Detection	Real-Time Adaptation	Closed-Loop Control	User Input Required
Chatbot/LLM-based systems	No	Limited	No	Yes
Affect monitoring systems	Yes	No	No	No
Multimodal emotion-aware systems	Yes	Limited	Partial	Sometimes
Proposed framework	Yes	Yes	Yes	No

, existing approaches either lack real-time adaptation capabilities or do not establish a direct feedback loop between emotion recognition and content adjustment. In contrast, the proposed framework integrates emotion detection with a closed-loop adaptation mechanism, enabling continuous and autonomous adjustment of the learning experience without requiring explicit user intervention.

Existing research on emotion-aware educational systems has largely focused on affect detection and monitoring, often employing multimodal approaches to estimate engagement or emotional states. However, in many cases, these systems either provide feedback to instructors or support post-hoc analysis, rather than directly modifying the learning content in real time. Furthermore, adaptation strategies, when present, are typically predefined or loosely coupled with the emotion recognition process.

The framework proposed in this study differs from prior approaches in several key aspects. First, it enables fully automated, real-time adaptation of activity difficulty based directly on the learner’s detected emotional state, without requiring explicit user input. Second, it establishes a closed-loop mechanism in which emotion recognition continuously influences the learning process, rather than serving as a passive monitoring tool. Third, it operates in a lightweight and modular manner, allowing integration into existing activity-based platforms without reliance on complex multimodal pipelines or external instructor intervention. These characteristics position the proposed approach beyond chatbot-based interaction and closer to truly adaptive, affect-driven educational systems.

3. Materials and Methods

The overall structure of the proposed framework is depicted in Figure 1. The innovative modules are highlighted in red to clearly distinguish them from standard computer vision components. In the following, we provide a detailed description of each stage, emphasizing the proposed contributions.

3.1. Face Detection

As a necessary preprocessing stage, the user’s face is detected from the camera input, enabling subsequent emotion estimation and adaptive decision-making. To this end, we employ a Haar Feature-based Cascade Classifier to detect and localize the face region [42]. This stage is considered a standard component of vision-based pipelines and is not part of the proposed contributions, but is required for real-time deployment.

In practical scenarios, face detection and subsequent emotion recognition may be affected by partial occlusion (e.g., hands, glasses), illumination variability, and non-frontal head poses. The selected Haar Cascade approach offers low computational cost and real-time performance, but may exhibit reduced robustness under such conditions. To mitigate this, the system operates continuously on a frame-by-frame basis, allowing transient detection failures to be smoothed over time through repeated sampling. As a result, short-term disruptions in face visibility or lighting inconsistencies have a limited impact on the overall emotion estimation used for adaptation decisions.

3.2. Neural Network Ensemble

The core contribution of this work lies in a dual-model ensemble for real-time affect inference, designed to improve robustness over single-model FER systems.

We employ two complementary deep learning models:

• The CAGE model [43], a transformer-based architecture with a MaxViT backbone [44], trained on both discrete emotion labels and continuous valence–arousal annotations.
• The DDAMFN++ model [45], a CNN-based architecture built on a Mixed Feature Network with a Dual-Direction Attention mechanism, producing 8-class emotion probabilities.

Both models are pretrained on the AffectNet dataset, ensuring exposure to large-scale in-the-wild facial variability.

The input is resized to 224 × 224 for CAGE and 112 × 112 for DDAMFN++. Emotion inference is performed at high frequency (0.01 s intervals), enabling fine-grained real-time affect tracking suitable for closed-loop adaptation.

Although both models are trained on diverse real-world datasets that include variations in lighting, pose, and partial occlusion, performance may still degrade under extreme conditions or atypical expressions outside the training distribution. This limitation is common to current FER systems and motivates the ensemble-based design adopted in the work.

3.3. Decision Logic

This stage introduces a novel probabilistic fusion mechanism that bridges regression-based and classification-based emotion representations, enabling a unified decision space.

The CAGE model outputs continuous valence–arousal (VA) values, while DDAMFN++ outputs categorical probabilities over eight emotions. To make both outputs comparable, we construct a probabilistic mapping from VA space to discrete emotion space using a Gaussian Mixture Model (GMM) [46].

The number of clusters is selected using the Bayesian Information Criterion (BIC) [47], which yields 12 clusters rather than a fixed 8-class assumption. This design choice is motivated by class imbalance in VA distributions. To further reduce bias, we apply random downsampling to equalize class cardinalities before cluster labeling.

Each cluster is mapped to an emotion class via majority voting, producing a probabilistic VA-to-emotion transformation (Figure 2).

The fused emotion distribution is then computed as follows:

$${\mathbf{E}}_{\mathbf{p}}=\alpha {\mathbf{C}}_{\mathbf{p}}+(1-\alpha ){\mathbf{D}}_{\mathbf{p}},\alpha \in [0,1]$$

(1)

where ${\mathbf{E}}_{\mathbf{p}}$, ${\mathbf{C}}_{\mathbf{p}}$ and ${\mathbf{D}}_{\mathbf{p}}$ are the fused probabilites, CAGE probabilities and DDAMFN probabilities vectors, respectively, and $\alpha$ controls the contribution of VA-based inference relative to classification-based prediction.

In this work, we set $\alpha =0.3$ based on empirical validation. This weighting prioritizes the stability of the DDAMFN++ classifier while using the VA-GMM pipeline as a corrective signal for ambiguous or borderline cases, particularly when categorical predictions are uncertain.

The final emotion estimate is obtained as follows:

$$E_{\mathrm{est}}=arg\underset{k}{max},{\left(E_p\right)}_k$$

(2)

This estimate is stored at the end of each activity and used for medium-term adaptation decisions, rather than instantaneous frame-level control.

3.4. Activity Adaptation

This stage introduces the closed-loop affect-driven adaptation mechanism, which is the second main contribution of this work.

Instead of reacting to instantaneous emotion predictions, the system computes a temporally aggregated affect score based on sustained emotional patterns during an activity.

Each detected emotion is assigned a pedagogically motivated weight (

Table 3. Emotion weights for the level change trigger.

Emotion	Weight
Happiness	5.0
Surprise	3.0
Neutral	0.1
Contempt	0.2
Sadness	−2.0
Fear	−2.0
Disgust	−2.0
Anger	−2.0

), where positive emotions encourage increased difficulty and negative emotions indicate overload or frustration.

The selection of weights is grounded in affective learning theory and empirical observations of emotional duration asymmetry in learning contexts. In particular, negative affect tends to persist significantly longer than positive “peak” states, such as momentary success or relief [48,49]. This motivates asymmetric weighting to avoid over-reacting to short-lived positive expressions.

Empirically, we observe that negative emotions persist approximately 2.5 times longer than positive ones ($f_{neg}=17.75\%$, $f_{hap}=7.1\%$), which is incorporated into the calibration of positive weights to ensure balanced adaptation behavior.

For example:

$$W_{hap}={\displaystyle \frac{f_{neg}·\left|W_{neg}\right|}{f_{hap}}}={\displaystyle \frac{0.1775·2.0}{0.071}}=5.0$$

(3)

$$W_{surp}={\displaystyle \frac{f_{neg}·\left|W_{neg}\right|}{f_{surp}}}={\displaystyle \frac{0.1775·2.0}{0.132}}=2.7\approx 3$$

(4)

The Average Emotion Score (AES) is then computed as follows:

$$AES={\displaystyle \frac{1}{N}}\sum _{i=1}^N{EW}_i$$

(5)

where ${EW}_i$ is the emotion weight at time step i.

A closed-loop adaptation rule is applied:

• Increase difficulty if $AES>0.05$;
• Decrease difficulty if $AES<-0.05$;
• Otherwise, maintain the current level.

The threshold selection is symmetric around a neutral baseline and is designed to prevent over-sensitivity to transient emotional fluctuations.

4. Results and Discussion

4.1. Experimental Setup

To evaluate the proposed framework in a controlled proof-of-concept scenario, alpha testing was performed using a desktop computer equipped with an Intel^® Core™ CPU (Intel64 Family 6 Model 140 Stepping 1, GenuineIntel) running Windows 10. A standard RGB webcam with a resolution of 640 × 480 at 30 FPS was used for real-time facial capture. The user was positioned at an approximate distance of 0.5 m from the camera to ensure stable face detection and reliable facial feature extraction. All tests were conducted under normal indoor lighting conditions.

This setup is intended to demonstrate system feasibility and real-time behavior under typical operating conditions rather than to provide a large-scale user study.

4.2. Software Adaptation

In our case study, among all the educational activities software referenced above, we selected eduActiv8 (version 4.25.10), an educational suite that includes math (basic operations, fractions, percentages, etc.) language (word building, word matching, etc.) and a logic, memory and general knowledge category (colour mixing, matching pairs of same images, mazes, etc.). The original eduActiv8 application allows the user to manually select the level of difficulty as depicted in Figure 3, using the arrows in the upper right corner.

In our version, the level selection feature has been removed, since difficulty is determined based on the user’s emotion. The respective snapshot of the modified eduActiv8 version is shown in Figure 4. The arrows for manual level change as well as the level number are not shown anymore.

After integrating our emotion detection framework, described in the previous section, with the modified version of eduActiv8, we performed alpha testing (testing by one of the authors) of the overall system, in order to showcase its features and significance.

First, as shown in Figure 5, we can examine how much time the user spent on each activity together with the maximum difficulty level. This might reveal which kind of activities the user found more intriguing so as to experience them for longer periods and at higher difficulty levels. Each G-XXX code represents a different activity (e.g., word matchers, shapes and solids, mixing colours for painting, etc.).

Next, we are able to check what the prevalent emotion was during each activity. This might in many cases, as in Figure 6, be neutral, which is considered, as stated above, a positive reaction towards the game, explaining the maximum level value for respective activities, as noticed in Figure 5.

The user can also examine the valence and arousal evolution throughout the game session, as depicted in Figure 7 and Figure 8. This is complemented by vertical dotted lines indicating the time instances at which a level change occurred, together with the direction of change, the new level and the triggering emotion, based on the computed AES. Also, dashed lines indicate the points at which an activity change took place. All activities start at a base level of value 1. We can clearly see that positive emotions such as happiness triggered a level rise, whereas the negative emotion of disgust triggered a level reduction. We can also notice that, in this specific occasion, there is an overall prevalence of negative valence and positive arousal values. This does not necessarily imply a negative emotional state, as is evident from Figure 2. Smoothed versions of the curves are also provided by averaging with a sliding window of length 5, in order to reduce the effect of instantaneous reactions, resulting in curves of lower valence and arousal magnitude, which are expected to represent the emotional state in a more reliable way. It should also be noted that the diagrams can be scaled up by the user in order to be examined in greater detail, especially when the total session duration is large.

Another metric that gives a good idea of the user’s emotional stability is fluctuation, which is defined as the sum of standard deviations of valence and arousal over a small period of time (typically five consecutive predictions) as shown in Equation (6):

$$Fluctuation=std\left(\mathbf{V}\right)+std\left(\mathbf{A}\right)$$

(6)

where $\mathbf{V}$ and $\mathbf{A}$ are the valence and arousal vectors taken in a sliding window basis. As depicted in Figure 9, fluctuation suggests a quite smooth transition between emotional states with its value, and especially the smoothed one, rarely exceeding a hard threshold of 0.5.

A pie-chart representing the total impact of emotions throughout the session is also provided, as in Figure 10. This is weighted in the same way that is used for calculating the AES, as defined above. In the specific example, happiness dominates the game session, followed by disgust and a much smaller contribution of the remaining emotions.

Another bar chart is produced depicting the average number of attempts until a correct answer is recorded per activity, as in Figure 11. This is an indicator of the level of difficulty as perceived by the user. A larger average number of attempts is consistent with a significant amount of time spent on the activity, as depicted in Figure 5 and suggests a higher level of engagement. In contrast, a zero number of attempts (no correct answer) together with a short time interval implies a lower engagement.

Finally, a weighted AES per activity is also derived, as shown in Figure 12. The respective emotion is also noted. A high positive value is consistent with a positive emotion like happiness, in contrast to a moderately low negative value corresponding to disgust. Quite low AES values are, as expected, paired with a neutral emotion.

The diagrams can also be inspected together at once, as subplots of a single diagram, so that faster conclusions may be drawn.

Based on the above experimentation, it is evident that a user/student or a tutor can greatly benefit from the use of our framework when integrated into an educational activities software. More specifically:

• The user’s emotional state is continuously and seamlessly monitored, without being affected by any extrinsic factors.
• The user is safely experiencing the activities without being online and without being video-recorded.
• The difficulty level, which may be manifested in different ways, depending on the game mechanism, can be modified during the session, so that the user remains satisfied and engaged for a longer period of time.
• The need for post-completion of questionnaires regarding the user’s experience is eliminated, together with the deviation that may be induced due to the delay between sessions and assessments.
• The user’s haptic intervention is minimized, since they are only required to select the type of activity, not its difficulty level.

During experimental testing, the system demonstrated stable behavior under typical indoor conditions, including moderate variations in lighting and natural user movement. Temporary failures in face detection or misclassifications due to brief occlusions did not significantly affect the adaptation outcome, as decisions are based on aggregated emotional estimates over time rather than instantaneous predictions. However, more challenging conditions, such as sustained occlusion or very low lighting, may reduce detection reliability and consequently affect adaptation quality.

Although the current implementation is demonstrated within the eduActiv8 platform, the proposed framework is not limited to activity-based learning environments. Its modular architecture allows integration into a wide range of educational systems, including Learning Management Systems (LMS), Intelligent Tutoring Systems (ITS), and interactive e-learning platforms. In such contexts, adaptation may extend beyond difficulty adjustment to include content sequencing, feedback generation, pacing control, or instructional strategy selection. This highlights the broader applicability of the proposed approach across diverse educational scenarios. In conclusion, any educational suite of this kind could be easily integrated and take advantage of the proposed framework.

4.3. Relation to Learning Effectiveness Indicators

The metrics derived from the proposed framework can be directly associated with widely used learning effectiveness indicators in educational technology research, including behavioral, cognitive, and affective dimensions of learning.

Time spent per activity, in conjunction with the maximum difficulty level achieved, serves as a proxy for behavioral engagement, time-on-task, and learner persistence. Extended interaction with an activity, particularly at higher difficulty levels, suggests sustained interest and progression within the learner’s zone of proximal development [50]. Time-on-task has consistently been shown to correlate with improved learning outcomes when associated with active engagement [51].

The prevalent emotion detected during each activity reflects the learner’s affective engagement. According to established engagement frameworks, emotional responses play a central role in shaping the learning experience [51,52]. Neutral and positive emotional states are generally associated with productive interaction and satisfaction, whereas persistent negative emotions may indicate frustration or disengagement.

The evolution of valence and arousal over time provides insight into the learner’s cognitive and affective dynamics. These dimensions align with core models of affect representation [14]. Different regions of the valence–arousal space correspond to well-established learning states, such as engagement or flow (high arousal, positive valence) [53], boredom (low arousal), or productive struggle (high arousal, negative valence) [54]. As such, these signals can be interpreted as proxies for cognitive load and engagement regulation.

The fluctuation metric, defined as the sum of the standard deviations of valence and arousal within a sliding window, is indicative of emotional stability and cognitive disequilibrium. Moderate variability in affective states has been linked to productive learning processes, where transitions between confusion and understanding support deeper learning [54].

The weighted distribution of emotions throughout a session provides an aggregate measure of the learner’s overall affective experience and user satisfaction. This aligns with prior work in affective computing and learning analytics, where emotional traces are used as indicators of user experience [55]. Unlike traditional post-session questionnaires, this approach enables continuous and unobtrusive monitoring.

The average number of attempts required to reach a correct answer constitutes a performance-based indicator related to perceived task difficulty and learning efficiency. This is consistent with the concept of desirable difficulty, where an optimal level of challenge enhances learning outcomes [56]. Extremely low or high values may indicate a lack of challenge or excessive difficulty, respectively.

The Average Emotion Score (AES) per activity acts as a composite metric that integrates multiple emotional signals into a single interpretable value. Similar composite indicators have been proposed in learning analytics to capture multidimensional engagement [57]. AES reflects both the quality and intensity of the learner’s emotional experience.

Finally, the observed difficulty adaptation events reflect the effectiveness of the system’s personalization mechanism. Adaptive learning systems aim to maintain learners within an optimal challenge zone, closely related to flow and improved learning outcomes [53,58].

Overall, the proposed framework provides a set of implicit and continuous measures that correspond to established learning effectiveness indicators, enabling real-time, non-intrusive, and fine-grained analysis of the learning process.

While the presented results demonstrate the effectiveness of the proposed framework within the specific context of the selected platform, it is important to note that different types of educational activities may elicit distinct affective and cognitive responses. For example, problem-solving tasks, memory-based exercises, and exploratory learning activities are associated with different engagement patterns and emotional dynamics, which may influence both emotion recognition accuracy and the resulting adaptation strategy. Consequently, further investigation is required to systematically analyze how activity type affects the relationship between detected emotions and optimal system adaptation. It should be noted that the reliability of these indicators is inherently linked to the accuracy of the underlying emotion recognition process, which may be affected by environmental and user-related factors.

5. Limitations

Although the proposed framework demonstrates the feasibility of real-time emotion-driven adaptive learning, several limitations should be acknowledged.

First, the current study is conducted as a proof-of-concept (PoC) implementation and is validated primarily through alpha testing in a controlled environment. As such, large-scale user studies with statistically significant sample sizes and cross-population evaluation are not yet included.

Second, the system relies on visual facial cues only, which inherently limits robustness under challenging conditions such as severe occlusion, extreme head poses, and low illumination. While pretrained models partially mitigate these issues, performance degradation may still occur in unconstrained real-world settings.

Third, several system parameters (e.g., fusion weight $\alpha$, emotion weights, and decision thresholds) are empirically selected based on preliminary observations rather than fully optimized through large-scale hyperparameter tuning. This may limit generalizability across different datasets or user populations.

Finally, the current evaluation is platform-specific (eduActiv8) and does not yet explore adaptation across different educational domains, age groups, or cultural contexts, which may influence both emotion expression and learning behavior.

6. Conclusions and Future Directions

In this work, we presented a novel architecture for continuous monitoring of learners’ emotional states within an educational application, enabling real-time adaptation of content to sustain positive user engagement. The proposed framework leverages two transformer-based deep neural network models, CAGE and DDAMFN++, whose outputs are fused to produce robust emotion estimates. These models are pretrained on the AffectNet dataset, which provides both discrete emotion labels and continuous valence–arousal annotations. A decision mechanism was designed to trigger adaptive difficulty adjustments based on aggregated emotional feedback over each activity period.

The results indicate that integrating affect-aware mechanisms into educational activity platforms can enhance user experience and provide valuable feedback to instructors. Beyond its current implementation, the proposed framework opens several avenues for future research.

Future work will explore more advanced fusion strategies for combining heterogeneous FER model outputs, aiming to improve reliability and decision accuracy. Additionally, extending the system to incorporate multimodal inputs—such as physiological signals (e.g., heart rate from wearable devices) or speech—represents a promising direction for achieving more comprehensive and robust affect detection.

Further research will also focus on integrating long-term learner modeling and data-driven personalization techniques, enabling the system to move beyond reactive adaptation toward predictive and individualized learning support.

Overall, the proposed framework contributes to the growing field of affect-aware intelligent tutoring systems and highlights the potential of combining deep learning with adaptive educational design. We believe that such approaches can play a key role in shaping the next generation of AI-powered educational technologies.