NAMI: A Neuro-Adaptive Multimodal Architecture for Wearable Human–Computer Interaction

Abstract

The increasing ubiquity of wearable computing and multimodal interaction technologies has created unprecedented opportunities for natural and seamless human–computer interaction. However, most existing systems adapt only to external user actions such as speech, gesture, or gaze, without considering internal cognitive or affective states. This limits their ability to provide intelligent and empathetic adaptations. This paper addresses this critical gap by proposing the Neuro-Adaptive Multimodal Architecture (NAMI), a principled, modular, and reproducible framework designed to integrate behavioral and neurophysiological signals in real time. NAMI combines multimodal behavioral inputs with lightweight EEG and peripheral physiological measurements to infer cognitive load and engagement and adapt the interface dynamically to optimize user experience. The architecture is formally specified as a three-layer pipeline encompassing sensing and acquisition, cognitive–affective state estimation, and adaptive interaction control, with clear data flows, mathematical formalization, and real-time performance on wearable platforms. A prototype implementation of NAMI was deployed in an augmented reality Java programming tutor for postgraduate informatics students, where it dynamically adjusted task difficulty, feedback modality, and assistance frequency based on inferred user state. Empirical evaluation with 100 participants demonstrated significant improvements in task performance, reduced subjective workload, and increased engagement and satisfaction, confirming the effectiveness of the neuro-adaptive approach.

1. Introduction

The evolution of human–computer interaction (HCI) has been marked by progressively sophisticated modalities aimed at mimicking the complexity and dynamism of natural human communication [1]. Advances in technology, from graphical user interfaces to speech recognition, gesture-based input, and haptic output, illustrate that modern interaction systems have moved beyond the confines of single modality input/output channels and are now essentially multimodal [2]. Furthermore, the advent of wearable computing and ubiquitous systems has extended the interaction field, enabling technology to be seamlessly embedded in everyday human contexts [3]. However, despite these developments, most multimodal systems continue to rely on perceptible behavioral cues to deduce user intentions and, consequently, fail to take into consideration the internal cognitive and affective states of users in real time [2]. This limitation inhibits such systems from reacting adaptively and empathetically to the needs of users.

Recent developments in the fields of neurotechnology and affective computing offer a feasible approach to overcome this limitation [4,5,6]. The development of cheap and non-invasive devices, such as wearable EEG headsets and functional near-infrared spectroscopy (fNIRS) sensors, has enabled the monitoring of brain activity in naturalistic environments. If these technologies are combined with physiological sensors monitoring variables like heart rate variability, skin conductance, and pupil dilation, a new generation of neuro-adaptive systems is under development. These systems aim to estimate users’ cognitive workload, attentional engagement, and emotional states in real time, and then adapt the interactive environment in a corresponding way. Nevertheless, the integration of such internal state information in a holistic, reproducible, and modular architecture for real-time multimodal wearable interaction is a vastly under-explored area of research [7,8,9,10,11].

The goal of the current study is to close the identified gap by proposing a neuro-adaptive multimodal architecture that integrates real-time electroencephalogram (EEG) and physiological signals into the wearable HCI interface control loop. Specifically, we describe and deploy an architectural framework made up of three interacting modules: sensing and data acquisition, cognitive-affective state estimation, and adaptive interaction control. We hypothesize that integrating neurophysiological and behavioral signals enables real-time adaptation that improves task performance and user experience; Section 5 reports the test of this hypothesis. We demonstrate the feasibility and benefits of this architecture through the development of a prototype system where users engage in an augmented reality (AR) task environment while wearing a lightweight EEG headband, a photoplethysmography (PPG) sensor, and a galvanic skin response (GSR) device. Multimodal inputs such as voice, head orientation, and hand gestures are combined with neurophysiological data to form a dynamic user profile. The system adjusts visual and auditory output channels, adapts task difficulty, and increases assistance features based on the calculated levels of cognitive load and emotional engagement.

Building on prior work in affective computing and multimodal HCI, the present study is grounded on two premises that define its conceptual background. First, behavioral and neurophysiological signals can be jointly integrated within a wearable multimodal architecture to infer users’ cognitive and affective states in real time. Second, adaptive control policies can translate these inferred states into effective adjustments of interaction parameters such as task difficulty, feedback modality, and assistance frequency. These considerations motivate the development of NAMI and lead to the central research question of this study:

RQ1: To what extent does a neuro-adaptive interface improve task performance, workload management, and user experience in a real educational setting compared with a baseline non-adaptive system?

To investigate this question comprehensively, three subordinate research questions are formulated:

RQ1.1: How can a wearable multimodal framework be formally specified to integrate behavioral (voice, gesture, gaze) and neurophysiological (EEG, GSR, HRV) inputs in real time?

RQ1.2: How can cognitive–affective state estimation be operationalized through lightweight, transparent modeling suitable for wearable systems?

RQ1.3: How do adaptive control mechanisms based on these estimates affect task performance, workload, engagement, and satisfaction relative to a non-adaptive baseline?

To address these research questions, this paper makes the following contributions:

• Formal architecture: We propose a principled, three-layer wearable framework that integrates multimodal behavioral inputs (voice, gesture, gaze) with neurophysiological data (EEG, GSR, HRV). These inputs are processed by a state estimation component (implemented as a ridge regression model, later referred to as CASEM) and an adaptive control component (implemented as a transparent rule-based controller, later referred to as AICM).
• Transparent control policy: We specify individualized percentile-based thresholds and the associated action vector, enabling faithful reproduction of the adaptation process.
• Real-time wearable implementation: We demonstrate closed-loop operation on AR hardware with end-to-end latency under 85 ms, confirming feasibility within interactive HCI limits.

Based on these contributions, we formulate the following hypothesis that is tested in this study:

H1. 

A wearable neuro-adaptive multimodal interface significantly improves task performance, reduces subjective workload, and enhances engagement and satisfaction compared with a non-adaptive baseline condition.

This hypothesis is empirically evaluated in Section 5, where results from the large-scale experiment with 100 postgraduate learners confirm its validity.

The significance of this work lies not only in its interdisciplinary integration of neuroscience, wearable technology, and HCI but in its potential for real-world, practical uses beyond the laboratory environment. The ability to create systems that are emotionally and cognitively aware is a critical step in the development of more humane and context-aware technology, including personalized learning and training, mental wellbeing monitoring, and cognitive assistance at the workplace.

2. Related Work

Multimodal interaction has been recognized as one of the defining features of next-generation human–computer interfaces for the past few decades as an avenue to enable more natural and efficient communication through the combination of complementary input and output modalities. Early studies of multimodal systems were centered on the combination of different modalities, including speech and gesture [12], haptic and visual feedback [13], and gaze and voice commands [14]. These efforts have given rise to robust multimodal systems that increase recognition accuracy and user experience across a wide range of applications, from desktop computing to virtual reality. However, most of these systems address the user as a black box, reacting to outward behavior but having no understanding of the user’s internal cognitive or affective state.

The affective computing research [4,15] attempts to overcome this limitation by giving systems the ability to perceive and respond to the emotions of their users. Several strategies used to infer affective states and then modify system behavior range from facial expression analysis [16,17], prosody analysis in speech [18,19], to physiology measurement [20,21]. However, these methods are often limited by their indirect and inherently ambiguous nature since they rely on outward expressions that are not necessarily reflective of the underlying cognitive or affective processes.

In parallel, the discipline of neuroergonomics [22] has developed through the application of brain–computer interfaces (BCIs) and neurophysiological monitoring in human–machine interfaces. Studies have shown that electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are useful measures for assessing mental workload, attention, and interest within controlled settings [23,24,25]. For example, in [26], the authors used EEG-based workload evaluations to dynamically adjust automation levels in the course of a flight simulation task, while in [27], the authors explored real-time fNIRS applications for enhancing speech recognition technology. These studies document the potential of neurophysiological signals to serve more sophisticated interaction paradigms by offering direct information about the internal state of users.

The development of wearable neurotechnology has enabled the transfer of such abilities from highly controlled laboratory settings to more ecologically reasonable ones. Currently, commercially marketed EEG headsets and fNIRS wearable devices allow for ongoing monitoring of brain activity during everyday activities [28,29,30]. This has encouraged research into neuroadaptive user interfaces, including real-time adjustment of system parameters in accordance with detected mental states [31,32,33]. However, the majority of current neuroadaptive systems are largely focused on a single modality (e.g., EEG) or a particular area of application (e.g., clinical rehabilitation or entertainment) [34], often without regard to the feasibility of integration into general multimodal interaction architectures.

Apart from the neurological signals received from the brain, physiological measures like heart rate variability (HRV), galvanic skin response (GSR), and pupilometry have been widely used in the detection of user states [35,36,37]. The fusion of such physiological measures with electroencephalogram (EEG) measures has been shown to improve the accuracy of mental state classification [38] where the authors demonstrated better engagement detection using EEG and GSR signals synergistically. This highlights the need for a multimodal framework for internal states that incorporates both central and peripheral signals related to cognitive and affective processes.

Despite these promising developments, a major shortfall remains in cohesive and formalized architectures that combine neurophysiological sensing and multimodal behavioral interaction in a wearable, real-time adaptive system. Current research often dissociates the development of multimodal interaction from its neuro-adaptive aspects, thus losing the potential value that can be achieved when these components are embedded in a reproducible and modular system. In addition, a substantial proportion of the existing literature reports findings from small-scale, highly controlled lab experiments, thus limiting the ecological soundness and generalizability of the findings.

This work seeks to advance the field by presenting a neuro-adaptive multimodal system that combines EEG, physiological signals, and traditional behavioral input modalities—namely voice, gesture, and gaze—into a unified, wearable, and real-time system. Unlike the existing body of work that mostly focuses on a single modality or scenario, our architecture is developed as a generalizable and extensible framework, which is demonstrated here in the context of augmented reality applications. This framework presents an important leap in overcoming the current divide in multimodal HCI research by proving that real-time measurements of internal states can be used systematically to enhance the control logic in pervasive computing environments and thus enable more adaptive and context-aware interaction experiences.

3. System Architecture

NAMI is a principled and modular framework, implemented as a working prototype, that endows wearable human–computer interfaces with the ability to respond adaptively to users’ affective and cognitive states in real time. The framework describes how multimodal behavioral and neurophysiological signals are transformed into control parameters that guide the adaptive interface. It emphasizes reproducibility, scalability, and computational efficiency, making it suitable even for wearable devices with limited resources. The subsequent sections present the system architecture with sufficient technical detail to ensure that the approach can be accurately reproduced and rigorously evaluated.

The architecture defines a deterministic function, defined as F, that maps the multimodal user state vector at time step t, defined as X_t ∈ R^m+n, to the control output vector u_t ∈ R³. The control output vector u_t contains elements that relate to task difficulty, feedback modality, and assistance frequency. The formal definition of this mapping is given below:

F: X_t⟼u_t

Mapping F is achieved by the successive application of three clearly specified modules, with each module performing a specific transformation:

$$X_t\stackrel{M_1}{\to }{Z}_t\stackrel{M_2}{\to }\widehat{y_t}\stackrel{M_3}{\to }{u}_t$$

We use early feature-level fusion, concatenating modality-specific features into Z_t = [z_voice, z_head, z_hand, z_gaze, z_EEG, z_GSR, z_HRV]. Early fusion was chosen over late fusion because it enables a unified feature space and reduces end-to-end latency, which is critical for real-time adaptation in AR. After per-participant baseline normalization, a ridge-regression model maps Z_t to continuous workload and engagement: [ŵ(t), ê(t)] = Z_tW + b, with L₂ regularization to prevent overfitting and maintain interpretability. Outputs are clipped to the range [0, 1] to facilitate integration with the rule-based controller, and smoothed over a two-second sliding window to attenuate sensor noise and transient fluctuations. If a modality is temporarily unavailable, a zero-mean imputed value (derived from baseline statistics) is used to keep Z_t stable, ensuring robust operation despite occasional signal loss.

The sensing architecture combines two synchronized subsystems: (i) a neurophysiological stack (4-channel frontal/temporal EEG sampled at 256 Hz; wrist-worn GSR and PPG sensors for HR/HRV) and (ii) a behavioral stack (automatic speech recognition for voice, depth-camera hand pose tracking, and 30 Hz eye-gaze capture). This design reflects prior evidence that neurophysiological signals capture workload and arousal while behavioral signals provide complementary information about attention and task engagement, thus justifying their joint inclusion in the fusion process. The sensors within the AR headset enable behavioral sensing by recording verbal commands using an automatic speech recognition (ASR) engine, recognizing hand gestures using a depth camera in conjunction with a pose estimation algorithm, and tracking gaze direction at a sampling rate of 30 Hz. These behavioral signals are then converted into a feature vector:

$$x_b\left(t\right)=\left[\begin{array}{c}{s}_1\left(t\right)\\ s_2\left(t\right)\\ ⋮\\ s_m\left(t\right)\end{array}\right]$$

where each s_i(t) is a scalar feature normalized to the unit interval [0, 1]. In parallel, neurophysiological sensing is performed using a lightweight EEG headband, providing four frontal and temporal channels at 256 Hz, and a wrist-mounted device providing GSR and HRV. EEG signals are bandpass-filtered between 1–40 Hz and decomposed into standard frequency bands, with power spectral density P_k(t) computed for each band. Peripheral signals include normalized SCL, SCR rate, and HRV measures. These are combined into the neurophysiological feature vector:

$$x_n\left(t\right)=\left[\begin{array}{c}{e}_1\left(t\right)\\ e_2\left(t\right)\\ ⋮\\ e_n\left(t\right)\end{array}\right]$$

Then, the complete normalized feature vector is created as explained below:

$$Z_t=\left[\begin{array}{c}{x}_b\left(t\right)\\ x_n\left(t\right)\end{array}\right]$$

Signals are synchronized with a common clock, normalized via z-score transformation, and exposed to adaptive noise-cancellation methods to obviate the effects of motion and ambient noise. All of these design choices have been tested empirically to ensure that the computational latency in this step is short (less than 5 ms per cycle) and feature variance is kept within reasonable limits (coefficient of variation < 10%) for 10 s.

CASEM is a supervised machine learning estimator that is trained offline using ridge regression on labeled workload and engagement data. Its goal is to map the multimodal feature vector Z_t to continuous estimates of workload $\widehat{C}(t)$ and engagement $\widehat{E}(t)$, which can be written as

$$\widehat{y_t}=\left[\begin{array}{c}\widehat{C}(t)\\ \widehat{E}(t)\end{array}\right]$$

where $\widehat{C}(t)$ is cognitive load and $\widehat{E}(t)$ is affective engagement, which are both continuous values in the range [0, 1]. These are estimated as:

$$\widehat{y_t}=W·Z_t+b$$

(1)

Here, W ∈ R^2×(m+n) is the weight matrix and b ∈ R² is the bias vector, both of which are learned offline by applying ridge regression to a calibration dataset with realistic workload and engagement labels. To cope with high-frequency noise, the estimates are smoothed over a sliding window T_w:

$$\widehat{y_t}={\displaystyle \frac{1}{T_w}}\sum _{T=t-T_w+1}^t\widehat{y_T}$$

(2)

Realistic workload is defined as the combination of (a) objective task-complexity features of the Java exercises (e.g., number of classes and bugs, branching depth) and (b) the NASA-TLX mental demand rating collected immediately post-trial. Both components are z-scored and averaged to form the workload target. Engagement is measured using the SAM arousal and pleasure indices [39].

The choice of linear regression over more complex nonlinear alternatives was driven by requirements for interpretability, reliability, and low computational latency, with the average inference time being under 1 millisecond.

AICM is the final stage that links $\widehat{y_t}$ with control parameters:

$$u_t=\left[\begin{array}{c}D\left(t\right)\\ M\left(t\right)\\ A\left(t\right)\end{array}\right]$$

(3)

Rules reflect human-factors principles (optimal arousal; managing mental workload; modality appropriateness). When ŵ is high, we reduce information density and increase scaffolding; when ê dips, we shift to multimodal prompts to regain attention. These rules are grounded in human-factors guidance on optimal arousal, mental workload management, and modality appropriateness [40,41,42,43,44,45,46,47].

Here, D(t) denotes the task difficulty level, M(t) the feedback modality (visual, auditory, or multimodal), and A(t) the assistance frequency (either low or high). These parameters are updated according to transparent, deterministic rules derived from human factors research. Specifically, task difficulty is updated as:

$$D\left(t+1\right)=\left\{\begin{array}{cc}D\left(t\right)-{\delta }_D,\hfill & \hfill if \widehat{C}\left(t\right)>{\theta }_C^{high}\\ D\left(t\right)+{\delta }_D,\hfill & \hfill if \widehat{C}\left(t\right)<{\theta }_C^{low}\\ \hfill D\left(t\right),& otherwise\hfill \end{array}\right.$$

Thresholds were individualized from each participant’s Baseline (BL) calibration using percentiles: θ^w_L = P₃₀(ŵ), θ^w_H = P₇₀(ŵ); θ^e_L = P₄₀(ê), θ^e_H = P₆₀(ê). When ŵ(t) > θ^w_H, task difficulty D decreased by one level; when it dropped below θ^w_L, difficulty increased. Likewise, if ê(t) < θ^e_L, the system switched to multimodal prompts and higher assistance frequency, reverting to default when ê(t) > θ^e_H. Percentile-based thresholds are commonly used in neuroadaptive HCI to normalize for inter-individual variability [31,43]. In the experimental setup, these thresholds were established individually for each participant during a brief baseline calibration phase. Participants first completed a standardized block of programming tasks without adaptive support, while concurrent workload and engagement measures (EEG, GSR, HRV, and behavioral markers) were recorded. The distributions of the estimated workload ŵ and engagement ê values obtained during this phase were used to compute participant-specific percentile thresholds (e.g., 30th/70th percentile for workload, 40th/60th percentile for engagement). This individualized approach ensured that thresholds reflected personal differences in physiology and interaction style rather than relying on fixed global cut-offs.

While the percentile-based calibration strategy is broadly applicable, the thresholds themselves are not entirely scenario-independent. In highly different application domains (e.g., safety-critical tasks, entertainment systems), recalibration would be necessary to account for differences in task demands, sensor noise, and baseline cognitive states. However, the calibration procedure is lightweight (5–7 min) and can be repeated with minimal burden, making it feasible to deploy across diverse real-world contexts.

The end-to-end latency, measured from the acquisition of X_t to the rendering of u_t, was consistently below 85 milliseconds across all trials, remaining well under the perceptual threshold for delays noticeable to users in interactive systems.

Figure 1 depicts an overall view of the architecture in the form of a logical data flow diagram that outlines the three-tier pipeline, inter-module interactions, and main data transformations. The design’s modularity allows for easy integration of more advanced machine learning models (e.g., recurrent neural networks for temporal processing) or alternative control mechanisms (e.g., reinforcement learning) without sacrificing the system’s essential operational integrity.

The architectural design has been carefully developed to eliminate unnecessary complexity while providing sufficient expressiveness required to cover the variety witnessed in real multimodal HCI systems. Every design choice, ranging from using linear regression for estimation to adopting rule-based policies for control, was driven by the twin mandates of real-time responsiveness and transparency, both of which are crucial in user-centric adaptive systems. Consequently, the architecture is not merely theoretically sound and mathematically correct but also empirically tested to be effective on wearable systems in realistic settings.

NAMI is a robust and flexible architecture that systematically maps perceivable user signals into responsive interface behavior. Its strengths in real-time performance, modularity, and technical soundness make it a platform for ongoing research and development in the area of context-aware, wearable human–computer interaction. We use transparent heuristics to ensure real-time reliability, while Section 6 discusses possible extensions to learned policies. In the current design, multimodal inputs are processed through CASEM to estimate workload and engagement [ŵ, ê], which are then passed to AICM to produce the control vector [D, M, A]^T. This loop closes the cycle from sensing to adaptation, enabling real-time adjustments with consistently low latency.

4. Experimental Setup and Procedure

4.1. Participants

To illustrate the application of the Neuro-Adaptive Multimodal Architecture (NAMI) in a real-world environment, consider a graduate student taking a Java Programming class as part of an Informatics Master’s program. The student is immersed in an AR learning system that overlays instructional material and interactive programming exercises in their field of view. In addition, this system has been equipped with NAMI in order to facilitate the dynamic modification in the student’s cognitive state and activities and improve the overall learning experience.

4.2. Apparatus and Data Acquisition

At the start of the session (labeled as time t₀, the student interacts with the AR headset, coupled with the neuro-adaptive system. This interaction triggers the Sensing and Data Acquisition Module (SAPM), starting the collection of multimodal streams of data. The behavioral indicators include student voice commands (e.g., “next question” or “display hint”), head and hand orientation while handling virtual code blocks, and fixation points of attention on important aspects of the interface. At the same time, neurophysiological sensors record electroencephalogram (EEG) signals via a four-channel headband, thus recording brain activity that captures mental workload, along with GSR and HRV indices recorded from a wrist-worn device, hence reflecting levels of physiological engagement and arousal. These are then synchronized, normalized, and assembled into the multimodal feature vector $Z_{t_0}$.

The system extracts multimodal features from all input channels and applies uniform preprocessing. Voice input contributes confidence measures and speaking rate, head and hand movements provide pose and gesture information, gaze data reflect fixation and saccade patterns, while EEG and GSR/HRV capture neural and physiological activity. All signals are resampled to a common 10 Hz rate, filtered for artifacts, and normalized per participant relative to the baseline.

4.3. Interaction Design and Gesture Integration

Within the AR interface, the code canvas, task panel, and feedback pane respond to these inputs: voice initiates high-level commands, gaze determines the active region, and hand gestures confirm or dismiss actions. Gesture probabilities from the depth camera are included in the multimodal feature vector, with actions triggered once a confidence threshold is exceeded. Gesture recognition was tightly integrated into the interaction flow as a primary input modality complementing voice and gaze. The AR system employed a depth camera in combination with a real-time pose estimation algorithm to track hand joints and classify dynamic gestures (e.g., swipe, pinch, rotate). Recognized gestures were mapped to high-level interaction commands, such as confirming a code insertion, dismissing a hint, or rotating a virtual code object for inspection. These commands were directly injected into the multimodal feature vector Z_t alongside voice and gaze features, ensuring synchronized processing across modalities. To reduce noise and false positives, gesture posteriors were accumulated over a 300 ms temporal window, and an action was executed only when the posterior probability exceeded a fixed confidence threshold. This design allowed gestures to operate seamlessly within the neuro-adaptive loop, where CASEM interpreted them as part of the behavioral state while AICM determined whether to reinforce or attenuate gesture-based commands depending on the inferred cognitive load and engagement levels.

4.4. System Overview

Figure 2 illustrates the overall interaction process of the NAMI-based system from a user-centered perspective. The learner, equipped with lightweight wearables (EEG headband, wrist sensor), interacts with the augmented reality interface through multimodal channels such as voice, gesture, and gaze. These behavioral inputs, together with neurophysiological signals (EEG, GSR, HRV), are continuously collected and processed. The Cognitive/Affective State Estimation Module (CASEM) integrates these inputs to infer workload and engagement levels, while the Adaptive Interaction Control Module (AICM) uses these estimates to regulate task difficulty, feedback modality, and assistance frequency. The resulting adaptations are delivered back to the AR interface, dynamically modifying the Task Panel, Code Canvas, and Feedback Panel. In this way, the figure complements the architectural pipeline (Figure 2) by emphasizing the user-facing interaction cycle and clarifying how the neuro-adaptive loop is experienced during real-time programming tasks.

4.5. Procedure and Adaptive Loop

During the course of an exercise that requires the use of a Java class hierarchy, SAPM updates the feature vector Z_t every 100 milliseconds. This vector contains several elements, such as confidence measures for speech recognition, confidence levels for gesture recognition, gaze fixation durations on each code element, EEG power in the theta and beta frequency bands, and normalized GSR levels. Prior affective-computing studies have established that EEG band power and electrodermal activity are reliable indicators of workload and arousal, and that multimodal fusion can improve the robustness of state estimation. Building on these findings, NAMI adopts an early-fusion strategy combined with a regularized linear estimator to balance interpretability with real-time performance. Unlike prior neuro-adaptive systems that (i) rely on a single sensing modality or domain-specific pipelines, (ii) target small, controlled samples, and (iii) omit a reproducible control policy, NAMI integrates feature-level fusion with a regularized linear estimator for low-latency inference, coupled with a transparent rule-based controller operating on individualized thresholds. The prototype achieves <85 ms closed-loop latency on wearable AR hardware and is evaluated with 100 learners, with full specification of features, the mapping F: X_t⟼u_t, and the adaptation policy to ensure reproducibility. This feature vector is then used as the input to CASEM, which uses the learned linear ridge regression model to infer the latent cognitive load $\widehat{C}(t)$ and engagement $\widehat{E}(t)$. To make these estimates reliable, they are smoothed over a two-second sliding window.

Around 10 min into the session (time t₁), CASEM detects a rising cognitive load: $\widehat{C}\left(t_1\right)=0.82$ exceeding the high workload threshold ${\theta }_C^{high}=0.75$. At the same time, engagement $\widehat{E}\left(t_1\right)=0.48$ indicates moderate disengagement, as the student spends longer gazing at irrelevant parts of the interface and shows elevated GSR response. These internal state estimates are fed into AICM.

The AICM utilizes predefined control tactics set within the specified framework. Since $\widehat{C}\left(t_1\right)>{\theta }_C^{high}$, the system reduces the difficulty of the task D(t₁) by one level, thus supporting the current activity through increased scaffolding and hiding unnecessary information. In response to moderate disengagement $\widehat{E}\left(t_1\right)<{\theta }_C^{low}$, the feedback modality M(t₁) shifts from a text-only format to a multimodal one, with verbal prompts and animated directional cues, while the assistance frequency A(t₁) is increased, leading to an increased offering of hints. In practice, three feedback modalities were supported: (i) visual feedback, consisting of on-screen textual hints, highlighted code segments, and animated arrows pointing to relevant UI elements; (ii) auditory feedback, consisting of short synthesized voice prompts that summarized the next step or emphasized errors; and (iii) multimodal feedback, which combined both visual and auditory cues to reinforce critical information. The choice of modality was governed by the AICM, which switched from unimodal to multimodal cues when engagement dropped below the individualized threshold. Assistance adaptation operated on two levels: the frequency of hints and the richness of their content. At low assistance levels, the system provided minimal guidance (e.g., highlighting only the location of an error), whereas at high assistance levels, it delivered step-by-step scaffolding, combining visual highlights with explanatory voice prompts. This adaptive mechanism ensured that support was increased when workload rose or engagement declined, but scaled back once the learner returned to an optimal cognitive state, thereby preventing over-reliance on assistance and maintaining user autonomy.

Therefore, the learner feels less intimidated by the assignment and starts to refocus. Soon after, CASEM detects an optimal cognitive load of $\widehat{C}\left(t_2\right)=0.61$ and an increased level of engagement at $\widehat{E}\left(t_2\right)=0.71$. Here, AICM makes sure that the adaptation stabilizes and stops making any more modifications.

4.6. Summary

Throughout the entire session duration, NAMI always operates in a closed-loop manner. SAPM sends updated feature vectors Z_t, whereas CASEM processes

$$\widehat{y_t}=\left[\begin{array}{c}\widehat{C}(t)\\ \widehat{E}(t)\end{array}\right]$$

Additionally, AICM changes

$$O_t=\left[\begin{array}{c}D\left(t\right)\\ M\left(t\right)\\ A\left(t\right)\end{array}\right]$$

accordingly. The modular, asynchronous design ensures that the system remains responsive, with total latency under 85 ms, imperceptible to the student.

This example illustrates how NAMI enables a context-aware and adaptive learning process in a wearable augmented reality environment, aligning task difficulty and feedback with the cognitive and affective state of postgraduate students learning Java programming. By enabling dynamic adjustment of task difficulty and levels of support for engagement, the architecture helps learners maintain an optimal learning zone, thus maximizing understanding and retention of complex programming principles. Notably, this architectural scheme demonstrates flexibility since it can be applied in any educational software systems in which dynamic adjustments are valuable, thus testifying to its universality and applicability in this context.

5. Evaluation and Results

In an effort to evaluate the effectiveness of the NAMI, an extensive empirical study was carried out within a real educational environment aimed at simulating the expected utilization of the system. This study included 100 postgraduate students (52 male, 48 females; mean age = 27.3, SD = 2.9) who were enrolled in the Postgraduate Program in Informatics and Applications, specifically in the Java Programming course. All participants reported having intermediate to advanced programming skills and previous experience with conventional AR-based interfaces. The main goal of this study was to quantify the effect of NAMI on different dimensions, including task performance, cognitive workload, engagement, satisfaction, and the system’s reliability in a wearable AR learning environment.

This study took place in a human–computer interaction laboratory equipped with workstations that supported AR. All the participants wore a Microsoft HoloLens 2 headset together with light-weight neurophysiological sensors, namely a four-channel EEG headband and a wrist device aimed at capturing GSR and HRV. The AR setting offered a range of interactive Java programming exercises, requiring the participants to perform activities such as object-oriented design, syntax debugging, and algorithm implementation using multimodal interaction, including voice, gaze, and gesture. The tasks were carefully designed to elicit measurable cognitive load while also keeping participants engaged for 40 min. The exercises were grouped into three categories: OO design, syntax/debugging, algorithmic implementation. More specifically, the object-oriented (OO) design tasks required learners to create and extend class hierarchies, correctly applying principles of inheritance, polymorphism, and encapsulation. Syntax/debugging exercises presented code snippets containing seeded errors (e.g., missing semicolons, type mismatches, uninitialized variables), and the students were asked to identify and correct them in real time within the AR environment. Algorithmic implementation exercises focused on writing and refining core algorithms, such as sorting routines (bubble sort, quicksort) and simple recursive methods (factorial, Fibonacci), with performance constraints on time complexity. Each exercise type was embedded in the AR interface through interactive code blocks and multimodal input (voice, gesture, and gaze), ensuring that tasks elicited both conceptual reasoning and procedural coding skills. This variety of tasks was deliberately chosen to balance conceptual design, low-level debugging, and algorithmic thinking, providing a comprehensive assessment of programming proficiency. Difficulty levels were pre-authored and validated in pilot runs: D1 (easy) fewer classes/branches and 1–2 seeded bugs; D2 (moderate) more classes/branches and 3–4 bugs; D3 (hard) deeper nesting and 5–6 bugs. Under BL the level remained fixed per block. Under NA, D(t) adapted online via the rules in Section 3 (e.g., hiding noncritical panels, adding hints or stepping to a simpler variant when ŵ exceeded θ^w_H).

To rigorously compare the baseline and adaptive versions of the system, we designed a within-subjects experiment in which each participant experienced both conditions in counterbalanced order (AB/BA) to control for learning and fatigue effects. Under the BL condition, task difficulty, feedback modality, and assistance frequency were predefined and held constant throughout the session, regardless of the participant’s state. In contrast, under the NA condition, NAMI continuously monitored cognitive load and engagement and adapted these parameters in real time, as described in Section 3. An “always-help” condition was not included, as prior studies in neuroergonomics and adaptive interface design suggest that excessive or continuous assistance may increase cognitive load and visual clutter rather than improving performance. Empirical research has shown that constant on-screen guidance can distract users, fragment attention, and hinder self-regulation when not contextually moderated [22,42,43,46]. Therefore, in the present study, assistance was provided adaptively, based on the estimated workload and engagement values [ŵ, ê], to balance supportive feedback with user autonomy. Each condition lasted 20 min and was followed by a 10 min interstitial break to minimize fatigue.

Objective performance measures included the time spent on each task in combination with the number of errors made. Subjective measures were collected through standardized instruments immediately after each condition: NASA-TLX workload scores (1–100 scale), Self-Assessment Manikin (SAM) engagement ratings (1–9 scale), and user satisfaction ratings (5-point Likert scale). System-level metrics, including adaptation latency and stability, were also automatically recorded by the platform. Statistical analysis made use of paired-sample t-tests with Bonferroni correction, as well as effect size calculation.

The results are summarized in

Table 1. Comparison of Baseline and Neuro-Adaptive Conditions.

Measure	Baseline (Mean ± SD)	Neuro-Adaptive (Mean ± SD)	Difference (%)
Task Completion Time (s)	312.4 ± 42.1	267.8 ± 38.7	−14.3%
Error Rate (%)	7.9 ± 2.3	4.8 ± 1.7	−39.2%
NASA-TLX Workload	64.2 ± 9.4	49.7 ± 8.1	−22.6%
SAM Engagement (1–9)	5.1 ± 1.0	7.0 ± 0.8	+37.3%
Satisfaction (1–5)	3.6 ± 0.6	4.7 ± 0.4	+30.6%

. Participants showed a significant improvement in task performance speed under the Neuro-Adaptive condition (mean = 267.8 s, SD = 38.7) compared to the BL condition (mean = 312.4 s, SD = 42.1), representing a mean improvement of about 14% (mean difference = 44.6 s, Cohen’s d = 1.10, p < 0.001). In line with prior works on adaptive learning systems [31,32,43], we also considered a practical significance threshold of 10% improvement to ensure that observed differences reflect meaningful gains rather than minor fluctuations. The observed reduction exceeded this threshold, thereby supporting both the statistical and the practical relevance of the effect. Error percentage reduced from 7.9% (SD = 2.3) to 4.8% (SD = 1.7), reflecting a reduction of almost 40% (mean difference = 3.1%, d = 1.53, p < 0.001). Subjective workload showed a significant reduction from a mean NASA-TLX score of 64.2 (SD = 9.4) to 49.7 (SD = 8.1), corresponding to an average drop of 14.5 points (d = 1.65, p < 0.001). Engagement and satisfaction measures also showed significant improvement (all p < 0.001). To account for the within-subjects design, statistics were performed on paired difference scores (NA–BL). Figure 3 visualizes these differences per participant, showing the overall positive trend while also highlighting individual variability, including a few cases with negligible or negative improvement. All reported effects reflected large effect sizes, confirming the robustness of the results.

Table 1 clearly illustrates that the neuro-adaptive changes brought about by NAMI ended up producing improvements that were both subjective and objective. Participants performed tasks more efficiently and accurately, while at the same time feeling a reduced workload with higher levels of engagement and satisfaction.

The results are also illustrated in Figure 4 and Figure 5. Figure 4 shows task completeness time and error rate under different conditions and highlights a significant reduction in both metrics when the NAMI system was active. Standard deviations are represented by error bars and depict both improvements and reduced variation in the neuro-adaptive system.

Figure 5 shows the subjective measures, that is, workload, engagement, and satisfaction. The workload scores significantly decreased under NAMI, but the scores for engagement and satisfaction improved, thus proving that users viewed the system adaptations positively instead of finding them distracting or intrusive.

Along with the main results, a system-level performance evaluation was made to determine both reliability and robustness. Average adaptation latency, from sensor input to the visible alteration in the interface, was found to be 82 milliseconds (SD: 6.3), within real-time constraints relevant for interactive systems. During all sessions, the system showed stability, with no failure or measurable degradation in performance.

Qualitative feedback collected directly from the participants conformed to findings drawn from the quantitative data. Over 90% of the participants preferred the neuro-adaptive system, describing it as “more natural,” “more in sync with my rhythm,” and “less stressful.” These findings are consistent with the engagement and satisfaction measures and highlight the acceptability of the architecture in the context of an actual learning environment.

6. Discussion

The findings of this study highlight the potential of neuro-adaptive multimodal interaction to address long-standing challenges in technology-enhanced learning. Beyond statistical improvements in performance and workload reduction, the results suggest that integrating behavioral and neurophysiological cues can create a more balanced learning experience, where cognitive demands are dynamically aligned with individual learner needs. This reflects a broader shift from static instructional design toward adaptive, learner-centered systems that respond to both observable behavior and latent mental states.

These findings also clarify why the chosen architectural trade-offs were effective in practice. The use of a simple linear estimator and rule-based adaptation policy proved sufficient to capture fluctuations in workload and engagement, as evidenced by the observed reduction in NASA-TLX scores by 22.6% and the 37.3% increase in engagement. This suggests that more complex models are not strictly necessary to obtain robust improvements, although future research may explore probabilistic or reinforcement learning–based controllers to determine whether further personalization can be achieved without sacrificing explainability or latency.

At the same time, the results highlight certain boundary conditions of the current evaluation. This study involved postgraduate informatics students working on programming tasks, where both workload and satisfaction improved consistently across participants. It remains to be tested whether the same architecture would generalize to different learner populations (e.g., novices, younger students, or professionals), to other types of cognitive tasks beyond programming, or to longer-term deployments where habituation or shifts in trust may occur. Such extensions follow naturally from the present findings, which demonstrate clear short-term benefits but do not yet establish how these effects evolve over time or across diverse contexts.

Placing NAMI in the wide-ranging context of relevant research, it is clear that this architecture extends prior work by unifying neurophysiology and multimodal interaction in a wearable, real-time architecture with explicit policies and large-sample evaluation. Multimodal interaction systems of today, such as those investigated by [48], have shown the benefits of combining speech, gesture, and gaze in joint augmented reality scenarios; however, they often treat the user as a black box, disregarding internal emotional or cognitive processes. Advances in affective computing, as found in studies such as [4,49], have shown that external markers, such as facial expressions, prosody, and posture, can affect adaptive behavior; however, these methods typically rely on indirect and imprecise measurements of mental states. While neuroergonomics studies, such as those outlined in [22,50], have confirmed the validity of EEG and fNIRS technologies as means for assessing workload and attention allocation, these systems remain largely limited to controlled laboratory settings and single-modal input streams. On the other hand, the NAMI combines behavioral and neurophysiological cues into a reproducible, modular, and real-time adaptive system, uniquely tailored for wearable devices. Such combination represents an innovative and pragmatic contribution that bridges theoretical models with practical human-centered interaction paradigms and thus enables the emergence of more humane and context-aware computing.

It should also be acknowledged that, despite the counterbalanced design, potential order effects may still have influenced the results. For example, the response time in a participant’s first session might differ from the second due to practice, fatigue, or adaptation to the AR environment, and similar considerations could apply to error rates, workload, and engagement. While the present study did not explicitly analyze first-versus-second session differences, future work could address such factors to further strengthen the validity of neuro-adaptive evaluations.

7. Conclusions and Future Work

This study presented the design, implementation, and evaluation of the Neuro-Adaptive Multimodal Architecture (NAMI) in a realistic programming education scenario. In a study with N = 100 postgraduate learners, the neuro-adaptive condition significantly reduced task completion time (−14%), errors (−39%), and subjective workload (−22%) while increasing engagement (+37%) and satisfaction (+30%) compared to Baseline. These results demonstrate that a simple linear estimator and rule-based controller can provide robust, low-latency adaptations that meaningfully improve both performance and subjective experience in augmented reality programming tasks.

Beyond these empirical findings, this study contributes to the state of the art by showing how neuro-adaptive methods can be applied to programming education, where maintaining optimal cognitive load is critical for learning complex abstractions. Unlike prior affective-computing approaches that rely on indirect behavioral proxies, NAMI integrates multimodal behavioral and neurophysiological signals in a transparent, reproducible pipeline, bridging theory and practice in wearable HCI.

While promising, the results should be interpreted within the boundaries of the present design. This study focused on postgraduate informatics students and short-term programming tasks; future research should investigate generalization to different learner groups, tasks, and longitudinal settings. Further work is also needed to compare the rule-based policy against always-help conditions, to incorporate learning gain measures, and to explore advanced control models such as reinforcement learning. Addressing ethical considerations of privacy, consent, and transparency will be essential for broader deployment.