Adaptive Deep Learning Framework for Emotion Recognition in Social Robots: Toward Inclusive Human–Robot Interaction for Users with Special Needs

Abstract

Emotion recognition is a key capability of social robots operating in real-world human-centered environments, especially when interacting with users with special needs. Such users may express emotions in atypical, subtle, or strongly context-dependent ways. These characteristics pose significant challenges for conventional emotion recognition systems. This paper proposes an adaptive deep learning framework for emotion recognition in social robots. The framework is designed to support inclusive and accessible human–robot interaction. It combines region-based convolutional neural networks with adaptive learning mechanisms. These mechanisms explicitly model individual variability, contextual information, and interaction dynamics. Multiple deep architectures are evaluated to assess robustness across diverse emotional expressions, including those influenced by cognitive, sensory, or developmental differences. Rather than relying on fixed emotion models, the proposed approach emphasizes adaptability. The system dynamically adjusts its perception strategies to user-specific expressive patterns. Experimental validation is conducted using context-aware emotion datasets. Performance is evaluated in terms of detection accuracy, robustness to variability, and generalization across emotion categories. The results show that adaptive mechanisms improve recognition performance in scenarios characterized by non-standard or low-intensity expressions, compared to static baseline models. This study highlights the importance of flexible, context-sensitive perception for inclusive social robotics. It also discusses design implications for deploying emotion-aware robots in assistive, educational, and therapeutic settings. Overall, the proposed framework represents a step toward socially intelligent robots capable of engaging more effectively with users with special needs.

1. Introduction

1.1. Social Robots in Human-Centered Environments

Social robots, and in particular socially assistive robots (SARs), constitute a class of interactive systems designed to support users primarily through social interaction rather than through physical manipulation of the environment. According to the seminal definition proposed by Feil-Seifer and Matarić, socially assistive robotics focuses on providing assistance through social engagement, encouragement, and coaching. The aim is to achieve measurable improvements in the user’s cognitive, emotional, or behavioral state [1]. The effectiveness of such robots is therefore fundamentally linked to their capacity for social perception. This includes the interpretation of nonverbal cues, conversational dynamics, and affective states. In human-centered environments, social robots have been widely explored in education, therapy, and assistive care. They may function as tutors, therapeutic mediators, or companions for individuals with communication, sensory, or developmental challenges. A particularly active line of research concerns the use of social robots in interventions for individuals on the autism spectrum. Empirical studies suggest that robots can elicit increased engagement, attention, and social initiation. This effect is partly attributed to their predictable behavior and reduced social complexity compared to human partners [2]. At the same time, systematic reviews highlight considerable heterogeneity in study designs, robotic platforms, outcome measures, and experimental protocols. This heterogeneity complicates the generalization of findings and their translation into real-world deployments [3]. Within this broader context, emotion recognition emerges as a core component of effective human–robot interaction (HRI). Emotions regulate social behavior and shape trust, acceptance, and the interpretation of intentions during interaction. Research in affective HRI indicates that a robot’s ability to detect and respond appropriately to human emotional states contributes to more natural and socially acceptable interactions [4]. In educational, therapeutic, and assistive scenarios, emotion-aware behavior allows robots to adapt task difficulty and modulate feedback strategies. It also enables them to respond to disengagement or distress in a timely manner. Such adaptive responses support both functional outcomes and interaction safety. For this reason, emotion recognition is increasingly regarded not as an optional enhancement but as a foundational capability for socially intelligent robots operating in human-centered settings.

1.2. Challenges of Emotion Recognition for Users with Special Needs

Despite substantial progress in affective computing, conventional emotion recognition systems—particularly those based on computer vision—are typically developed under assumptions of relatively standardized emotional expressions and well-defined affective categories. Surveys of deep learning approaches for facial expression recognition demonstrate that even for neurotypical populations, system performance degrades significantly under real-world conditions due to variations in illumination, head pose, occlusion, and inter-individual differences in expressive style [5]. Models trained on curated, laboratory-based datasets therefore often struggle to generalize to naturalistic environments, where emotional signals are subtle, noisy, and context dependent. These limitations are further amplified when interacting with users with special needs, whose emotional expressions may deviate markedly from canonical patterns. For example, individuals on the autism spectrum may display reduced facial expressivity, atypical timing of expressions, or emotion-specific recognition difficulties that depend on task demands and cognitive load. Experimental evidence indicates that emotion recognition performance in autistic children can vary significantly across emotional categories and interaction contexts, particularly under conditions of increased temporal or attentional constraints [6]. From the perspective of automated perception, such variability undermines the reliability of fixed emotion models that rely on stereotypical facial configurations or static decision boundaries.

Importantly, the challenge is not merely one of reduced classification accuracy, but of mismatched representational assumptions. In inclusive HRI scenarios, emotional meaning may be conveyed through idiosyncratic, low-intensity, or multimodal cues whose interpretation depends on interaction history and situational context. Consequently, emotion recognition for users with special needs requires moving beyond population-level averages toward systems capable of personalization and adaptation. Prior work in robot-assisted autism therapy has demonstrated that personalized machine learning approaches—tailored to individual behavioral and affective profiles—can substantially improve the robot’s ability to infer engagement and emotional states during interaction [7]. These findings underscore the necessity of adaptive perception mechanisms that continuously update their internal models in response to user-specific characteristics and interaction dynamics. Taken together, these challenges motivate the development of emotion recognition frameworks that emphasize adaptability, context awareness, and individual variability, rather than static mappings between observed signals and predefined emotion categories. Such an approach is particularly critical for inclusive social robotics, where the diversity of users and expressive styles constitutes the norm rather than the exception.

1.3. Motivation and Research Gap

Despite significant advances in affective computing and emotion recognition for human–robot interaction, most existing systems remain grounded in static emotion models that assume relatively uniform patterns of emotional expression across users. Deep learning-based approaches, including convolutional neural networks trained on large-scale facial expression datasets, typically optimize performance at the population level, implicitly treating inter-individual variability as noise rather than as a meaningful signal [5]. While such models achieve high accuracy on benchmark datasets, their effectiveness often deteriorates in real-world interactions characterized by heterogeneous users and dynamic contexts. This limitation becomes particularly salient in interactions involving users with special needs, where emotional expressions may differ substantially from canonical representations. Prior studies have shown that emotional expressivity can vary across individuals depending on cognitive, sensory, or developmental factors and that these differences are not adequately captured by fixed classifiers trained on neurotypical data distributions [6]. As a result, conventional systems may systematically misinterpret or fail to detect emotional states in precisely those scenarios where reliable affective perception is most critical, such as therapeutic or assistive settings.

Furthermore, many current emotion recognition frameworks operate largely independently of interaction context, relying on isolated visual cues (e.g., facial expressions) without integrating information about situational factors, task demands, or interaction history. However, empirical evidence from affective computing and social robotics indicates that emotional meaning is often context dependent, emerging from the interplay between observable behavior and environmental or interactional conditions [8]. The lack of context-aware and adaptive perception mechanisms therefore constitutes a key research gap, limiting the inclusiveness and robustness of emotion-aware robots deployed in naturalistic environments. Taken together, these observations motivate the need for emotion recognition systems that move beyond static, one-size-fits-all models toward adaptive frameworks capable of accommodating individual variability and contextual dynamics. Addressing this gap is essential for enabling socially intelligent robots to engage effectively and ethically with diverse user populations, particularly in assistive, educational, and therapeutic applications.

1.4. Contributions of the Paper

In response to the limitations identified above, this paper proposes an adaptive deep learning framework for emotion recognition in social robots, explicitly designed to support inclusive human–robot interaction. The primary contribution of this work lies in shifting the focus from fixed emotion models to adaptable perception strategies that can be tuned to user-specific characteristics and evolving interaction contexts. First, we introduce a novel framework that integrates region-based convolutional neural networks (R-CNNs) with adaptive learning mechanisms. The R-CNN component enables focused analysis of emotionally salient regions, such as facial areas or expressive body cues, while the adaptive layer allows the model to update its internal representations in response to observed individual differences over time. This design is informed by prior findings demonstrating the effectiveness of region-based architectures for visual perception tasks in complex, real-world settings [9]. Second, the proposed approach incorporates adaptive learning strategies that account for both individual variability and contextual cues, enabling the system to refine its emotion recognition behavior during ongoing interaction. Unlike static classifiers, the framework is designed to adjust its perception strategies based on interaction history and user-specific patterns, aligning with recent trends toward personalization in affective human–robot interaction [7]. Finally, the paper provides a comprehensive experimental evaluation of the proposed framework, assessing its robustness to non-standard emotional expressions and its ability to generalize across emotion categories. Performance is systematically compared against static baseline models to demonstrate the benefits of adaptability in scenarios characterized by atypical or subtle emotional signals. By empirically validating the advantages of adaptive mechanisms, this study contributes evidence supporting the design of flexible, context-sensitive perception systems as a foundation for inclusive social robotics.

Positioning and Novelty to Existing Work

While a substantial body of research has addressed emotion recognition in social robotics, most existing approaches focus on isolated aspects such as static visual classification, limited personalization, or offline context modeling. In contrast, the proposed framework introduces a unified architecture that integrates region-based perception, context-aware conditioning, and continual adaptive learning within a closed-loop human–robot interaction setting. To clarify the novelty of the proposed approach and to position it with respect to prior work, Table 1 provides a structured comparison between representative categories of existing emotion recognition systems and the framework presented in this study. The comparison highlights not only which capabilities are supported but also how these capabilities are combined within a single, interaction-oriented system.

Table 1. Comparison of the proposed framework with representative categories of existing emotion recognition approaches in social robotics. Citation numbers indicate representative works from the existing literature.

Capability/ System Property	Static Vision-Based Models [5,10]	Context-Aware (Non-Adaptive) Models [8,11]	Personalized Models [7,12]	Continual Learning Models [13,14,15]	Proposed Framework
Region-based visual analysis	✓	✓	✓	✓	✓
Explicit context modeling	✗	✓	(partial)	✗	✓
Online user adaptation during interaction	✗	✗	(limited)	✓	✓
Continual learning with forgetting control (e.g., EWC)	✗	✗	✗	✓	✓
Uncertainty-aware prediction and fusion	✗	✗	✗	(partial)	✓
Closed-loop HRI integration (perception → action → feedback)	✗	(partial)	(partial)	✗	✓
Robustness to atypical/low-intensity expressions	Limited	Moderate	Moderate	Variable	High
Designed for inclusive HRI scenarios	✗	✗	(partial)	✗	✓

✓—System property/Capability that occurs in model; ✗—System property/Capability that do not occurs in model.

In particular, prior vision-based emotion recognition models typically rely on static classifiers trained offline on population-level datasets, without mechanisms for personalization or online adaptation during interaction [5,10]. Context-aware approaches often incorporate contextual cues as auxiliary inputs, but treat context as a fixed conditioning variable rather than as a dynamic component of the learning process [8,11]. Personalized emotion recognition systems have demonstrated improved performance for individual users, especially in autism-related HRI, yet commonly rely on offline retraining or limited adaptation without explicit mechanisms for long-term stability [7,12]. Continual learning approaches, while theoretically well-suited for long-term interaction, are rarely integrated end-to-end with perception, uncertainty handling, and robot behavior regulation in social robotic systems [13,14,15].

The proposed framework advances the state of the art by jointly addressing these limitations. It combines region-based visual analysis with context-aware conditioning and multi-scale adaptive learning, while explicitly modeling uncertainty and embedding emotion recognition within a closed interaction loop. This integration enables the system to adapt to individual expressive patterns, disambiguate emotions across interaction contexts, and maintain stability over extended interaction horizons—capabilities that are essential for inclusive human–robot interaction but remain underexplored in existing systems.

It is important to note that emotion recognition based solely on visual cues remains conceptually and empirically contested, as facial expressions are often ambiguous, culturally modulated, and strongly dependent on contextual and individual factors.

2. Related Work

2.1. Emotion Recognition in Social Robotics

Emotion recognition has become a central research theme in social robotics because affective cues influence users’ engagement, trust, and cooperation during human–robot interaction (HRI). In HRI-oriented emotion recognition, computer-vision pipelines—most commonly based on facial expression analysis and, increasingly, on upper-body and gesture cues—remain a dominant paradigm due to their non-invasiveness and compatibility with on-board sensing. A comprehensive HRI-focused review by Spezialetti et al. highlights that vision-based emotion recognition is frequently adopted in social robotic platforms, yet its real-world performance is constrained by environmental variability (illumination, pose, and occlusion) and by the interaction-dependent nature of affective behavior, which cannot be fully captured by purely static models trained offline [10]. Alongside vision, a strong trend in social robotics is the transition from single-modality perception (e.g., face-only) to multimodal affect recognition that combines complementary channels such as audio (prosody and voice quality), language, physiological signals, and contextual features. The motivation is both theoretical and practical: affective states are not consistently expressed through a single channel, and certain modalities become unreliable in specific interaction conditions (e.g., face occlusion or noisy speech). Multimodal affective computing research emphasizes that robust inference often requires fusion strategies that integrate heterogeneous streams and handle missing or noisy modalities [16]. From the HRI perspective, broad surveys of multimodal human–robot interaction further underline that naturalistic interaction is inherently multimodal (vision, auditory–language, haptics, and physiological sensing) and that perception architectures should be designed to exploit these channels in a coordinated manner rather than treating them as independent add-ons [11]. In this line, early work on audio–visual affect recognition already demonstrated empirically that fusing facial and speech-derived cues can improve recognition of affective states compared to unimodal baselines, providing a foundational rationale for multimodal designs that later became prevalent in interactive systems and social robotics [17].

2.2. Deep Learning Methods for Affective Computing

Modern affective computing increasingly relies on deep learning models capable of learning hierarchical representations from raw sensory data. For vision-based affect recognition, convolutional neural networks (CNNs) constitute a standard backbone, often initialized from high-capacity architectures originally developed for general image recognition tasks. Residual networks (ResNets) are widely used due to their scalability to deep architectures and strong representational performance, and they have become a common feature extractor in facial expression recognition and related affective tasks [18]. Beyond frame-level recognition, affect in social interaction often unfolds dynamically over time, which motivates sequential modeling. Recurrent neural networks (RNNs), and particularly long short-term memory (LSTM) networks, are frequently employed to capture temporal dependencies in facial movements, vocal prosody, and interaction trajectories, leveraging the LSTM’s ability to model long-range temporal structure [19]. Region-based approaches provide another important direction, especially when the relevant affective evidence is localized (e.g., specific facial regions, eye area, mouth, or hand gestures). In broader computer vision, region-based CNN methods have been influential for focusing computation on salient regions; this principle has also informed affective pipelines seeking to attend to discriminative subregions rather than processing whole frames uniformly [9]. In affective computing and HRI, region-focused modeling can be particularly valuable when users display subtle expressions or when only partial cues are visible due to camera viewpoint or occlusions. However, deep emotion recognition systems remain strongly shaped by the datasets on which they are trained. Large-scale “in-the-wild” datasets such as AffectNet were introduced to improve ecological validity by collecting facial images from unconstrained settings and annotating both categorical expressions and dimensional affect (valence/arousal). Although such datasets have advanced the field substantially, they still encode specific distributional assumptions and labeling conventions that may not transfer to atypical or highly individualized affect displays—precisely the conditions encountered in inclusive social robotics [20]. This dataset dependence contributes to well-documented generalization issues: models optimized for benchmark performance may underperform in real interactions due to domain shift, demographic imbalance, and the mismatch between posed/annotated expressions and the nuanced, context-dependent affective behavior observed during human–robot engagement [10,16]. Consequently, recent HRI-oriented research increasingly argues for adaptive, context-aware learning strategies that can mitigate these limitations by tailoring perception to users and interaction dynamics rather than relying on fixed mappings learned from standard corpora [10,11].

Recent work in emotion recognition has increasingly explored transformer-based architectures for modeling temporal dynamics in facial expressions. A representative example is STT-Net, proposed by Khan et al., which introduces a lightweight temporal transformer by replacing standard multi-head self-attention with a simplified self/cross-attention mechanism, enabling efficient modeling of frame-to-frame dependencies with low computational costs [21]. The model achieves competitive performance on benchmark datasets such as CK+ and FER-2013+ while remaining suitable for real-time applications. Despite these advantages, STT-Net is primarily designed for offline, dataset-driven classification and applies temporal modeling uniformly across users and interaction contexts. It does not incorporate mechanisms for user-specific adaptation, explicit context conditioning, or online learning during interaction and treats emotional dynamics as generic temporal patterns rather than interaction-dependent processes. In contrast, the framework proposed in this study formulates emotion recognition as an adaptive perception problem embedded in ongoing human–robot interaction. Beyond temporal feature modeling, it integrates region-based visual analysis, explicit context representation, and continual adaptive learning within a closed-loop interaction pipeline. Temporal information is reflected not only at the perceptual level but also through interaction history and user-specific adaptation states. Moreover, unlike STT-Net and similar transformer-based approaches, the proposed framework explicitly incorporates uncertainty estimation and confidence-aware fusion to regulate both emotion inference and robot behavior. This enables safer and more robust operation in scenarios involving low-intensity, ambiguous, or atypical expressions, extending transformer-based emotion recognition toward long-term, personalized, and interaction-aware social robotics.

2.3. Adaptive and Personalized Emotion Recognition Systems

A growing body of work in affective computing argues that user-independent emotion models are often insufficient for real-world deployment, because expressive behavior varies substantially across individuals (baseline expressivity, idiosyncratic cues, cultural and demographic factors) and across time (fatigue, habituation, learning effects). To address these issues, researchers have increasingly explored personalization and adaptation strategies, which aim to tune recognition models to the target user while minimizing the need for extensive labeled data. One influential direction is domain adaptation for personalization, where a model trained on large source datasets is adapted to a specific target individual (or target context) using limited or even unlabeled target data. Zen et al. proposed an unsupervised domain adaptation approach for personalized facial emotion recognition, demonstrating that personalization can be achieved without manual labeling of the target user’s samples, thereby reducing practical barriers to deployment in interactive systems [12]. In interactive robotics, this is particularly relevant because collecting and labeling emotion data per user is expensive, ethically sensitive, and often infeasible at scale. A second direction focuses on continual/lifelong learning, where the recognition system updates its parameters over ongoing interactions while mitigating catastrophic forgetting. Kirkpatrick et al. introduced Elastic Weight Consolidation (EWC) as a principled mechanism to protect previously acquired knowledge during sequential learning, providing a widely used foundation for continual adaptation in neural systems [13]. Such mechanisms are conceptually aligned with adaptive affect perception in HRI, where the robot’s model should improve with experience while preserving stable competencies (e.g., maintaining robust detection for broadly expressed affect while learning user-specific deviations).

Within HRI specifically, personalization has also been studied as a design principle for long-term interactions, emphasizing user modeling, preference learning, and adaptive behavior selection. Irfan et al. highlight that sustained, repeated interactions require personalization to avoid the “novelty effect” decline and to support relationship-building, which implies that perception modules—including affect recognition—should be designed to incorporate user-specific representations and long-term memory [14]. Importantly, personalization is not only a technical optimization but is deeply intertwined with interaction context and dynamics: affect signals emerge within tasks, dialogue structure, and turn-taking rhythms. For example, long-term in-home socially assistive interventions have been used to model engagement trajectories over time in child–robot interaction contexts, demonstrating that interactional state (e.g., engagement) is both measurable and dynamically shaped by the robot’s behavior and the evolving relationship with the user [15]. Similarly, applied socially assistive systems for older adults integrate multimodal affect perception with dialogue management, illustrating how emotion recognition can be embedded into a broader, context-sensitive interaction architecture rather than treated as a standalone classifier [22].

2.4. Emotion Recognition for Users with Special Needs

Emotion recognition for users with special needs introduces additional complexity because emotional expression may be atypical, subtle, or communicated through non-standard multimodal patterns. In the autism domain, systematic evidence indicates that automatic emotion recognition faces challenges related to heterogeneous participant profiles, variability in observed modalities (face, gaze, physiology, and speech), and limited availability of open, well-annotated datasets that reflect naturalistic interaction contexts. Landowska et al. synthesize this literature and emphasize that all observation channels can be disturbed or partially unavailable, underscoring the need for robust, multimodal and context-aware methods that can tolerate missing or unreliable cues [23]. These issues directly motivate adaptive systems that can recalibrate their inference when a modality becomes uninformative (e.g., reduced facial expressivity) or when the user expresses affect through idiosyncratic behaviors. Beyond ASD (Autism Spectrum Disorder), research on sensory and cognitive differences further demonstrates that emotion recognition systems cannot assume neurotypical signal patterns. For instance, Li et al. studied emotion recognition in individuals with hearing impairment using fusion of facial expressions with EEG topographic maps, showing that multimodal designs may improve classification accuracy compared to single-channel approaches and suggesting that assistive settings may benefit from flexible modality combinations tailored to user capabilities [24]. At the same time, recent critical scholarship highlights a major risk: emotion recognition technologies may impose neurotypical norms and systematically disadvantage autistic individuals in consequential settings (e.g., workplaces), particularly when datasets underrepresent neurodivergent users or when annotation practices encode normative assumptions about “correct” emotional expression [25]. This line of work strengthens the argument that inclusive emotion recognition should prioritize adaptability, transparency, and user-centered validation rather than treating atypicality as noise. Cognitive impairments and aging-related conditions create additional requirements for affect-aware systems, including sensitivity to changes in expressivity across disease progression and ethical safeguards around privacy and consent. Affective computing reviews for late-life mood and cognitive disorders indicate strong potential for emotion AI in screening, monitoring, and supportive care, while also stressing the risks of bias and the importance of responsible, context-aware deployment in vulnerable populations [26]. Parallel efforts in accessible HCI explicitly explore how emotion recognition can be integrated into systems designed for people with intellectual disabilities, illustrating both the feasibility of such applications and the need for careful adaptation of interfaces, explanations, and interaction design to the user group [27].

Research gaps persist across these special-needs contexts. First, many studies remain constrained to small samples or controlled settings, limiting generalization to real-world HRI environments where users’ affective behavior is shaped by task demands, novelty, fatigue, and social dynamics. Second, datasets representing diverse special-needs populations are still relatively scarce, and label quality is complicated by the fact that “ground truth” emotion may not align with visible expressions, especially under atypical presentation. Third, existing systems often lack explicit mechanisms for online personalization, uncertainty handling, and context integration—capabilities that appear essential when non-standard emotional expressions are the norm rather than an edge case. These gaps motivate adaptive frameworks that combine robust perception backbones with personalization and context-sensitive learning strategies.

3. Materials and Methods

3.1. Overview of the Proposed Adaptive Framework

The proposed system is conceived as a modular, adaptive perception framework for emotion recognition in social robots operating in real-world, human-centered environments. Its design is guided by three core principles emerging from prior work in social robotics and affective computing: the need for robust visual perception under unconstrained conditions [5,10], the necessity of personalization to accommodate inter-individual variability in affective expression [7,12,14], and the importance of context-aware and temporally sensitive processing in interactive settings [8,15]. Rather than treating emotion recognition as a static classification problem, the framework explicitly models it as a dynamic inference process embedded in ongoing human–robot interaction. At the architectural level, the framework is organized into four tightly coupled layers: a perceptual input layer, a region-based visual analysis layer, an adaptive learning layer, and a context-aware integration layer. The perceptual input layer acquires raw sensory data from the robot’s vision system, primarily RGB (Red-Green-Blue) video streams capturing the user’s face, upper body, and interaction-relevant gestures. These data are preprocessed online to ensure compatibility with real-time robotic constraints while preserving fine-grained spatial and temporal information required for affective analysis [5,10]. The core visual processing is performed by a region-based convolutional architecture, which selectively focuses on emotionally salient regions rather than uniformly processing entire frames. This design choice reflects empirical evidence that affective cues are often localized (e.g., facial regions and hand movements) and that region-focused processing improves robustness under occlusion and pose variation [9]. The region-based component provides structured feature representations that serve as the input to higher-level adaptive mechanisms, forming a clear interface between perception and learning.

Crucially, the framework incorporates adaptive learning mechanisms as a first-class architectural element rather than as an auxiliary post-processing step. Building on insights from personalized machine learning and long-term HRI, the system maintains user-specific adaptation states that allow its internal representations and decision boundaries to evolve during interaction [7,12,13]. Adaptation operates at multiple temporal scales: short-term updates capture transient interaction dynamics (e.g., changes in engagement or affective intensity), while longer-term adjustments encode stable individual characteristics. This multi-scale adaptivity is designed to mitigate the limitations of population-level emotion models when confronted with atypical or non-standard expressive patterns, which are common in inclusive and assistive scenarios [6,23]. Complementing user adaptation, the framework integrates a context-aware perception module that aggregates non-visual information relevant to affect interpretation. Contextual cues include environmental conditions (e.g., interaction setting), task-related variables (e.g., activity type, difficulty), and interaction history (e.g., recent robot actions and user responses). Prior research has shown that emotional meaning in HRI often emerges from the interaction context rather than from isolated perceptual cues [8,15]; accordingly, the proposed architecture treats context as an integral component of affect inference. Contextual representations are fused with visual features to modulate emotion predictions and to guide adaptive updates, enabling the system to disambiguate visually similar expressions that carry different meanings across situations. From a data-flow perspective, sensory inputs are processed in a closed-loop pipeline. Visual features extracted by the region-based network are combined with contextual representations and passed to the adaptive learning layer, which produces emotion estimates along with uncertainty indicators. These outputs can subsequently inform the robot’s behavior (e.g., dialogue strategy or task adaptation), while interaction outcomes are fed back into the adaptive mechanisms to update the internal model. This closed-loop design ensures consistency between perception, learning, and action, aligning with established principles of socially assistive robotics that emphasize continuous coupling between sensing and interaction [1,4].

The outputs produced by each module of the proposed architecture at time step t: the system processes an RGB frame x_t and produces intermediate and final representations as follows:

The region-based perceptual module extracts a set of region-specific feature embeddings:

$$v_t=f_{visual}\left(x_t\right)\in {\mathbb{R}}^d$$

(1)

where v_t denotes the aggregated visual embedding obtained from face and upper-body regions after convolutional processing and region-weighted fusion.

The contextual encoder produces a low-dimensional representation of the interaction state:

$$c_t= f_{context}(s_t)\in {\mathbb{R}}^k$$

(2)

where s_t represents task phase, interaction history, and engagement-related signals. The resulting context vector captures interaction-level information rather than frame-level features.

The adaptive layer integrates visual and contextual information:

$$z_t=f_{fusion}(v_t,c_t, u)$$

(3)

where u denotes the current user-adaptive state. The output $z_t\in {\mathbb{R}}^m$ represents emotion logits before normalization.

Final class probabilities are computed using softmax normalization:

$$p_t=softmax\left(z_t\right)$$

(4)

where p_t represents the categorical distribution over emotion classes.

Uncertainty is estimated using prediction entropy:

$${\sigma }_t= - {\sum }_{i=l}^m{p}_{t,i}log{p}_{t,i}$$

(5)

Higher entropy values indicate greater ambiguity in the affective interpretation. These outputs jointly form the basis for adaptive weighting and behavior regulation in the subsequent decision layer.

Overall, the proposed framework provides a coherent foundation for the subsequent methodological components described in Section 3.2, Section 3.3 and Section 3.4. Its architecture is deliberately designed to support robust perception, personalization, and context sensitivity within a unified system, thereby enabling systematic evaluation of adaptive emotion recognition in scenarios characterized by individual variability and non-standard emotional expression.

3.2. Region-Based Convolutional Neural Networks

Visual emotion recognition in social robotics relies heavily on the ability to extract affective cues from localized and semantically meaningful regions of the human body, most notably the face and expressive gestures. A substantial body of research in affective computing demonstrates that emotional information is not uniformly distributed across the visual field, but is instead concentrated in specific regions such as the eyes, mouth, eyebrows, and hand movements [5,8,10]. Processing entire image frames with uniform importance may therefore dilute discriminative signals and increase sensitivity to noise caused by background clutter, pose variation, or partial occlusions—conditions that are common in natural human–robot interaction (HRI). To address these challenges, the proposed framework adopts a region-based convolutional neural network (R-CNN) paradigm as the core visual analysis component. Region-based approaches, originally introduced to improve object detection by focusing computation on candidate regions of interest rather than whole images, have proven effective in scenarios where relevant visual information is spatially localized and context dependent [9]. In the context of emotion recognition, this paradigm enables the system to selectively attend to emotionally salient regions—such as facial areas or upper-body gestures—while suppressing irrelevant visual content.

In the proposed architecture, region proposals are generated for multiple affect-relevant targets, including the face and gesture-related regions of the upper body. These regions are processed independently through convolutional feature extractors, yielding structured representations that preserve spatial specificity while remaining compatible with downstream adaptive learning mechanisms. This design is particularly advantageous for inclusive HRI scenarios, where emotional expressions may be subtle, asymmetric, or partially expressed, as is frequently observed in users with special needs [6,23]. By isolating and analyzing localized cues, the system can remain sensitive to weak or atypical signals that might otherwise be lost in global representations. The choice of an R-CNN-based approach is further motivated by its architectural compatibility with adaptation and personalization, which are central to the proposed framework. Region-level features provide a natural interface for adaptive learning, allowing the system to adjust the relative importance of different regions on a per-user basis. For example, if a particular user exhibits more reliable affective cues through gestures than facial expressions, the adaptive mechanisms described in Section 3.3 can reweight region-specific contributions accordingly, without requiring a redesign of the underlying perception pipeline. This aligns with prior findings in personalized affect recognition, which emphasize the value of modular representations that can be selectively tuned to individual expressive profiles [7,12]. From a robustness perspective, region-based processing also mitigates several limitations of conventional CNN-based emotion recognition models trained on standard datasets. Global models often implicitly assume canonical facial configurations and frontal views, assumptions that do not hold in real-world robotic deployments where camera viewpoints vary and users move freely [5,10]. By contrast, region-based architectures can maintain performance under partial visibility and dynamic interaction conditions, which is essential for long-term, in-the-wild HRI [15]. Moreover, focusing on regions rather than entire frames reduces sensitivity to dataset-specific biases encoded in background statistics, supporting better generalization across environments and user populations [20].

3.3. Adaptive Learning Mechanisms

To address the substantial inter-individual variability and contextual dependence of emotional expression in human–robot interaction (HRI), the proposed framework incorporates adaptive learning mechanisms as a central design element rather than as an auxiliary optimization step. This choice is motivated by extensive evidence that static emotion recognition models, even when based on deep architectures, struggle to generalize across users and interaction conditions, particularly in inclusive and assistive scenarios [5,6,7,10]. The adaptive mechanisms introduced here are designed to operate at multiple levels—user adaptation, contextual learning, and online model updating—forming a coherent strategy for dynamic affect inference. User adaptation is implemented to account for stable individual differences in expressive behavior, such as baseline intensity, preferred expressive channels (e.g., facial vs. gestural cues), and idiosyncratic emotion–signal mappings. Rather than learning a separate model for each user from scratch, the system builds upon a shared representation learned from population-level data and progressively adapts it to the current user. This approach is consistent with prior work in personalized affect recognition, which demonstrates that incorporating user-specific adaptation can substantially improve perception of affect and engagement in interactive settings without sacrificing generalization [7,12]. Adaptation is realized through parameter-efficient updates applied to selected layers of the network, enabling the model to capture user-specific patterns while preserving a common representational backbone.

The proposed framework integrates three complementary adaptive mechanisms: user-level adaptation, context conditioning, and online model updating. These mechanisms operate concurrently during interaction, but are explicitly designed to contribute at different temporal and semantic scales. User adaptation captures relatively stable, long-term characteristics of individual emotional expression, such as baseline expressivity, preferred expressive channels, and idiosyncratic emotion–signal mappings, as supported by prior work on personalized affect recognition in HRI [7,14]. Context conditioning operates at a shorter temporal scale, modulating emotion interpretation based on situational variables such as task type, interaction phase, and recent interaction history, which are known to strongly influence affective meaning in social interaction [8,10,15]. Online updating provides a continuous adjustment mechanism that allows the system to refine its internal representations as new interaction data become available, while maintaining stability through continual learning constraints [13]. To prioritize and combine these mechanisms during interaction, the framework employs an uncertainty-aware weighting strategy. Each adaptive component produces both an emotion prediction and an associated confidence estimate derived from temporal consistency and model uncertainty. Final emotion inference is obtained through weighted fusion, where predictions with higher uncertainty are automatically down-weighted in favor of more reliable components.

Aggregation in the proposed framework is performed through uncertainty-aware dynamic weighting of perceptual and contextual contributions. Rather than treating visual and contextual signals as equally reliable, the system modulates their influence based on confidence estimates.

Let

p_t^(v) denote the emotion distribution inferred from visual features,

p_t^(c) denote the context-conditioned distribution, and

σ_t^(v) and σ_t^(c) denote their corresponding uncertainty measures.

The final prediction is computed as:

$${\widehat{p}}_t= w_v\left(t\right)p_t^{\left(v\right)}+ w_c\left(t\right)p_t^{\left(c\right)}$$

(6)

where the weights are inversely proportional to uncertainty:

$$w_i\left(t\right)= {\displaystyle \frac{\frac{1}{{\sigma }_t^{\left(i\right)}}}{\sum j\frac{1}{{\sigma }_t^{\left(j\right)}}}}$$

(7)

This formulation ensures that signals with lower uncertainty exert greater influence on the final decision. When contextual signals are unreliable or inconsistent, σ_t^(c) increases, automatically reducing the weight assigned to contextual conditioning. Conversely, when visual features are ambiguous (e.g., low-intensity or asymmetric expressions), contextual information receives higher weighting. This dynamic aggregation mechanism allows the system to resolve conflicts between personalization, contextual cues, and immediate perceptual signals in a mathematically principled manner.

In practice, this design ensures that long-term user models dominate when contextual signals are noisy or ambiguous, while strong and reliable contextual cues can transiently override historical patterns when confidence is high. Potential conflicts between personalization and context—such as cases where the user-specific model suggests one emotional state while the current interaction context implies another—are resolved through this confidence-driven prioritization rather than fixed precedence rules. This avoids brittle behavior and allows the system to dynamically adapt to changing interaction conditions without overfitting to either short-term fluctuations or long-term priors. Similar uncertainty-guided fusion principles have been shown to improve robustness in adaptive affective systems and long-term human–robot interaction scenarios [7,10,13]. As a result, the proposed framework achieves a balanced integration of stability and flexibility, enabling reliable emotion recognition in interactions characterized by both individual variability and strong contextual dependence.

Beyond personalization, the framework supports contextual learning, recognizing that emotional meaning in HRI is shaped by situational factors such as task demands, interaction phase, and recent robot behavior. Empirical studies in affective computing emphasize that emotion cannot be reliably inferred from perceptual cues alone, but must be interpreted in relation to context and interaction dynamics [8,10]. Accordingly, contextual variables are incorporated as conditioning signals that modulate the adaptive learning process, allowing the model to adjust its interpretation of identical perceptual inputs across different situations. For example, reduced facial expressivity may indicate disengagement in a learning task, but may be a neutral or expected response in a cognitively demanding phase of interaction. By explicitly integrating context, the adaptive mechanisms reduce ambiguity and improve robustness in real-world scenarios. A critical requirement for adaptive emotion recognition in long-term human–robot interaction is the ability to update the model during ongoing interaction without catastrophic degradation of previously acquired knowledge. Continuous online learning in deep neural networks is known to be susceptible to catastrophic forgetting, whereby adaptation to newly observed data overwrites representations learned from earlier interactions. To address this issue, the proposed framework adopts a continual learning strategy inspired by Elastic Weight Consolidation (EWC) [13].

Within this framework, model parameters are partitioned according to their functional role in the perception pipeline. Lower-level convolutional layers responsible for generic visual feature extraction are kept frozen after initial training, ensuring stability of fundamental perceptual representations across users and interaction sessions. Online adaptation is restricted to a designated set of higher-level layers responsible for affective representation and decision boundaries, which are more directly associated with user-specific and context-dependent variation. This architectural separation allows the system to adapt to individual expressive patterns while preserving robust, population-level visual features. To further constrain online updates, EWC-based regularization is applied during adaptation. Parameters that are estimated to be important for previously learned affective representations are penalized during gradient updates using a Fisher-information-based importance measure. As a result, updates are biased toward parameters that are less critical for earlier knowledge, reducing interference between newly observed interaction data and previously acquired emotion recognition capabilities. This mechanism enables gradual personalization while mitigating catastrophic forgetting, in line with established continual learning principles [13]. From a long-term perspective, this design supports stable adaptation across extended interaction horizons spanning multiple sessions, days, or weeks. Rather than accumulating unbounded drift, the model converges toward a user-specific operating regime in which adaptation updates decrease in magnitude as confidence in the learned representation increases. At the same time, the regularization constraints ensure that exposure to short-term fluctuations or transient emotional states does not destabilize the overall perception model. Consequently, the proposed strategy achieves a balance between plasticity and stability, which is essential for sustained, reliable emotion recognition in long-term assistive and social robotic deployments.

In assistive and inclusive human–robot interaction (HRI), emotion recognition systems must operate safely from the very first moments of interaction, before sufficient user-specific data are available for personalization. This cold-start phase is particularly critical when interacting with users with special needs, as incorrect or overly confident affective feedback may lead to misunderstanding, distress, or reduced trust. Accordingly, the proposed framework explicitly incorporates precautionary mechanisms that govern system behavior prior to and during the early stages of adaptation. At interaction onset, the emotion recognition model operates in a population-level mode initialized from offline training data, without applying user-specific adaptation updates. During this phase, predictions are treated as provisional and are systematically accompanied by uncertainty estimates derived from temporal consistency and model confidence. Rather than committing to strong affective interpretations, the system maintains conservative decision boundaries that favor neutral or low-intensity interpretations when confidence is insufficient. This design ensures that early interactions remain socially appropriate even in the presence of expressive ambiguity or atypical affective signals.

Uncertainty estimation plays a central role in regulating system behavior during cold-start and subsequent adaptation phases. When prediction uncertainty exceeds a predefined threshold, emotion labels are either softened (e.g., mapped to broader affective categories) or replaced with a neutral/unknown state. This prevents the robot from responding as if a specific emotion were reliably detected when evidence is weak or inconsistent. Similar confidence-aware strategies have been recommended in affective HRI to avoid inappropriate or misleading robot behavior, particularly in sensitive interaction contexts [4,10]. During periods of high uncertainty or insufficient adaptation, the robot’s expressive and interactive responses are intentionally constrained. Instead of displaying strong emotion-contingent behaviors, the robot adopts neutral, low-arousal responses, such as reduced facial expressivity, minimal head movements, or generic verbal feedback. These fallback behaviors are designed to be non-intrusive and socially safe, minimizing the risk of misaligned emotional feedback. Importantly, behavioral constraints are relaxed gradually as the system accumulates interaction data and confidence in its user-specific model increases, enabling a smooth transition from cautious to more personalized interaction.

The proposed cold-start strategy reflects a broader safety-oriented design philosophy: emotion recognition is treated as an assistive perceptual capability rather than as a definitive assessment of the user’s internal state. By prioritizing uncertainty awareness, conservative inference, and constrained robot responses, the framework minimizes the potential harm associated with early misclassification. This approach is consistent with ethical guidelines and prior findings emphasizing that emotion-aware systems interacting with vulnerable populations should err on the side of underreaction rather than overreaction, particularly when affective signals are subtle, atypical, or context dependent [3,25].

3.4. Context-Aware Perception Module

Emotion expression in human–robot interaction (HRI) is inherently context dependent, emerging from the interplay between observable behavior, environmental conditions, and the temporal structure of the interaction. Consequently, the proposed framework incorporates a dedicated context-aware perception module that augments visual affect recognition with information about the interaction setting, history, and emotional dynamics. This design choice is grounded in extensive evidence from affective computing and social robotics showing that isolated perceptual cues are often insufficient for reliable emotion inference, particularly in naturalistic and assistive scenarios [4,8,10]. The first component of the module captures environmental and situational information, including task type (e.g., learning, therapeutic exercise, or free interaction), interaction phase, and basic environmental characteristics (e.g., session duration and repetition of tasks). Such variables provide essential priors for interpreting affective signals, as the same observable behavior may convey different emotional meanings depending on situational demands. Prior research has demonstrated that task context and cognitive load significantly influence both emotional expression and recognition performance, underscoring the necessity of incorporating contextual descriptors into affective models [6,8]. A second component models the interaction history, representing the temporal sequence of robot actions and user responses. Rather than treating emotion recognition as an independent frame-level decision, the framework explicitly encodes short- and medium-term interaction trajectories, allowing affective inferences to be smoothed and interpreted over time. This approach aligns with findings from long-term HRI studies, which show that engagement and emotional states evolve gradually and are shaped by prior interaction outcomes rather than instantaneous stimuli alone [14,15]. Interaction history also provides a basis for resolving ambiguity in perceptual cues by exploiting temporal consistency and change patterns. Finally, the module represents emotional dynamics, capturing trends, transitions, and variability in inferred affective states across time. Emotional dynamics are modeled as latent temporal processes that reflect how emotions intensify, attenuate, or shift in response to interaction events. End-to-end architecture is presented on Figure 1. This perspective is consistent with theoretical and empirical work emphasizing that affect in interaction is a dynamic phenomenon rather than a sequence of independent categorical states [8]. By explicitly modeling emotional trajectories, the system can distinguish transient noise from meaningful affective change and provide more stable inputs to adaptive learning mechanisms. Contextual information was encoded using a low-dimensional feature vector comprising task type (one-hot encoding), interaction phase (ordinal encoding), and recent interaction history summarized over a sliding temporal window.

Figure 1. End-to-end architecture of the proposed adaptive emotion recognition framework for social robots. The pipeline integrates region-based visual feature extraction with context-aware conditioning and multi-scale adaptive learning, producing both emotion predictions and uncertainty estimates that regulate robot behavior and drive continual online updates via a feedback loop.

Context Acquisition in Real-World Interaction

Reliable context acquisition in real-world human–robot interaction (HRI) is challenging due to noise, partial observability, and segmentation errors. To address these constraints, the proposed framework adopts a non-invasive and conservative approach to context extraction, relying exclusively on interaction-level and system-internal signals rather than physiological or intrusive sensing. Task-related context and interaction phase are derived from the robot’s internal task manager and dialogue state, which provide stable indicators of interaction structure. Engagement-related cues are estimated indirectly through behavioral proxies such as continuity of interaction, visual attention persistence, and response consistency, which can be obtained robustly in unconstrained environments without explicit affect labeling [10,15]. Contextual features are aggregated over sliding temporal windows to reduce sensitivity to transient noise and short-term misdetections.

Importantly, contextual information is treated as a probabilistic conditioning signal rather than a deterministic input. When contextual cues are inconsistent, missing, or unreliable, their influence on emotion inference is automatically down-weighted through uncertainty-aware fusion. In such cases, the system relies more strongly on visual cues and longer-term user adaptation, while overall prediction uncertainty increases. Elevated uncertainty directly triggers conservative robot behavior, as described in Section 3.3, preventing overconfident or inappropriate responses. By explicitly modeling uncertainty and enabling graceful degradation under context errors, the proposed framework ensures that context enhances emotion recognition when reliable, while failing safely when noisy or unavailable. This design supports robust, real-time operation in messy real-world HRI scenarios, where perfect context sensing cannot be assumed [4,10].

3.5. Datasets and Data Preprocessing

The experimental evaluation was conducted using a multi-dataset setup designed to reflect the conditions of real-world human–robot interaction and to explicitly include contextual variability and non-standard emotional expressions. All datasets were processed following a unified preprocessing pipeline to ensure consistency across training, adaptation, and evaluation stages.

3.5.1. Datasets

Three categories of datasets were used: AffectNet (N ≈ 425,000 images) as the primary ‘in-the-wild’ dataset, plus interaction-oriented datasets such as HRI-Emotion [25] (N ≈ 5000 frames) and the MEED (Multimodal Emotion Expression Dataset) [10] corpus for continuity in sequential user–robot tasks.

AffectNet was employed as the primary large-scale dataset for pretraining the visual backbone of the emotion recognition system. AffectNet contains over one million facial images collected from the Internet and annotated for both categorical emotions and continuous valence–arousal dimensions [20]. From this corpus, only samples with verified annotations and sufficient image quality were retained, resulting in a subset of 287,401 images spanning seven basic emotion categories (neutral, happiness, sadness, anger, fear, disgust, surprise). This dataset provided broad coverage of facial appearance variability, pose, illumination, and demographic diversity.

To ensure alignment with interactive and temporal aspects of HRI, additional datasets containing video-based emotional expressions with temporal continuity were incorporated. These datasets include sequences where emotional expressions evolve over time rather than appearing as isolated frames, enabling the modeling of emotional dynamics and interaction history. All sequences were segmented into interaction episodes of 3–10 s, preserving temporal coherence and contextual continuity as required by the proposed framework [10,15].

To evaluate robustness to non-standard emotional expressions, targeted subsets were constructed that emphasize reduced-intensity, asymmetric, or ambiguous facial expressions. These subsets were curated based on annotation confidence scores and expression intensity indicators, ensuring the inclusion of samples that are typically underrepresented or filtered out in standard affective computing pipelines. This design choice reflects empirical evidence that such expressions are common in users with special needs and pose significant challenges for conventional recognition systems [6,23].

3.5.2. Data Preprocessing Pipeline

All datasets were processed using a deterministic, reproducible preprocessing pipeline, applied uniformly across training, adaptation, and testing phases. First, region-based detection was performed on all images and video frames to localize affect-relevant regions. Facial regions were detected using a standard face detection model, and upper-body regions were extracted when available to capture gesture-related cues. Detected regions were cropped and resized to a fixed spatial resolution of 224 × 224 pixels, consistent with the convolutional backbone requirements [9,18]. Second, geometric normalization was applied to facial regions to reduce variability unrelated to affect, including scale differences and in-plane rotation. Importantly, no aggressive landmark-based warping was used, as such transformations were found to suppress idiosyncratic expressive patterns that are critical for personalization and inclusive modeling [5,6]. Pixel intensities were normalized channel-wise using statistics computed from the training split only. Third, for video-based data, temporal structuring was preserved. Frames were grouped into fixed-length sequences, and no temporal shuffling was performed. This ensured that temporal dependencies and emotional trajectories remained intact for downstream modeling of interaction history and emotional dynamics [15,19]. Fourth, label handling and quality control were explicitly designed to avoid bias toward exaggerated expressions. Samples with ambiguous or low-confidence annotations were retained and labeled according to their original annotations rather than being discarded. This contrasts with common practices in facial expression recognition, where such samples are often excluded, leading to overly optimistic performance estimates and reduced generalization to real-world users [5,20].

3.5.3. Data Splits and Usage

The data were partitioned into training (70%), validation (15%), and test (15%) sets at the subject or sequence level where applicable, ensuring that no identity leakage occurred between splits. AffectNet was used primarily for initial model training, while interaction-oriented and atypical-expression subsets were emphasized during adaptive learning and evaluation phases. This separation allowed a clear assessment of how well the adaptive mechanisms generalize beyond the distributions seen during initial training. Overall, this dataset design and preprocessing strategy ensured that the evaluation environment faithfully reflected the challenges of inclusive human–robot interaction. By combining large-scale in-the-wild data, temporally structured interaction data, and explicitly preserved atypical expressions, the experimental setup provided a rigorous and realistic basis for assessing adaptability, robustness to variability, and generalization across emotion categories.

3.5.4. Experimental Conditions and Evaluation Protocol

In contrast to static image-based emotion datasets, the interaction-oriented datasets used in this study consist of temporally structured video sequences that explicitly preserve the dynamics of human–robot interaction. Each sequence represents a continuous interaction episode rather than an isolated facial expression, with durations ranging from approximately 3 to 10 s. These datasets include variations in interaction phase, task demands, and expressive dynamics, enabling the modeling of emotional trajectories and short-term interaction history. Such temporal and contextual structure is critical for affect recognition in HRI, where emotional meaning emerges over time and is shaped by interaction context rather than by single-frame cues alone [10,15]. All interaction-oriented sequences were processed without temporal shuffling, ensuring that sequential dependencies required for adaptive learning and context-aware perception were preserved throughout training and evaluation.

To evaluate robustness under conditions representative of real-world HRI, controlled perturbation scenarios were explicitly defined along three dimensions: pose variation, occlusion, and expression intensity. Pose perturbations were introduced by selecting samples with non-frontal head orientations, including moderate and large yaw and pitch deviations, reflecting natural user movement during interaction. Occlusion conditions included partial visibility of facial regions due to self-occlusion (e.g., hand-to-face gestures) or environmental factors, consistent with observations in unconstrained social robotic settings [5,10]. Expression intensity perturbations targeted samples annotated with low-confidence or low-intensity affective signals, resulting in subtle, asymmetric, or ambiguous facial expressions. These conditions are known to disproportionately affect recognition performance, particularly in users with special needs, and therefore provide a meaningful stress test for adaptive emotion recognition systems [6,23]. Performance under perturbation was assessed relative to standard test conditions, allowing quantification of degradation and robustness across models.

Scenario transfer refers to the evaluation of model performance when applied to interaction contexts that differ from those observed during base training. In this study, scenarios were defined by combinations of task structure (e.g., structured task-oriented interaction vs. free interaction), interaction phase, and expressive dynamics. Scenario transfer evaluation therefore measures the ability of the model to generalize across qualitatively different interaction settings while maintaining stable emotion-category discrimination. Importantly, training and evaluation scenarios were separated at the sequence level to avoid identity or temporal leakage. This protocol reflects realistic deployment conditions in social robotics, where robots are expected to operate across diverse tasks and interaction styles without retraining [10,15]. Robust performance under scenario transfer indicates that the model captures interaction-invariant affective patterns rather than overfitting to specific dataset configurations.

3.5.5. Users with Special Needs

Given the focus of this study on inclusive human–robot interaction, it is important to clarify how users with special needs are represented in the experimental data and how this representation relates to the evaluation of adaptive emotion recognition mechanisms. The experimental validation of the proposed framework relied primarily on publicly available affective datasets and interaction-oriented video data, which do not provide explicit demographic annotations regarding disability status or clinical diagnoses. As is common in affective computing research, these datasets predominantly reflect heterogeneous populations without systematic labeling of special-needs profiles [5,23]. Consequently, the study does not claim comprehensive demographic representation of users with special needs across all datasets. To partially address this limitation and to inform early-stage validation, an initial round of pilot interaction tests was conducted with a small group of adult student participants who self-reported having formal educational or clinical diagnoses related to cognitive, developmental, or sensory differences. These exploratory sessions were conducted in an academic setting as part of regular coursework and research activities, without collecting sensitive health data or personal identifiers, and were used solely to observe interaction dynamics and expressive variability rather than to derive quantitative performance claims. The insights gained from these pilot interactions informed the design of adaptive mechanisms and experimental conditions, but the resulting data were not treated as a statistically representative sample of special-needs populations.

To ensure conceptual clarity, this study explicitly distinguishes between participants with special needs and atypical or low-intensity emotional expressions observed in the data. The latter refers to expressive patterns characterized by reduced intensity, asymmetry, ambiguity, or non-canonical timing of facial expressions. Such patterns are frequently reported in users with special needs, particularly in autism spectrum and cognitive impairment contexts [6,23], but they are not exclusive to these populations. Importantly, atypical expressions can also occur in neurotypical users due to situational factors such as cognitive load, fatigue, social inhibition, or task constraints. Accordingly, the experimental evaluation focuses on expression-level characteristics rather than attempting to infer or label participant-level clinical categories. Subsets emphasizing atypical and low-intensity expressions were constructed based on annotation confidence and expression intensity indicators, without associating these samples with specific diagnostic labels. This approach avoids conflating expressive variability with disability status and aligns with ethical recommendations cautioning against normative assumptions in emotion recognition research [25]. The adopted strategy reflects a deliberate methodological choice: instead of treating “users with special needs” as a homogeneous or explicitly labeled group, the framework is evaluated based on its ability to handle expressive variability and deviation from canonical emotion patterns. This choice is consistent with prior work arguing that inclusive affective systems should prioritize adaptability to individual expressive baselines rather than rely on demographic or diagnostic categorization [7,23].

3.6. Experimental Setup

All experiments were conducted following a controlled and reproducible training protocol designed to isolate the effects of adaptivity and context awareness on emotion recognition performance. The experimental pipeline comprised three phases: base model training, adaptive fine-tuning during interaction, and evaluation under controlled variability conditions. The region-based convolutional backbone was initialized using weights pretrained on large-scale facial expression data and subsequently fine-tuned on the training split described in Section 3.5. Training was performed using the Adam optimizer (Adaptive Moment Estimation optimizer) with an initial learning rate of 1 × 10⁻⁴, exponential decay, and a batch size of 32 for image-based inputs and 8 for sequence-based inputs. Cross-entropy loss was used for categorical emotion recognition, while auxiliary regularization terms were applied to stabilize adaptation updates and prevent overfitting to short-term interaction data, in line with established practices in adaptive and continual learning [13]. Early stopping based on validation loss was employed to ensure convergence without excessive specialization to the training distribution. Adaptive updates were performed at fixed temporal intervals corresponding to every 50 consecutive video frames (approximately 2 s of interaction), using accumulated interaction data rather than single-frame observations.

After base training, the model entered an online adaptive phase, during which user-specific and context-conditioned updates were enabled. Adaptation parameters were restricted to designated layers responsible for high-level affect representation, while lower-level convolutional filters remained frozen. This configuration ensured that adaptation captured individual and contextual variation without degrading the robustness of learned visual features [7,12]. Model updates were triggered at fixed temporal intervals during interaction sequences, using accumulated interaction data rather than single frames, thereby aligning adaptation with emotional dynamics rather than instantaneous fluctuations [15].

To assess the contribution of the proposed adaptive framework, several baseline models were implemented and evaluated under identical data splits and preprocessing conditions:

Static CNN baseline—a standard convolutional network trained once and deployed without adaptation, representing conventional emotion recognition pipelines [5,10].

Region-based static model—an R-CNN-based architecture identical to the proposed perceptual backbone but without adaptive or context-aware components, isolating the effect of region-based processing [9].

Personalization without context—a model incorporating user-level adaptation but excluding contextual conditioning, enabling analysis of the contribution of interaction context beyond personalization alone [7,14]. All baselines were trained using the same optimization settings and evaluated on the same test sets to ensure fair comparison. The experimental design was intentionally structured to balance ecological validity with controlled evaluation, ensuring that reported performance reflects both realistic interaction conditions and reproducible experimental settings.

Real-Time Feasibility and Computational Cost

For social robots operating in real-world human–robot interaction (HRI) scenarios, real-time performance and computational efficiency are critical design constraints. Emotion recognition modules must operate within tight latency budgets to ensure timely and socially appropriate robot responses, while adaptive mechanisms must not compromise interaction responsiveness. This section analyzes the computational cost and real-time feasibility of the proposed adaptive framework. All real-time performance measurements were conducted on a standard consumer-grade computing platform representative of typical robotic deployments. The system was executed on a workstation equipped with an Intel Core i7-class CPU, 32 GB of RAM (Intel Corporation, Santa Clara, CA, USA), and an NVIDIA RTX-series GPU (NVIDIA Corporation, Santa Clara, CA, USA). The affective perception pipeline was implemented in Python 3.11.4 and executed using GPU acceleration for convolutional inference, while robot control and communication were handled on the CPU. Input video streams were processed at a nominal frame rate of 25–30 FPS (frames per second), consistent with common RGB camera configurations used in social robotic platforms.

End-to-end inference latency was measured as the elapsed time between acquisition of an RGB video frame and the availability of an emotion prediction and uncertainty estimate. Across interaction-oriented test sequences, the average inference latency ranged between approximately 200 and 350 ms per frame, depending on scene complexity and the number of active regions of interest. This corresponds to an effective throughput of 8–12 FPS for the full perception–adaptation pipeline. Importantly, inference latency remained stable across interaction phases and did not exhibit cumulative drift over extended interaction sequences. These results indicate that the proposed framework satisfies the temporal constraints required for real-time social interaction, where response delays below approximately 500 ms are generally considered acceptable for maintaining perceived responsiveness in HRI [4,10].

Online adaptive updates introduce additional computational overhead beyond standard inference. To balance adaptivity and responsiveness, model updates were not performed at every frame, but were instead scheduled at fixed temporal intervals using accumulated interaction data. In the implemented configuration, adaptive updates were triggered approximately every 50 video frames (corresponding to roughly 2 s of interaction). Each update involved backpropagation through a restricted subset of higher-level network layers, while lower-level convolutional layers remained frozen. The average duration of a single adaptation update was approximately 120–180 ms, depending on the size of the accumulated data window and the active regularization constraints.

Crucially, adaptation updates were executed asynchronously with respect to robot actuation, ensuring that inference and behavior generation were not blocked by learning operations. As a result, the perceived responsiveness of the robot remained unaffected during online learning phases. This design aligns with established recommendations for real-time adaptive systems, which emphasize decoupling perception, learning, and control loops to avoid interaction disruptions [13,15].

The proposed framework explicitly addresses the trade-off between adaptive learning and real-time interaction requirements. Frequent updates increase sensitivity to short-term expressive changes but risk introducing latency spikes, whereas infrequent updates favor stability at the cost of slower personalization. By employing interval-based updates with uncertainty-aware regularization, the framework achieves a balanced operating regime in which adaptivity improves emotion recognition robustness without degrading interaction fluidity. Empirically, no statistically significant increase in inference latency or frame drop rate was observed when adaptive learning was enabled, compared to a non-adaptive region-based baseline. Additional implementation details, hyperparameter configurations, and extended training specifications are provided in Appendix A.

3.7. Evaluation Metrics

Performance was evaluated using a multi-dimensional metric framework designed to capture not only classification accuracy but also robustness and generalization—properties that are critical for inclusive human–robot interaction. Primary performance was measured using overall classification accuracy and macro-averaged F1-score across emotion categories. Macro-averaging was selected to account for class imbalance and to ensure that performance on less frequent or lower-intensity expressions contributed equally to the evaluation [5,20]. Confusion matrices were additionally analyzed to identify systematic misclassification patterns, particularly between emotions known to be difficult to distinguish in real-world settings (e.g., fear vs. surprise). Robustness was assessed by evaluating performance under controlled perturbations reflecting real interaction conditions, including variations in pose, partial occlusion, and reduced expression intensity. Test subsets emphasizing atypical and low-intensity expressions were used to quantify performance degradation relative to standard test conditions. Robustness was reported as the relative change in F1-score between standard and perturbed conditions, following established evaluation practices in affective computing [5,10]. To evaluate generalization, models were tested on emotion categories and expression styles underrepresented during base training, including expressions exhibiting atypical timing or asymmetry. Category-wise recall and balanced accuracy were computed to assess whether adaptive mechanisms improved sensitivity to diverse affective patterns rather than optimizing performance for dominant classes only. This analysis directly addresses concerns raised in prior work regarding bias toward neurotypical or exaggerated expressions in emotion recognition systems [6,23,25].

4. Results

4.1. Quantitative Evaluation Results

The quantitative evaluation demonstrates clear and consistent advantages of the proposed adaptive, context-aware framework over static emotion recognition models across all experimental conditions. Table 2 summarizes the main performance metrics obtained on the held-out test set, averaged across five independent runs to account for stochastic variability in training and adaptation. Differences between the proposed adaptive framework and all baseline models were statistically significant, as the observed performance gains consistently exceeded the corresponding standard deviation margins across five independent runs (p < 0.01, paired t-test), with effect sizes (Cohen’s d) indicating medium to large practical significance. The adaptive model achieved an overall classification accuracy of 78.6% and a macro-averaged F1-score of 0.76, outperforming all baseline approaches. In comparison, the static CNN baseline reached an accuracy of 68.2% (F1 = 0.66), while the region-based but non-adaptive model achieved 71.4% accuracy (F1 = 0.69). The personalization-only baseline (user adaptation without contextual conditioning) improved performance to 74.1% accuracy (F1 = 0.72) but still remained below the full adaptive framework. These results indicate that both region-based perception and adaptive learning contribute positively to performance, with the largest gains observed when adaptation is combined with contextual conditioning. The improvement over the static CNN baseline exceeds 10 percentage points in accuracy, which is substantial given the difficulty of in-the-wild emotion recognition tasks reported in prior studies [5,10]. The proposed adaptive framework achieves the highest accuracy and macro-F1 score across all evaluated models, demonstrating that the combination of region-based perception, user adaptation, and contextual conditioning yields consistent improvements over static and partially adaptive baselines. Emotion-wise recall analysis revealed that the adaptive framework consistently improved recognition of negatively valenced and easily confused emotions, such as fear and sadness, which are known to be challenging in unconstrained settings [5,20]. For these categories, recall improvements ranged from +9% to +14% relative to the static baseline. Importantly, no degradation was observed for well-recognized categories such as happiness, indicating that adaptation did not trade off performance on dominant expressions in favor of rarer ones.

Table 2. Quantitative comparison of emotion recognition performance across models (Accuracy and Macro-F1).

Model	Accuracy (%)	Macro-F1	Std. Dev. (Accuracy)
Static CNN baseline	68.2	0.66	±0.9
Region-based CNN (static)	71.4	0.69	±0.8
Personalized model (no context)	74.1	0.72	±0.7
Proposed adaptive framework	78.6	0.76	±0.6

Overall, these results confirm that adaptivity and context awareness yield systematic and statistically meaningful improvements over static emotion recognition pipelines, validating the architectural choices described in Section 3.

4.2. Robustness to Non-Standard Emotional Expressions

Robustness was evaluated using test subsets specifically constructed to emphasize non-standard emotional expressions, including reduced intensity, asymmetric facial movements, and temporally irregular expression patterns, results are shown in Table 3. These subsets are representative of expressive behaviors frequently observed in users with special needs and are known to cause significant performance degradation in conventional emotion recognition systems [6,23]. On the atypical-expression subset, the static CNN baseline exhibited a pronounced drop in performance, with accuracy decreasing from 68.2% to 54.7% (−13.5%). In contrast, the proposed adaptive framework showed a substantially smaller degradation, decreasing from 78.6% to 71.2% (−7.4%). This difference demonstrates that the adaptive mechanisms effectively compensate for expressive variability that violates the assumptions of population-level emotion models. The region-based static model showed intermediate robustness (−10.1% drop), indicating that localized feature extraction alone improves resilience to atypical cues, but is insufficient without adaptation. The personalization-only baseline reduced performance loss to −8.9%, confirming that user adaptation is beneficial, yet still less effective than the combined user-and-context adaptive strategy. The adaptive framework substantially improves recall for emotions that are typically difficult to recognize under non-standard expressive conditions, particularly fear, sadness, and disgust. These gains confirm that adaptivity mitigates systematic failures observed in static models when expressive intensity is reduced or atypical.

Table 3. Recall (%) per emotion category evaluated on the atypical-expression test subset ¹.

Emotion	Static CNN	Region-Based (Static)	Personalized (No Context)	Proposed Adaptive
Neutral	58.3	61.9	65.2	70.8
Happiness	76.4	78.1	80.5	82.7
Sadness	49.7	54.6	58.9	66.3
Anger	57.2	60.8	63.1	68.5
Fear	44.9	50.3	54.6	61.2
Disgust	52.6	56.7	59.4	64.8
Surprise	63.1	66.9	69.8	73.6

¹ Definition of atypical expressions. In this study, atypical expressions refer to emotional displays that deviate from canonical, high-intensity facial expression patterns commonly represented in standard affective datasets. Operationally, samples were classified as atypical if they met at least one of the following criteria: low expression intensity, indicated by low annotation confidence or reduced valence/arousal magnitude; spatial asymmetry, reflected in uneven activation of facial regions (e.g., unilateral mouth or eyebrow movement); high model uncertainty, quantified by elevated prediction entropy or low temporal consistency across consecutive frames.

When evaluation was restricted to sequences characterized by low-intensity or ambiguous expressions—conditions often reported in studies involving users with developmental or cognitive differences [6,23]—the adaptive framework achieved a relative F1-score improvement of 15–18% over the static baseline. Notably, improvements were most pronounced for neutral–sad and neutral–fear confusions, which are commonly reported failure modes in emotion recognition systems trained on standard datasets [5].

These results provide strong empirical evidence that the proposed framework is not merely more accurate under ideal conditions but fundamentally more robust to expressive variability. This robustness is critical for inclusive social robotics, where non-standard emotional expressions are not edge cases but a defining characteristic of the target user population [3,25].

Figure 2 illustrates that while all models experience performance degradation as expressive atypicality increases, the proposed adaptive framework exhibits significantly greater robustness. The accuracy drop between standard and highly atypical expressions is reduced by more than 45% relative to the static baseline, confirming the effectiveness of adaptive and context-aware mechanisms in handling non-standard emotional expressions.

Figure 2. Accuracy degradation under increasing levels of expressive atypicality.

4.3. Generalization Across Emotion Categories

Generalization across emotion categories is a critical requirement for emotion recognition systems deployed in real-world human–robot interaction, where emotional expressions vary not only in intensity and form but also in their contextual manifestation across scenarios and users. To evaluate the generalization capability of the proposed adaptive framework, we conducted a series of analyses focusing on cross-category stability, scenario transfer, and consistency of performance under distribution shift. The models were evaluated across multiple interaction scenarios differing in task structure and affective demands, including structured task-oriented interactions and less constrained, spontaneous interaction segments. Results indicate that the proposed adaptive framework maintains stable performance across these scenarios, with accuracy variations limited to a narrow range (±3.1%), whereas static baselines exhibited substantially larger fluctuations (up to ±8.7%). This stability suggests that the integration of contextual information and adaptive updates enables the model to preserve emotion-category discrimination even when surface-level expressive cues change across scenarios [10,15]. Table 4 demonstrates low variance across scenarios, supporting claims of category-level stability and robust generalization.

Table 4. Recall (%) per emotion category evaluated across different interaction scenarios.

Emotion	Structured Task	Semi-Structured	Free Interaction	Std. Dev.
Neutral	71.2	69.8	70.5	0.7
Happiness	83.1	82.4	81.9	0.6
Sadness	65.9	64.7	66.2	0.8
Anger	68.4	67.1	69.0	1.0
Fear	60.8	59.6	61.2	0.8
Disgust	64.1	63.3	65.0	0.9
Surprise	74.5	73.2	75.1	1.0

To further assess generalization, we analyzed category-wise performance consistency across scenarios. The adaptive framework demonstrated balanced recall across all emotion categories, with no single category disproportionately dominating performance gains. In contrast, static and partially adaptive baselines showed strong dependence on highly expressive categories (e.g., happiness), while performance for negatively valenced or ambiguous emotions (e.g., fear, sadness) degraded markedly when evaluated outside the training distribution. This finding aligns with prior observations that static models tend to overfit to dominant expression patterns present in standard datasets [5,20]. An additional evaluation was performed using test subsets characterized by expression styles underrepresented during base training, including atypical timing and reduced expressivity. Under these conditions, the adaptive framework retained an average of 92% of its in-distribution macro-F1 score, compared to 81% for the region-based static model and 76% for the plain CNN baseline. These results confirm that adaptive learning and context-aware conditioning mitigate the effects of distribution shift, enabling reliable emotion recognition beyond the original training domain [7,13].

4.4. Ablation Study

To quantify the contribution of individual components of the proposed framework, an ablation study was conducted in which key adaptive and contextual modules were selectively disabled while keeping all remaining elements unchanged. This experimental design allows for a precise assessment of how each component influences overall emotion recognition performance, robustness, and generalization. The ablation analysis focuses on three core components: user-level adaptation, context-aware conditioning, and online model updating during interaction. Each ablated variant was trained and evaluated using the same datasets, preprocessing pipeline, and evaluation metrics, ensuring direct comparability.

Removing user-specific adaptation resulted in a consistent performance drop across all evaluation conditions. Overall accuracy decreased by 4.5 percentage points, with the most pronounced degradation observed for emotions characterized by subtle or low-intensity expressions (e.g., sadness and fear), as shown in Table 5. This confirms that personalization plays a critical role in capturing stable individual differences in expressive behavior, as previously suggested by personalized affect recognition studies [7,12]. Disabling contextual inputs while retaining user adaptation led to a reduction in robustness rather than a uniform drop in accuracy. Although in-distribution performance decreased modestly (−2.8%), performance under scenario changes and atypical-expression conditions degraded substantially (−6.1%). This indicates that context primarily supports interpretive stability across interaction scenarios, enabling the model to disambiguate similar perceptual cues that carry different emotional meanings depending on task and interaction state [8,10]. When online adaptation during interaction was disabled, the model’s performance remained comparable to the full framework in early interaction segments but deteriorated over longer sequences. Accuracy decreased progressively as interaction length increased, resulting in a 7.3% drop in long-session evaluations. This finding demonstrates that continual updating is essential for maintaining performance in long-term human–robot interaction, where expressive patterns and engagement evolve over time [13,15]. Importantly, the ablation results reveal that adaptive components do not contribute additively but synergistically (as shown in Table 6). While each component individually improves performance, the full framework consistently outperforms all ablated variants across accuracy, robustness, and generalization metrics. This synergy underscores the necessity of integrating personalization, context awareness, and continual learning into a unified perception architecture rather than treating them as independent enhancements.

Table 5. Overall accuracy and macro-F1 score for the full model and ablated variants.

Model Variant	Accuracy (%)	Macro-F1
Full adaptive framework	78.6	0.76
User adaptation	74.1	0.72
Context-aware conditioning	75.8	0.73
Online model updating	71.3	0.69
User adaptation & context	69.9	0.67

Table 6. Accuracy (%) on atypical-expression test data for ablated models.

Model Variant	Standard	Atypical	Drop (%)
Full adaptive framework	78.6	71.2	−7.4
User adaptation	74.1	64.9	−9.2
Context-aware conditioning	75.8	63.7	−12.1
Online model updating	71.3	60.4	−10.9

The ablation study provides strong empirical evidence that adaptivity is not a monolithic feature but a composite of complementary mechanisms. User adaptation captures stable individual differences, context-aware conditioning ensures interpretive consistency across scenarios, and online updating sustains performance in long-term interactions.

5. Experimental Validation

To strengthen the validation of the proposed adaptive emotion recognition framework beyond dataset-driven evaluation, we designed and conducted an embodied experiment involving a physical robotic platform—the OHbot v.2.2 programmable robot head (Ohbot Ltd., Bristol, England, UK). OHbot is an expressive, servo-actuated robot head designed to support educational and research uses, featuring multiple servo motors, facial actuation primitives, and Python programmability for real-time sensor and actuator control.

5.1. Robotic Platform and Experimental Configuration

The objective of this experiment was to integrate the adaptive affective perception pipeline directly with a robotic embodiment and evaluate performance in live human–robot interaction (HRI). This section describes hardware configuration, software integration, interaction protocol, and evaluation metrics.

Robot Platform: The OHbot robot head equipped with eight servo motors enabling smooth control of head orientation, eye movement, jaw motion, and facial expressions.

Sensing: A USB RGB camera was mounted on the robot frame to capture participants’ facial expressions during interaction. The camera stream was processed in real time for emotion inference.

Computing Environment: A host computer running Python 3.9 and the OHbot Python library v2.2, connected via USB for controlling servos and querying sensor input.

The adaptive framework’s affective perception model was deployed as a Python module that received camera frames, produced emotion predictions, and communicated these predictions to the robot control script. The OHbot Python API was used to actuate facial expressions and head movements based on model outputs.

To illustrate how the proposed system operates in practice, consider the following interaction scenario observed during experimental sessions. A user engages in a structured task phase involving cognitive load. During task execution, the user exhibits a subtle, low-intensity smile. The visual module extracts region-based features and produces a moderately confident prediction of “happy” with elevated entropy due to weak activation intensity. Simultaneously, the context module identifies the current interaction phase as task-focused, historically associated with concentration rather than overt positive affect. The adaptive fusion layer integrates these signals. Because the visual entropy is relatively high and contextual consistency is strong, the aggregation mechanism increases the weight assigned to contextual conditioning. The resulting distribution shifts from “happy” toward a lower-intensity positive or neutral-affect interpretation. The uncertainty estimate remains above the confidence threshold, triggering conservative behavioral output. Instead of producing an exaggerated happy response, the robot performs a mild acknowledgment gesture (slight head movement, neutral tone) without explicit emotional labeling. As the interaction continues and similar patterns recur, the adaptive state u gradually adjusts to the user’s expressive baseline. Over time, the system reduces uncertainty for subtle smiles and becomes more confident in interpreting them appropriately within the user-specific context. This example demonstrates how perception, context modeling, adaptive learning, uncertainty estimation, and behavioral regulation jointly operate within a closed-loop interaction.

5.2. Interaction Protocol and Evaluation Metrics

A structured interaction protocol was employed to elicit a variety of affective expressions from participants and is summarized in Figure 3.

Figure 3. Three-phase experimental protocol. The model is first trained offline (base training), then adapted online during interaction using user- and context-conditioned updates with continual-learning constraints, and finally evaluated in two complementary settings: offline dataset-based benchmarks and embodied HRI validation with the OHbot robot head.

Baseline Calibration: Participants first sat facing the robot without instructions to establish a neutral baseline.

Emotion Elicitation Tasks: Standardized prompts were shown on a monitor, requesting participants to display basic emotions (e.g., happiness, sadness, surprise, neutral) for 10-s intervals.

Contextual Engagement: Participants engaged in a short dialog task with the robot where they responded to questions requiring affective reactions (e.g., “Tell me something that makes you happy”).

Adaptive Feedback: Robot reactions were driven online by the affective perception model outputs, dynamically influencing robot facial expressions and head movements.

During each phase, both raw camera frames and internal predictions with confidence values were logged for analysis.

Metrics evaluated the embodied system along three primary dimensions:

Recognition Accuracy in Ecological Setting: Comparison of real-time predictions during HRI with benchmark annotation of recorded videos by external coders.

Response Latency: Measurement of the time elapsed between camera frame acquisition and robot action execution, providing insight into real-time feasibility.

User-Perceived Naturalness: Participants completed a brief post-interaction questionnaire assessing perceived responsiveness, naturalness, and engagement (Likert scale 1–5).

To minimize interference between learning and control, online model updates are scheduled on a separate thread/process when possible. When asynchronous execution is not available, updates are executed only at pre-defined interaction boundaries (e.g., between prompts) to avoid visible response stalls. For each video frame (or frame batch), the logger records timestamp, predicted emotion, per-class probabilities, uncertainty/confidence value, selected actuation command (servo targets), and the current adaptation state (e.g., update counter and last update time). In addition, raw frames are optionally stored (or encoded to a compressed stream) to enable offline re-annotation by external coders and to compute agreement metrics. The embodied experiment was executed as a single Python application that orchestrates video acquisition, region-based preprocessing, model inference with uncertainty estimation, optional online updates at fixed intervals, and OHbot actuation via the vendor Python API. To ensure traceability, the system logs both perceptual and control-side signals.

In practice, the policy implements three layers of control: a nominal expressive template for each emotion class (head pose, eye direction, jaw movement), an intensity scaling factor proportional to confidence, and a safety gate that clamps or replaces emotion-contingent actuation with a neutral response when uncertainty exceeds a threshold. Robot behavior was generated using a lightweight emotion-to-actuation policy that maps the predicted emotion label and its uncertainty to OHbot’s facial actuation primitives. The mapping was intentionally conservative during cold-start and under high uncertainty, following the interaction safety principles described in Section 3.3.

5.3. Experimental Results

The embodied experiment demonstrated that the adaptive model maintained competitive accuracy in real-time HRI compared to offline dataset evaluations, with mean recognition accuracy exceeding 78% across basic emotions. Response latency averaged approximately 200–350 ms per frame, indicating the feasibility of real-time interaction loops on standard consumer hardware. Questionnaire results revealed positive assessments of robot responsiveness (mean 4.2/5) and naturalness of interaction (mean 3.9/5), underscoring the practical value of embodiment in perception tasks. Importantly, the integration of adaptive emotion recognition with OHbot’s expressive capabilities enabled a richer interactive experience than dataset-only evaluation. The robot’s ability to modulate its own expressions in concert with user affect provided participants with immediate, perceivable social responses.

This real-world validation bridges the gap between purely dataset-driven performance metrics and embodied interactive behavior, showing that the proposed adaptive system can operate effectively in live HRI contexts. Employing an affordable, programmable platform such as OHbot allows researchers to test affective perception and adaptive responses in tangible robotic systems, aligning with recommendations for embodied AI research.

6. Discussion

6.1. Interpretation of Results: The Role of Adaptivity in Emotion Perception

The presented results provide converging evidence that adaptivity is a decisive factor for reliable emotion perception in social robots operating in human-centered and inclusive environments. The proposed adaptive framework consistently outperformed static and partially adaptive baselines. A central finding is that performance gains are not limited to in-distribution conditions, but persist under expressive variability and distribution shift. The adaptive framework reduced accuracy degradation under atypical expressions by nearly half compared to the static CNN baseline. This result directly addresses a well-documented limitation of conventional emotion recognition systems, which tend to assume prototypical, high-intensity expressions and therefore fail when confronted with subtle or atypical affective signals [5,6,23]. The observed robustness indicates that adaptive mechanisms enable the model to recalibrate its internal representations in response to user-specific expressive patterns rather than forcing observations into fixed emotion templates. Equally important is the stability across emotion categories and interaction scenarios. The low variance in category-wise recall across structured, semi-structured, and free interaction contexts suggests that the framework captures emotion-related regularities that generalize beyond specific tasks or settings. This finding aligns with interaction-oriented theories of affect, which conceptualize emotion as a dynamic process embedded in context and history rather than as an isolated perceptual event [8,15]. By integrating contextual cues and emotional dynamics into the perception pipeline, the system maintains consistent discrimination between emotion categories even when surface-level expressions vary. The ablation study further clarifies the mechanistic sources of these improvements. User adaptation emerged as essential for capturing stable individual differences, while context-aware conditioning primarily supported robustness across scenarios. Online model updating proved critical for long-term interaction stability, preventing the gradual performance decay observed when adaptation was disabled. Together, these results demonstrate that adaptivity functions as a multi-layered capability, where personalization, contextual interpretation, and continual learning jointly support reliable emotion perception.

From a broader perspective, these findings reinforce the argument that emotion recognition in social robotics should be framed as an adaptive perceptual process, rather than a one-shot classification problem. This shift is particularly consequential for assistive, educational, and therapeutic applications, where users’ emotional expressions are shaped by cognitive, sensory, or developmental differences and may evolve over time [2,3,22].

6.2. Comparison with Existing Approaches: Advantages over Static Models

When compared with existing approaches to emotion recognition in human–robot interaction, the proposed framework demonstrates several systematic advantages over static models commonly reported in the literature. Traditional deep learning approaches, including state-of-the-art CNN-based facial expression recognizers, have achieved impressive performance on benchmark datasets but exhibit limited transferability to interactive and inclusive settings [5,10]. As confirmed by the baseline results in Section 4.1, such models experience substantial performance degradation when evaluated under non-ideal conditions typical of real-world HRI. Region-based models partially address these limitations by focusing on emotionally salient areas, improving robustness to background noise and pose variation [9,10]. This effect is evident in the improved performance of the region-based static baseline relative to the plain CNN. However, the results clearly show that region-based perception alone is insufficient: without adaptive learning, these models remain constrained by population-level assumptions and fail to accommodate individual variability.

Personalized approaches reported in prior work, particularly in autism-focused HRI, have highlighted the benefits of adapting affect recognition models to individual users [7]. The personalization-only baseline in this study confirms these findings, yielding consistent improvements over static models. Nevertheless, the comparative analysis demonstrates that personalization without contextual awareness leads to reduced robustness when interaction scenarios change, as reflected in the larger performance drops under atypical-expression and distribution-shift conditions (Section 4.2 and Section 4.3). The proposed framework advances beyond existing approaches by integrating personalization, context awareness, and continual adaptation within a single architecture. This integration differentiates it from multimodal or context-agnostic affect recognition systems surveyed in the literature [10,11,16], which often treat context as auxiliary information or restrict adaptation to offline retraining phases. In contrast, the present results show that online, interaction-driven adaptation is crucial for sustaining performance over time and across scenarios, a requirement emphasized in long-term HRI studies but rarely operationalized in perception modules [14,15].

Importantly, the comparative advantages observed here are not limited to aggregate metrics. Improvements are most pronounced for emotion categories and expressive styles that are traditionally underrepresented or misclassified, such as low-intensity fear or sadness (Table 3). This directly addresses concerns raised in recent critical discussions about bias and exclusion in emotion recognition technologies, particularly for neurodivergent users [25]. By reducing reliance on fixed normative models of expression, the adaptive framework supports a more inclusive and ethically aligned approach to affect-aware robotics. In summary, relative to existing static and partially adaptive approaches, the proposed framework offers a qualitative shift in how emotion recognition is conceptualized and implemented for social robots. Rather than optimizing performance on idealized datasets, it prioritizes flexibility, stability, and user-centered adaptation—properties that are essential for deploying emotion-aware robots in real-world assistive, educational, and therapeutic environments.

The ethical implications of deploying emotion recognition in social robots are closely tied to uncertainty, adaptation, and interaction safety. In the proposed framework, ethical considerations are not treated as an external constraint but are embedded directly into system design through uncertainty-aware inference, conservative cold-start behavior, and constrained robot responses. By explicitly coupling prediction confidence with behavioral modulation, the system prioritizes underreaction over overreaction when affective evidence is weak, ambiguous, or contextually unreliable. This design choice is particularly important in assistive and inclusive HRI scenarios, where misinterpretation of subtle or atypical expressions may negatively affect user trust or well-being. As interaction history accumulates and uncertainty decreases, personalization is introduced gradually, enabling a controlled transition from cautious to more expressive interaction. This integration of technical uncertainty handling with ethical risk mitigation highlights the role of adaptive perception as a safety mechanism rather than a purely performance-driven component.

6.3. Implications for Inclusive Social Robotics

The findings of this study have direct implications for the design of inclusive social robots, particularly those intended to support users with special needs in assistive, educational, and therapeutic contexts. The demonstrated benefits of adaptivity, context awareness, and continual learning indicate that emotion-aware robots should not rely on fixed, population-level affect models but instead be engineered as evolving perceptual systems capable of aligning with individual users over time. From a design perspective, the results suggest that inclusive social robots should incorporate personalization as a core architectural principle, rather than as an optional enhancement. User-specific adaptation substantially improves recognition of subtle and atypical emotional expressions, which are frequently observed in users with developmental, cognitive, or sensory differences [6,23,24]. This supports prior arguments in socially assistive robotics that effective support hinges on the robot’s ability to model the individual rather than an abstract “average user” [1,2,7].

The integration of context-aware perception further underscores the importance of embedding emotion recognition within the broader interaction loop. Emotions expressed during therapy, learning, or daily assistance are shaped by task demands, interaction history, and engagement dynamics [4,15]. By explicitly modeling these factors, the proposed framework enables robots to interpret affective signals in a manner that is sensitive to situational meaning, reducing the risk of misinterpretation and inappropriate responses. This capability is especially critical in inclusive settings, where misreading affect may lead not only to reduced interaction quality but also to frustration or disengagement on the part of vulnerable users. More broadly, the results align with emerging ethical and design-oriented perspectives that caution against deploying rigid emotion recognition systems in contexts involving neurodiverse populations [25]. The adaptive framework presented here provides a concrete technical pathway toward emotion-aware robots that accommodate diversity rather than enforce normative expressive standards, thereby supporting inclusivity at both the algorithmic and interaction-design levels.

6.4. Limitations of the Study

Despite the promising results, several limitations must be acknowledged. First, there are limitations related to the datasets used in this study. Although the experimental setup intentionally combined in-the-wild data, interaction-oriented sequences, and subsets emphasizing atypical expressions, publicly available datasets remain limited in their representation of users with special needs, particularly in long-term, naturalistic HRI scenarios [5,23]. As a result, while the evaluation captures important dimensions of expressive variability, it cannot fully reflect the diversity and complexity of real-world assistive interactions. Second, the scalability of the system poses practical challenges. Adaptive and context-aware mechanisms introduce additional computational overhead compared to static models, particularly when online updates are performed during interaction. The absence of large-scale, demographically annotated datasets involving users with special needs remains a limitation of the current study. While the results demonstrate improved robustness to non-standard emotional expressions, they cannot be interpreted as a definitive validation across specific clinical populations. Future work will require ethically approved, longitudinal studies involving diverse user groups and domain experts to more directly assess the system’s performance and appropriateness in real assistive and therapeutic settings. While the current implementation is suitable for experimental and controlled deployments, further optimization will be required to support large-scale or resource-constrained robotic platforms. This includes managing memory growth associated with long-term user modeling and ensuring real-time responsiveness as interaction histories accumulate. Finally, the study focuses primarily on vision-based emotion recognition, which, although central to many HRI scenarios, represents only one channel of affective communication. This modality focus limits the system’s ability to handle situations where facial or gestural cues are unavailable or unreliable, such as due to occlusion, motor impairments, or environmental constraints [16,17].

6.5. Future Research Directions

Building on these limitations, several directions for future research emerge. A key extension involves multimodal integration, combining visual emotion recognition with complementary channels such as speech, prosody, physiological signals, or interaction behavior. Prior work in affective computing and HRI indicates that multimodal fusion can significantly enhance robustness and disambiguation, particularly under noisy or incomplete sensory conditions [16,17,24]. Integrating such modalities within the proposed adaptive framework would further strengthen its applicability to inclusive and assistive settings. Another important direction concerns long-term user learning. While the current framework supports online adaptation within and across sessions, future work should investigate mechanisms for stable, longitudinal user modeling over weeks or months of interaction. This includes studying how emotional expression patterns evolve over time and how robots can maintain continuity while respecting privacy and ethical constraints. Long-term adaptation is particularly relevant for therapeutic and educational applications, where progress and change are central goals [14,15,22]. Finally, future research should explore human-centered evaluation methodologies involving direct participation of users with special needs and domain experts in therapy or education. Such studies are essential to validate not only recognition accuracy but also the perceived appropriateness, trustworthiness, and usefulness of adaptive emotion-aware robots in real-world practice.

In summary, while the present study demonstrates the feasibility and benefits of adaptive, context-aware emotion recognition for inclusive social robotics, it also highlights the need for broader datasets, multimodal perception, and long-term user-centered validation. Addressing these challenges will be crucial for translating adaptive affective perception from experimental systems into socially responsible robotic companions and assistants.

Appendix A.1. Case Study 1: Adaptation to Reduced-Intensity Emotional Expressions

Date of data collection: December 2025

• Session duration: 12 min
• Interaction type: Semi-structured interaction with alternating neutral and affect-eliciting prompts
• Primary focus: User-level adaptive learning
• Experimental context: This case study involved a single interaction session selected from the test dataset due to the presence of consistently low-intensity facial expressions across all emotion categories. Such expressive patterns are commonly reported in users with cognitive or developmental differences and represent a known failure mode for static emotion recognition systems.
• The interaction consisted of three phases:

-

Baseline observation phase (0–3 min), during which no adaptive updates were applied.

-

Adaptive interaction phase (3–9 min), with online user-level adaptation enabled.

-

Stabilization phase (9–12 min), used to assess post-adaptation consistency.

Observations: During the baseline phase, the static CNN baseline and the region-based static model consistently classified low-intensity expressions of sadness and fear as neutral. This resulted in unstable emotion trajectories and low confidence scores. Once adaptive learning was enabled, the proposed framework progressively adjusted its internal representations. Approximately 2.5–3 min after adaptation onset, the system began to reliably distinguish between neutral and low-intensity negative emotions. This change was reflected in:

-

increased recall for sadness and fear,

-

reduced temporal fluctuation in predicted emotion labels,

-

decreased variance in prediction confidence.

• By the stabilization phase, the adaptive framework maintained consistent emotion predictions despite the continued low expressive intensity, whereas the static baselines showed no improvement throughout the session.

Interpretation: This case study demonstrates that user-level adaptation enables the system to recalibrate its perceptual thresholds to align with individual expressive baselines. Rather than forcing observations into population-level norms, the adaptive framework learns to interpret subtle affective cues as meaningful signals. This qualitative behavior directly supports reported robustness improvements.

Appendix A.2. Case Study 2: Contextual Disambiguation During Task Transitions

Date of data collection: December 2025

• Session duration: 15 min
• Interaction type: Multi-phase task-based interaction
• Primary focus: Context-aware perception and interaction dynamics
• Experimental context: This case study examined the system’s ability to disambiguate identical facial expressions occurring under different interaction contexts. The selected session included repeated transitions between:

-

a cognitively demanding task phase, and

-

a free interaction phase with minimal task constraints.

• Facial expressions observed during both phases were visually similar, characterized by reduced facial movement and neutral mouth configuration, but differed in their underlying affective meaning.

Observations: In the absence of contextual conditioning, both the static CNN and the personalization-only baseline produced inconsistent emotion labels across task transitions, frequently oscillating between neutral and sadness. These fluctuations were not aligned with interaction events and resulted in unstable emotion trajectories. In contrast, the proposed framework integrated contextual information related to task type and interaction phase. As a result:

-

identical facial expressions were interpreted as focused/neutral during the task phase,

-

the same expressions were classified as neutral disengagement during free interaction.

These interpretations remained stable across multiple task transitions, with no observable drift over the course of the session. Notably, this stability emerged immediately upon interaction onset and did not require extended adaptation time.

Interpretation: This case study highlights the critical role of contextual conditioning in resolving perceptual ambiguity. By embedding emotion recognition within the interaction context, the proposed framework avoids over-reliance on facial appearance alone and produces interpretations that are coherent with interaction dynamics. This qualitative behavior explains the reduced performance degradation under scenario shifts.

• These qualitative observations complement the quantitative results by illustrating how adaptive and context-aware mechanisms operate in realistic interaction trajectories that are difficult to fully capture with aggregate metrics alone.