Multimodal Classification Algorithms for Emotional Stress Analysis with an ECG-Centered Framework: A Comprehensive Review

Abstract

Emotional stress plays a critical role in mental health conditions such as anxiety, depression, and cognitive decline, yet its assessment remains challenging due to the subjective and episodic nature of conventional self-report methods. Multimodal physiological approaches, integrating signals such as electrocardiogram (ECG), electrodermal activity (EDA), and electromyography (EMG), offer a promising alternative by enabling objective, continuous, and complementary characterization of autonomic stress responses. Recent advances in machine learning and artificial intelligence (ML/AI) have become central to this paradigm, as they provide the capacity to model nonlinear dynamics, inter-modality dependencies, and individual variability that cannot be effectively captured by rule-based or single-modality methods. This paper reviews multimodal physiological stress recognition with an emphasis on ECG-centered systems and their integration with EDA and EMG. We summarize stress-related physiological mechanisms, catalog public and self-collected databases, and analyze their ecological validity, synchronization, and annotation practices. We then examine preprocessing pipelines, feature extraction methods, and multimodal fusion strategies across different stages of model design, highlighting how ML/AI techniques address modality heterogeneity and temporal misalignment. Comparative analysis shows that while deep learning models often improve within-dataset performance, their generalization across subjects and datasets remains limited. Finally, we discuss open challenges and future directions, including self-supervised learning, domain adaptation, and standardized evaluation protocols. This review provides practical insights for developing robust, generalizable, and scalable multimodal stress recognition systems for mental health monitoring.

1. Introduction

Emotional stress has become a global public health challenge and is closely associated with anxiety, depression, and cognitive decline, exerting negative impacts on both daily life and work performance [1]. Recent epidemiological reports indicate a sustained rise in mental health disorders worldwide, with anxiety and depressive disorders affecting more than 300 million people globally, and accounting for a substantial proportion of years lived with disability [2,3]. Moreover, chronic stress exposure has been increasingly linked to accelerated cognitive decline and elevated risk of neurodegenerative disorders, particularly in aging populations.

Current stress assessment approaches predominantly rely on self-report questionnaires, which are highly subjective and insufficient for objective, continuous, and real-time monitoring [4,5]. In this context, physiological signals such as electrocardiogram (ECG), electromyography (EMG), and electrodermal activity (EDA) have gained increasing attention for their ability to reflect dynamic changes in the autonomic nervous system [6]. Multimodal physiological fusion techniques integrate complementary information across different signal channels, and combining these modalities with subjective scales is of great importance for enhancing psychological health monitoring in daily life.

In this review, emotional stress is adopted as an operational concept referring to stress-related physiological responses elicited or modulated by emotionally salient stimuli. Unlike general psychological stress, which often involves prolonged exposure to external demands and coping processes, emotional stress is characterized by short-term affective arousal accompanied by measurable autonomic nervous system responses. From a physiological perspective, emotional stress shares substantial overlap with high-arousal emotional states, particularly along the arousal dimension, which explains why datasets originally developed for emotion recognition are frequently utilized in stress-related research. Accordingly, this review focuses on studies in which stress labels are defined through emotionally driven tasks or stimuli and validated via physiological responses, thereby bounding the scope of the survey to emotional stress rather than chronic or purely cognitive stress paradigms.

In recent years, several surveys and review studies have summarized progress in physiological signal-based emotion and stress recognition. Existing reviews have examined stress detection from wearable sensors [7], affective computing with multimodal physiological signals [8], and machine learning or deep learning methods for emotion analysis. These studies provide valuable overviews of signal characteristics, modeling techniques, and application scenarios. However, most prior reviews either focus on a single modality or treat multimodal fusion at a high level, without systematically analyzing fusion strategies across different integration stages or explicitly addressing challenges such as modality heterogeneity, temporal misalignment, and cross-dataset generalization. In addition, limited attention has been paid to the interaction between physiological mechanisms, data characteristics, and algorithmic design choices. These gaps motivate the need for a more structured and mechanism-aware review that connects physiological foundations with multimodal fusion strategies and learning-based classification methods.

Despite significant progress has been made in emotional stress analysis, major gaps remain at both data and algorithmic levels. Public databases are limited by small sample sizes, narrow demographic coverage, inconsistent synchronization protocols, and coarse annotation standards [9,10]. Existing classification models often suffer from overfitting and poor cross-dataset generalization, as many fusion strategies fail to account for modality heterogeneity and temporal misalignment [11,12]. Therefore, a comprehensive review is needed to organize current knowledge, compare methodologies across different stages of multimodal fusion, and clarify emerging challenges and research trends. This work provides a systematic analysis of physiological mechanisms, multimodal databases, preprocessing pipelines, feature extraction techniques, fusion strategies, and classification algorithms, offering insights to guide the development of accurate, interpretable, and generalizable emotional stress recognition systems.

2. Physiological and Psychometric Basis of Emotional Stress

2.1. Physiological Foundations of Multimodal Stress Assessment

In this review, emotional stress is used as an operational concept referring to stress-related physiological responses that are elicited or modulated by emotional stimuli [13]. Unlike general psychological stress, which often involves prolonged exposure to external demands and coping processes, emotions typically describe short-term affective states characterized by dimensions such as valence and arousal [14]. Emotional stress lies at the intersection of these constructs, where emotionally driven arousal acts as a controlled stressor and produces measurable autonomic responses. From a physiological perspective, emotional stress shares substantial overlap with high-arousal emotional states, which explains why datasets originally designed for emotion classification are frequently adopted in stress-related research [15].

Emotional stress activates the autonomic nervous system, leading to measurable changes in cardiovascular activity, muscle tension, and sweat gland secretion. These physiological pathways can be captured through ECG, EMG, and EDA, which represent complementary modalities reflecting sympathetic and parasympathetic responses. Unlike subjective questionnaires, these signals provide continuous, real-time, and objective indicators of stress-related changes, forming the physiological foundation for multimodal emotional stress detection. Understanding the mechanisms and characteristics of these signals is essential for selecting appropriate features and designing effective classification models for stress recognition.

ECG is a fundamental physiological modality for emotional stress detection due to its sensitivity to autonomic nervous system regulation [16]. A standard ECG waveform is composed of the P wave, QRS complex, and T wave, collectively referred to as the PQRST morphology, which reflects successive phases of cardiac electrical activity. Specifically, the P wave corresponds to atrial depolarization, the QRS complex represents rapid ventricular depolarization, and the T wave reflects ventricular repolarization.

Under stress, increased sympathetic activation and reduced parasympathetic modulation can alter this morphology, most notably through changes in QRS duration and QT interval. Prolonged or shortened QRS duration indicates modified ventricular conduction dynamics, while QT interval variation reflects stress-induced changes in ventricular repolarization timing [17,18]. These alterations are associated with autonomic imbalance and have been consistently observed under both acute and chronic stress conditions [19]. Although indices such as the LF/HF ratio remain debated due to sensitivity to respiration and methodological variance [20], they are still widely used as non-invasive indicators of stress-related autonomic shifts [21]. The characteristic structure of the PQRST complex and its stress-related variations are illustrated in Figure 1.

EMG records the electrical activity of skeletal muscles and reflects neuromuscular responses regulated by both voluntary control and autonomic modulation during emotional stress [22,23]. At the physiological level, EMG signals arise from the superposition of motor unit action potentials (MUAPs), where each motor unit consists of a motor neuron and the muscle fibers it innervates. Stress-induced sympathetic activation increases muscle tension and alters motor unit firing patterns, leading to elevated EMG amplitude and shifts in spectral content, particularly in facial and upper limb muscles [24].

Typical EMG waveforms may exhibit biphasic or triphasic MUAP shapes, which reflect the spatial and temporal propagation of action potentials along muscle fibers and are commonly used to characterize neuromuscular activity under both normal and stress-related conditions. These waveform characteristics are illustrated in Figure 2, demonstrating EMG’s capability to capture subtle stress-related muscle activation. However, EMG signals are susceptible to motion artifacts, electrode placement variability, and inter-individual differences in muscle physiology, which may reduce reliability when EMG is used as a single modality [25]. Consequently, EMG is most effective when integrated with complementary physiological signals such as ECG and EDA in multimodal stress detection systems.

EDA reflects sweat gland activity regulated by the sympathetic nervous system and is highly sensitive to emotional and cognitive stress [26,27]. When individuals experience psychological arousal, increased sympathetic activation leads to elevated skin conductance, characterized by tonic skin conductance level (SCL) and phasic skin conductance responses (SCR) [28]. These dynamic features, including rise time, amplitude, and recovery rate, are illustrated in Figure 3 and are widely used as indicators of short-term stress reactivity [29]. However, EDA is easily influenced by environmental factors and individual variations in sweat gland activity, which may reduce its reliability when used alone [30]. Consequently, EDA is often integrated with ECG and EMG to leverage its high sympathetic specificity and improve the robustness of multimodal stress recognition systems [31].

In emotional stress research, single-modality physiological signals rarely capture the complex relation between psychological states and bodily responses [32]. Among non-invasive indicators, ECG, EMG, and EDA, respectively, index cardiovascular regulation, muscle activation, and sympathetic arousal. ECG reflects autonomic balance via heart-rate variability and waveform morphology; EMG quantifies facial and limb muscle tension; EDA provides rapid phasic responses to short-term arousal. Each modality, however, has characteristic limitations, including motion artifacts, electrode-placement sensitivity, and environmental dependence, which together constrain standalone reliability [33,34]. A side-by-side appraisal (

Table 1. Comparative summary of ECG, EEG, EMG, and EDA in emotional stress detection.

Dimension	ECG	EEG	EMG	EDA
Key Feature Indicators	HRV (SDNN, LF/HF), QTc, PQRST morphology	Band power (delta, theta, alpha, beta), frontal asymmetry, coherence, entropy	RMS, mean frequency, facial/limb MUAP waveforms	SCL (baseline), SCR (amplitude, rise time, recovery time)
Stress-Related Changes	HRV reduction, LF/HF increase, QTc fluctuation, PQRST waveform changes	Increased frontal theta and beta activity, reduced alpha power, altered connectivity	Muscle tension increase, higher amplitude and frequency, irregular contractions	SCR frequency and amplitude increase, SCL elevation
Advantages	Clinically mature, well established medical basis	High temporal resolution, direct reflection of central nervous system activity	Directly reflects muscle tension and facial expression changes	Highly sensitive to sympathetic activity, effective for short-term stress detection
Limitations	Strongly affected by inter-individual variability and motion artifacts [33], limited generalizability [36]	Sensitive to noise and artifacts, requires careful preprocessing and calibration, limited portability	Sensitive to electrode placement, prone to noise and artifacts [34], non-specific muscle tension changes	Strongly influenced by temperature [37] and humidity and skin properties [38], nonlinear and time-varying [39]
Applicable Scenarios	Clinical stress monitoring, cardiovascular stress studies, wearable devices for health	Laboratory-based stress experiments, cognitive workload assessment, affective computing	Facial EMG in emotion recognition, workplace stress detection, muscle fatigue studies	Cognitive load tasks, affective computing, psychophysiological stress research

) clarifies comparative strengths and weaknesses and motivates modality-complementary designs. Accordingly, this synthesis provides the rationale for multimodal feature fusion and classification frameworks [35].

While ECG, EMG, and EDA provide the physiological basis for stress recognition, signal-only models are constrained by noise, inter-individual variability, and construct validity. Incorporating psychometric scales as labels and auxiliary features anchors physiological patterns to perceived stress, improving both accuracy and interpretability. The next subsection introduces commonly used questionnaires and how they complement physiological modalities in multimodal assessment.

Although EEG is included in Table 1 for completeness, it is not treated as a primary modality in this review. The purpose of including EEG is to provide a comparative reference that highlights fundamental trade-offs between central and peripheral physiological signals in emotional stress research. As summarized in Table 1, EEG offers high temporal resolution and direct access to central nervous system activity, making it valuable in controlled laboratory settings. However, its practical deployment in stress monitoring is constrained by susceptibility to noise, complex preprocessing requirements, and limited portability, which restrict its applicability in wearable and real-world scenarios.

By contrast, ECG, EMG, and EDA provide more feasible and scalable solutions for continuous stress assessment, particularly under ambulatory or long-term monitoring conditions. Therefore, EEG is discussed here primarily to contextualize modality selection and to justify the scope of this review, rather than to advocate its inclusion in ECG-centered multimodal frameworks. This synthesis clarifies that the focus on ECG, complemented by EMG and EDA, reflects a deliberate balance between physiological relevance and deployment feasibility.

2.2. Common Stress-Related Questionnaires in Stress Assessment

Psychometric assessment provides standardized and quantifiable tools for evaluating psychological stress, offering greater objectivity than clinical interviews [40]. These questionnaires allow for systematic evaluation of mental states and serve as valuable complements to physiological monitoring. They are also used to evaluate the effectiveness of psychological interventions, providing evidence-based guidance for mental health management. Commonly employed instruments are summarized in

Table 2. Summary of psychological questionnaires for stress assessment.

Type	Measurement Dimensions	No. of Items	Scoring	Evaluation Objective	Application Scenarios	Advantages
PSS [41]	Perceived stress	10/14 (PSS-10/PSS-14)	5-point Likert (0–4)	Assess subjective stress perception	Mental health research; stress management	Easy to implement; suitable for general population
STAI [42]	State and Trait Anxiety (SA and TA)	20 each (40 total)	4-point Likert (1–4)	Differentiate SA and TA	Clinical anxiety assessment; screening	Separately assesses SA and TA
PANAS [43]	Positive or Negative affect	10 each (20 total)	5-point Likert (1–5)	Assess affective states	Emotion research; therapy evaluation	Distinguishes affect valence
SRQ [44]	Physiological, psychological, and behavioral responses to stress	24	5-point Likert (1–5)	Assess comprehensive stress responses	Stress research; psychophysiological studies	Suited for multimodal physiological signal studies

, which highlights their assessment dimensions, scoring systems, and application scenarios.

Questionnaires such as the PSS and SRQ provide structured frameworks for quantifying perceived and physiological stress responses [45]. However, their reliance on self-reports introduces potential biases related to recall accuracy, emotion regulation, and situational context. As a result, their diagnostic precision benefits from integration with objective physiological indicators such as ECG, EMG, and EDA, which capture continuous autonomic responses.

Integrating questionnaire-derived scores with physiological signals enhances interpretability and generalization of stress detection models [46]. When employed as both labels and auxiliary features, psychometric data bridge subjective perception and physiological measurement, enabling multimodal frameworks to more accurately capture emotional stress. This integration establishes a conceptual foundation for developing comprehensive, data-driven approaches to emotional stress assessment.

3. Multimodal Stress Databases

3.1. Public and Self-Collected Stress Databases

Emotional stress databases play a crucial role in stress research by providing standardized multimodal data for developing and validating recognition algorithms [47]. Currently, multimodal signals are primarily obtained from two types of databases: public and self-collected datasets [48]. Representative public databases are summarized in

Table 3. Comparison of widely used multimodal stress databases.

Database	WESAD [50]	AMIGOS [51]	MAHNOB-HCI [52]	DREAMER [53]	PhysioNet (Stress Rec.) [54]
Physiological Modalities	ECG, EDA, Respiration, Motion	ECG, EDA, EEG, Facial expressions	ECG, EEG, EDA, Respiration	ECG, EEG	ECG, EDA, Respiration
No. of Subjects	15	40	27	23	40
Stress Induction Method	Wearable sensor-based protocol	Video-induced emotions	Video-induced emotions	Video-induced emotions	Driving simulation
Primary Application Domain	Wearable stress or affect detection	Affective computing; multimodal analysis	Human–computer interaction	VR-based emotion recognition	Driver stress evaluation
Annotation Type	Task phases; self-reports	Self-reports (arousal/ valence)	Self-reports; event markers	Self-reports (arousal/ valence)	Driving states; coarse labels
Recording Duration/ Protocol	Baseline task recovery blocks (minutes)	Multiple short video clips (minutes/ clip)	Video clips with synchronization	Video sessions (multi-trial)	Long continuous driving sessions
Acquisition Device/ Environment	Wearable sensors (lab/semi-free)	Lab; biosensors + video	Lab; medical grade devices	Lab; biosensors	Driving simulator; semi-realistic
Synchronization Alignment	Provided; multi-sensor timestamps	Requires alignment across modalities	Provided across channels	Provided for physiological signals	Provided; per-segment labels
Key Limitations	Limited sample size	Modality alignment required	Lack of high-stress tasks	Restricted modality coverage	High inter-individual variability

, which include WESAD, AMIGOS, MAHNOB-HCI, DREAMER, and PhysioNet. These datasets provide opportunities to explore physiological and behavioral responses under controlled emotional or stress-inducing conditions [49].

As shown in Table 3, most public multimodal databases focus on core physiological signals such as ECG, EDA, and EEG, while few include complementary modalities like respiration, motion, or facial features [55,56]. Video-induced paradigms dominate experimental designs due to their convenience and reproducibility, though driving simulation and wearable scenarios are also emerging [57]. Annotations typically rely on subjective self-reports (e.g., valence/arousal) or coarse task phases, providing basic temporal structure but limited event-level precision.

Despite their wide use, several common challenges remain. First, modality imbalance continues to be a key limitation because behavioral, EMG, and environmental signals are insufficiently represented, thereby restricting the thorough assessment of robust fusion techniques [58]. Second, ecological validity is constrained, as most stressors involve short, video-based tasks with moderate intensity, which restrict generalization to real-world conditions. Thirdly, annotation inconsistency and lack of unified synchronization protocols make cross-database comparisons difficult. Furthermore, many datasets include small, homogeneous cohorts, reducing their representativeness and fairness in modeling population diversity.

To mitigate data scarcity and annotation imbalance in public stress databases, data augmentation and oversampling strategies have been increasingly explored. Conventional approaches include random oversampling, class-balanced reweighting, and noise-based perturbations such as time warping, amplitude scaling [59], and frequency domain augmentation [60], which aim to improve class balance without altering label distributions. However, these techniques often fail to capture the complex temporal and cross-modal dependencies inherent in physiological stress responses, particularly under extreme low-sample regimes.

More recently, generative modeling has emerged as a promising direction for synthesizing realistic physiological data. Generative adversarial networks (GANs) and variational autoencoders (VAEs) have been applied to generate synthetic ECG, EDA, and multimodal samples that preserve marginal distributions and temporal characteristics [61]. In scenarios of extreme data scarcity, Generative Adversarial Network Synthesis and Markov Random Fields for Oversampling (GANSO) has been proposed to jointly model local temporal structure and global statistical dependencies, enabling more stable oversampling when training data are very limited [62]. Despite their potential, generative augmentation methods require careful validation to avoid distributional drift, mode collapse, or amplification of annotation noise, and synthesized data should therefore be treated as complementary rather than replacement resources for real recordings [63].

Consequently, public stress databases are best positioned as benchmarks and baselines for algorithm evaluation rather than as comprehensive training resources. Their standardized formats facilitate reproducibility and algorithmic comparison, yet they cannot fully capture the complexity of naturalistic stress responses [64]. Researchers are increasingly developing their own datasets that broaden the range of signal modalities, incorporate ecologically valid paradigms, and enhance annotation granularity [47]. This complementary strategy, which leverages public datasets for benchmarking and self-collected datasets for contextual extension, provides a balanced foundation for advancing multimodal stress recognition research and enhancing model robustness across diverse environments [48].

3.2. Experimental Paradigms and Methodological Insights

Existing public stress databases suffer from limited modality coverage, small sample sizes, and inconsistent annotations, restricting their ability to model real-world stress dynamics [65]. To address these shortcomings, researchers increasingly construct self-collected databases with customized task designs and flexible labeling schemes [66]. Such databases enable higher ecological validity by allowing diverse experimental settings and multimodal signal acquisition [67]. They have thus become a vital data source for advancing multimodal emotional stress recognition research.

Self-collected databases often employ carefully designed paradigms to elicit measurable stress responses across cognitive, emotional, social, and ecological dimensions [68]. Cognitive tasks such as the Stroop or MIST tests emphasize standardized control but induce limited stress intensity. Emotional induction via videos or music provides reproducibility yet suffers from individual variability [69]. Social paradigms like the Trier Social Stress Test (TSST) and immersive VR simulations offer strong ecological realism but involve ethical and technical challenges [70].

Table 4. Comparison of typical experimental tasks in self-collected stress databases.

Task Type	Specific Task	Stress Induction Mechanism	Main Measured Signals	Advantages	Limitations
Cognitive	Stroop [71]	Color–word conflict and cognitive load	ECG, EDA, EEG	Simple and highly standardized	Limited stress intensity
	MIST [72]	Time pressure and negative feedback	ECG, EDA, EEG	Strong induction and widely used	Artificial setting and low ecological validity
Emotional Induction	Video stimuli	Emotionally evocative film clips	ECG, EDA, EEG, facial expression	Easy to implement and well controlled	Large interindividual variation
	Audio stimuli	Music and affective sounds	ECG, EDA	Simple and noninvasive	Narrow stimulus range
	IAPS image set [73]	Standardized affective pictures	ECG, EDA, EEG	High standardization and replicable	Monotony and weak persistence
Social Stress	TSST [74]	Public speaking and interview causing social evaluation pressure	ECG, EDA, cortisol, EMG	High ecological validity and strong stress	Hard to standardize and ethical concerns
Real-World Simulation	Driving simulation [75]	Traffic complexity and performance pressure	ECG, EDA, respiration, motion	High ecological validity	High cost and strong individual variability
	Immersive VR tasks [76]	Virtual environments that elicit stress	ECG, EDA, EEG, motion	High immersion and realism	Technology dependence and expense

summarizes their mechanisms, measured signals, and trade-offs.

Compared with public datasets such as WESAD or AMIGOS, self-collected databases offer greater flexibility in modality selection, annotation granularity, and task customization [77]. While public databases ensure accessibility and benchmarking consistency, they often rely on low-stress or single-task conditions that limit generalizability [50]. In contrast, self-collected datasets enable richer multimodal acquisition under controlled yet realistic scenarios [78]. However, the cost of equipment, time, and standardization remains a barrier to large-scale replication and open sharing.

Recent self-collected databases exhibit diverse paradigms and signal modalities, integrating physiological, behavioral, and contextual data [79]. Representative examples include Stress-ID and ADA Base, which provide comprehensive multimodal coverage, and WEMAC, which introduces gender-focused and VR-based designs [80]. Others, such as EmpathicSchool and Multi-PENG, emphasize naturalistic tasks and fine-grained psychological labeling [81]. Despite these advances, key challenges remain unresolved, such as small sample sizes, subjective labeling, and inconsistent data quality across modalities [82].

Table 5. Summary of self-collected stress databases.

Name	Sample Size	Task Type	Specific Tasks	Modalities	Annotation Scheme	Advantages	Limitations
Stress-ID [79]	65 participants (18F and 47M, age 21–55)	Cognitive, Emotional, Social, Relaxation Control	Breathing baseline; emotional videos; seven interactive stress tasks; public speaking; relaxation	ECG, EDA, Respiration, Video, Audio	Self-assessments (0 to 10 stress); SAM; binary and three-class labels	Comprehensive multimodal coverage; large dataset; detailed annotations; baseline models provided and publicly available	Controlled lab setup; possible sensor-induced stress; subjective bias; gender imbalance; partial data loss
Muse [83]	28 college students (during and after final exams)	Emotional induction and monologue elicitation	Baseline + monologues + emotional videos; stress level assessed via PSS scale	Video, Thermal camera, Audio, Heart rate, Skin conductance and temperature	Self-reports (PSS, SAM); external annotations via AMT	Naturalistic stress context; multimodal coverage; synchronized baseline and task data	Small sample and student-only cohort; contextual rather than controlled stress elicitation
Empathic School [84]	20 students (aged 21 to 35)	Cognitive, Emotional, Social, Relaxation	Magazine reading, presentation prep, IQ and Stroop, music, funny video, breathing, rest sessions	Facial video, EDA, HR, BVP, IBI, Skin temperature, ACC	NASA-TLX scores; video-based expression labels	Comprehensive wearable and facial coverage; multiple task types; public code	Lab-only environment; coarse temporal resolution; no chronic stress analysis
VERRD [85]	34 participants (final 26 used)	Real-world simulation	360° VR video environment (12 clips)	Eye tracking, ECG, GSR, Self-reports	SAM and VAS scales with circumplex valence–arousal model	High immersion and ecological validity; multimodal integration; publicly available features	Small sample; no EEG data; raw video stimuli unavailable
ADA Base [86]	51 participants	Cognitive + Driving Simulation	n-back (1–3 back, single and dual) and semi-autonomous driving tasks with secondary infotainment load	ECG, EDA, EMG, PPG, Respiration, Skin Temp, Eye tracking, Facial video, Cortisol	Baseline and load levels (low, medium, high); subjective questionnaires (NASA-TLX, PSS, PANAS)	Broad multimodal coverage; realistic simulation; continuous load labels with synchronization	Missing EEG; limited sample; privacy restrictions in video data; artifact-prone sessions
WEMAC [80]	100 women (20–77 years)	Emotional induction (VR fear vs neutral)	Immersive VR videos eliciting fear and neutral states	BVP, GSR, Skin Temp, Resp, EMG, Motion, Speech features	Discrete (12 emotions) and dimensional (VAD) ratings; speech annotations	High ecological validity; large female cohort; VR-based emotion elicitation	Order effects; limited stimuli range; gender-specific sample
Multi-PENG [81]	39 participants (30M and 9F, mean age 24.3)	Video game tasks with graded difficulty	Sports and fighting games (FIFA’23, Street Fighter V) with round-level surveys and pauses	EEG, Eye tracking, Heart rate, Controller inputs, Facial video, Gameplay footage, Surveys	Self-reports; third-party annotations subset	Rich multimodal data; precise temporal alignment; public availability on Kaggle	Limited games; motion artifacts; partial missing modalities; small sample
ForDigit-Stress [82]	40 participants (57.5% F, 40% M, 2.5% diverse; mean age 22.7 ± 3.2)	Social stress task (digital mock interview)	Remote job interviews with 14 stages (self-intro, motivation, logic and math questions, etc.)	PPG, EDA, Cortisol, Facial AUs, Eye tracking, Body skeleton, Speech, HD video, Audio	Frame-by-frame annotations (2 psychologists + self-reports + cortisol validation)	Ecological interview scenario; comprehensive modalities; continuous labels and baseline features	Limited sample; missing eye tracking; EDA delay vs labels; single scenario context

summarize their main features and limitations.

Future database development should balance standardization with ecological realism, combining traditional paradigms with immersive VR or AR environments [87]. Expanding demographic diversity through multi-institutional collaboration will improve generalizability and fairness. Annotation frameworks should integrate self-reports with behavioral and physiological cues for more objective, multi-layered labeling [88]. Technical improvements in synchronization, noise control, and long-term stability will enhance data reliability [89]. Ultimately, standardized acquisition protocols and open data sharing will enable reproducibility and advance multimodal stress research [90].

4. Preprocessing and Feature Extraction

This section summarizes the theoretical foundations and methodological considerations of preprocessing and feature extraction, thereby establishing a unified basis for reproducible, comparable, and physiologically meaningful analyses in emotional stress research. Because ECG, EMG, and EDA differ markedly in sampling rates, dynamic ranges, latencies, and noise characteristics, insufficient preprocessing can bias core indices, inflate model variance, and ultimately undermine the robustness and generalizability of multimodal fusion and downstream classification. Accordingly, we first outline a modality-aware preprocessing workflow and parameter-selection principles for multimodal settings, with emphasis on reporting standards and the avoidance of data leakage. We then summarize representative features across the time domain, frequency domain, and time–frequency domain, as well as nonlinear descriptors for each modality, and further extend the discussion to cross-modal coupling measures, feature selection, and dimensionality reduction, highlighting the physiological interpretability and task relevance of the resulting feature representations.

4.1. Preprocessing Overview

Preprocessing plays a foundational role in multimodal emotional stress analysis, ensuring that ECG, EMG, and EDA signals collected from heterogeneous devices are temporally aligned, denoised, and comparable before feature extraction. Due to differences in sampling rates, dynamic ranges, and noise characteristics, unprocessed signals can distort physiological indices and reduce model robustness. Recent studies emphasize that standardized pipelines integrating synchronization, denoising, and normalization substantially improve data reliability and model generalization, particularly in cross-device and real-world environments [91]. Proper preprocessing thus bridges the gap between raw physiological recordings and reproducible, interpretable multimodal analytics. Figure 4 illustrates the complete signal-processing workflow adopted in multimodal emotional stress analysis, highlighting the sequential steps from data acquisition to classification.

In addition to the standard preprocessing steps illustrated in Figure 4, data augmentation techniques are frequently incorporated in the literature to mitigate limited sample sizes and class imbalance in stress datasets. Common augmentation strategies for physiological signals include time-domain perturbations such as window cropping, time shifting, amplitude scaling, and additive noise injection, which preserve label semantics while increasing data diversity. Frequency-domain augmentation, including spectral masking or random frequency scaling, has also been applied to enhance robustness against device-specific noise and recording variability. For multimodal settings, modality-consistent augmentation and random modality dropout are sometimes adopted to improve fusion robustness under partial signal loss. These augmentation techniques are typically applied after segmentation and normalization, and prior to model training, serving to enhance generalization without altering the underlying physiological interpretation [59,92].

The typical workflow includes four steps: time synchronization and resampling, which correct clock drift and align modalities to a unified time base, often 250–500 Hz for ECG, 100–200 Hz for EMG envelopes, and 10–32 Hz for EDA [93]; denoising and filtering, using zero-phase bandpass and notch filters with adaptive refinement to suppress motion and power-line interference [94]; segmentation and label alignment, partitioning recordings into fixed or event-locked windows while compensating physiological latency and avoiding cross-trial leakage [95]; and normalization and quality control, applying z-scoring or baseline correction to reduce individual variability without eroding physiological meaning [96].

Effective preprocessing follows several key principles. First, physiological interpretability must be preserved; filters should suppress structured noise without distorting waveform morphology or event timing. Second, statistical transparency and reproducibility are critical: studies should report sampling rates, filter bands, missing data handling, and synchronization error statistics to enable replication [97]. Third, intersubject and interdevice harmonization mitigates nonphysiological variance using calibration or batch-correction methods such as ComBat and mixed-effects residualization [98]. Finally, data augmentation can expand training variability through controlled perturbations, including baseline wander, power-line residue, and minor time shifts, provided that physiological plausibility and label alignment are maintained [99].

In summary, preprocessing in emotional stress studies aims to create a reproducible and physiologically coherent foundation for multimodal fusion. By combining time alignment, adaptive denoising, normalization, and careful documentation, researchers can minimize bias from heterogeneous acquisition and improve cross-study comparability. Despite advances in adaptive filtering and automated pipelines, several challenges still exist in managing real-time drift, uncertain latencies, and device variability. Future work should prioritize adaptive and self-supervised preprocessing frameworks that integrate quality indices and uncertainty estimation to deliver more robust multimodal emotional stress analysis [100].

4.2. Feature Extraction for Emotional-Stress Classification

Feature extraction is a critical step that bridges raw physiological recordings and interpretable stress-related patterns. Its goal is to extract compact and meaningful descriptors that reflect autonomic and muscular responses under emotional stress. Recent studies emphasize that time, frequency, time–frequency, and nonlinear domains capture complementary physiological dynamics, each revealing distinct aspects of autonomic modulation and psychophysiological arousal [10]. Properly engineered features not only enhance model accuracy but also improve generalization across subjects and contexts. With the proliferation of wearable and multimodal sensors, robust feature representations play an increasingly central role in achieving high-precision emotional stress detection [101].

ECG, EMG, and EDA remain the principal physiological modalities for stress recognition, each reflecting a different facet of autonomic or somatic activation. ECG features focus on heart rate variability (HRV), encompassing time-domain indices such as SDNN and RMSSD, spectral metrics like LF and HF power, and nonlinear measures including entropy and Poincaré geometry [102,103]. Recent works combine multiple feature scales through graph neural networks or wavelet scattering to better capture morphological and frequency correlations [104,105]. EMG features quantify muscle activation using root mean square, waveform length, and median frequency; hybrid time–frequency and attention-based representations have improved robustness under motion artifacts and intersubject variability [106,107]. EDA features describe sympathetic arousal through skin conductance level (SCL) and response (SCR) characteristics, with energy- and entropy-based descriptors providing additional discrimination. Transformer and CNN–LSTM frameworks have recently automated EDA feature learning and artifact detection for more reliable classification [108].

Subjective scales such as the Perceived Stress Scale (PSS) and Stress Response Questionnaire (SRQ) provide complementary self-reported dimensions of stress that can be quantitatively integrated with physiological features. Correlation and regression analyses reveal consistent relationships between questionnaire scores and HRV, SCR, or EMG-derived activity, confirming their physiological validity [109]. In multimodal learning, feature-level fusion directly concatenates questionnaire and physiological descriptors for unified modeling [110], while decision-level fusion aggregates independent model outputs through weighted voting or probabilistic rules to enhance robustness under heterogeneous noise [111]. Hybrid or cross-modal attention approaches further align temporal structures between discrete questionnaire scores and continuous biosignals, improving interpretability and robustness to subjective bias.

Overall, physiological and subjective features jointly underpin multimodal stress recognition. Carefully selected descriptors across ECG, EMG, and EDA capture distinct but complementary responses of the autonomic nervous system and muscle activation, while their fusion with psychological measures enhances contextual understanding. Future research should emphasize unified feature embeddings and adaptive multimodal fusion mechanisms that integrate cross-modal correlations and uncertainty awareness to achieve more interpretable and generalizable emotional stress classification.

5. Multimodal Feature Fusion and Cross-Modal Representation

Multimodal fusion and cross-modal representation are essential because no single modality can capture the asynchronous and complementary responses of ECG, EMG, and EDA. We compare fusion paradigms and introduce cross-modal modeling that aligns and shares representations to cope with temporal asynchrony, physiological lags, domain shift, noise, and missing data. The section then addresses robustness and external generalization in real-world use, using rigorous splits, external validation, ablation, and significance testing to balance physiological interpretability with reproducible modeling.

5.1. Multimodal Feature Fusion Strategies

From a methodological perspective, multimodal physiological fusion is motivated by the fact that emotional stress manifests through multiple, partially independent physiological pathways, including cardiovascular, electrodermal, and neuromuscular responses. No single modality can fully capture these dynamics due to differences in temporal resolution, noise characteristics, and physiological sensitivity. Prior studies have shown that fusing complementary modalities can improve robustness and discriminability by reducing uncertainty and compensating for modality-specific limitations, especially under real-world conditions with noise or partial signal loss [112]. However, fusion is not universally beneficial, as performance gains depend on modality complementarity, data quality, synchronization accuracy, and sample size, and improper fusion may introduce redundancy or exacerbate overfitting [113]. These observations motivate the need for principled fusion strategies and provide the rationale for the systematic comparison of feature-level, hidden-layer, and decision-level fusion paradigms in this section.

Figure 5 and Table 6 provide a structured overview of the three mainstream multimodal fusion strategies commonly used in emotional stress recognition, explicitly illustrating their fusion stages, interaction mechanisms, and practical trade-offs.

Multimodal feature fusion integrates complementary sources such as ECG, EMG, EDA, and subjective questionnaire data to enhance emotional stress classification. Differences in temporal resolution, data type, and physiological linkage across modalities motivate three mainstream strategies: feature-level, hidden-layer, and decision-level fusion. Each presents distinct trade-offs—feature-level fusion requires strict temporal and statistical alignment, hidden-layer fusion strengthens cross-modal dependency learning at higher computational cost, and decision-level fusion offers robustness to missing modalities at the expense of deep inter-modal interactions.

Feature-level fusion concatenates ECG, EMG, and EDA features with questionnaire data into a unified vector for classification. It captures interactions between physiological and psychological responses but is sensitive to redundancy and asynchrony. Techniques such as Z-score normalization, PCA, and Autoencoder-based mapping improve alignment and compactness [114,115]. Advanced variants employ Deep CCA and modality dropout to enhance robustness to missing data [116]. Despite simplicity and interpretability, early fusion may suffer from overfitting and modality imbalance; therefore, reporting should include feature dimensionality, missing-data proportion, and cross-session robustness to ensure reproducibility.

Hidden-layer fusion operates at intermediate network stages, enabling shared representation learning through mechanisms such as cross-modal attention, latent-space projection, and mixture-of-experts gating [117,118]. CNN-, LSTM-, and Transformer-based encoders independently process each modality before interaction, facilitating complementary feature extraction under asynchronous or noisy conditions [119,120]. To address physiological delays and sampling disparities, learnable offset encoding, visibility masks, and modality dropout can be adopted. While more computationally intensive, this strategy enhances semantic correlation and cross-device generalization when sufficient data and tuning are available.

Decision-level fusion independently trains classifiers for each modality and aggregates their outputs through weighted voting, probability averaging, or attention-weighted ensembles [121,122]. It excels in handling partial or missing modalities and maintains high interpretability by preserving unimodal transparency [123]. However, its limited ability to capture deep inter-modal dependencies constrains accuracy in strongly coupled tasks. Transformer-based decision weighting has recently improved adaptability and interpretability under real-world variability [124].

From a theoretical perspective, whether multimodal fusion improves over single-modality inference depends on information complementarity and sample efficiency. Although complementary modalities can reduce uncertainty and lower achievable error bounds, fusion is not guaranteed to be beneficial in practice because it also increases feature dimensionality, noise sources, and estimation burden. Pereira et al. theoretically compared early and late fusion under Gaussian assumptions by analyzing error probabilities with perfect model knowledge and finite training data [125]. Their results indicate that early fusion is optimal with sufficient data, whereas under finite-sample regimes, its performance degrades more severely due to dimensionality expansion, making late fusion preferable in data-limited settings. This theoretical insight explains why empirical gains from fusion in stress recognition vary across datasets: fusion helps when modalities are complementary and well aligned, but may degrade performance when additional modalities are noisy, weakly informative, or poorly synchronized.

Figure 5 and Table 6 jointly illustrate the architectural and algorithmic trade-offs among the three fusion paradigms. Feature-level fusion emphasizes simplicity and interpretability; hidden-layer fusion captures deep semantic complementarity across asynchronous modalities; decision-level fusion prioritizes flexibility and robustness. Future directions include unified fusion architectures integrating early–mid–late interactions, self-supervised and contrastive learning for cross-modal consistency, modality-aware completion and lightweight deployment for wearable devices, and standardized evaluation with interpretable metrics to enhance reproducibility and practical adoption [100].

Table 6. Comparative summary of mainstream multimodal fusion strategies and representative studies for emotional stress recognition.

Strategy	Key Properties	Typical Methods	Strengths	Limitations	When to Use
Early fusion (feature-level)	Fusion at input feature stage; shallow interaction; sensitive to misalignment; low compute; high interpretability.	PCA [126], Z-score [127], CCA/DCCA [128], autoencoder mapping [129].	Simple and deployable; transparent feature attribution; effective when signals are well synchronized.	Feature redundancy and modality imbalance; overfitting risk under high dimensionality; weak under asynchrony or missing signals.	Well-aligned datasets, stable acquisition, and sufficient sample size with mature feature engineering.
Mid fusion (hidden-layer)	Fusion at intermediate layers; deep interaction via shared latent space; moderate missing-modality tolerance; high compute; moderate interpretability.	Cross-modal attention [130], latent projection [131], MoE or gating [118,132], Transformer fusion [133].	Learns explicit cross-modal dependencies; handles heterogeneous feature spaces; improved robustness under noise with proper regularization.	Data-hungry; tuning-sensitive; possible training instability (modality dominance or gradient imbalance); heavier deployment cost.	Asynchronous or heterogeneous signals where learned interactions are required and sufficient training data or resources are available.
Late fusion (decision-level)	Fusion at output stage; minimal interaction; strong robustness to missing modalities; moderate compute; high interpretability per modality.	Weighted voting [134], probability averaging [135], stacking ensembles [136], attention-weighted decision fusion [137].	Fault-tolerant and modular; easy to add or remove modalities; works under partial signal loss; strong reproducibility with unimodal baselines.	Limited deep inter-modal semantics; performance depends on unimodal model quality; may underperform when strong cross-modal coupling exists.	Real-world deployment with missing data, sensor dropouts, or when modularity and reliability are priorities.

Table 6. Comparative summary of mainstream multimodal fusion strategies and representative studies for emotional stress recognition.

Strategy	Key Properties	Typical Methods	Strengths	Limitations	When to Use
Early fusion (feature-level)	Fusion at input feature stage; shallow interaction; sensitive to misalignment; low compute; high interpretability.	PCA [126], Z-score [127], CCA/DCCA [128], autoencoder mapping [129].	Simple and deployable; transparent feature attribution; effective when signals are well synchronized.	Feature redundancy and modality imbalance; overfitting risk under high dimensionality; weak under asynchrony or missing signals.	Well-aligned datasets, stable acquisition, and sufficient sample size with mature feature engineering.
Mid fusion (hidden-layer)	Fusion at intermediate layers; deep interaction via shared latent space; moderate missing-modality tolerance; high compute; moderate interpretability.	Cross-modal attention [130], latent projection [131], MoE or gating [118,132], Transformer fusion [133].	Learns explicit cross-modal dependencies; handles heterogeneous feature spaces; improved robustness under noise with proper regularization.	Data-hungry; tuning-sensitive; possible training instability (modality dominance or gradient imbalance); heavier deployment cost.	Asynchronous or heterogeneous signals where learned interactions are required and sufficient training data or resources are available.
Late fusion (decision-level)	Fusion at output stage; minimal interaction; strong robustness to missing modalities; moderate compute; high interpretability per modality.	Weighted voting [134], probability averaging [135], stacking ensembles [136], attention-weighted decision fusion [137].	Fault-tolerant and modular; easy to add or remove modalities; works under partial signal loss; strong reproducibility with unimodal baselines.	Limited deep inter-modal semantics; performance depends on unimodal model quality; may underperform when strong cross-modal coupling exists.	Real-world deployment with missing data, sensor dropouts, or when modularity and reliability are priorities.

5.2. Cross-Modal Representation and Consistency Modeling

In emotional stress recognition, ECG, EDA, and EMG capture complementary but heterogeneous facets of autonomic responses, differing in physiological origin, temporal dynamics, and noise characteristics. Simple concatenation often fails to align, transform, or compensate information across modalities. We therefore organize cross-modal representation learning around three objectives: temporal alignment, distributional alignment, and semantic alignment to obtain modality-invariant yet physiologically meaningful features [138].

Cross-modal temporal alignment aims to mitigate inconsistencies in representation caused by differences in sampling frequency and physiological response latency across modalities. In emotional stress scenarios, ECG signals typically respond in near real time, EDA signals exhibit a delay of several seconds due to the slow dynamics of sweat gland activation, and EMG signals are highly sensitive to rapid changes in muscle activity [139]. To address these disparities, learnable temporal shift parameters and delay-gating mechanisms can be incorporated during the encoding stage to establish trainable phase relationships across modalities [140]. This can be complemented with temporal positional encoding or phase-aware embedding to capture rhythmic characteristics at multiple time scales [141]. For scenarios requiring fine-grained segment-level alignment, dynamic time warping or its differentiable variants may be employed to accommodate local temporal distortions [142], while anti-aliasing resampling and causal convolution can be used to avoid artificial correlations and future information leakage induced by interpolation [127]. Collectively, these mechanisms enable the construction of a comparable temporal baseline across modalities without disrupting intrinsic physiological timing patterns.

Statistical alignment targets discrepancies in feature distributions and domain shifts that arise from modality characteristics or device heterogeneity [143]. Its goal is to reduce modality discriminability in the representation space and promote domain invariance. Representative approaches include kernel-based statistical matching methods such as maximum mean discrepancy to constrain higher-order moment convergence [144]; adversarial domain alignment to suppress modality-specific information through a discriminator network [145]; and correlation or covariance alignment techniques to achieve second-order distribution matching without additional adversarial training [146]. In cases involving batch effects or device-specific biases, recalibrated normalization, whitening transformations, or hierarchical normalization schemes may be combined to attenuate session- and hardware-specific variability [147]. When used in combination with temporal alignment, statistical alignment reinforces the assumption that identical emotional states should exhibit similar distributional structures across modalities, thereby improving external generalization across devices and recording sessions.

Semantic alignment seeks to ensure that different modalities express the same emotional construct within a unified high-level representation space [148]. It operates by guiding the model to cluster cross-modal samples of the same emotional category into a shared semantic region. Typical strategies include supervised or semi-supervised contrastive learning and prototype-based metric learning [149], which enforce intra-class compactness and inter-class separability to enhance discriminability and modality interchangeability [150]; mutual information maximization [151] and deep canonical correlation analysis [152], which strengthen cross-modal correlations and facilitate the formation of a common subspace representation; and curriculum-based weighting or confidence-aware learning in scenarios with noisy or incomplete labels [153]. Compared with statistical alignment alone, semantic alignment is more directly task-driven and is particularly critical for cross-task transfer and cross-subject generalization. It also provides a strong constraint for the development of shared latent space representations, ensuring consistent and interpretable downstream inference across heterogeneous modalities.

Table 7. Comparative analysis of cross-modal alignment strategies in multimodal physiological stress recognition.

Aspect	Objective	Representative Algorithms	Theoretical Assumption	Advantages	Limitations	Scenarios
Temporal Alignment	To correct variations in sampling frequency and physiological response latency, enabling temporal correspondence across modalities.	Learnable time-shift modules [154]; temporal position encoding [155]; dynamic time warping (DTW) and differentiable variants [156]; causal temporal convolution [157]; delay-aware attention mechanisms [158].	Modality-specific discrepancies can be modeled through learnable temporal shifts and local temporal scaling.	Directly accounts for physiological latency and asynchronous sampling, improving segment-level alignment and temporal consistency.	Excessive alignment may introduce artificial temporal distortions; global alignment is difficult under causal or real-time constraints.	Suitable when modalities exhibit distinct temporal dynamics (e.g., near-instant ECG response, delayed EDA activation, rapid EMG fluctuations).
Distribution Alignment	To reduce inter-modality divergence in feature distributions and mitigate domain shift caused by device, session, or subject variability.	Maximum Mean Discrepancy (MMD) [159]; Domain-Adversarial Neural Networks (DANN) [160]; Correlation Alignment (CORAL) [161]; whitening transformation [162]; layer-wise normalization recalibration [163].	Modalities share alignable statistical moments that can be matched through explicit distributional constraints.	Mitigates domain shift without requiring strong supervision, enhancing robustness to inter-subject or inter-device variability.	Statistical alignment may overlook semantic structure; adversarial optimization can be unstable.	Preferred in scenarios with device heterogeneity, batch effects, or cross-session variability.
Semantic Alignment	To map heterogeneous modalities into a unified semantic space in which representations of the same emotional state converge.	Contrastive learning frameworks (InfoNCE, SupCon) [164]; triplet and prototypical networks [165]; mutual information maximization [166]; Deep Canonical Correlation Analysis (DCCA) [167].	Different modalities approximate a shared semantic manifold and can be aligned through discriminative or correlation-based objectives.	Enhances discriminability and modality-invariant representation, facilitating external generalization and cross-modal transfer.	Requires reliable labels or pseudo-labels; risk of semantic collapse or dominance of a single modality.	Most effective when semantic consistency across modalities is critical for modality-agnostic inference or transfer learning.

provides a comparison of temporal, distributional, and semantic alignment strategies with respect to their objectives, underlying mechanisms, and applicability. The analysis indicates that although these strategies offer complementary advantages for addressing modality heterogeneity, they remain constrained by localized alignment effects, susceptibility to noise, and limited capacity to model real-world dynamics. Building on shared latent-space modeling, future research should prioritize: developing scalable latent representations that adapt to time-varying modality characteristics; introducing semantics-preserving and invariance constraints that generalize across devices and scenarios; and strengthening adaptive alignment and robust inference under low signal-to-noise ratios, nonstationarity, and partial-modality conditions. Progress along these directions is critical for enhancing the stability and deployability of multimodal physiological systems in complex real-world settings and provides the theoretical and technical basis for robustness-oriented modeling in such environments.

5.3. Robustness and Generalization in Real-World Scenarios

Although multimodal emotion and stress recognition has achieved steady progress under controlled laboratory settings, real-world deployment remains limited. Physiological signals in natural conditions often suffer from unstable quality, incomplete modalities, and device or subject variability, making simple fusion strategies insufficient for robust inference and cross-domain generalization [168]. Ensuring robustness and external generalization has therefore become essential for translating experimental success into practical systems.

Real-world motion, posture shifts, and sensor slippage inject nonstationary artifacts that overlap true physiology, e.g., ECG R-peak distortions and EDA spikes or drifts from contact changes [169], complicating denoising and alignment [170]. Task interaction further couples motion and autonomic dynamics, confounding classification boundaries [171]. Low SNR during everyday activities degrades synchrony across modalities [172]; deep models then overfit spurious cues and destabilize under modality dropout or interruptions [173,174]. Beyond noise, strong inter and intra subject variability and context dependence (fatigue, attention) impede a stable decision boundary and harm cross-task generalization [175,176,177]. Distribution shifts across users [178], devices [179], and stress paradigms further erode transferability.

To evaluate external generalization, a standardized benchmark suite should be established, incorporating leave-one-subject-out validation for person-level transfer and additional cross-device, cross-session, and cross-dataset evaluations to reveal distribution and label shifts [180,181]. Define splits that prevent temporal and trial leakage by disallowing adjacent windows across splits, and document modality availability, including the percentage of windows with dropouts and the rate of alignment failures. Beyond accuracy and AUROC, report class imbalance aware metrics such as balanced accuracy, Matthews correlation coefficient, and area under the precision–recall curve, as well as runtime, latency, and memory footprint to reflect deployability. Quantify uncertainty and calibration using measures such as predictive entropy, Brier score, and expected calibration error, and pair these with statistical confidence through bootstrap confidence intervals, permutation tests, and effect sizes for key comparisons. Conduct robustness stress tests, including synthetic noise corruption, modality ablation, and inference under missing modalities, and include ablations that remove alignment losses or fusion blocks to attribute gains. Finally, release random seeds, preprocessing and alignment parameters, code versions, and device metadata to ensure full reproducibility.

Prioritize invariance and adaptation at three layers. Signal level: adaptive filtering, blind source separation, multi-sensor artifact decoupling, plus baseline normalization and dynamic recalibration to reduce drift. Representation level: domain invariant learning [182], contrastive or self-supervised pretraining for label-efficient stability [183], and prototype alignment or normalization to tighten class structure across hardware and sessions [184]. Learning level: meta and few-shot personalization for rapid subject adaptation [185], adversarial or noise injection training with uncertainty modeling for robustness [186], and test time adaptation to track input shifts online [187].

In summary, instability, modality disruption, and domain shifts remain the core barriers to real-world stress recognition. Robustness and generalization should thus be treated as core design principles rather than afterthoughts or post-hoc refinements. The next chapter classifies current algorithms by architectural paradigm and generalization capability, providing a structured framework for developing deployable and scalable multimodal systems.

6. Classification Algorithms for Emotional Stress Recognition

After cross-modal representation and fusion, the goal is classification, which involves mapping integrated features to stable and generalizable emotion and stress labels. Current work follows two tracks: traditional pipelines built on handcrafted features and shallow classifiers, and deep models that learn spatiotemporal representations and align modalities end-to-end. This section reviews representative advances and limits of both paradigms, comparing feature dependence, generalization, and deployability, and highlighting practice-oriented improvements for real-world use.

6.1. Traditional Machine Learning-Based Approaches

Traditional machine learning provides the methodological foundation for multimodal emotion and stress recognition, combining manually designed features with supervised classifiers in a standardized analysis framework. As shown in Figure 6, physiological signals such as ECG, EDA, and EMG are preprocessed, transformed into discriminative features, and refined through feature selection or dimensionality reduction to reduce redundancy and improve generalization. The resulting feature set is then used to train conventional classifiers and validated under subject-dependent and subject-independent settings to assess robustness and generalization. This framework, widely adopted in early research, established a reproducible benchmark and revealed key limitations in robustness, fusion efficiency, and transferability, forming the foundation for subsequent deep learning developments.

Traditional machine learning-based approaches rely on handcrafted feature representations that encode domain-specific physiological knowledge prior to classification [188]. For ECG signals, commonly used features are derived from heart rate variability and waveform morphology, including time-domain indices such as SDNN and RMSSD, frequency-domain measures such as LF and HF power, and nonlinear descriptors such as entropy and Poincaré plot statistics [189]. These features are designed to capture autonomic nervous system regulation under stress conditions.

For EMG signals, handcrafted features typically quantify muscle activation intensity and spectral characteristics, including root mean square, waveform length, zero-crossing rate, and median or mean frequency, reflecting stress-related changes in muscle tension and neuromuscular activity [190]. EDA features mainly characterize sympathetic arousal and are often extracted from both tonic and phasic components, such as skin conductance level, skin conductance response amplitude, rise time, and recovery metrics.

Feature selection plays a key role in traditional machine learning frameworks, serving as an intermediate stage between feature extraction and classification to identify informative subsets from high-dimensional multimodal physiological data. Common strategies include statistical filtering, wrapper-based evaluation, and embedded importance estimation during model training. Based on the refined features, studies typically apply classifiers such as support vector machines, k-nearest neighbors, decision trees, random forests, logistic regression, and naive Bayes, while ensemble learning methods have been introduced to enhance robustness. For sequential physiological data, probabilistic models such as hidden Markov models and conditional random fields have been explored to capture temporal dependencies. Overall, this framework is valued for its interpretability and engineering practicality but remains sensitive to feature design, data quality, and parameter selection, which limits generalization under asynchronous or noisy multimodal conditions.

Table 8. Traditional multimodal stress classification methods with preprocessing and feature engineering details.

Author	Datasets	Signals	Extracted Features	Feature Selection	Classifier	Accuracy	F1 Score	Limitations
Torres-Valencia et al. [191]	DEAP, MAHNOB-HCI	EEG (main), GSR, HR, Resp., Temp., EMG, EOG	EEG spectral features, HRV indices, statistical descriptors	Not specified	SVM	75.17%	79.25%	Limited to SVM; binary classification only; shallow fusion; no temporal modeling; no real-time validation
Hao et al. [192]	Self-collected	EEG, PPG, EOG	Time–frequency and statistical features	Random Forest Selection (RFS)	1D-CNN + RFS	90.67%	91.47%	Small sample size; all-male cohort; limited diversity of feature selection methods
Gunawan et al. [193]	Self-built	ECG, EMG, EEG	Statistical and time-domain features	–	KNN	73.33%	–	Small sample size; unbalanced data; no F1/recall reported; no model comparison
Patil et al. [194]	BioVid Heat Pain	ECG, EDA, EMG	HRV features, EDA statistical features, EMG amplitude descriptors	Not specified	Logistic Regression	83.20%	–	F1 and recall not reported; sensitive to feature dimensionality; limited nonlinear modeling ability
Dutsinma et al. [195]	Self-built	Heart Rate, Blood Pressure	Heart rate and blood pressure statistics	–	Decision Tree	95%	–	Small sample size; limited modalities; restricted applicability
Abadi et al. [196]	Self-built	Finger pressure, hand motion, facial expression	Pressure and motion statistical descriptors	Late fusion-based aggregation	Semi-Naive Bayesian (Late Fusion)	93%	93%	Facial data loss; limited physiological integration; robustness to sensor failure needs improvement

summarizes representative studies employing traditional machine learning approaches, outlining datasets, signal modalities, classifier configurations, and key limitations to provide a comparative overview of their performance and applicability.

As shown in Table 8, reported performance varies markedly across datasets and signal combinations, which reflects instability introduced by heterogeneous feature engineering and fusion choices as well as inconsistent data splits and preprocessing [197]. Early fusion that directly merges multimodal features can inflate or depress results through redundancy and misalignment, and many studies omit key metrics such as F1 score and recall, further hindering fair comparison [198]. To improve reliability, future work should adopt standardized protocols that prevent temporal and trial leakage, use a common set of metrics, and fully disclose preprocessing pipelines, feature inventories, fusion settings, and random seeds, with code and data made accessible for replication [92].

Traditional machine learning methods rely on handcrafted descriptors such as heart rate variability and power spectral density that often transfer poorly across datasets and tasks, and most pipelines use short or static windows that miss long term dynamics and delayed physiological responses. Fusion remains sensitive to sampling heterogeneity and timing mismatches, which introduce redundancy and noise that obscure cross-modal relations. Under subject independent evaluation, such as leave one subject out, performance typically declines, revealing weak generalization to inter-individual variability and domain shifts [175,180]. Robustness is further limited by motion artifacts and intermittent modalities in real-world recordings, which degrade stability during deployment [172]. Promising directions include more physiologically grounded feature construction with redundancy control, multiscale temporal modeling, alignment-aware and missing-aware fusion, and rigorous validation under the standardized protocols outlined above.

In summary, improvements to traditional pipelines converge on three directions: constructing physiologically meaningful features with careful selection to curb redundancy and improve transferability, introducing temporal modeling with rigorous evaluation to capture nonstationary and delayed responses, and adopting adaptive fusion to handle modality heterogeneity and missing data. These steps yield incremental gains but do not overcome the core dependence on handcrafted features and shallow architectures, which limits learning of high-level nonlinear relations across modalities. As a result, deep learning has become the leading paradigm by enabling end-to-end feature learning, explicit temporal modeling, and integrated multimodal representation within a single framework.

6.2. Deep Learning-Based Approaches

Deep learning has become the primary direction for multimodal emotion and stress recognition because traditional pipelines based on handcrafted features, limited temporal modeling, and weak adaptability to modality heterogeneity cannot meet real-world requirements. Compared with manual features and shallow classifiers, end-to-end models learn temporal dynamics, spatial structure, and cross-modal interactions within a single trainable system, which improves robustness and generalization under inter-individual variability. Recent progress concentrates on attention and Transformer architectures, adaptive cross-modal alignment, self-supervised pretraining with transfer learning, and compression for edge deployment. As shown in Figure 7, the workflow integrates signal representation, hierarchical feature extraction, multimodal fusion, model training, and evaluation in one pipeline, and shared encoders with cross-modal attention enable the integration of heterogeneous physiological signals.

Deep learning approaches for multimodal emotion and stress recognition are commonly categorized into four major families: convolutional neural networks, recurrent neural networks, hybrid convolutional–recurrent models, and attention-based Transformer architectures. Emerging directions include state space models exemplified by Mamba [199], mixture of experts with dynamic routing [132], generative models such as GAN and VAE [200], self-supervised and contrastive learning, knowledge guided networks including Kolmogorov Arnold Networks [201], and graph neural networks. Convolutional models specialize in capturing local morphological patterns and are computationally efficient on short temporal windows. Recurrent architectures capture temporal dependencies across sequences. Hybrid convolutional–recurrent models integrate both local and long-range dynamics. Transformer-based architectures learn long-distance relationships and adaptively weight cross-modal interactions, although they typically require larger datasets and greater computational resources. Taken together, these model classes present distinct trade-offs with respect to accuracy, robustness, data requirements, computational cost, and practical deployability.

With respect to optimization and convergence, the reviewed classification models are commonly trained by minimizing an empirical risk objective, yet their optimization properties differ markedly across paradigms. Traditional machine learning pipelines built on handcrafted features often employ convex or quasi-convex objectives, where optimality can be more explicitly characterized and convergence is typically stable under standard solvers. In contrast, deep multimodal models are optimized using stochastic gradient-based methods on highly non-convex objectives. As a result, optimality in deep learning-based stress recognition is generally understood in a practical sense, referring to convergence toward stable solutions with good generalization performance rather than guarantees of global optima [202].

Convergence behavior in multimodal deep learning is further affected by modality heterogeneity. Differences in signal scale, sampling frequency, physiological latency, and data availability across modalities can induce gradient imbalance and modality dominance, leading to unstable training or suboptimal fusion [203]. In practice, convergence is commonly evaluated through smooth loss reduction, bounded generalization gaps on validation sets, robustness across random seeds, and the absence of gradient explosion or collapse. To improve convergence stability, existing studies adopt strategies such as modality-wise normalization, balanced or curriculum sampling, alignment-aware fusion, regularization and early stopping, gradient clipping, and uncertainty- or attention-weighted training to prevent a single modality from overwhelming the fused representation [204]. Transparent reporting of these optimization settings and convergence diagnostics is essential for reproducibility and fair comparison across multimodal stress recognition studies.

Table 9. (a) Comparison of multimodal stress classification algorithms based on deep learning (Part A). (b) Comparison of multimodal stress classification algorithms based on deep learning (Part B, continued from (a)).

(a)
Author	Datasets	Signals	Classifier	Accuracy	F1 Score	Limitations
Bahar et al. [205]	KMED, DEAP	EEG + facial image	AAT + SIFT + LBP + FLF + SVM	KMED: 89.95%, DEAP: 92.44%	–	Signal-to-image processing required; only binary classification is used; sensitive to data synchronization and frame selection
Chouinard et al. [206]	RECOLA, LM-TSST	Facial video, ECG, EDA	Decoupled Mamba Network + Time Relaxation Reconstruction Mechanism + Transformer	Only Concordance correlation coefficient: RECOLA: 0.3921, LM-TSST: 0.3774	–	Sensitive to modal differences; Limited handling of modal imbalance; Generalization depends on training distribution; Lack of multi-scenario validation
Xingchao Wang et al. [207]	DEAP, Self-built, Million Song Dataset	EEG, ECG, facial video	Multimodal LSTM + Emotional Markov Chain	Valence: 86.75%, Arousal: 83.24%	–	The granularity of emotion modeling is limited, the computational complexity is high, there are large differences between individuals, and personalized adjustments are required. Real-time performance is affected by hardware
Le Fang et al. [208]	Emo-MG, IEMOCAP, EMOTIC	EEG + micro-gesture, voice + video, image	multimodal model based on Kolmogorov v-A Arnold Network, Transformer variant with KAN attention mechanism	Emo-MG: 83.54%, IEMOCAP: 72.16%, EMOTIC (Valence dim.): 73.41%	–	The test speed is slightly slower; the IEMOCAP data is still lower than some SOTA models (such as CORRECT); the Transformer version is more complicated to calculate
Sathi-shkumar Moorthy et al. [209]	AffWild2, AFEW-VA, IEMOCAP	Speech audio, spectrogram, MFCC, facial images video	Hybrid Multi-Attention Network, Contains CSSA + HASPCM modules	IEMOCAP: 75.39%	–	The model structure is complex; it consumes a lot of resources; it takes a long time to train; it depends on the quality of annotations
Erdem et al. [210]	DEAP	EEG+ Facial expression video	GRU(main), LSTM, Transformer	Single modality GRU: 91.8%,Multimodality: 97.8%	GRU: 97%	The Transformer model is complex and requires a large amount of data; facial expression data is affected by the quality of the recording, some of it needs to be synthesized and supplemented
Ao Li et al. [131]	DEAP, WESAD	EEG, ECG, EMG, EOG, GSR	CovNet + Attention Fusion Model + Conditional Self-Attention GAN + CovNet Classifier	DEAP: 96.06%, WESAD: 95.70%	DEAP: 95.80%, WESAD: 95.45%	High reliance on generative models; requires high-computing equipment; lacks verification in real complex environments; scalability and cross-domain adaptability need further research
(b)
Author	Datasets	Signals	Classifier	Accuracy	F1 Score	Limitations
Kaveti Pavan et al. [211]	IIT Hyderabad in-house controlled driving dataset	ECG, EDA	1D-CNN with cross squeeze-and-excitation attention; LOSOCV	76.54% ECG→EDA attention; 76.37% LOSOCV mean	73.02% (best)	Small, single-site, all-male cohort; two-class setup; short windows; device- and scenario-specific; no benchmarking on public datasets; limited evidence for real-world generalization and edge deployment latency/energy.
Axel Gedeon et al. [118]	ASCERTAIN and KEMDy20	ECG, EEG, EDA, TEMP, IBI, EMO, Audio signal, Text	Cross-Attention Gated Mixture of Experts.	Arousal: 99.71%, Valence: 99.71%	Arousal: 99.83%, Valence: 99.76%	The model is highly complex, has a large number of parameters, and requires high hardware resources; it requires multimodal and complete input; its generalization ability has not been verified in real environments; and its model interpretability is limited.
Yekta et al. [212]	SWEET, DAPPER, LabToDaily	EDA, PPG, TEMP, ACC	CNN-complex	–	SWEET: 97.80%	The signal is severely affected by motion artifacts, the model is insensitive to motion state, the Transformer module is not effective when the data scale is insufficient, requires a long time window, and is not suitable for real-time emotion detection. Self-supervision requires a large amount of unlabeled data, and the training cost is high.
Thaduri et al. [213]	Stress-Lysis	ECG, EDA, Behavioral signals, Environmental signals	GNN-T-GCN	92.3%	–	The generalization ability is not verified. It requires a lot of computing resources. The mixing of environmental signals and physiological signals may introduce noise. It does not deal with motion artifacts and lost data in real wearable devices. It has not been verified under real-time monitoring conditions.

a,b indicate that recent multimodal stress recognition studies include a wide variety of datasets, signal combinations, and architectural paradigms. Overall, deep learning models demonstrate clear advantages in automatically learning hierarchical and cross-modal representations without manual feature engineering. Convolutional neural networks effectively extract local morphological features and are suitable for short-window and edge-oriented analysis, while recurrent and hybrid CNN–RNN architectures capture temporal dependencies in non-stationary physiological signals. Attention-based and Transformer models achieve adaptive weighting and long-range dependency learning, facilitating dynamic cross-modal alignment and feature interaction. State-space models such as Mamba enable efficient long-sequence modeling with reduced memory consumption, and mixture-of-experts or dynamic gating structures enhance robustness under modality imbalance and individual variability. Self-supervised and contrastive learning approaches improve data efficiency and cross-device generalization, whereas graph neural networks and generative models contribute to structural modeling and data augmentation, addressing label scarcity and class imbalance in complex multimodal scenarios.

Despite these advances, deep learning models still face challenges in data dependence, generalization, and practical deployment. Their performance often relies on large, high-quality datasets and extensive computational resources, which limit reproducibility and accessibility across research environments. Cross-subject and cross-device variability remains a key obstacle to robust generalization, and overfitting to specific experimental settings is still prevalent. Moreover, most frameworks require complete and synchronized modalities, reducing their reliability under signal loss or asynchronous acquisition. Future development should therefore emphasize unified cross-modal representation learning, adaptive model calibration, and efficiency-oriented architectures that balance accuracy with resource constraints. Additionally, incorporating explainability and uncertainty estimation will be critical to improving transparency and trustworthiness in real-world applications.

In summary, deep learning has become the dominant paradigm for multimodal emotion and stress recognition, offering superior feature abstraction and adaptive fusion capabilities compared with traditional methods. However, its benefits are not absolute, as data requirements, computational cost, and interpretability remain open issues. The following section systematically compares traditional and deep learning paradigms in terms of performance, generalization, complexity, and practicality, providing a structured foundation for subsequent model selection and optimization.

6.3. Comparison and Analysis

In order to deepen the comparison of classification methods for multimodal stress recognition, this section juxtaposes traditional machine learning with deep learning based models. Building on the representative studies summarized in Table 8 and Table 9a,b, we evaluate both paradigms across data efficiency, interpretability, fusion depth, alignment capability and deployability.

Table 10. Comparison between traditional and deep learning-based methods for multimodal stress recognition.

Modals	Feature Method	Data Scale	Interpre-Tability	Fusion Type	Alignment	Deployment Cost	Generalization	Sensitivity
Traditional Machine Learning	Manual feature extraction	Can work with small datasets	High	Mostly early or decision-level fusion	Requires manual synchronization or preprocessing	Low computational resources required; suitable for edge devices	Relatively robust in low-variance tasks	More reliant on individual feature quality
Deep Learning	Automatic feature learning via deep networks	Requires large-scale, labeled data for optimal performance	Low; needs XAI tools such as SHAP or Grad-CAM	Supports mid-level fusion via attention or KAN mechanisms	Can model inter-modal dynamics and delays	High computational cost; lightweight models needed for deployment	Sensitive to overfitting without regularization or transfer learning	Better at capturing high-order correlations between modalities

distills the key contrasts and offers practical guidance for model selection and future research.

Recent studies increasingly reveal a widening gap between traditional machine learning and deep learning for multimodal emotion and stress recognition. Niu et al. demonstrated an interpretable machine learning framework that integrates EEG, ECG, and clinical variables and highlights the value of transparent physiological modeling [214]. On consumer devices, Khuntia et al. achieved high accuracy by fusing wearable ECG with speech using a deep model [215]. For cross modal correlation learning, Zhao and collaborators introduced a contrastive autoencoder-based fusion approach that strengthens inter-modality representation consistency [129]. In parallel, Feng and colleagues explored deep learning with Internet of Things pipelines for real-time monitoring and personalized ECG analysis [216]. Methodological guidance further suggests accounting for redundancy and mutual correlation when designing multimodal fusion strategies [217]. Together, these studies underscore the advantages of deep models in cross-modal modeling and fusion.

Deep learning offers clear benefits in representation learning and cross modal integration, yet important gaps remain in practice. Generalization is limited under small samples and inter subject variability, robustness is fragile under missing or asynchronous modalities and motion artefacts, interpretability and uncertainty are weak, and latency, energy, and memory costs challenge edge deployment. Contributing factors include scarce labels, heterogeneous preprocessing and evaluation protocols, and barriers to reproducibility. Addressing these issues points to several directions: self supervised pretraining to mitigate data scarcity, domain adaptation and personalization for cross subject transfer, robust objectives with explicit missing modality learning for reliability, model compression and scheduling for resource constrained deployment, and physiology informed modeling with explainability and calibrated uncertainty under standardized and reproducible protocols.

7. Key Challenges in Multimodal Deep Learning

Although multimodal deep learning enhances emotion and stress recognition by exploiting complementary physiological signals, it still faces fundamental challenges in robustness, generalization, and real-time applicability. These arise from intrinsic modality differences, asynchronous dynamics, limited data availability, and opaque model behavior. Understanding these challenges is essential for guiding future improvements in multimodal affective computing.

Despite remarkable advances in multimodal deep learning, emotion and stress recognition systems still face persistent challenges that restrict their robustness and practical scalability. First, modal heterogeneity and structural incompatibility among physiological signals lead to information redundancy, feature misalignment, and negative transfer across modalities. Second, temporal asynchrony between fast-response signals and delayed modalities causes semantic mismatches and unstable fusion, further aggravated by device-level sampling discrepancies. Third, the field remains highly dependent on large-scale labeled datasets, yet physiological data collection and annotation are costly and subjective, resulting in small-sample bias, label imbalance, and limited cross-dataset transferability. Moreover, poor generalization and inter-subject variability expose models to distribution shifts across individuals, devices, and experimental contexts, while fusion dependency makes model performance fragile under modality dropout, noise, or domain drift. In addition, the lack of interpretability in deep multimodal architectures hinders transparency and clinical trust, and real-time deployment is constrained by computational complexity, latency, and energy limitations in wearable environments. Collectively, these challenges highlight the gap between laboratory accuracy and real-world applicability, emphasizing the need for adaptive, explainable, and resource-efficient multimodal frameworks.

Table 11. Summary of key challenges in multimodal deep learning for emotion and stress recognition.

Challenge	Manifestation	Impact	Key Points	Solutions
Modal heterogeneity and structural incompatibility	Inconsistent sampling rates, temporal–spectral characteristics, and dynamic ranges across modalities cause misaligned features and redundant information.	Reduces discrimination and generalization; leads to negative transfer and unstable convergence.	Heterogeneous signals differ in temporal response and noise; shared encoders fail to capture modality-specific semantics.	Modality-specific encoders with shared–private embedding spaces; adaptive normalization and attention weighting; confidence-aware fusion; calibration-aware models [218].
Temporal asynchrony between modalities	Physiological and device-level delays cause unsynchronized data streams.	Leads to semantic mismatches, feature redundancy, and cross-modal interference; affects temporal generalization.	Static alignment cannot adapt to dynamic tasks; physiological lags and rating delays persist.	Dynamic and Soft-DTW alignment; asynchronous attention; adaptive delay modeling; cross-modal temporal calibration [219].
Dependence on large-scale labeled datasets	Physiological data collection and annotation are costly and subjective, leading to small-sample imbalance and missing modalities.	Overfitting and unstable training; poor scalability under real-world data scarcity.	Annotation subjectivity and limited public datasets restrict model capacity; multimodal imbalance worsens with missing data.	Self-supervised and contrastive learning; few-shot learning; incremental and federated adaptation; multi-expert Transformer with sparse gating [118].
Poor generalization and small-sample adaptability	Strong inter-subject variability and inconsistent experimental paradigms cause domain shift and poor cross-dataset transfer.	Degraded performance under unseen subjects, devices, or contexts.	Differences in physiological baselines and sensor configurations hinder subject-independent modeling.	Domain adaptation; invariant representation learning; meta-learning for personalization; contrastive reinforcement transfer learning [220].
Fusion dependency	Model performance heavily depends on complete, synchronized, high-quality multimodal inputs.	Missing or noisy modalities cause cascading attention mismatches and unstable predictions.	Deep fusion assumes modality complementarity and shared latent space; lacks physiological priors.	Uncertainty-aware fusion; modality dropout; reliability weighting; self-supervised multimodal contrastive learning [221].
Lack of interpretability	Deep models act as black boxes; decisions lack physiological or psychological explainability.	Limits trust, clinical adoption, and ethical approval.	SHAP and LIME improve transparency but remain locally inconsistent and computationally costly.	Embedded explainable modules; multimodal causal analysis; model-agnostic and differentiable interpretability frameworks [222].
Real-time deployment and efficiency	High latency, energy consumption, and limited compute capacity on edge devices hinder real-time response.	Prevents large-scale and wearable implementation; increases delay and power usage.	Multi-branch architectures and cross-modal attention raise complexity; quantization may reduce precision.	Model pruning, quantization, knowledge distillation; dynamic inference; edge–cloud co-inference; lightweight CNN or Transformer architectures [223].

summarizes these prominent challenges, their manifestations, impacts, and representative solution directions.

Building on Table 11, recent research converges on structure-aware fusion, which combines modality-specific encoders with shared and private embeddings, cross-modal attention, and adaptive gating to address heterogeneity, as well as on adaptive temporal alignment to accommodate latency mismatches. Data-efficient learning and cross-domain generalization approaches target small samples and distribution shift. To curb fusion dependency, uncertainty-aware reliability weighting, modality dropout, and robustness- or causal-oriented regularization help stabilize decisions under noise and missing inputs. Interpretability is enhanced through embedded explanatory modules and multimodal causal analysis, while real-time deployment leverages lightweight architectures alongside pruning, quantization, knowledge distillation, and edge–cloud co-inference to meet latency and energy budgets. Collectively, these directions prioritize adaptive, explainable, and resource-aware design, narrowing the gap between laboratory performance and real-world application.

In summary, the effectiveness of multimodal deep learning for emotion and stress recognition depends not only on achieving high accuracy but also on ensuring robustness, interpretability, and efficiency under real-world conditions. Future research should prioritize adaptive fusion architectures, data-efficient learning, and hardware-aware deployment to bridge the gap between laboratory performance and practical applications. Establishing standardized benchmarks across devices, subjects, and environments will be essential for the sustainable advancement of multimodal affective computing.

8. Discussion and Future Directions

This review has systematically examined recent advances in multimodal emotion and stress recognition, with a particular focus on physiological signals such as ECG, EMG, and EDA and their integration through machine learning and deep learning-based fusion strategies. By organizing existing studies across physiological foundations, public and self-collected datasets, preprocessing pipelines, feature extraction, fusion paradigms, and classification models, we highlighted both the strengths and limitations of current approaches. Despite notable progress, persistent challenges—including modality heterogeneity, temporal asynchrony, limited data scale, weak cross-subject generalization, and deployment constraints—continue to restrict real-world applicability and reproducibility.

Overall, the reviewed evidence indicates that multimodal fusion can substantially enhance stress recognition performance when modalities are complementary and well aligned, but may degrade performance under noisy, weakly informative, or data-limited conditions. Deep learning-based models offer superior representation learning and adaptive fusion capabilities compared with traditional machine learning pipelines, yet their benefits are often offset by high data dependence, computational cost, and limited interpretability. These findings underscore the importance of principled fusion design, transparent evaluation, and robustness-oriented modeling in physiological stress analysis.

Looking forward, future research should prioritize a small number of key directions to strengthen translational impact: the development of standardized and ecologically valid multimodal datasets; robust and lightweight fusion architectures that balance accuracy with efficiency; alignment-aware and uncertainty-aware learning to handle modality heterogeneity and missing data; and evaluation protocols that explicitly consider generalization, robustness, and deployability.

In conclusion, multimodal emotion and stress recognition remains a promising yet challenging field at the intersection of affective computing and intelligent healthcare. By consolidating current knowledge and identifying critical methodological gaps, this review provides a structured foundation for developing reliable, interpretable, and scalable stress recognition systems that can move beyond laboratory settings toward real-world deployment.