Cross-Modal Attention Fusion: A Deep Learning and Affective Computing Model for Emotion Recognition

Abstract

Artificial emotional intelligence is a sub-domain of human–computer interaction research that aims to develop deep learning models capable of detecting and interpreting human emotional states through various modalities. A major challenge in this domain is identifying meaningful correlations between heterogeneous modalities—for example, between audio and visual data—due to their distinct temporal and spatial properties. Traditional fusion techniques used in multimodal learning to combine data from different sources often fail to adequately capture meaningful and less computational cross-modal interactions, and struggle to adapt to varying modality reliability. Following a review of the relevant literature, this study adopts an experimental research method to develop and evaluate a mathematical cross-modal fusion model, thereby addressing a gap in the extant research literature. The framework uses the Tucker tensor decomposition to analyse the multi-dimensional array of data into a set of matrices to support the integration of temporal features from audio and spatiotemporal features from visual modalities. A cross-attention mechanism is incorporated to enhance cross-modal interaction, enabling each modality to attend to the relevant information from the other. The efficacy of the model is rigorously evaluated on three publicly available datasets and the results conclusively demonstrate that the proposed fusion technique outperforms conventional fusion methods and several more recent approaches. The findings break new ground in this field of study and will be of interest to researchers and developers in artificial emotional intelligence.

1. Introduction

Artificial emotion recognition is a subfield of affective computing that empowers machines to perceive, interpret, and respond to human emotions. Ekman’s Theory of Basic Emotions identifies six universal emotions: anger, fear, sadness, disgust, happiness, and surprise [1]. As an essential component of human–computer interaction, emotion recognition plays a pivotal role in diverse applications, such as teaching and learning [2], online gaming [3], medical diagnostics [4], and decision-making assistance [5]. Conventional emotion recognition techniques have relied on handcrafted feature extraction processes applied to unimodal inputs, such as speech signals [6] or facial expressions [7]. Standard features such as pitch [8], energy, spectral properties [9], facial geometric features, facial landmarks [10], expression descriptors, and facial action coding units [11] have achieved some successes, but in general have often failed to represent the complexity and variability in human emotions in real-world scenarios.

The advent of deep learning has transformed this field of study by enabling models to extract hierarchical and discriminative features from raw data automatically. Popular deep learning architectures such as Convolutional Neural Networks (CNNs) [12], Recurrent Neural Networks (RNNs) [13], Long Short-Term Memory (LSTM) [14], and Bidirectional LSTM (Bi-LSTM) [15] have demonstrated strong capabilities in capturing spatial and temporal dynamics [16]. In particular, cross-modal emotion recognition, which integrates information from heterogeneous modalities, has emerged as a promising direction due to the complementary nature of these signals.

Cross-modal emotion recognition typically involves two core components: feature extraction [17] and feature fusion [18]. Feature extraction aims to derive salient and high-level representations from raw inputs, while fusion seeks to integrate these heterogeneous features into a unified representation. While traditional fusion strategies, such as early fusion (feature-level concatenation) [19], late fusion (decision-level concatenation) [20], and hybrid fusion [21] have achieved reasonable success, they are fundamentally limited in that they often rely on shallow architectures and struggle with noise or incomplete data from any one modality. Feature fusion preserves the information from each modality to a great extent, but it often introduces challenges such as temporal misalignment between modalities and a heightened risk of overfitting, due to the high dimensionality of the combined feature space.

In this context, this paper addresses the following research questions (RQs):

RQ1. To what extent do affective computing and recent advancements in deep learning improve cross-modal fusion for audio and image-based emotion recognition?

RQ2. Can a new model be developed and validated to improve correlations between heterogeneous modalities (such as audio and visual data) to detect and interpret human emotional states?

Following this brief introduction, Section 2 outlines the research methodology and more specifically describes the experimental design, deep-learning model architecture, and hyperparameter configurations. Section 3 sets out the main results of this study, whilst emergent issues are analysed and discussed in Section 4. Finally, Section 5 summarises the contribution of this research, points out limitations, and notes possible future research initiatives in this field of study.

2. Materials and Methods

This study comprises two main phases which combine different research methods and philosophies (Figure 1). In Phase 1, an interpretivist philosophy is assumed, combined with an inductive qualitative approach to assess the extant literature and develop a provisional conceptual framework as context for the subsequent primary research phase. Gill and Johnson [22] suggest that such an approach is most appropriate when the research aim is exploratory in nature, as was the case in Phase 1 of the project when the objective was to explore the relevant literature on affective computing. Then, in Phase 2, a positivist stance is adopted for the conduct and evaluation of the experiment. More detail on these two phases is provided below.

2.1. Narrative Review

Phase 1 comprised a narrative review of the relevant literature to identify key themes and provide the basis for development of a provisional conceptual framework (PCF) for the conduct of the experimental phase. Phase 1 was a “broad scan of contextual literature” through which “topical relationships, research trends, and complementary capabilities can be discovered” [23] (p. 351), allowing for the identification of the relevant sources and mapping of key concepts [24]. Such reviews are also sometimes termed “scoping reviews” [25] or “integrative reviews” [26], “with the aim to assess, critique, and synthesize the literature on a research topic in a way that enables new theoretical frameworks and perspectives to emerge” [26] (p. 335).

This review thus aimed to map recent developments in cross-modal fusion methods for audio–visual emotion recognition. Between May and September 2025, searches were conducted across IEEE Xplore, ACM Digital Library, Scopus, Web of Science, SpringerLink, and ScienceDirect, covering the period 2014–2025. Keywords included “cross-modal fusion”, “multimodal fusion”, “audio–visual deep learning”, “attention fusion”, “low-rank fusion”, “tensor fusion”, “tensor decomposition”, and “bilinear pooling”. This review focused on peer-reviewed studies aligned with audio, visual, multimodal, and cross-modal emotion recognition, particularly those employing deep learning, attention mechanisms, and low-rank tensor approximation techniques. Non-peer-reviewed works, unimodal studies, non-English papers, and studies without machine-learning-based fusion were excluded. A total of 178 records were identified, 28 duplicates were removed, 150 articles were screened, and 92 full-text papers were examined. Of these, 68 studies met the inclusion criteria, and these provided references to other studies, which were then assessed, which, in turn, provided further relevant sources. Studies of electrophysiological data from the brain (EEG) and heart (ECG) to detect emotional state were not considered. Such studies require specialized sensors and controlled environments, whilst this research focuses on dialog-based audio–video modalities that analyze external behavioral cues, such as facial expressions and vocal tone, for naturalistic emotion recognition. These represent distinct research domains with different objectives and data characteristics, and so EEG/ECG-based studies were intentionally excluded from the scope of the literature review.

An assessment of the literature supported the development of the PCF. A PCF not only gathers concepts but also integrates them into one single structure. The goal is to find factors, attributes, variables, behavior, processes, etc. that provide an initial analytical frame for subsequent primary research. According to Levering [27], a PCF is a good starting point to explain and examine a wider subject area. Similarly, Jabareen [28] suggested that a conceptual framework can provide an analytical network for investigating a particular phenomenon. The selection of concepts is based on the number of occurrences in a text, their meaning, and relevance to the research objectives or questions.

2.2. Experimental Design and Validation

Phase 2 focused on experimental development and evaluation of a novel cross-modal fusion using Tucker decomposition. Such experiments can be seen as “a method of gathering information and data on a subject through observation in controlled settings” [29]. Symbols and notations included in the equations and algorithms are shown in the abbreviation section at the end of this article. In emotion recognition, cross-modal fusion is motivated by the fact that emotional cues are distributed across multiple modalities. Relying on a single modality can lead to ambiguity, whereas combining audio and visual information enables the model to exploit complementary emotional signals for improved accuracy and robustness [30].

However, direct outer-product fusion of multimodal features produces a high-dimensional tensor with significant redundancy. To address this, Tucker decomposition [31] was employed, this being a powerful multilinear factorization technique that compresses the cross-modal tensor into a low-rank core tensor and learnable projection matrices. This allows the model to capture the most salient inter-modal correlations while maintaining computational efficiency. The Tucker framework provides flexibility in rank selection per mode and supports end-to-end learning, making it particularly suitable for fine-grained audio–visual emotion classification [30,32,33].

Cross-modal feature interaction and the associated feature extraction methods were used, combining information from two heterogeneous sources: audio and video frames used as modalities. Audio features capture vocal tone, referring to pitch, timbre, and modulation patterns that convey emotional cues such as anger, sadness, or happiness, while intensity reflects the loudness or energy level of speech, often associated with emotional arousal. In contrast, video frames convey facial expressions and gestures. Since audio and visual features represent distinct modalities, combining their features enables the model to capture richer and more complete information than using either alone in emotion recognition tasks. Appendix A provides details on the model architecture and hyperparameter configuration.

Figure 2 depicts the experimental design. Regarding audio feature extraction (left side in Figure 2), the 2D convolutional neural network processes the audio modality, specifically designed to extract high-level features while preserving their temporal sequence. This network operates on Mel-spectrogram inputs to learn emotion-relevant time–frequency patterns. The 2D convolutional layers apply small learnable filters that slide across the spectrogram to detect localized changes in pitch, energy, and spectral shape associated with emotional expression. Layers with different kernel sizes and strides capture patterns at multiple temporal and spectral scales, while batch normalization ensures training stability and accelerates convergence. The ReLU activation introduces non-linearity, allowing the model to learn complex mappings between acoustic cues and emotional states. Unlike models that collapse an entire clip into a single representation, this CNN preserves the sequential structure of the audio, producing a sequence of high-level features that reflect the evolution of emotion over time. The network comprises a series of 2D convolutional layers with varying kernel sizes and strides, followed by batch normalization and a non-linear activation function (ReLU). The detailed architecture is provided in Appendix A (

Table A1. Hyperparameters for 2D CNN and ResNet-18 model.

Audio Modality	Video Modality
2D CNN	Resnet-18
Input size = (Batch size, 128 × 256)	Input size = 224 × 224 × 3
Kernel size = 3	Kernel (conv layers) = 7 × 7, 3 × 3, 1 × 1
Activation function = ReLU	Activation function = ReLU
Max pooling = 3 × 3, stride = 2	Max pooling = 3 × 3, stride = 2
Batch size =128	Batch size = 128
Epochs = 50–100	Epochs = 50–100
Learning rate = 0.0001 to 0.001	Learning rate = 0.0001 to 0.001
Optimizer = Adam	Optimizer= Adam
Loss Function = Categorical Cross-Entropy	Loss Function = Categorical Cross-Entropy
Audio Augmentation = Pitch shifting, time stretching, background noise	Video Augmentation Horizontal flip, rotation, random cropping
Synchronization = Temporal alignment of audio and video segments	Synchronization = Temporal alignment of audio and video segments
Output dimension = 128	Output dimension = 512
Latent Space dimension = 60 × 256	Latent Space dimension = 90 × 256
Random 80:20 Split
Number of Attention_head = 4
Dropout_rate = 0.5
Weight_decay= 1 × 10−4

). This architecture ensures that the output for each audio segment is not a single aggregated vector, but rather a high-dimensional audio feature sequence, as expressed in Equation (1):

$$F_{audio\_seq}\in {\mathbb{R}}^{T_{audio}\times D_{audio\_per\_step}}$$

(1)

Here, $T_{audio}$ represents the number of temporal steps (or frames) remaining after convolutions and pooling, and $D_{audio\_per\_step}$ represents the feature dimension at each of these time steps. The 2D convolutional neural network processes the audio modality, extracting high-level time–frequency features while preserving their temporal order. Raw audio segments are input as tensors of shape (batch size, sequence length), and the convolutional layers operate along both time and frequency axes. Rather than collapsing the entire segment into a single feature vector, the network outputs a sequence of frame-level feature embeddings. Preserving this temporal structure allows the model to represent variations in pitch, intensity, and rhythm are key aspects of emotional prosody that unfold over time. Such sequential representations are crucial for capturing dynamic emotional transitions and for synchronizing temporal cues with the corresponding visual frames in the subsequent fusion stage.

In terms of visual feature extraction (right side in Figure 2), individual video frames are processed as tensors of shape (batch size, height, width, channels). Batch size denotes the number of samples processed simultaneously during training or inference. Height and width correspond to the spatial dimensions of each feature map. Channels indicate the number of feature maps or filters, capturing different aspects of the input (e.g., color channels in RGB images or feature channels in convolutional feature maps). A pre-trained ResNet-18 model [34], initialized with ImageNet weights [35], is the backbone for extracting robust spatial features from each frame. The pre-training on the large-scale ImageNet dataset enables the network to learn generic low- and mid-level visual features (e.g., edges, textures, shapes), which are then fine-tuned on the emotion recognition dataset to adapt to domain-specific facial expression patterns.

The final classification layer of the pre-trained ResNet-18 model was removed because the objective is to extract discriminative visual features, rather than performing ImageNet classification. After removing the last fully connected (FC) layer, a 512-dimensional embedding from the global average pooling layer is obtained, which represents each input frame in a compact and informative manner. This feature vector encodes high-level spatial and semantic information (e.g., eye movement, mouth shape, and facial muscle tension), which is subsequently aligned and fused with the corresponding audio features for emotion classification. This approach enables transfer learning, where the lower layers capture general visual patterns while the final embedding reflects task-specific emotional cues.

The final classification layer of ResNet-18 is removed, allowing the network to output a high-dimensional visual feature vector for each frame. To prepare for temporal fusion, these frame-level features are then explicitly stacked along a temporal dimension to form a visual feature sequence for the entire video utterance, as shown in Equation (2):

$$F_{image\_seq}\in {\mathbb{R}}^{T_{image}\times D_{image\_per\_frame}}.$$

(2)

Here, $T_{image}$ represents the number of frames in the utterance, and $D_{image\_per\_frame}$. It is the feature dimension extracted by ResNet-18 for each frame.

The sequence of frame-level visual features is subsequently processed by a Bidirectional Gated Recurrent Unit (Bi-GRU) to model temporal dependencies across the video frames. The Bi-GRU enables the network to capture how facial expressions evolve over time—for example, how a neutral expression gradually transitions into a smile or frown. By incorporating information from both past and future frames, the Bi-GRU learns contextual patterns such as intensity buildup, duration, and smooth emotional transitions, which cannot be inferred from isolated frames alone. This temporal modeling complements the spatial representations extracted by ResNet-18, thereby strengthening the system’s ability to recognize emotions in continuous video sequences. Instead of using only the final hidden state, the sequence of hidden states produced by the Bi-GRU is retained to preserve frame-wise temporal context for downstream fusion. The resulting temporally refined visual feature sequence is denoted in Equation (3):

$${F’}_{image\_seq}\in {\mathbb{R}}^{T_{image}\times D_{image\_per\_frame}}$$

(3)

This output represents the temporally refined visual feature sequence, which is subsequently used for cross-modal alignment and tensor construction. The cross-modal alignment step synchronizes the temporally refined visual features with their corresponding audio features at each common time step, ensuring consistent pairing across modalities. The aligned features are then used to construct a cross-modal tensor that captures fine-grained interactions between the two modalities for Tucker decomposition–based fusion (see Appendix B, Algorithm A2 for detailed steps and formulations).

Independent deep neural networks, namely a 2D CNN for audio and Resnet-18 for video, extract high-level, discriminative features from each modality. The deeper convolutional layers of ResNet-18 generate abstract, high-level representations that capture semantic and expression-specific cues, such as facial muscle activation patterns, eye and mouth region dynamics, and texture variations associated with different emotions. These features are termed “discriminative” because they enable the model to distinguish between visually similar emotional states (e.g., happiness vs. surprise) by emphasizing subtle yet meaningful spatial patterns. Moreover, these high-level visual embeddings facilitate cross-modal alignment with audio features during the fusion stage, as they encode semantically rich information that complements vocal attributes such as tone, intensity, and rhythm. This semantic alignment between modalities enhances the model’s ability to form consistent and context-aware multimodal representations, leading to improved emotion recognition accuracy. In the proposed work, the extracted feature vectors from the audio ${(F}_{audio})$ and visual ${(F}_{image})$ modalities are fused using an outer product, which overcomes the limitations of linear fusion by modeling all pairwise correlations between audio and visual features. This results in a richer and more discriminative joint representation for emotion classification.

This proposed work enhances the fusion process by utilizing a cross-modal attention mechanism, which enables the model to dynamically learn which parts of one modality are most relevant when considering the other. The motivation for using this mechanism is to allow for context-aware feature refinement, where the model selectively emphasizes emotionally salient cues and suppresses noisy or less informative ones. This process helps ensure better temporal and semantic alignment between modalities before constructing the joint tensor for Tucker decomposition. It refines the modality-specific feature vectors, producing contextually richer representations: ${(F’}_{audio})$ and ${(F’}_{image})$. The model selectively emphasizes emotionally salient cues and suppresses noisy or less informative ones. The proposed model incorporates a cross-modal attention mechanism to facilitate interaction between the audio and visual modalities. This mechanism is designed to allow the model to learn the relevance of features across modalities, such that information from one stream can modulate the representation of the other. Through this process, the audio and visual feature vectors ${(F’}_{audio})$ and ${(F’}_{image})$ are contextually adjusted based on inter-modal correlations prior to fusion.

The attention operation serves to focus computational resources on the most informative regions in each modality, which theoretically contributes to more coherent multimodal feature representations before the subsequent Tucker decomposition. Additionally, the mechanism is structured to mitigate the influence of less informative or noisy features by adaptively weighting their contributions, thereby improving the stability of the fusion process.

This approach helps create a more contextually rich interaction before the subsequent Tucker decomposition, which is employed to efficiently model high-order correlations between the refined audio and visual features. The Tucker framework decomposes the cross-modal tensor into a low-rank core tensor and projection matrices, thereby capturing fine-grained interactions while mitigating the high dimensionality produced by the outer product fusion. This step provides a compact and expressive joint representation for emotion classification.

Another key advantage of the cross-modal attention mechanism is its ability to handle situations where one modality (e.g., audio or visual) becomes noisy or less informative. For example, in spontaneous or acted emotion datasets, visual cues may be compromised by poor lighting conditions, head movements, or partial facial occlusion, while audio signals may contain background noise, overlapping speech, or inconsistent vocal intensity.

In such cases, the attention mechanism adaptively down-weights the contribution of the unreliable modality, relying more heavily on the complementary one to preserve emotional consistency and enhance robustness. During training, attention coefficients are computed using a similarity function between the query and key representations, followed by a Softmax normalization that ensures the weights sum to one. Features associated with noisy or less informative inputs naturally receive lower attention scores, thereby reducing their influence in the fused representation. For example, when background noise affects an audio segment or a frame suffers from facial occlusion, the corresponding attention weights for that modality decrease, enabling the model to emphasize the cleaner, more informative modality. This dynamic weighting enhances both the robustness and interpretability of the emotion recognition process.

Moreover, the learned attention weights can provide interpretive insight into which features or modality interactions contribute most to the model’s emotion recognition decision. Attention weights do not offer full interpretability, but can provide indicative insights into the regions or modalities the model focuses on during decision-making. Such qualitative observations can help identify which audio–visual interactions are emphasized during emotion recognition, offering a limited but useful perspective on the model’s internal behavior.

In the proposed cross-modal attention mechanism, the sequential features from one modality act as the Query (Q), while those from the other modality serve as Key (K) and Value (V). Rather than merely exchanging information, this mechanism enables the model to compute a relevance score between every element of the two modalities using a similarity function (e.g., the dot product), which is then normalized through a Softmax function to produce attention weights. These weights determine how strongly each element in one modality should attend to the other, allowing the model to emphasize emotionally consistent cues and suppress irrelevant or noisy signals. The process is performed in both directions, with visual attention to audio and audio attention to visuals, to generate context-enriched feature sequences that are temporally aligned and semantically refined before the Tucker decomposition stage. A detailed description of this mechanism, including attention head configuration and embedding dimensions, is provided in Appendix A.3.

The audio feature sequence ($F_{audio\_seq})$ acts as the Query (Q) and attends to the visual feature sequence ($F_{image\_seq})$ (Key/Values), producing the attention-weighted visual feature representation (${F’}_{image\_seq}$). Here, “querying” means that each audio time-step feature evaluates the relevance of all visual frame features to determine which visual cues correspond most closely to that segment of audio (e.g., matching voice pitch or tone with facial expression). The attention mechanism computes these similarity scores using a dot-product between query and key representations, normalized through Softmax, producing attention weights that emphasize the most relevant visual features for each audio step. This allows the model to highlight visual cues and determine which ones correspond most closely to a specific segment of audio (e.g., matching voice pitch or tone with facial expression). The model computes attention scores that represent the similarity between each visual frame and all audio time-step features. These scores quantify the strength of the relationship between each visual feature and each audio feature, based on learned semantic and temporal correlations.

Conversely, the visual feature sequence $F_{\mathrm{image}\_\mathrm{seq}}$ serves as the Query and attends to the audio feature sequence $F_{\mathrm{audio}\_\mathrm{seq}}$, generating the attention-weighted audio representation ${F’}_{\mathrm{audio}\_\mathrm{seq}}$. The model computes attention scores that represent the similarity between each visual frame and all audio time-step features.

This process allows every visual time-step to selectively emphasize the most relevant portions of the audio sequence, for instance, associating lip movements or facial tension with corresponding pitch, intensity, or rhythm variations. In this manner, the audio representation is refined in a context-aware and temporally aligned manner before Tucker decomposition.

The attention mechanism in the model is designed to re-weight features across time and modality without altering their dimensional structure. Maintaining the same temporal ${(T}_{common})$ and feature $(D_{audio}$, $D_{image})$ dimensions ensures that:

• Temporal alignment between audio and visual modalities is preserved, allowing for one-to-one correspondence across time steps.
• Tensor construction for Tucker decomposition remains consistent, since outer product fusion requires matching time-step alignment, and
• Feature interpretability and spatial integrity are retained, avoiding information loss from dimensional projection.

These attention-weighted feature sequences $({F’}_{audio\_seq}\mathrm{and}{F’}_{image\_seq}$) retain the original sequential and feature dimensions because the attention mechanism modifies the importance weighting of features rather than their dimensionality. Preserving these dimensions ensures that both modalities remain temporally aligned ${(T}_{common})$ and structurally compatible for subsequent tensor construction in the Tucker decomposition stage. This design allows the model to encode inter-modal dependencies while maintaining the temporal order and feature-space integrity of each modality, enabling accurate outer-product fusion without loss of correspondence. Specifically, the refined audio and visual sequences are defined in Equations (4) and (5):

$${(F’}_{audio\_seq}\in {\mathbb{R}}^{T_{common}\times D_{audio\_per\_timestep}})$$

(4)

$${(F’}_{image\_seq}\in {\mathbb{R}}^{T_{common}\times D_{image\_per\_frame}})$$

(5)

However, the attention mechanism dynamically learns and updates its weighting values based on the similarity between feature representations from the two modalities. At each time step, attention scores are computed as the dot-product similarity between the query and key projections and are normalized using Softmax to form a probability distribution over all possible interactions. These weights are optimized during training, allowing the model to gradually emphasize strongly correlated audio–visual pairs (e.g., raised pitch aligned with smile intensity) and de-emphasize weak or irrelevant ones. Consequently, the attention mechanism highlights inter-modal correlations and captures contextual relevance over time, producing refined feature sequences that are then fused to construct the higher-order tensor for Tucker decomposition.

Tucker decomposition is applied to this cross-modal tensor to reduce redundancy and computational complexity arising from the high-dimensional outer-product representation. The model isolates the most informative latent components that capture essential cross-modal correlations by decomposing the tensor into a low-dimensional core tensor and factor matrices while discarding irrelevant or redundant variations. This compression not only yields a compact and discriminative joint representation but also facilitates end-to-end optimization as a learnable layer within the network, enhancing both efficiency and generalization for the emotion recognition task (see Appendix A for implementation details).

The core tensor encapsulates the most salient and statistically significant audio–visual interactions by jointly modeling temporal, spectral, and spatial dependencies. This compact latent representation has been shown to be highly discriminative and informative for affective tasks because it retains the strongest cross-modal correlations while filtering redundant noise [36]. Empirically, the improved performance of the model over baseline fusion methods further supports that the Tucker-derived core serves as a representative joint feature for emotion recognition. All components of the proposed architecture are jointly trained in an end-to-end manner using backpropagation, allowing gradients to flow through the feature extraction, cross-modal attention, and Tucker fusion layers. The network parameters are updated iteratively using the categorical cross-entropy loss computed over emotion labels, ensuring that the model learns discriminative representations directly optimized for the emotion classification task.

The flexibility of Tucker decomposition is leveraged to enable a richer tensor representation beyond simple vector fusion (Figure 3). In previous studies on multimodal fusion, such as the Tensor Fusion Network [30], MUTAN [32], and BLOCK Fusion [37], the batch dimension of the input tensor $(A^{\left(Batch\right)}$ or $A^{\left(N\right)})$ is typically preserved but not explicitly decomposed. These approaches perform per-sample cross-modal fusion, where each tensor represents the interactions among modalities for an individual sample while maintaining batch processing for parallel training efficiency. These approaches perform per-sample cross-modal fusion, where each tensor represents interactions among modalities for an individual sample, while batch processing is maintained for parallel training efficiency.

These approaches perform per-sample cross-modal fusion, where each tensor represents interactions among modalities for an individual sample, while batch processing is maintained for parallel training efficiency. The primary goal is to find a shared, lower-dimensional representation for the interaction between modalities within each sample. However, the proposed cross-modal tensor construction in this study is distinct in that it integrates attention-weighted audio and visual feature sequences through a time-aligned outer-product operation, forming a high-order tensor that explicitly retains both temporal and feature modes of each modality. This design captures fine-grained spatiotemporal dependencies beyond simple vector concatenation. The initial input tensor, $X$, is defined with more modes directly relevant to the temporal and feature dimensions of audio and visual data, enabling a more granular and powerful fusion. The detailed formulation and step-by-step construction process for the tensor are provided in Appendix B.

Extensive experiments are conducted on three widely used datasets- IEMOCAP [38], RAVDESS [39], and CREMA-D [40] to validate the effectiveness of the fusion framework. Profiles of these datasets are provided in Appendix C. All experiments were conducted on a high-performance system detailed in Appendix D. The effectiveness of the proposed cross-modal fusion framework was evaluated using the standard evaluation metrics: precision, recall, F1-score, and accuracy, as formally defined in Equations (6)–(9).

Precision: Precision measures the proportion of correctly predicted positive instances among all instances predicted as positive. High precision indicates that the model makes few false positive errors when predicting the positive class.

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}}$$

(6)

Recall (Sensitivity): Recall measures the proportion of correctly predicted positive instances among all actual positive instances. High recall indicates that the model successfully detects the true positive instances.

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}}$$

(7)

F1 Score: F1 Score is a reliable metric for evaluating classification performance, particularly when the dataset has imbalanced class distributions. F1 score provides a more balanced measure of a model’s accuracy by combining precision and recall.

$$\mathrm{F}1\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}={\displaystyle \frac{2\ast \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(8)

Accuracy: Accuracy provides a straightforward indicator of the model’s consistency throughout all classes.

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}}$$

(9)

3. Results

This section presents the main results of this study, directly addressing the two RQs in Section 3.1 and Section 3.2. Other emergent issues are discussed and analysed in Section 4.

3.1. RQ1. To What Extent Do Affective Computing and Recent Advancements in Deep Learning Improve Cross-Modal Fusion for Audio and Image-Based Emotion Recognition?

Affective computing—which combines elements of computer science, psychology, and cognitive science—enables the development of computing devices that can analyse voice, text and facial expressions, thereby comprehending emotional states and customising responses accordingly. Previous studies in the field primarily focused on unimodal approaches [41,42,43,44], leveraging either audio or image data independently for emotion recognition. These traditional methods, while effective to an extent, often lacked robustness due to their reliance on a single modality. Recent research interest has progressively shifted toward cross-modal emotion recognition [45,46,47,48], where two or more modalities are integrated to enhance recognition accuracy and contextual understanding. This evolution has been underpinned by advances from conventional machine learning to more sophisticated deep learning architectures in recent years. Based on an analysis of the extant literature, recent progress on audio and image-based fusion mechanisms is assessed, focusing on cross-modal alignment designed to capture complementary spatiotemporal information from both modalities.

The extraction of audio signal features has seen various advancements since the mid-20th century. The earliest and most fundamental features were extracted from the time domain, with notable progress continuing until the late 1950s [49]. Time-domain features, such as the Zero Crossing Rate [50], Root Mean Square (RMS) Energy, and Waveform Shape [51], provided foundational tools for analyzing audio signals. Researchers then began exploring the frequency domain, introducing features like spectral centroid, bandwidth, contrast, flatness, and roll off [52] from the spectrograms [53], which offered insights into the spectral characteristics of signals.

To address the limitation of both time and frequency domain approaches, a joint time-frequency [54], and domain features such as Mel-Frequency Cepstral Coefficients (MFCCs) [55], Chroma Features, Mel Spectrogram, and Constant-Q Transform (CQT) [56], were developed. These features enabled a more prominent representation of audio signals by capturing temporal and spatial dynamics [57]. Additionally, advanced features like Pitch and Harmonics [8], Formants, and Temporal Modulation [58] explored and expanded statistical capabilities, supporting more complex applications in audio signal processing.

Despite these advancements, some limitations persist. For example, while time-domain features such as short-term energy, zero-crossing rate, or waveform amplitude are computationally efficient, they often fail to capture frequency-specific cues like pitch contour, harmonic structure, and formant shifts elements which strongly correlate with emotional tone and intensity [59]. Conversely, frequency-domain representations such as spectrograms, MFCCs, or chroma features effectively capture these acoustic dynamics but are computationally more demanding, making them less suitable for real-time or resource-constrained applications. These trade-offs continue to drive innovation in audio feature extraction for emotion recognition tasks.

Feature extraction from video data involves processing spatial and temporal information to capture meaningful patterns. Spatial features [57] are typically extracted frame-by-frame using convolutional neural networks, identifying objects, textures, and expansions [60]. Temporal features [58], crucial for understanding motion and dynamics, are captured by optical flow [61] or any deep neural network [62]. The process poses several challenges, including the high computational cost of processing large volumes of frames and maintaining temporal coherence.

Facial feature extraction is a key challenge and plays a key role in emotion recognition. It leverages some techniques, such as the Facial Action Coding System (FACS) [63], handcrafted features [64], and facial expression features [65]. Handcrafted features have garnered significant attention due to their effectiveness in representing facial features. Some traditional techniques include Gabor filters [66], which are adept at capturing texture and orientation information; Scale-Invariant Feature Transform (SIFT) [67], which excels in identifying distinctive key points; Local Binary Patterns (LBP) [68], renowned for their texture representation capability.

The feature extraction step is critical in reducing the dimensionality of input data, thereby enhancing model accuracy and mitigating the risks of overfitting. By efficiently extracting meaningful patterns, these techniques lay the foundation for the emotion recognition task. However, handcrafted features often struggle to generalize across diverse datasets, as their performance may depend heavily on several environmental factors, and overall, they struggle to capture subtle facial expressions.

Thus, fusion mechanisms for cross-modal emotion recognition are required to integrate features from multiple heterogeneous data sources or modalities (Figure 4). This approach is particularly advantageous in scenarios where information from various domains must be combined to understand the underlying patterns or behaviors comprehensively. The fusion process typically involves several strategies: early fusion [18], late fusion [20], hybrid fusion [21], model-level fusion [69], hierarchical fusion [70], rule-based decision-level fusion [71], and estimation-based fusion [72]. These fusions are mainly subdivisions of Intra-modal fusion and Cross-modal fusion [45]. Early fusion combines features from different modalities at the initial stages, creating a unified feature set before applying machine learning algorithms. Ortega et al. [73] proposed an approach using early fusion; the study emphasizes the importance of meticulous hyperparameter optimization to address challenges such as premature overfitting. Conversely, late fusion processes each modality separately and then merges their outputs at the decision level [74]. Hybrid fusion combines early and late fusion elements, integrating features at multiple stages to capitalize on their advantages [75]. Model-level fusion involves combining different models trained on separate modalities by averaging their outputs or using more sophisticated techniques such as stacking or boosting [69].

Hierarchical fusion employs a multi-layered approach, where data is fused at various levels of abstraction, allowing for a more nuanced and detailed information integration [76].

The cross-modal fusion mechanism [75,77,78] effectively integrates and interacts with contextual relevance among features from different modalities. These existing works provided the foundation for the design of an interpretability method for audio and video-based emotion recognition to justify the significance of every input visual characteristic and every vocal segment. By leveraging the complementary strengths of different data types, fusion mechanisms enhance robustness and reliability. This approach is extensively applied for emotion recognition, but also in other fields including medical diagnostics and autonomous driving, where integrating diverse data sources results in more accurate and insightful outcomes. Recent research in cross-modal emotion recognition has focused on using cross-attention feature-level techniques to improve the merging of audio and image features. Praveen et al. [78] developed a joint cross-attention model that effectively integrates audio and visual cues for dimensional emotion detection, allowing for a more comprehensive understanding of complicated emotional states. Similarly, Mocanu et al. [45] demonstrated how cross-modal audio-video fusion, attention processes, and deep metric learning improved identification accuracy by aligning modality-specific representations. The study by Zhou et al. [79] described a cross-attention and hybrid feature weighting neural network, emphasizing the importance of balanced feature contributions in emotion recognition from large-scale video clips.

Furthermore, Lee et al. [80] investigated speech emotion identification using cross-modal fusion methods that align audio and textual information, demonstrating the synergy of linguistic and acoustic modalities. These developments demonstrate cross-modal effectiveness in capturing detailed interactions across modalities, overcoming the limitations of cross-modal data fusion, and improving emotion recognition ability. Despite these advances, limitations persist. Transformer-based models [81] frequently require significant processing resources, which might be prohibitive for real-time applications. At the same time, relying on large-scale and balanced datasets for training may result in lower performance in domains with limited or imbalanced data. These problems underscore the importance of the search for new efficient and scalable techniques for cross-modal emotion identification.

3.2. RQ2. Can a New Model Be Developed and Validated to Improve Correlations Between Heterogeneous Modalities (Such as Audio and Visual Data) to Detect and Interpret Human Emotional States?

This section presents the experimental results of the cross-modal fusion mechanism for emotion recognition using the Tucker decomposition framework, demonstrating its effectiveness across three datasets: RAVDESS, CREMA-D, and IEMOCAP (see Appendix C). The performance is evaluated using standard classification metrics, including precision, recall, and F1-score, with macro-averaged and per-emotion.

Table 1. Model performance on the RAVDESS dataset.

Emotions	Angry	Calm	Disgust	Fearful	Happy	Neutral	Sad	Surprise
Precision	92.12	92.8	91.47	93.2	93.85	91.47	91.17	93.6
Recall	92.8	93.91	91.64	91.06	93.12	93.5	91.55	91.55
F1-Score	92.46	93.35	91.55	92.12	93.49	92.47	91.36	92.56

illustrates the per-emotion performance of the model on the RAVDESS dataset, which includes 8 distinct emotional classes.

The model achieves consistently high performance across all emotions on RAVDESS, reflecting its robust ability to distinguish between a wide range of expressive states: in particular, “Happy”, “Calm”, and “Surprise” are recognized with effective accuracy. Even the lowest F1-score for “Sad” remains remarkably high, indicating a well-balanced classification capability without significant bias towards specific emotions. This balanced performance illustrates the effectiveness of the approach to Tucker decomposition in leveraging complementary cues from audio and video modalities.

Table 2. Model performance on the CREMA-D dataset.

Emotions	Anger	Disgust	Fear	Happy	Neutral	Sad
Precision	86.5	88.8	87.93	87.39	85.62	85.62
Recall	85.23	88.46	87.4	87.83	85.08	88.88
F1-Score	85.86	88.63	87.67	87.61	85.35	87.22

illustrates the model’s strong performance on the CREAM-D dataset. “Disgust” achieves the highest F1-score, closely followed by “Fear” and “Happy”, while “Neutral” and “Anger” show slightly lower, yet still very competitive, predictions. The overall high F1-score across all emotions confirms the model’s ability to effectively generalize and classify emotions in this dataset.

Table 3. Model performance on the IEMOCAP dataset.

Emotions	Anger	Happiness	Sadness	Neutral	Excited	Frustration	Surprise	Fear
Precision	83.87	84.22	82.85	83.14	81.38	85.87	84.75	82.62
Recall	85.08	82.42	84.45	84.64	82.11	82.89	82.41	83.94
F1-Score	84.47	83.31	83.64	83.88	81.74	84.36	83.56	83.27

illustrates the model’s robust performance on the more complex IEMOCAP dataset. F1-scores for “Anger” and “Frustration” are among the best predicted emotions, highlighting the model’s capability to handle highly distinct and subtle cue-based emotional states. While “Excited” shows the lowest F1-score, it still represents a strong performance given the inherent variability and often subtle nature of this emotion in spontaneous audio waves. The overall balanced F1-Scores across the diverse set of emotions in IEMOCAP further validate the model’s effectiveness in capturing complex cues.

Table 4. Model performance evaluation: weighted and unweighted accuracy metrics on RAVDESS, CREMA-D, and IEMOCAP datasets.

Dataset	Weighted Accuracy	Unweighted Accuracy
RAVDESS	92.46	88.74
CREMA-D	87.31	84.27
IEMOCAP	83.22	79.63

provides a comprehensive result of the model’s overall evaluation performance, presenting both weighted and unweighted accuracy metrics across the three datasets. Weighted accuracy accounts for class imbalance by averaging the accuracy of each class, while unweighted accuracy is simply the total number of correct predictions divided by the total number of samples. The result in Table 4 illustrates the model’s robust performance. The consistently high weighted accuracy scores across all datasets confirm the model’s ability to perform well even in the presence of potential class imbalances, highlighting its balanced classification capability for each emotion class.

To validate the efficacy of the cross-modal fusion framework, a comprehensive analysis was conducted against various baseline fusion methods and several state-of-the-art cross-modal emotion recognition approaches.

Table 5. Performance comparison with other state-of-the-art cross-modal approaches on the RAVDESS dataset.

Source	Fusion	Accuracy	Remarks
[82]	Self-attention	75.76	The analysis relies solely on one dataset
[83]	Late fusion	86.70	The analysis relies solely on one dataset
[46]	Concatenation	66.90	Simple concatenation, high dimensionality
[45]	Cross-Attention	89.25	Poor sensitivity to micro-expressions
[84]	Cross-Attention	82.42	Computationally expensive and performed on a single dataset
Proposed Model	Cross-modal attention	92.46	Reduces dimensionality while employing three established benchmarks

presents a detailed performance comparison, specifically focusing on the RAVDESS dataset, a key benchmark for this domain.

Table 5 demonstrates superior performance by the proposed model on the RAVDESS dataset, achieving an accuracy of 92.46%. This significantly surpasses other contemporary methods, highlighting the effectiveness of the cross-modal attention fusion mechanism and Tucker decomposition-based multi-linear fusion. For instance, approaches utilizing self-attention [82] or simple concatenation [46] yielded substantially lower accuracies of 75.76% and 66.90%, respectively, often due to limitations such as reliance on single datasets or inherent high-dimensionality challenges. This comparative analysis indicates distinct advantages of the proposed model in the field of cross-modal emotion recognition.

Table 6. Performance comparison with other state-of-the-art cross-modal approaches on the CREMA-D dataset.

Source	Fusion	Accuracy	Remarks
[48]	Multi-branch attention	72.45	Speaker-dependent model
[45]	Cross Attention	84.57	Poor sensitivity to micro-expressions
[47]	Conformer encoder	77.9	Versatile learning model
Proposed Model	Cross-modal attention	87.31	Common temporal dynamics along with the GRU layer

and

Table 7. Performance comparison with other state-of-the-art cross-modal approaches on the IEMOCAP dataset.

Source	Fusion	Accuracy	Remarks
[85]	Hybrid Multi-Attention Fusion	75.39	Parallel co-attention mechanism
Proposed Model	Cross-modal attention	83.22	Intra-modal temporal refinement

illustrate that the model consistently demonstrates superior performance across both the CREMA-D and IEMOCAP datasets when compared to other state-of-the-art cross-modal approaches. On CREMA-D, the model achieved an accuracy of 87.31%, surpassing methods like those set out by Mocanu et al. [45] and Goncalves et al. [47], and significantly outperforming speaker-dependent models such as that of John and Kawanishi [48]. This highlights the effectiveness of the approach put forward here in capturing common temporal dynamics and leveraging the GRU layer for robust feature learning on this dataset. Similarly, on the highly challenging IEMOCAP dataset, the model secured an accuracy of 83.22%. This result is notably higher than that achieved by Moorthy and Moon [85]. The consistent outperformance on IEMOCAP underscores the critical role of our framework’s intra-modal temporal refinement and sophisticated cross-modal attention fusion mechanism using Tucker decomposition in handling the complexities of spontaneous, cross-modal emotional expressions.

The confusion matrices are presented in Figure 5, Figure 6 and Figure 7 for the RAVDESS, CREMA-D, and IEMOCAP datasets. These provide a granular insight into the model’s per-emotion classification performance. These matrices visually complement the aggregate accuracy metrics by showing the number of correctly classified instances for each emotion (represented by strong diagonal elements) and identifying any specific emotions that are frequently confused with others (off-diagonal elements).

The lower accuracy observed on the IEMOCAP and CREMA-D datasets compared to RAVDESS primarily stems from the inherent variability and complexity of these datasets. RAVDESS consists of acted, high-quality, and well-balanced recordings captured under controlled conditions, resulting in clearer emotional cues and consistent expressions across subjects. In contrast, IEMOCAP and CREMA-D contain spontaneous, heterogeneous samples with variations in speaker identity, emotional intensity, accent, and recording conditions, which increase the difficulty of emotion classification. Figure 5 represents the confusion matrix for the RAVDESS dataset, illustrating class-wise performance of the proposed model on that specific dataset.

4. Discussion

The above results raise three related issues worthy of further analysis and discussion. First, as regards the audio processing segment for emotion recognition in the model, not all parts of the spectrogram are equally important. Figure 8 provides a visualization of cross-attention weight matrices between audio and visual modalities. Each heatmap shows how visual frame features (queries) attend to audio frame features (keys/values). The horizontal axis represents audio time steps, the vertical axis visual frames, and the color intensity denotes normalized attention weights. The left panel depicts randomly distributed weights before convergence (no clear temporal structure); the middle panel shows focused attention along the diagonal, indicating temporal alignment between modalities; and the right panel illustrates selective peaks corresponding to key emotional moments. These visualizations qualitatively demonstrate how the attention mechanism tends to concentrate on temporally or contextually relevant segments, rather than uniformly attending across the entire sequence.

Darker shades indicate higher attention scores, highlighting specific-time frequency regions that are more silent for the model’s emotion recognition.

Figure 9 demonstrates the approach of the attention mechanism in video data. Unlike static images, videos contain a temporal dimension, requiring the model to identify both spatially and temporally salient regions. Relevance is determined through the attention weights, which are computed by measuring the dot-product similarity between the query and key feature representations for each frame. These scores are then normalized using a Softmax function to assign higher weights to frames or regions that exhibit stronger correlations with the emotional context. The resulting visualization therefore highlights the areas or segments that the model assigns higher attention weights to during emotion inference.

Brighter regions indicate higher attention, showing the model’s focus on particular spatial areas and temporal frames within a video sequence.

Figure 10 illustrates the cross-modal attention mechanism that allows elements from one modality to attend to elements in another, revealing inter-modal dependencies. This visualization is important for understanding the cross-modal integrated information and resolving ambiguities by leveraging complementary cues from diverse data sources, ultimately leading to more robust and accurate predictions.

Secondly, and more specifically, the results provided examples of audio-visual cross-attention, where the flow of attention is from the visual modality to the audio modality. This means that features extracted from the visual stream are “querying” or attending to specific parts of the audio stream to enhance their understanding. For example, in a scenario where a person is speaking, the visual representations of their lips might attend to the corresponding speech sounds in the audio track. This type of attention helps the model to establish correspondence between what is seen and what is heard, allowing it to identify temporal alignments or causal relationships. It is particularly useful in refining visual interpretations.

In audio–video emotion recognition, the query–key–value (QKV) mechanism in the attention model enables each modality to selectively attend to the most relevant features in the other, effectively integrating complementary emotional cues across audio and video. Modality-specific encoders (CNNs for video and audio) extract high-level features, including facial expressions, speech tone, and rhythm. During cross-modal attention, the feature sequence from one modality (for example, audio) serves as the Query (Q), while the other modality (video) provides the Keys (K) and Values (V). The attention module computes relevance scores via dot-product similarity between Q and K, and uses these scores to weight the Values. The resulting weighted features highlight where emotional information in one modality aligns most strongly with the other. The mechanism operates bidirectionally: audio features attend to visual features, and visual features attend to audio. This bidirectional exchange enables the model to associate prosodic variations (pitch, tone, rhythm) with facial expressions and muscle dynamics, yielding richer multimodal representations for emotion understanding.

The cross-modal attention mechanism contributes three important functional advantages within the proposed framework. First, it facilitates temporal synchronization by aligning audio and visual cues that may express the same emotion slightly out of phase—for example, a shift in speech tone often occurs just before a corresponding facial expression, such as a smile or frown. Second, it performs feature selection, automatically assigning higher weights to emotionally salient patterns such as variations in voice pitch, vocal intensity, or facial muscle tension while suppressing background noise or frame-level distractions. Third, it enhances robustness by emphasizing cross-modal signals that consistently co-occur across audio and visual streams, allowing the model to remain effective even when one modality contains missing, degraded, or ambiguous information.

Together, these properties help the attention mechanism generate more coherent and context-aware multimodal representations prior to the Tucker fusion stage. Overall, cross-modal attention provides a data-driven way to connect vocal prosody and facial expressions, thereby producing semantically aligned, discriminative features before the Tucker-decomposition-based fusion stage [86,87].

The results also illustrate how, in the context of visual-audio cross-attention, the audio modality is attending to the visual modality, meaning that each audio feature vector computes attention weights over all visual frame features to identify which visual cues are most relevant to that audio segment. Here, the audio features “query” the visual information to gain a richer context. In the context of audio-to-visual cross-attention, the audio modality actively attends to relevant visual cues to gain contextual information. In this mechanism, each audio feature vector acts as a Query (Q) that computes a similarity score with all visual feature vectors (Keys, K) to identify which visual frames are most relevant to the current sound segment. The resulting attention weights are then used to form a weighted sum of the corresponding visual features (Values, V), producing an audio representation enriched with visual context. For example, when processing a low-pitched, slow vocal tone associated with sadness, the model may attend to visual cues such as drooping eyelids or a downturned mouth to reinforce its emotional interpretation. This cross-attention process enables the model to resolve ambiguities in vocal cues and recognize nuanced emotional states more accurately. In other words, the model learns to associate specific acoustic patterns, such as changes in pitch, tone, or intensity, with corresponding facial expressions or movements. This process produces an attention-weighted visual context that is fused back into the audio representation, allowing the model to interpret vocal emotions with the aid of visual cues. For instance, when a sad tone is detected, the audio features may focus more on visual indicators of sadness (e.g., drooping eyelids, downturned mouth), helping the model to disambiguate or refine emotional interpretation.

This attention mechanism allows the model to leverage visual cues to disambiguate or enhance the interpretation of emotional vocalizations. It is vital in identifying the emotion where visual context is crucial for understanding nuanced emotions, such as distinguishing genuine and feigned emotional expressions, or interpreting subtle emotional states, where audio cues alone might be ambiguous.

Thirdly, a deeper understanding of different extracted features can be displayed in a feature correlation heatmap (Figure 11). This visualization indicates the relationships and interdependencies between these features, whether they are acoustic (pitch, intensity, formants) or visual (facial landmarks, mouth openness). Each cell in the heatmap represents the Pearson correlation coefficient between two specific features. A high positive correlation, represented by warmer colours like red or orange, suggests that as the value of one feature increases, the other tends to increase as well. Conversely, a high negative correlation, represented by cooler colours like blue or dark blue, indicates that as one feature increases, the other tends to decrease. A value close to zero is often represented by white or light grey, indicating little to no linear relationship. Analyzing this heatmap helps the identification of redundant features, highly influential features, and potential dependencies that can inform feature selection, model architecture design, and improve the accuracy of emotion recognition. For instance, a strong correlation between vocal pitch and eyebrow movement might suggest a shared underlying emotional expression.

5. Conclusions

Emotion recognition centres on the ability to precisely infer human emotions from numerous sources and modalities including physical signals such as speech and facial expression as discussed in this article. However, as recently noted by Mengara Mengara and Moon [88], “multimodal emotion recognition faces substantial challenges due to the inherent heterogeneity of data sources, each with its own temporal resolution, noise characteristics, and potential for incompleteness” [88] (p. 1). In this context, this article put forward and evaluated a novel audio-image-based cross-modal fusion model which leverages a Tucker decomposition-based factorization layer designed to explicitly capture and efficiently fuse high-order spatiotemporal feature interactions. This method reduces dimensionality while preserving complex multi-linear relationships within the fused representation, enhancing interaction expressiveness and emotion recognition accuracy. Cross-attention mechanisms are incorporated to further improve alignment, allowing the model to emphasize relevant information across modalities dynamically.

This research clearly has its limitations, in that it focuses on a limited number of modalities and just three datasets for experimentation. Nevertheless, the authors believe the methodology deployed here represents a robust model for cross-model emotion recognition by extracting modality-specific spatiotemporal features. As Praveen et al. [78] recently noted “most state-of-the- art audio-visual (A-V) fusion methods rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities” [78] (p. 2486), yet the model presented here facilitates the transformation of heterogeneous inputs, originating from the same emotional expression, into a unified latent representation space. The results consistently demonstrate that the proposed fusion approach outperforms the traditional fusion approach and several state-of-the-art models, achieving superior performance in emotion recognition tasks across these diverse datasets. This opens up several promising avenues for future research. For example, incorporating additional modalities and expanding datasets to include a broader spectrum of diverse emotional expressions could lead to more comprehensive and nuanced emotion recognition models. Further exploration could focus on implementing real-time emotion recognition systems in a range of organisational contexts, including enhanced customer service, adaptive educational tools, and supportive healthcare interventions.