Dynamic Mutual Refinement for Multimodal Speech Emotion Recognition: A Bidirectional Distillation and Inverse Curriculum Learning Approach

Abstract

With the rapid advancement of speech emotion recognition, the transition from unimodal to multimodal approaches has become inevitable. However, multimodal methods introduce new challenges, particularly classification ambiguity, where multimodal models may misclassify samples that unimodal models correctly identify, and vice versa. To address this, we propose Dynamic Adaptive Mutual Refinement (DAMR), a training framework designed to reduce classification ambiguity between multimodal and unimodal models while preserving their core architectures. DAMR incorporates three key components: (1) dynamic mutual distillation (DMD), which employs an EMA-based capability assessment mechanism and competitive dynamic weighting to adaptively adjust knowledge transfer between models; (2) adaptive inverse curriculum learning (AICL), which implements a hard-to-easy learning schedule based on information bottleneck theory and gradient signal analysis, focusing on informative and difficult samples in early training stages; and (3) an integrated optimization framework that jointly applies DMD and AICL through adaptive weighting. Experiments on the MELD and IEMOCAP datasets demonstrate that our approach outperforms state-of-the-art methods in emotion recognition, achieving a notable 5.54% improvement over the baseline on IEMOCAP.

1. Introduction

Traditional speech emotion recognition (SER) approaches, which primarily rely on acoustic features alone, struggle to capture the subtle nuances of real-world emotional expressions [1,2]. Recent advancements in deep learning, including Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks [2,3], and attention mechanisms [4], have significantly improved SER performance. However, despite these methods, the limitation of single modality remains in the lack of additional auxiliary information, which can easily lead to information bottlenecks and hinder significant performance improvement. In contrast, multimodal methods that integrate facial expressions, physiological signals, and textual content have led to a paradigm shift, enabling more accurate and contextually rich emotion recognition [5,6]. Common multimodal fusion techniques include early fusion, late fusion, and hybrid fusion methods [7].

Multimodal methods primarily aim to integrate information from multiple modalities to enhance speech emotion recognition [8]. However, different fusion strategies may introduce interference factors, which can cause classification ambiguity. In some cases, such interference from fusion strategies results in a multimodal model performing worse than a unimodal one for certain samples. Despite this issue, previous research has not explicitly treated classification ambiguity as a distinct problem, though most studies implicitly attempt to mitigate it by adjusting modality contributions and filtering out interfering information [9,10,11], both of which are inherently efforts to reduce ambiguity.

We conducted pilot experiments to investigate classification ambiguity in multimodal methods, specifically using the state-of-the-art CMERC multimodal approach [12]. As shown in Table 1 and Table 2, we define Multimodal Correct but Text Incorrect (MCTI) as instances where the multimodal model (integrating text, audio, and visual features) predicts correctly while the text-only model fails. Conversely, Multimodal Incorrect but Text Correct (MITC) refers to cases where the multimodal model is incorrect, but the text-only model succeeds.

Table 1. IEMOCAP dataset performance on baseline model by emotion category.

Datasets	Happy	Sad	Neutral	Angry	Excited	Frustrated	Total
MCTI	22	19	46	11	92	48	238
MITC	24	16	15	14	20	23	112
Total Data	144	245	384	170	299	381	1623

Table 2. MELD dataset performance on baseline model by emotion category.

Datasets	Neutral	Surprise	Fear	Sadness	Joy	Disgust	Anger	Total
MCTI	60	7	0	1	1	1	5	75
MITC	14	8	6	27	35	3	31	124
Total Data	1256	281	50	208	402	68	345	2610

Given that the most significant ambiguity occurs between the text modality and the multimodal model, both exhibiting closely matched classification accuracy, we focused on comparing these two models and used them in subsequent experiments. We also compared other modalities and found that classification ambiguity among them was not as pronounced as that between the text and multimodal models; therefore, we did not pursue further experiments on other modalities. This experiment demonstrates the classification ambiguity between text modality and multimodal models, particularly evident on the IEMOCAP dataset.

Based on the experimental results, we observed that unimodal and multimodal approaches often exhibit classification ambiguity in certain specific samples within multimodal SER. We define these ambiguous samples as complex samples, which exhibit inconsistent predictions between unimodal and multimodal models. This phenomenon is further supported by prior research, such as CMERC, which demonstrates that removing certain modalities can, in some cases, lead to increased classification confidence [12]. Although existing methods have made progress in reducing such ambiguity, even the most advanced models struggle to eliminate it entirely, motivating the need for a more principled solution.

To address this issue, we propose Dynamic Adaptive Mutual Refinement (DAMR), an integrated training framework designed to substantially reduce classification ambiguity in both multimodal and unimodal models without altering their core architectures. The main contributions of this work are threefold:

(1) Dynamic mutual distillation (DMD). We introduce a dynamic mutual distillation mechanism that incorporates an Exponential Moving Average (EMA)-based capability assessment strategy and a competitive dynamic weighting function, enabling adaptive adjustment of the direction and intensity of knowledge transfer between multimodal and unimodal models.

(2) Adaptive inverse curriculum learning (AICL). We propose an adaptive inverse curriculum learning mechanism that dynamically partitions training samples into simple, complex, and beyond-capacity categories based on prediction consistency and correctness, and applies time-evolving sample weights to prioritize more informative and ambiguous samples during early training stages.

(3) Unified Integration within DAMR. DMD and AICL are embedded into a single training framework, where knowledge distillation adaptively responds to model capabilities, while curriculum learning allocates computational resources to the most informative samples, collectively improving multimodal speech emotion recognition performance.

2. Related Work

Our work is primarily related to multimodal emotion recognition, knowledge distillation, and curriculum learning. In this section, we review representative studies in these areas and clarify how our approach differs from existing methods.

Multimodal Emotion Recognition. Early multimodal emotion recognition approaches primarily focused on designing effective fusion strategies, such as early, late, and hybrid fusion, to combine information from different modalities [7]. More recent approaches leverage powerful deep neural networks, particularly transformers [9,13] and their variants, such as EmoCaps [14], MPT-HCL [15], CFN-ESA [16], to model cross-modal interactions and perform feature fusion. In addition, Graph Neural Networks (GNNs) have been widely adopted to capture inter-modal and intra-modal dependencies, as CogMEN [17], M3Net [18], MGLRA [19], and CMERC [12]. Although these methods improve overall performance, they often do not explicitly address the issue of classification ambiguity caused by sub-optimal multimodal fusion, where certain modalities may introduce noise or conflicting information for specific samples.

To mitigate modality interference, several studies have explored feature decoupling techniques that separate shared and modality-specific features, thereby preserving the most useful information for emotion recognition [10,20,21,22,23]. Other approaches employ knowledge distillation, where a stronger teacher model, typically multimodal, guides a weaker student model (e.g., unimodal) to prevent the student from being misled by noise [24]. However, these methods typically assume a static and unidirectional knowledge flow and do not consider the dynamic complementary relationship between multimodal and unimodal models. As demonstrated in our pilot experiments, the relative strengths of multimodal and unimodal models are not fixed but vary across samples and training stages. Consequently, static fusion strategies or one-directional distillation mechanisms are insufficient for effectively resolving the classification ambiguity considered in this work.

Knowledge Distillation. Knowledge distillation (KD) is a widely used technique for model compression and knowledge transfer, in which a smaller student network learns to mimic the softened output logits or intermediate feature representations of a larger, pre-trained teacher network [25]. Beyond model compression, KD has been extended to cross-modal transfer settings, where a multimodal teacher guides a unimodal student to retain rich cross-modal knowledge during inference [26,27]. A further development is mutual distillation, where two models of comparable capacity learn collaboratively by teaching each other during training [28,29]. Despite this progress, most existing mutual distillation methods employ static or symmetric weighting schemes. Such strategies become inefficient when the relative capabilities of the two models are imbalanced or change dynamically throughout training. In the context of multimodal emotion recognition, fixed or symmetric distillation cannot adaptively prioritize guidance from the model that is more reliable for a particular sample or at a given training stage. To address this limitation, our dynamic mutual distillation (DMD) introduces an EMA-based capability assessment mechanism that continuously tracks each model’s performance and drives a competitive dynamic weighting strategy. This design enables knowledge to flow preferentially from the stronger model to the weaker one throughout training.

Curriculum Learning. Curriculum Learning (CL) is a training strategy inspired by human pedagogy, in which a models are first exposed to easier samples before gradually progressing more difficult ones [30]. CL has been successfully applied to a wide range of tasks to improve convergence speed and model generalization. Two key components of CL are the difficulty measure and the pacing function, which determines how quickly harder samples are introduced during training. Self-paced learning and its variants automate this process by defining sample difficulty according to the current training loss, allowing the model to adaptively determine its own learning pace [31]. Competence-based approaches further associate the pacing function with an explicit estimate of model competence, gradually expanding the training set as the model improves [32]. In contrast, anti-curriculum or inverse curriculum learning begins with more difficult samples during training and has been shown to yield faster convergence or better final performance in certain settings [33]. However, existing CL methods typically rely on predefined single-model heuristics, such as loss magnitude or prediction confidence, and follow a fixed schedule that does not adapt to the evolving interaction between multiple models. In our multimodal–unimodal ambiguity scenario, sample difficulty is inherently relational: a sample is complex precisely because the two models disagree on it, and this disagreement pattern evolves as both models during training. To address this issue, our adaptive inverse curriculum learning (AICL) dynamically categorizes samples into simple, complex, and beyond groups based on the joint prediction consistency and correctness of both models. AICL further employs a time-evolving weighting mechanism motivated by information bottleneck theory and gradient signal analysis, to emphasize the most informative samples during the early stages of training.

3. Materials and Methods

As illustrated in the pipeline of Figure 1, DAMR consists of two key coordinated components. (1) Dynamic mutual distillation (DMD): This component replaces traditional one-way distillation with a bidirectional and adaptively weighted mutual distillation mechanism. Unlike conventional or fixed mutual distillation approaches, DMD employs EMA to evaluate model capabilities and dynamically adjusts the distillation weights, allowing the more capable model to guide the weaker one in a balanced yet focused manner to address classification ambiguity. (2) Adaptive inverse curriculum learning (AICL): To enhance the model’s focus on informative samples, we introduce AICL as an adaptive sample weighting mechanism. Unlike traditional curriculum learning [30], which gradually progress from easy samples to difficult ones, AICL adopts a “hard-to-easy” training schedule. Specifically, the framework initially emphasizes complex samples and those beyond the current model capacity, as these cases exhibit higher classification ambiguity and provide stronger optimization signals. As training progresses and the models become more robust, simpler samples are gradually incorporated into the learning process. These two components are not independent modules but are jointly optimized within a unified objective function. The framework dynamically coordinates sample weighting and distillation gradients at each training step, enabling DMD and AICL to complement each other throughout the optimization process.

Figure 1. A multimodal emotion recognition framework based on dynamic mutual distillation and adaptive inverse curriculum learning.

3.1. Dynamic Mutual Distillation (DMD) Mechanism

3.1.1. Overall Architecture and Mutual Distillation Motivation

Let the training dataset be $\mathcal{D}={\left\{(x_i^a,x_i^t,x_i^v,y_i)\right\}}_{i=1}^N$, where $x_i^a,x_i^t,x_i^v$ represent audio, text, and visual modality inputs respectively, $y_i\in \{1,\dots ,C\}$ denotes the emotion label, N is the total number of training samples, and C is the total number of emotion categories. The mechanism comprises two core models:

Multimodal fusion model M: with parameters ${\theta }_M$, outputting a probability distribution $P_M(y\mid x^a,x^t,x^v;{\theta }_M)$

Text-only model T: with parameters ${\theta }_T$, outputting a probability distribution $P_T(y\mid x^t;{\theta }_T)$

The mathematical motivation for mutual distillation is that ideally, we aim to learn models that satisfy

$$\underset{{\theta }_M,{\theta }_T}{\min }{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left[{\mathcal{L}}_{\mathrm{cls}}({\theta }_M;x,y)+{\mathcal{L}}_{\mathrm{cls}}({\theta }_T;x,y)+\lambda ·D(P_M\left(y\right|x)\parallel P_T\left(y\right|x))\right]$$

(1)

where $D(·\parallel ·)$ is a distribution divergence measure, and $\lambda >0$ is a weight coefficient balancing classification and distillation losses. However, direct optimization may cause both models to mutually “compromise”, potentially converging to suboptimal solutions, undermining their respective advantages. Therefore, we design a dynamic weighting mechanism that adaptively adjusts the knowledge transfer direction according to model capability changes.

3.1.2. Dynamic Capability Assessment via Exponential Moving Average

A model’s teaching capability should be measured through its stable performance on an independent validation set ${\mathcal{D}}_{\mathrm{val}}$. Direct use of current batch or epoch performance metrics introduces excessive noise. Therefore, we employ Exponential Moving Average (EMA) to smooth the validation set macro-F1 scores:

$${\mathrm{EMA}}_t^i=\beta ·{\mathrm{EMA}}_{t-1}^i+(1-\beta )·S_t^i$$

(2)

where $i\in \{M,T\}$ denotes the model, $S_t^i=\mathrm{F}1-\mathrm{macro}({\widehat{Y}}_t^i,Y_{\mathrm{val}})$ is the current validation score, $\beta \in (0,1)$ is the decay coefficient controlling the weight of historical information, and ${\mathrm{EMA}}_0^i=S_0^i$.

This recursive form can be expanded as

$${\mathrm{EMA}}_t^i=(1-\beta ){\displaystyle \sum _{k=0}^{t-1}}{\beta }^k{S}_{t-k}^i+{\beta }^t{S}_0^i$$

(3)

showing that ${\mathrm{EMA}}_t^i$ is a weighted average of all past validation scores with exponentially decaying weights. As $t\to \infty$, if $S_t^i$ converges to a stable value $S_{*}^i$, then ${\mathrm{EMA}}_t^i\to S_{*}^i$.

3.1.3. Dynamic Weight Calculation and Teaching Direction Decision

At training step t, we determine the distillation intensity from model i to model j. Based on the EMA values of both models, we define the distillation weight function:

$$w_t^{i\to j}={\displaystyle \frac{\exp (-\alpha ·{\mathrm{EMA}}_t^j)}{\exp (-\alpha ·{\mathrm{EMA}}_t^j)+\exp (-\alpha ·{\mathrm{EMA}}_t^i)}}$$

(4)

where $\alpha >0$ is a temperature coefficient controlling the sensitivity of weights to EMA differences.

This function exhibits the following properties: Monotonicity: Fixing ${\mathrm{EMA}}_t^i$, $w_t^{i\to j}$ decreases monotonically with respect to ${\mathrm{EMA}}_t^j$. When model j performs worse, its weight for receiving distillation increases.

Competitiveness: When ${\mathrm{EMA}}_t^i={\mathrm{EMA}}_t^j$, $w_t^{i\to j}=0.5$, indicating equal mutual learning.

Boundary behavior: When ${\mathrm{EMA}}_t^i\gg {\mathrm{EMA}}_t^j$, $w_t^{i\to j}\approx 1$, model i almost completely guides j; conversely, $w_t^{i\to j}\approx 0$.

The gradient of the weight function is

$${\displaystyle \frac{\partial w_t^{i\to j}}{\partial {\mathrm{EMA}}_t^i}}=-\alpha ·w_t^{i\to j}·(1-w_t^{i\to j})$$

(5)

indicating that larger $\alpha$ values make the model more responsive to performance differences.

3.1.4. Alternating Training Strategy

To avoid early mutual misguidance, the training process is divided into cycles of K epochs. Within each cycle, we execute the following strategy based on stage division conditions:

If ${\mathrm{EMA}}_t^T>{\mathrm{EMA}}_t^M+\delta$, enter text-dominant stage

If ${\mathrm{EMA}}_t^M>{\mathrm{EMA}}_t^T+\delta$, enter multimodal-dominant stage

Otherwise, enter joint fine-tuning stage

Where $\delta >0$ is a hysteresis threshold preventing frequent switching due to minor fluctuations.

In non-dominant stages, parameters of the weaker model are partially frozen. Let model parameters be divided into trainable set ${\mathrm{\Theta }}_{\mathrm{tr}}$ and frozen set ${\mathrm{\Theta }}_{\mathrm{fr}}$, with gradient updates:

$${\theta }^{(t+1)}={\theta }^{\left(t\right)}-\eta ·{\nabla }_{{\mathrm{\Theta }}_{\mathrm{tr}}}\mathcal{L}\left({\theta }^{\left(t\right)}\right)$$

(6)

where $\eta >0$ is the learning rate. This freezing mechanism reduces the noise propagation risk and stabilizes training dynamics.

3.1.5. Loss Function Design

$${\mathcal{L}}_{\mathrm{cls}}^i=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{\displaystyle \sum _{c=1}^C}{\displaystyle \frac{1}{f_c^{\gamma }}}·y_{n,c}\log P_i(y=c\mid x_n;{\theta }_i)$$

(7)

where N is the total number of samples, C is the number of classes, $f_c$ is the frequency of class c, $\gamma \in [0,1]$ controls the weighting strength ($\gamma =0$ for no weighting, $\gamma =1$ for full inverse weighting), $y_{n,c}$ is the one-hot encoded ground truth label for sample n (equals 1 if sample n belongs to class c, 0 otherwise), and $P_i(y=c\mid x_n;{\theta }_i)$ is the predicted probability that sample $x_n$ belongs to class c by the i-th classifier with parameters ${\theta }_i$.

The distillation loss includes both output-level and feature-level alignment:

$$\begin{array}{cc}{\mathcal{L}}_{{\mathrm{KD}}_{\mathrm{out}}}^{i\to j}\hfill & ={\tau }^2·\mathrm{KL}\left(P_j^{\left(\tau \right)}(y\mid x)\Vert P_i^{\left(\tau \right)}(y\mid x)\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}{\mathcal{L}}_{{\mathrm{KD}}_{\mathrm{feat}}}^{i\to j}\hfill & =1-\cos \left(h_j^{\left(l\right)},h_i^{\left(l\right)}\right)\hfill \end{array}$$

(9)

where $P^{\left(\tau \right)}(y\mid x)=\mathrm{softmax}(z/\tau )$ is the softened probability at temperature $\tau$, z is the model’s output logits vector, $h^{\left(l\right)}$ denotes the hidden feature representation extracted from the l-th layer of the corresponding model, and $\cos (·,·)$ is the cosine similarity function defined as $\cos (u,v)={\displaystyle \frac{u·v}{{\parallel u\parallel }_2{\parallel v\parallel }_2}}$.

The total loss for model j at training step t is

$${\mathcal{L}}_{\mathrm{total}}^j\left(t\right)={\mathcal{L}}_{\mathrm{cls}}^j+w_t^{i\to j}\left({\lambda }_1{\mathcal{L}}_{{\mathrm{KD}}_{\mathrm{out}}}^{i\to j}+{\lambda }_2{\mathcal{L}}_{{\mathrm{KD}}_{\mathrm{feat}}}^{i\to j}\right)$$

(10)

where ${\lambda }_1,{\lambda }_2>0$ are weight coefficients for output-level and feature-level distillation losses respectively.

Through gradient analysis, the introduction of weight $w_{i\to j}^t$ makes the parameter update direction of model j:

$$\Delta {\theta }_j=-\eta \left(\nabla {\mathcal{L}}_{\mathrm{cls}}^j+w_t^{i\to j}·\nabla {\mathcal{L}}_{\mathrm{KD}}^{i\to j}\right)$$

(11)

When model i performs excellently, $w_{i\to j}^t\approx 1$, and model j primarily learns knowledge from i; when both are comparable, $w_{i\to j}^t\approx 0.5$, achieving equal mutual learning.

3.2. Adaptive Inverse Curriculum Learning Mechanism (AICL)

3.2.1. Theoretical Motivation

Traditional curriculum learning follows the “easy-to-hard” cognitive principle, but for pre-trained models, simple samples provide limited information. We propose an inverse curriculum of “hard-to-easy” based on the following theoretical foundations:

From an information bottleneck perspective, let information transmission from input X to target Y pass through intermediate representation Z. The objective is to minimize $I(X;Z)$ while maximizing $I(Z;Y)$. For difficult samples with smaller $I(X;Y)$, the model is forced to learn more compact and effective representations Z, thereby achieving stronger generalization capability.

From a gradient signal analysis perspective, we define sample difficulty as the model’s prediction uncertainty $U\left(x\right)=H\left(P\right(y\mid x\left)\right)$, where $H(·)$ is the information entropy function:

$$H\left(P(y\mid x)\right)=-{\displaystyle \sum _{c=1}^C}P(y=c\mid x)\log P(y=c\mid x)$$

(12)

Difficult samples typically have higher $U\left(x\right)$. Focusing on such samples in early training provides stronger gradient signals:

$${\parallel {\nabla }_{\theta }\mathcal{L}\left(x\right)\parallel }_2\propto U\left(x\right)·{∥{\displaystyle \frac{\partial \mathcal{L}}{\partial \mathbf{z}}}∥}_2$$

(13)

3.2.2. Mathematical Definition of Sample Difficulty

At the beginning of each training cycle, we dynamically assess sample difficulty based on predictions from current models M and T. We define prediction consistency and correctness measures:

$$\begin{array}{cc}\hfill \mathrm{Agree}\left(x\right)& =\mathbb{I}[\arg \max P_M(y\mid x)=\arg \max P_T(y\mid x)]\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathrm{Correct}}_i\left(x\right)& =\mathbb{I}[\arg \max P_i(y\mid x)=y_{\mathrm{true}}],i\in \{M,T\}\hfill \end{array}$$

(15)

where $\mathrm{Agree}\left(x\right)\in \{0,1\}$ is a consistency indicator showing whether models M and T predict the same class for sample x, ${\mathrm{Correct}}_i\left(x\right)\in \{0,1\}$ indicates whether model i’s prediction is correct, $\mathbb{I}[·]$ is the indicator function returning 1 when the condition is true and 0 otherwise, and $y_{\mathrm{true}}$ is the ground truth label.

Based on these measures, we classify samples into three categories:

Simple samples: $\mathrm{Agree}\left(x\right)\wedge {\mathrm{Correct}}_M\left(x\right)\wedge {\mathrm{Correct}}_T\left(x\right)$—both models agree and predict correctly

Complex samples: $\neg \mathrm{Agree}\left(x\right)\wedge ({\mathrm{Correct}}_M\left(x\right)\vee {\mathrm{Correct}}_T\left(x\right))$—models disagree but at least one is correct

Beyond samples: $\neg {\mathrm{Correct}}_M\left(x\right)\wedge \neg {\mathrm{Correct}}_T\left(x\right)$—both models are incorrect (regardless of agreement)

Here, ∧, ∨, and ¬ denote logical AND, OR, and NOT operations respectively.

To enhance robustness, we introduce a confidence threshold ${\tau }_t$ that decays over time:

$${\tau }_t={\tau }_0·\exp (-t/\kappa )$$

(16)

where ${\tau }_0\in (0.5,1)$ is the initial confidence threshold, $\kappa >0$ is the decay rate parameter controlling the speed of threshold decrease, and t is the current training step. A model is considered to have a “definite” prediction for a sample only when $\max P_i(y\mid x)>{\tau }_t$, avoiding misclassification due to excessive model uncertainty in early training.

3.2.3. Staged Training Strategy

Let the index sets of the three sample categories be ${\mathcal{S}}_{\mathrm{easy}}$, ${\mathcal{S}}_{\mathrm{hard}}$, and ${\mathcal{S}}_{\mathrm{beyond}}$ respectively. We define three time-dependent weight functions ${\gamma }_{\mathrm{easy}}\left(t\right)$, ${\gamma }_{\mathrm{hard}}\left(t\right)$, and ${\gamma }_{\mathrm{beyond}}\left(t\right)$ that satisfy ${\sum }_{k\in \{\mathrm{easy},\mathrm{hard},\mathrm{beyond}\}}{\gamma }_k\left(t\right)=1$ for all t. During training, ${\gamma }_{\mathrm{easy}}\left(t\right)$ increases from ${\gamma }_{\min }$ to 1, while ${\gamma }_{\mathrm{beyond}}\left(t\right)$ decreases from $1-{\gamma }_{\min }$ to 0. The hard sample weight is computed as ${\gamma }_{\mathrm{hard}}\left(t\right)=1-{\gamma }_{\mathrm{easy}}\left(t\right)-{\gamma }_{\mathrm{beyond}}\left(t\right)$.

Specifically, ${\gamma }_{\mathrm{easy}}\left(t\right)$ is defined as

$${\gamma }_{\mathrm{easy}}\left(t\right)=\left\{\begin{array}{cc}{\gamma }_{\min }\hfill & \mathrm{if}t<T_1\hfill \\ {\gamma }_{\min }+{\displaystyle \frac{t-T_1}{T_2-T_1}}({\gamma }_{\mathrm{mid}}-{\gamma }_{\min })\hfill & \mathrm{if}{T}_1\le t<T_2\hfill \\ {\gamma }_{\mathrm{mid}}+{\displaystyle \frac{t-T_2}{T_3-T_2}}(1-{\gamma }_{\mathrm{mid}})\hfill & \mathrm{if}{T}_2\le t<T_3\hfill \\ 1\hfill & \mathrm{if}t\ge T_3\hfill \end{array}\right.$$

(17)

where $0<T_1<T_2<T_3$, ${\gamma }_{\min }\in (0,1)$ is the minimum weight for easy samples, and ${\gamma }_{\mathrm{mid}}\in ({\gamma }_{\min },1)$ is the intermediate stage weight value.

At step t, the model’s loss function is adjusted to

$$\begin{array}{cc}\hfill {\mathcal{L}}_i^{\left(t\right)}=& {\gamma }_{\mathrm{easy}}\left(t\right)·{\displaystyle \frac{1}{|{\mathcal{S}}_{\mathrm{easy}}|}}\sum _{x\in {\mathcal{S}}_{\mathrm{easy}}}{\mathcal{L}}_i(x;{\theta }_i)\hfill \\ \hfill & +{\gamma }_{\mathrm{hard}}\left(t\right)·{\displaystyle \frac{1}{|{\mathcal{S}}_{\mathrm{hard}}|}}\sum _{x\in {\mathcal{S}}_{\mathrm{hard}}}{\mathcal{L}}_i(x;{\theta }_i)\hfill \\ \hfill & +{\gamma }_{\mathrm{beyond}}\left(t\right)·{\displaystyle \frac{1}{|{\mathcal{S}}_{\mathrm{beyond}}|}}\sum _{x\in {\mathcal{S}}_{\mathrm{beyond}}}{\mathcal{L}}_i(x;{\theta }_i)\hfill \end{array}$$

(18)

where $i\in \{M,T\}$ represents the model index, and ${\mathcal{L}}_i$ denotes the total loss of model i as defined in Equation (11). The gradient of the sample weighting coefficient $w_t$ is then given by   

$${\displaystyle \frac{\partial w_t}{\partial t}}=\eta ·\left({\mathbb{E}}_{x\sim {\mathcal{S}}_{\mathrm{hard}}}\left[{\nabla }_{w_t}{\mathcal{L}}_i(x;{\theta }_i)\right]-{\mathbb{E}}_{x\sim {\mathcal{S}}_{\mathrm{easy}}}\left[{\nabla }_{w_t}{\mathcal{L}}_i(x;{\theta }_i)\right]\right),$$

(19)

which adaptively regulates the contribution of hard and easy samples during training.

3.2.4. Adaptive Adjustment Mechanism

Stage transition decisions are based on a W-step sliding average of validation set performance:

$${\overline{S}}_t={\displaystyle \frac{1}{W}}{\displaystyle \sum _{w=0}^{W-1}}{S}_{t-w}$$

(20)

where $W>0$ is the sliding window size. A stage transition is trigged when the following condition is satisfied:   

$$\exists \Delta t\ge \Delta t_{\min }:{\overline{S}}_t-{\overline{S}}_{t-\Delta t}<ϵ$$

(21)

This condition indicates that the performance improvement has stagnated. Here $\Delta t_{\min }>0$ represents the minimum stage duration used to prevent premature switching, and $ϵ>0$ denotes the stagnation threshold.

Under standard regularity assumptions and appropriate learning rate scheduling, the proposedAICL strategy facilitates stable optimization and improved convergence behavior. By prioritizing to difficult samples during the early stages of training, AICL encourages the model to focus on instances near decision boundaries, which typically provides stronger and more informative gradient signals. This emphasis helps the model learn discriminative representations more effectively during the initial optimization phase.

3.3. Integration of DMD and AICL Within the DAMR Framework

The DAMR framework integrates the DMD mechanism and the AICL mechanism into a unified optimization framework. Algorithm 1 summarizes the overall training procedure within DAMR.

This integrated approach ensures that knowledge distillation adapts to model capabilities while curriculum learning focuses computational resources on informative samples, leading to improved emotion recognition performance in multimodal speech tasks.

Algorithm 1 Training procedure of the DAMR framework

Require: Training dataset D, validation dataset $D_{\mathrm{val}}$, initial models M and T Ensure: Optimized models $M^{*}$ and $T^{*}$   1: for each training step t do   2:    Evaluate current model capabilities: compute ${\mathrm{EMA}}_t^M$ and ${\mathrm{EMA}}_t^T$   3:    Determine teaching direction and compute dynamic weights: $w_t^{M\to T}$ and $w_t^{T\to M}$   4:    Classify samples based on current predictions: partition dataset into ${\mathcal{S}}_{\mathrm{easy}}$, ${\mathcal{S}}_{\mathrm{hard}}$, and ${\mathcal{S}}_{\mathrm{beyond}}$   5:    Apply stage-specific sample weights: compute ${\gamma }_{\mathrm{easy}}\left(t\right)$, ${\gamma }_{\mathrm{hard}}\left(t\right)$, and ${\gamma }_{\mathrm{beyond}}\left(t\right)$   6:    Update model parameters using the combined loss function ${\mathcal{L}}^{\left(t\right)}$   7:    Monitor validation performance ${\overline{S}}_t$ and adjust training stage when convergence criteria are met   8: end for   9: return Optimized models $M^{*}$ and $T^{*}$

4. Experiments and Analysis

4.1. Datasets

We conduct extensive experiments on two widely used multimodal emotion recognition benchmark datasets to validate the effectiveness of our proposed DAMR framework.

4.1.1. IEMOCAP Dataset

The Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset [34] is a multimodal corpus comprising approximately 12 h of audiovisual data from dyadic conversations. The dataset contains recordings from 10 professional actors (5 male and 5 female) engaged in scripted and improvised scenarios designed to elicit emotional expressions. Following the standard protocol in emotion recognition research [12], we focus on six primary emotion categories from the IEMOCAP dataset: Happy, Sad, Angry, Neutral, Frustrated, and Excited. The dataset contains 5531 utterances in total, with the distribution shown in Table 3. We adopt the standard 5-fold cross-validation protocol, where for each fold, 4 sessions are used for training and 1 session is used for testing, ensuring that no speaker overlap exists between training and testing sets.

Table 3. Emotion distribution in IEMOCAP dataset.

Emotion	Happy	Sad	Neutral	Angry	Excited	Frustrated
Training	595	860	1360	878	828	1472
Testing	149	215	340	220	207	368

4.1.2. MELD Dataset

The Multimodal EmotionLines Dataset (MELD) [35] is an extension of the EmotionLines dataset, comprising approximately 13,000 utterances from 1433 dialogues extracted from the TV series “Friends”. The dataset encompasses seven emotion categories: neutral, surprise, fear, sadness, joy, disgust, and anger. MELD presents a more challenging scenario due to its conversational context and natural emotional expressions in real-world dialogue settings. The dataset is pre-split into training (9989 utterances), validation (1109 utterances), and testing (2610 utterances) sets as shown in Table 4.

Table 4. Emotion distribution in MELD dataset.

Split	Neutral	Surprise	Fear	Sadness	Joy	Disgust	Anger
Training	4710	1205	268	683	1743	271	1109
Validation	470	150	40	111	163	22	153
Testing	1256	281	50	208	402	68	345

4.2. Evaluation Metrics

To comprehensively evaluate the performance of our proposed method and ensure fair comparison with baseline approaches, we employ multiple evaluation metrics capturing different aspects of model performance:

Weighted Accuracy (WA): This is computed as the weighted proportion of correctly classified samples, where each class is weighted by its sample size:

$$\mathrm{WA}={\displaystyle \sum _{c=1}^C}{\displaystyle \frac{N_c}{N}}·{\mathrm{Acc}}_c={\displaystyle \sum _{c=1}^C}{w}_c·{\mathrm{Acc}}_c$$

(22)

where C is the number of classes, $N_c$ is the number of samples in class c, $w_c=N_c/N$ is the weight for class c, and ${\mathrm{Acc}}_c$ is the accuracy for class c.

Weighted-F1 Score: This is the harmonic mean of precision and recall, averaged across all classes with weights proportional to each class’s support, thereby accounting for class imbalance:

$$\mathrm{F}1-\mathrm{weighted}={\displaystyle \sum _{c=1}^C}{\displaystyle \frac{n_c}{N}}·{\displaystyle \frac{2·{\mathrm{Precision}}_c·{\mathrm{Recall}}_c}{{\mathrm{Precision}}_c+{\mathrm{Recall}}_c}}$$

(23)

Per-class F1 Scores: Individual F1 scores for each emotion category, enabling detailed analysis of model performance across different emotional states and identifying specific strengths and weaknesses.

4.3. Implementation Details

4.3.1. Experimental Environment

All experiments are conducted on a computing cluster equipped with NVIDIA RTX 3090 GPUs (24 GB memory). The software environment comprises Python 3.8, PyTorch 1.12.0, CUDA 11.6, and cuDNN 8.4.0. For reproducibility, we implement our framework using the Hugging Face Transformers library (version 4.25.0) and ensure deterministic behavior through proper seed initialization.

4.3.2. Feature Extraction

Following the state-of-the-art CMERC approach [12], we extract features from three modalities:

Text Features: We utilize the pre-trained RoBERTa-large model [36] to extract contextual text representations. Specifically, we apply layer normalization and average the final four hidden layers to obtain a 1024-dimensional feature vector for each utterance. The RoBERTa model is fine-tuned during training with a learning rate of 2 × 10⁻⁵.

Audio Features: We employ the OpenSMILE toolkit [37] with the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [38] configuration to extract 88-dimensional acoustic features, including prosodic, spectral, and voice quality characteristics.

Visual Features: We use a pre-trained DenseNet-121 model [39] to extract facial expression features from video frames. Faces are detected using MTCNN [40] and aligned before feature extraction. We sample 3 frames per second and average the resulting 1024-dimensional features across all frames in each utterance.

4.3.3. Hyperparameter Configuration

To ensure reproducibility and fair comparison, we carefully tune and fix all hyperparameters through grid search on the validation set. Table 5 summarizes the key hyperparameters used in our experiments.

Table 5. Hyperparameter configuration for DAMR framework.

Hyperparameter	IEMOCAP	MELD
General Training
Batch size	32	64
Learning rate	1 × 10−4	5 × 10−5
Weight decay	1 × 10−5	1 × 10−5
Optimizer	AdamW	AdamW
Max epochs	100	80
Early stopping patience	15	12
Random seed	42	42
DMD Parameters
EMA decay coefficient ($\beta$)	0.9	0.9
Temperature coefficient ($\alpha$)	2.0	2.5
Hysteresis threshold ($\delta$)	0.02	0.02
Distillation temperature ($\tau$)	4.0	4.0
Output distillation weight (${\lambda }_1$)	0.5	0.5
Feature distillation weight (${\lambda }_2$)	0.3	0.3
Cycle length (K)	5	5
AICL Parameters
Initial confidence threshold (${\tau }_0$)	0.7	0.7
Decay rate ($\kappa$)	50	40
Minimum easy weight (${\gamma }_{\min }$)	0.1	0.1
Mid-stage weight (${\gamma }_{\mathrm{mid}}$)	0.5	0.5
Stage 1 duration ($T_1$)	20	15
Stage 2 duration ($T_2$)	50	40
Stage 3 duration ($T_3$)	80	65
Sliding window size (W)	5	5
Stagnation threshold ($ϵ$)	0.001	0.001
Min stage duration ($\Delta t_{\min }$)	10	8

4.3.4. Reproducibility Settings

To ensure reproducibility across different runs and hardware configurations, we set random seeds to 42 for Python, NumPy, and PyTorch. We also enable deterministic algorithms in PyTorch via torch.use_deterministic_algorithms(True), configure CUBLAS_WORKSPACE_CONFIG=:4096:8 for deterministic CUDA operations, and disable benchmark mode while enabling deterministic mode in cuDNN.

All experiments are repeated 5 times with different random seeds (42, 123, 456, 789, 1024), and we report the mean and standard deviation of the results.

4.4. Baseline Methods

We compare our proposed DAMR framework with several advanced multimodal emotion recognition methods, including graph-based approaches and transformer-based architectures. The baselines include: M3Net [18], a multivariate, multimodal, and multitask learning network; MGLRA [19], which leverages masked graph learning with recurrent attention; TelME [41], a temporal-enhanced learning framework; CFN-ESA [16], a cross-modal fusion network with emotion-specific attention mechanisms; EmoCaps [14], an emotion capsule network with dynamic routing; MPT-HCL [15], a multimodal prompt tuning method with hierarchical contrastive learning; and CMERC [12], a cross-modal emotion recognition method with confidence calibration that serves as our base model.

4.5. Comparative Experiments

4.5.1. Overall Performance Comparison

Table 6 presents the comprehensive comparison of our DAMR framework against all baseline methods on both IEMOCAP and MELD datasets. Our method achieves state-of-the-art performance across all evaluation metrics.

Table 6. Overall performance comparison on IEMOCAP and MELD datasets. Best results are in bold; second-best results are underlined. Results are averaged over 5 runs with standard deviations.

Method	IEMOCAP		MELD
	WA	F1-Weighted	WA	F1-Weighted
M3Net [18]	68.2	69.24	–	–
MGLRA [19]	71.3	70.10	66.4	64.90
TelME [41]	–	70.48	–	67.37
CFN-ESA [16]	70.78	71.04	67.85	66.70
EmoCaps [14]	–	71.77	–	64.00
MPT-HCL [15]	72.83	72.51	65.86	65.02
CMERC [12]	–	71.98	–	66.85
DAMR (Ours)	78.4±0.3	78.05±0.3	70.2±0.2	69.01±0.2
Improvement over SOTA	–	+5.54	–	+2.16

To ensure a fair and rigorous comparison, all baseline methods are evaluated under identical preprocessing and evaluation protocols. Specifically, we use the same data splits, feature extraction pipelines, and evaluation metrics (weighted-average F1) commonly used in prior studies on IEMOCAP and MELD, without any dataset-specific tuning that could favor our method. For baselines where official implementations are publicly available, we re-run them in our environment; for the remaining methods, we directly cite the results reported in their original publications, as they were obtained using the same standard splits.

Regarding the performance improvement over CMERC on IEMOCAP, we acknowledge that the improvement is substantial and therefore requires careful validation. We attribute this improvement to three main factors: (1) our framework employs richer contextual encoding to capture longer conversational dependencies, which is particularly beneficial for IEMOCAP, where dialogues tend to be longer than those in MELD; (2) we introduce an auxiliary alignment objective that was absent in CMERC; and (3) although CMERC uses a confidence calibration mechanism to reduce overconfident predictions, it does not explicitly model inter-speaker dynamics, which are explicitly considered in our framework. To verify that the observed gains are statistically reliable rather than artifacts of random initialization or data ordering, all experiments are repeated using five different random seeds, and the standard deviation are reported. We further conduct paired t-tests between our method and CMERC across the five runs. The performance improvement on IEMOCAP is statistically significant ($p<0.01$), confirming that the gains are unlikely to be attributable to random variation.

Key Observations:

• Consistent superiority: DAMR achieves the best performance across all metrics on both datasets, demonstrating its effectiveness and generalizability. On IEMOCAP, we obtain 78.4% WA and 78.05% weighted F1, outperforming the previous best method MPT-HCL by 5.54% in weighted F1. On MELD, we achieve 70.2% WA and 69.01% weighted F1, representing a 2.16% improvement in weighted F1 over CMERC and a 2.35% improvement in WA over CFN-ESA. This highlights the ability of DAMR to maintain strong classification performance even when accounting for class imbalance.
• Reduced variance: Our method shows lower standard deviations (±0.2–0.3%) compared to typical baseline variations, indicating more stable training and better convergence properties. This stability stems from the dynamic mutual distillation mechanism that prevents both models from falling into poor local optima.
• Substantial gains on IEMOCAP: The larger improvement on IEMOCAP (+6.07% weighted F1) compared to MELD (+2.16% weighted F1) suggests that DAMR is particularly effective for conversational emotion recognition where contextual understanding and class distribution nuances are crucial. The bidirectional knowledge transfer between text and multimodal models better captures the nuanced emotional dynamics and handles imbalanced categories more effectively.
• Multimodal Advantage: The listed multimodal methods exhibit varying performance levels. However, the substantial gap between DAMR and the suboptimal method CMERC indicates that simply fusing modalities is insufficient—addressing classification ambiguity through dynamic mutual distillation and adaptive curriculum learning is crucial for achieving optimal performance, especially when evaluated with the weighted F1 metric that reflects real-world class distributions.

4.5.2. Per-Class Performance Analysis

To provide deeper insights into our method’s effectiveness across different emotion categories, Table 7 presents the per-class F1 scores on both datasets.

Table 7. Comparison of different methods on the IEMOCAP and MELD datasets, best results are in bold.

Methods	IEMOCAP							MELD
	Happy	Sad	Neutral	Angry	Excited	Frustrated	W-F1	Neutral	Surprise	Fear	Sadness	Joy	Disgust	Anger	W-F1
M3Net [18]	52.74	79.39	67.55	69.30	74.39	66.58	69.24	79.31	58.76	20.51	40.46	63.21	26.17	52.53	65.47
MGLRA [19]	63.50	81.50	71.50	61.10	76.30	67.80	70.10	80.80	59.50	0.00	27.80	66.50	0.00	48.40	64.90
TelME [41]	49.46	83.48	67.42	68.49	77.38	68.63	70.48	80.22	60.33	26.97	43.45	65.67	26.42	56.70	67.37
CFN-ESA [16]	53.67	80.60	71.65	70.32	74.82	68.06	71.04	80.05	58.78	21.62	41.82	66.50	26.92	54.18	66.70
EmoCaps [14]	71.91	85.06	64.48	68.99	78.41	66.76	71.77	77.12	63.19	3.03	42.52	57.50	7.69	57.54	64.00
MPT-HCL [15]	58.13	85.97	66.75	69.96	74.06	69.06	72.51	77.82	58.26	21.52	45.15	60.18	30.36	59.25	65.02
CMERC (baseline) [12]	60.73	81.89	71.65	69.51	77.45	67.02	71.98	80.18	60.42	24.69	40.48	65.30	32.31	54.16	66.85
DAMR (ours)	68.86	89.67	75.58	73.84	85.48	72.58	78.05	81.59	62.72	32.94	43.71	67.15	33.33	58.04	69.01

Key Observations:

• Consistent state-of-the-art performance: DAMR achieves the highest weighted F1 scores on both IEMOCAP (78.05%) and MELD (69.01%), significantly exceeding previous leading methods such as CMERC by 6.07% and 2.16%, respectively. This demonstrates the strong effectiveness and broad generalizability of DAMR across distinct multimodal emotion recognition benchmarks.
• Robust recognition across emotions: DAMR consistently outperforms competitors on a wide range of emotion categories, including challenging and less frequent classes like Fear, Disgust, and Frustrated, reflecting its superior ability to model subtle and complex emotional cues in conversational settings.
• Improved performance on IEMOCAP indicates strength in dialogue modeling: The larger gain on IEMOCAP suggests DAMR excels at capturing the nuanced emotional dynamics within multi-turn conversations, likely due to its enhanced semantic fusion and bidirectional knowledge transfer between modalities.
• Effectiveness beyond simple fusion: Despite all methods leveraging multimodal inputs, the significant performance gap between DAMR and baseline models such as CMERC highlights that advanced mechanisms—like dynamic mutual distillation and adaptive curriculum learning—are essential to resolve classification ambiguity and maximize recognition accuracy.

4.5.3. Confusion Matrix Analysis

Figure 2 presents confusion matrices for CMERC and DAMR on both IEMOCAP and MELD datasets.

Figure 2. Confusion matrices for (a,b) IEMOCAP and (c,d) MELD test sets comparing CMERC baseline and DAMR. Darker colors indicate higher values. DAMR consistently shows reduced off-diagonal confusion across both datasets.

IEMOCAP Key Improvements:

• Happy–Neutral confusion: CMERC confuses 39 Happy samples as Neutral. DAMR reduces this to 21, a 46% reduction, demonstrating significantly improved detection of positive emotional cues beyond the neutral baseline.
• Excited–Frustrated confusion: CMERC misclassifies 28 Excited samples as Frustrated. DAMR reduces this to 17, a 39% improvement, showing better discrimination between high-arousal positive and negative states.
• Frustrated–Sad confusion: CMERC confuses 47 Frustrated samples as Sad, while DAMR reduces this to 38, indicating improved modeling of the distinction between active frustration and passive sadness.
• Sad–Frustrated confusion: DAMR reduces confusion from 69 to 61 cases while improving Sad recognition (254→275 correct predictions), demonstrating better capture of acoustic and contextual differences between these negative emotions.
• Diagonal strengthening: DAMR shows substantial improvements in diagonal values: Happy (92→109, +18%), Excited (199→212, +7%), Neutral (283→296, +5%), Angry (114→125, +10%), and Sad (254→275, +8%).

MELD Key Improvements:

• Neutral–Surprise confusion: CMERC confuses 79 Neutral samples as Surprise, while DAMR reduces this to 59, a 25% reduction, showing improved distinction between neutral expressions and subtle surprise cues in conversational contexts.
• Neutral–Sadness confusion: CMERC misclassifies 44 Neutral samples as Sadness. DAMR reduces this to 34, a 23% improvement, demonstrating better discrimination between emotionally neutral content and subdued sadness.
• Joy–Fear confusion: DAMR reduces confusion from 33 to 29 cases, indicating improved modeling of the acoustic and contextual differences between positive and anxious emotional states.
• Diagonal strengthening: DAMR achieves consistent improvements: Neutral (1050→1098, +5%), Joy (171→177, +4%), Anger (270→271), and Sadness (179→184, +3%).

Cross-Dataset Observations:

The confusion matrix analysis reveals that the DAMR improvements are consistent across both datasets, with particularly strong performance in reducing confusion between emotions with similar valence but different arousal levels (Happy–Neutral in IEMOCAP, Neutral–Surprise in MELD). The dynamic attention mechanism effectively captures subtle multimodal cues that distinguish closely related emotional states, validating the model’s generalization capability across different conversational contexts.

4.5.4. Analysis of Classification Ambiguity Reduction

To directly validate our method’s effectiveness in reducing classification ambiguity, we analyze the MCTI and MITC statistics before and after applying DAMR. Table 8 shows the results.

Table 8. Classification ambiguity analysis: MCTI and MITC sample counts.

Method	IEMOCAP		MELD
	MCTI	MITC	MCTI	MITC
CMERC (Baseline)	238	112	75	124
DAMR (Ours)	276	68	103	79
Change	+38	−44	+28	−45
Percentage	+16.0%	−39.3%	+37.3%	−36.3%

Key Findings:

• Significant MITC reduction: DAMR reduces MITC samples by 39.3% on IEMOCAP and 36.3% on MELD. This indicates that the multimodal model makes fewer mistakes compared to the text-only model, demonstrating effective utilization of complementary multimodal information.
• MCTI increase: The increase in MCTI samples (16.0% on IEMOCAP, 37.3% on MELD) shows that the multimodal model now correctly classifies more samples that the text-only model fails on, further validating the effectiveness of multimodal fusion under DAMR.
• Net ambiguity reduction: The net change (MCTI increase minus MITC decrease) is positive, indicating an overall reduction in classification ambiguity. Specifically, DAMR resolves 44 ambiguous cases on IEMOCAP and 45 on MELD where the baseline multimodal model is incorrect but the text model is correct.
• Mutual refinement effect: These results directly validate our hypothesis that dynamic mutual distillation enables both models to learn from each other’s strengths, with the multimodal model benefiting more from the text model’s linguistic understanding while maintaining its multimodal advantage.

4.6. Ablation Studies

To thoroughly understand the contribution of each component in DAMR, we conduct comprehensive ablation studies by systematically removing or replacing key modules.

4.6.1. Component-Wise Ablation

Table 9 presents the results of removing individual components from the full DAMR framework.

Table 9. Ablation study on key components of DAMR, best results are in bold.

Configuration	IEMOCAP
	WA	UA	F1-Weighted
CMERC (Baseline)	72.3	71.5	71.98
Fixed Mutual Distillation	73.8	73.1	73.52
DMD only	75.2	74.6	74.91
AICL only	74.6	73.9	74.28
DMD + AICL (Full DAMR)	78.4	77.6	78.05

Analysis:

• Fixed vs. Dynamic Distillation: Comparing “Fixed Mutual Distillation” (+1.54% weighted F1) with “DMD only” (+2.93% weighted F1) shows that dynamic weighting based on model capabilities is crucial. Fixed weights cannot adapt to the changing relative strengths of the two models during training, limiting their effectiveness in handling class imbalance and ambiguity.
• Individual contributions: Both DMD (+2.93% weighted F1) and AICL (+2.30% weighted F1) independently improve performance over the baseline, validating the effectiveness of each component in increasing classification accuracy while considering class distribution. DMD shows slightly larger gains, suggesting that addressing model-level ambiguity through mutual distillation is particularly important for improving weighted classification metrics.
• Synergistic effect: The full DAMR framework achieves a +6.07% weighted F1 improvement, which is greater than the sum of the individual contributions from DMD (+2.93%) and AICL (+2.30%). This indicates a positive synergy between the two components, as the combined effect surpasses simple additive expectations. The synergy arises because AICL effectively identifies complex, potentially imbalanced samples that benefit most from the DMD adaptive knowledge transfer, leading to more balanced and robust recognition performance.

4.6.2. DMD Design Choices

We investigate the impact of different design choices in the DMD framework. Table 10 shows the results.

Table 10. Ablation study on DMD design choices (IEMOCAP weighted F1), best results are in bold.

Configuration	F1-Weighted
Full DMD	78.05
w/o EMA (use current validation score)	77.21
w/o Dynamic Weighting (fixed $w=0.5$)	76.52
w/o Alternating Strategy	77.38
w/o Feature-level Distillation	77.61
One-way Distillation (M→T only)	76.18
One-way Distillation (T→M only)	76.84

Key Insights:

• EMA importance: Removing EMA and using current validation scores directly (−0.84% weighted F1) introduces noise and instability. The smoothing effect of EMA is crucial for reliable capability assessment, preventing the model from making erratic distillation decisions based on temporary performance fluctuations, which is especially important when evaluating imbalanced class distributions.
• Dynamic weighting necessity: Using fixed weights (−1.53% weighted F1) significantly degrades performance, confirming that adaptive adjustment based on relative model capabilities is essential. Fixed weights fail to capture the evolving complementary strengths of the two models throughout training, reducing the effectiveness of knowledge transfer under varying class imbalance.
• Alternating strategy value: Removing the alternating training strategy (−0.67% weighted F1) leads to degradation, validating this design to avoid mutual misguidance when both models are initially unreliable, thereby improving stable convergence in the weighted F1 metric.
• Feature-level distillation: Removing feature-level distillation (−0.44% weighted F1) indicates that aligning intermediate representations enhances output-level knowledge transfer, promoting more consistent internal feature learning that reinforces classification robustness across classes.
• Bidirectional superiority: Both one-way distillation variants perform worse than the full bidirectional DMD. Notably, T→M distillation (−1.21% weighted F1) outperforms M→T (−1.87% weighted F1), implying the text model’s linguistic knowledge strongly benefits the multimodal model. Yet, allowing mutual exchange produces the best overall weighted F1 by leveraging complementary model strengths.

4.6.3. AICL Design Choices

We examine different aspects of the AICL strategy. Table 11 presents the results.

Table 11. Ablation study on AICL design choices (IEMOCAP weighted F1), best results are in bold.

Configuration	F1-Weighted
Full AICL	78.05
w/o Confidence Threshold Decay	77.42
w/o Sample Categorization (uniform weights)	76.31
Traditional CL (easy-to-hard)	76.58
Random Curriculum	76.12
Static Hard Sample Focus (no scheduling)	77.19

Analysis:

• Confidence threshold decay: Removing the time-decaying confidence threshold (−0.63% weighted F1) hurts performance. Early in training, a high threshold helps avoid overfitting uncertain predictions, while its gradual decay lets the model leverage improved confidence calibration, enhancing performance especially on minority or complex classes.
• Sample categorization value: Using uniform weights for all samples (−1.74% weighted F1) demonstrates that distinguishing simple, complex, and beyond-threshold samples is critical. Different sample types yield different learning signals, and weighting accordingly helps the model better handle class imbalance and difficult cases.
• Inverse vs. traditional curriculum: Traditional easy-to-hard curriculum learning (−1.47% weighted F1) underperforms our inverse curriculum. For pre-trained models, simple samples contribute limited novel information, while emphasizing complex samples yields stronger gradients and encourages learning more discriminative features across all classes.
• Curriculum vs. random: Randomizing the curriculum (−1.93% weighted F1) performs worst, confirming that structured sample ordering is essential. Random weighting fails to provide the focused learning path that curriculum learning supplies, leading to suboptimal handling of class imbalance.
• Dynamic scheduling necessity: Static hard sample focus without scheduling (−0.86% weighted F1) indicates that gradually reintroducing easier samples prevents overfitting on difficult cases and supports balanced performance across categories.

4.6.4. Hyperparameter Sensitivity Analysis

We analyze the sensitivity of DAMR to key hyperparameters. Figure 3 shows the results.

Figure 3. Hyperparameter sensitivity analysis on IEMOCAP dataset. (a) EMA decay coefficient $\beta$, (b) temperature coefficient $\alpha$, (c) distillation temperature $\tau$, (d) stage duration configuration.

Key Observations:

• EMA decay coefficient ($\mathit{\beta }$): Performance is relatively stable for $\beta \in [0.85,0.95]$, with optimal value around 0.9. Being too small $\beta$ (<0.8) introduces excessive noise, while being too large $\beta$ (>0.95) makes the model slow to respond to capability changes.
• Temperature coefficient ($\mathit{\alpha }$): The model shows a clear optimum at $\alpha =2.0$. Lower values ($\alpha <1.5$) make the weighting function too insensitive to performance differences, while higher values ($\alpha >3.0$) cause overly aggressive switching between models.
• Distillation temperature ($\mathit{\tau }$): Performance peaks at $\tau =4.0$. Lower temperatures ($\tau <3.0$) produce overly sharp distributions that are difficult to learn from, while higher temperatures ($\tau >5.0$) over-smooth the distributions and lose important information.
• Stage duration: The three-stage configuration $(T_1=20,T_2=50,T_3=80)$ works best. Shorter stages lead to insufficient learning in each phase, while longer stages waste computational resources on less informative samples.

4.6.5. Modality Selection Ablation

To justify our choice of using text as the unimodal model in DMD, we compare different modality combinations. Table 12 shows the results.

Table 12. Unimodal baseline performance on test set, best results are in bold.

Unimodal Model	WA	UA	F1-Weighted
Text-only	64.14	62.83	64.42
Audio-only	49.76	47.92	48.51
Visual-only	37.92	36.18	36.85

Justification:

• Text superiority: Text-only models achieve substantially superior performance (64.42% weighted F1) compared to audio-only (48.51% weighted F1, −15.91%) and visual-only (36.85% weighted F1, −27.57%) configurations. This performance gap reflects that linguistic content carries the most explicit and robust emotional cues in conversational contexts, making text the most effective unimodal teacher for distillation.
• Performance-ambiguity correlation: Text-only models, with the highest baseline weighted F1 (64.42%), are the most reliable and capable teachers. In contrast, weaker modalities like audio and visual, with substantially lower weighted F1 scores, risk transferring noisy or unreliable signals, reducing distillation effectiveness.
• Complementarity and information richness: Text provides rich semantic and linguistic information that fundamentally complements lower-level audio prosody and visual expression cues. The marked superiority of text over audio and visual, as measured by weighted F1, highlights its stronger emotional semantic representation. Audio and visual modalities alone demonstrate limited discriminative strength for emotion recognition.
• Ambiguity characteristics: As indicated by modality disagreement counts (MCTI + MITC = 350 on IEMOCAP), text-multimodal conflicts occur more frequently than other pairs, offering richer opportunities for mutual learning. Together with text’s strong weighted F1 baseline, these disagreements represent meaningful, informative signals rather than noise, enabling effective bidirectional knowledge transfer.

4.7. Convergence and Training Dynamics Analysis

To understand how DAMR affects the training process, we analyze convergence behavior and training dynamics.

4.7.1. Dynamic Weight Evolution

Figure 4 visualizes how the distillation weights $w_t^{M\to T}$ and $w_t^{T\to M}$ evolve during training.

Figure 4. Evolution of dynamic distillation weights during training on IEMOCAP. The weights adapt to changing model capabilities, with the multimodal model gradually becoming the stronger teacher.

Analysis:

• Initial phase (epochs 1–20): Both weights are relatively balanced ($w_t^{M\to T}\approx w_t^{T\to M}\approx 0.5$), indicating similar capabilities. The text model sometimes provides stronger guidance due to its pre-trained linguistic knowledge.
• Transition phase (epochs 20–50): $w_t^{M\to T}$ gradually increases while $w_t^{T\to M}$ decreases, showing that the multimodal model is learning to effectively leverage complementary information and becoming the stronger model.
• Mature phase (epochs 50+): $w_t^{M\to T}$ stabilizes around 0.7, indicating that the multimodal model consistently outperforms the text model and provides stronger guidance. However, $w_t^{T\to M}\approx 0.3$ remains non-zero, showing that bidirectional learning continues to be beneficial.
• Adaptive response: The weights show small fluctuations responding to validation performance changes, demonstrating the dynamic nature of the mechanism. These fluctuations are smoothed by EMA, preventing erratic behavior.

4.7.2. Sample Weight Distribution

Figure 5 shows how sample weights change across training stages under AICL.

Figure 5. Distribution of sample weights across training stages. AICL progressively shifts focus from complex/beyond samples to include more simple samples.

Observations:

• Stage 1 (epochs 1–20): Complex and beyond samples receive high weights (0.8–1.0), while simple samples are down-weighted (0.1–0.3). This forces the model to focus on informative, challenging cases.
• Stage 2 (epochs 20–50): Weights become more balanced, with complex samples at 0.6–0.8 and simple samples increasing to 0.4–0.6. This gradual transition prevents abrupt changes in training dynamics.
• Stage 3 (epochs 50–80): All sample types receive relatively uniform weights (0.7–0.9), ensuring comprehensive learning across the entire dataset. Simple samples are now fully incorporated to refine decision boundaries.
• Sample reclassification: As training progresses, many samples transition from complex to simple categories (shown by the increasing proportion of simple samples from 35% to 62%), indicating improved model confidence and reduced ambiguity.

4.8. Qualitative Analysis

To provide intuitive understanding of the effectiveness of DAMR, we present qualitative examples and visualizations.

4.8.1. Case Studies

Table 13 shows representative examples where DAMR successfully resolves classification ambiguity.

Table 13. Case studies of ambiguity resolution by DAMR.

ID	Utterance	Ground Truth	Text-Only	CMERC	DAMR
IEMOCAP Examples
1	“That’s exactly what I wanted to hear!”	Happy	Happy √	Excited ✕	Happy √
2	“I can’t believe you did that to me…”	Sad	Angry ✕	Sad √	Sad √
3	“Whatever, I don’t really care anymore.”	Frustrated	Neutral ✕	Neutral ✕	Frustrated √
4	“Oh my god, this is terrible!”	Angry	Sad ✕	Angry √	Angry √
MELD Examples
5	“Really? You’re kidding me!”	Surprise	Surprise √	Neutral ✕	Surprise √
6	“I’m so scared right now…”	Fear	Sadness ✕	Fear √	Fear √
7	“This is absolutely disgusting.”	Disgust	Anger ✕	Anger ✕	Disgust √
8	“Ha ha, very funny.” [sarcastic tone]	Anger	Joy ✕	Neutral ✕	Anger √

Analysis:

• Example 1 (MCTI case): The text model correctly identifies “Happy” from the positive linguistic content, but CMERC confuses it with “Excited” due to high-energy acoustic features. DAMR resolves this by learning from the text model’s linguistic understanding through DMD, correctly predicting “Happy”.
• Example 2 (MCTI case): The text “I can’t believe you did that to me” could indicate anger, but the sad vocal tone and facial expression lead CMERC to correctly predict “Sad.” DAMR maintains this correct multimodal prediction, while the text-only model fails.
• Example 3 (Beyond case): Both text and CMERC fail on this ambiguous utterance where the neutral words mask underlying frustration. DAMR, through the focus of AICL on such complex cases and the mutual refinement of DMD, learns to detect subtle acoustic and visual cues indicating “Frustrated”.
• Example 8 (Sarcasm case): Sarcasm is particularly challenging, as the literal text meaning contradicts the intended emotion. DAMR successfully leverages acoustic cues (sarcastic tone) and visual cues (facial expression) to correctly identify “Anger”, while both text-only and CMERC fail.

4.8.2. Failure Case Analysis

We analyze the 68 remaining MITC cases on IEMOCAP where the DAMR multimodal model is incorrect but the text model is correct:

• Noisy modalities (32%): Poor audio quality or occluded faces lead the multimodal model astray while text remains reliable.
• Sarcasm and irony (24%): Complex linguistic phenomena where literal multimodal cues contradict intended meaning.
• Ambiguous ground truth (18%): Cases where the gold label itself is debatable, and the multimodal prediction may actually be reasonable.
• Rare emotion patterns (15%): Unusual emotional expressions not well-represented in training data.
• Other factors (11%): Miscellaneous cases including annotation errors and edge cases.

4.9. Discussion

The comprehensive experimental results validate the effectiveness of DAMR across multiple dimensions:

Theoretical validation: The ablation studies confirm our theoretical motivations. DMD’s dynamic weighting based on EMA-smoothed capabilities enables effective bidirectional knowledge transfer, while the AICL inverse curriculum provides stronger learning signals by focusing on informative samples.

Practical effectiveness: DAMR achieves state-of-the-art performance on both IEMOCAP and MELD datasets with consistent improvements across all metrics and emotion categories. The particularly strong gains on minority classes and ambiguous cases demonstrate the method’s ability to address classification ambiguity.

Interpretability: The visualizations of weight evolution, sample weight distribution, and feature spaces provide clear insights into how DAMR works. The dynamic adaptation of distillation weights and progressive curriculum scheduling are observable and interpretable.

Efficiency trade-offs: While DAMR introduces moderate training overhead (approximately 1.3× training time compared to the baseline), its faster convergence and zero inference overhead make it practical for real-world applications. The significant performance gains justify the additional computational cost during training.

Generalizability: Strong cross-dataset performance and consistent improvements across different emotion categories suggest that DAMR learns generalizable representations rather than dataset-specific patterns.

Future directions: The remaining challenges point to several promising research directions: (1) incorporating explicit cultural and contextual modeling, (2) developing better strategies for handling multimodal inconsistency, (3) extending to fine-grained emotion recognition, and (4) integrating with large language models for improved linguistic understanding.

Limitations: We acknowledge that the current study has not explore all possible configurations of the proposed framework. DAMR is specifically designed and evaluated in a multimodal setting where text, audio, and video modalities are jointly available. Consequently, its applicability to unimodal scenarios—such as speech-only or vision-only emotion recognition—remains an open question. The proposed dynamic mutual distillation mechanism relies on capability differences between complementary modalities to facilitate bidirectional knowledge transfer. In the absence of complementary modalities, this mechanism may not operate as effectively as intended. Therefore, extending DAMR to unimodal or modality-incomplete settings, represents an important direction for future research. In particular, future work could investigate whether similar dynamic weighting and mutual refinement strategies can be established within a single modality (e.g., across different acoustic feature streams or alternative visual representations).

In summary, the extensive experiments demonstrate that DAMR effectively reduces classification ambiguity in multimodal speech emotion recognition through the synergistic combination of dynamic mutual distillation and adaptive inverse curriculum learning, achieving new state-of-the-art results while maintaining computational efficiency and interpretability.

5. Conclusions

In this paper, we proposed DAMR, a multimodal speech emotion recognition framework designed to address classification ambiguity through two complementary mechanisms: dynamic mutual distillation (DMD) and adaptive inverse curriculum learning (AICL). DMD enables bidirectional knowledge transfer between a text-only model and a multimodal model, with distillation weights that adapt dynamically to their relative capabilities via EMA-smoothed performance tracking. AICL further enhances learning by prioritizing ambiguous and difficult samples during the early stages of training and gradually incorporating simpler samplesas optimization progresses, thereby providing more informative training signals throughout the training process.

Extensive experiments on the IEMOCAP and MELD datasets demonstrate that DAMR consistently outperforms state-of-the-art baseline methods across all evaluation metrics, achieving weighted F1 scores of 78.05% on IEMOCAP and 69.01% on MELD. Ablation studies show that both DMD and AICL contribute independently and synergistically to overall performance improvement, with the complete framework outperforming variants that contain only individual components. In addition, statistical significance testing ($p<0.01$) confirms that the observed improvements are reliable and not attributable to random variation. Qualitative analysis and confusion matrix comparisons show that DAMR is particularly effective at resolving confusion between emotionally similar categories and handling challenging, including sarcasm and minority emotion classes.

Despite these improvements, several limitations remain. Noisy or occluded modalities, sarcastic expressions, and ambiguous ground-truth annotations continue to account for many of the remaining failure cases. Future work will focus on explicit modeling of multimodal inconsistency, integration with large language models for richer linguistic grounding, and extension to fine-grained as well as cross-cultural emotion recognition scenarios.