MBS: A Modality-Balanced Strategy for Multimodal Sample Selection

Abstract

With the rapid development of applications such as edge computing, the Internet of Things (IoT), and embodied intelligence, massive multimodal data are continuously generated on end devices in a streaming manner. To maintain model adaptability and robustness in dynamic environments, incremental learning has gradually become the core training paradigm on edge devices. However, edge devices are constrained by limited computational, storage, and communication resources, making it infeasible to retain and process all data samples over time. This necessitates efficient data selection strategies to reduce redundancy and improve training efficiency. Existing sample selection methods primarily focus on overall sample difficulty or gradient contribution, but they overlook the heterogeneity of multimodal data in terms of information content and discriminative power. This often leads to modality imbalance, causing the model to over-rely on a single modality and suffer performance degradation. To address this issue, this paper proposes a multimodal sample selection strategy based on the Modality Balance Score (MBS). The method computes confidence scores at the modality level for each sample and further quantifies the contribution differences across modalities. In the selection process, samples with balanced modality contributions are prioritized, thereby improving training efficiency while alleviating modality bias. Experiments conducted on two benchmark datasets, CREMA-D and AVE, demonstrate that compared with existing approaches, the MBS strategy achieves the most stable performance under medium-to-high selection ratios (0.25–0.4), yielding superior results in both accuracy and robustness. These findings validate the effectiveness of the proposed strategy in resource-constrained scenarios, providing both theoretical insights and practical guidance for multimodal sample selection in learning tasks.

1. Introduction

With the rapid development of applications such as edge computing, the Internet of Things (IoT), and embodied intelligence, intelligent systems are gradually shifting from centralized cloud computing toward more distributed and real-time deployment paradigms. In addition to relying on the cloud for centralized model training and updates, an increasing number of tasks require incremental local training on edge devices to quickly adapt to dynamic environments; meanwhile, inference and decision-making must be executed directly on end devices to meet the demands of low latency, privacy preservation, and energy efficiency. This collaborative mode of “training on edge devices + inference on end devices” enables intelligent systems to achieve continual learning and rapid response under resource constraints, but it also imposes stricter requirements on data processing efficiency, model lightweighting, and sample management. In such scenarios, massive multimodal data from sensors, cameras, and interactive devices are continuously generated in a streaming manner, encompassing both high-dimensional information such as vision and speech, and low-dimensional signals such as environmental perception. These data provide rich semantic support for continual learning and adaptation of intelligent systems, but they also introduce new challenges: on the one hand, the computational, storage, and communication resources of edge devices and end nodes are limited, making it difficult to sustain long-term storage and training on full datasets; on the other hand, the data distribution evolves dynamically over time, requiring models to be continuously updated in order to maintain stable performance and generalization in dynamic environments. Therefore, in resource-constrained intelligent systems, how to efficiently process continuously arriving multimodal streaming data becomes a core challenge.

Traditional deep learning relies on one-time large-scale data training, where powerful computational resources and massive datasets enable the development of high-performance models that have achieved breakthrough progress in tasks such as image recognition, speech understanding, and natural language processing. However, this centralized, full-data-driven paradigm struggles to adapt to the continuous changes in data distribution under dynamic environments. Once trained, models remain relatively fixed and lack the ability for real-time updates. In contrast, Incremental Learning can continuously update models as new data arrive, thereby avoiding catastrophic forgetting and better aligning with the needs of distributed architectures. Particularly on edge devices, lightweight models can be incrementally trained to quickly adapt to local environmental changes, thus supporting inference and real-time decision-making on end devices. Nevertheless, edge devices are constrained by limited computational and storage resources, making it infeasible to retain all historical samples over the long term. This raises a critical task: how to correctly and efficiently select the most valuable data samples under limited resources.

Taking a UAV swarm system as an example, as illustrated in Figure 1, the UAVs continuously collect images, audio, and environmental sensor data, which are uploaded to edge nodes for incremental training. However, due to limitations in storage capacity, communication bandwidth, and edge computing power, it is infeasible to store all streaming data in full or to process all multimodal information in a timely manner. More importantly, different modalities exhibit significant disparities in data volume, semantic richness, and noise levels—for instance, the image modality typically contains more detailed semantic information, while audio or environmental sensing modalities may be sparser or more susceptible to interference. These differences can lead to modality imbalance during training, where the model gradually becomes biased toward the “dominant modality” with richer information, while neglecting the “weaker modalities.” This imbalance not only hinders effective cross-modal fusion, but also introduces prediction bias and degrades overall model performance. In resource-constrained edge environments, such issues are further exacerbated. This representative application scenario highlights a more general technical issue: how to design a sample selection strategy in resource-constrained incremental learning environments that balances efficiency and modality fairness. Such a strategy must not only reduce the overhead caused by redundant data under limited computational and storage conditions, but also correctly and efficiently identify and retain the samples most valuable for model convergence, thereby preventing the model from over-relying on a single modality and ensuring that training efficiency is maintained alongside cross-modal balance and generalization capability.

Existing sample selection methods have shown some effectiveness in mitigating data redundancy under resource-constrained conditions. Typically, these approaches are based on sample difficulty or gradient contribution, and can significantly improve training efficiency in single-modality or homogeneous data scenarios. However, their limitations become increasingly apparent when applied to multimodal incremental learning. The phenomenon of modality imbalance not only weakens cross-modal information fusion, but may also lead to long-term model bias and performance degradation. More critically, in resource-constrained edge environments, if the selection strategy fails to account for modality-level balance, such bias can be further amplified, making it difficult for the model to maintain stable and robust performance in dynamic tasks.

This highlights the urgent need for a new sample selection paradigm—one that can simultaneously balance efficiency and modality fairness under limited resources, and correctly and efficiently identify the samples most valuable for model convergence. To this end, we propose a modality-balanced multimodal sample selection strategy. This strategy operates in an online manner: either during early training or continuously, it computes the Modality Balance Score (MBS) for each data sample to quantify the relative contributions of dominant and weaker modalities to model convergence. Based on the distribution of MBS values, the strategy selects the subset of samples for subsequent training.

Specifically, we construct a non-parametric classifier based on a prototypical network, where the confidence score at the modality level is computed by measuring the distance between each modality-specific feature and its corresponding class prototype. This score characterizes the discriminative strength of each modality for the current sample and reflects the model’s reliance on that modality’s information. Take a dual-modality audio-visual learning task as an example: if a sample’s confidence score in the audio modality is significantly higher than that in the visual modality, it indicates that the sample is already highly discriminative in the audio modality, while its contribution from the visual modality is limited—and vice versa. However, such intuitive comparisons are insufficient for systematically assessing the balance of modality contributions across samples. To address this, we introduce MBS, which normalizes the confidence differences across modalities to produce a unified metric that reflects the degree of modality balance for each sample. Based on this score, we can perform real-time selection of samples with relatively balanced modality contributions, prioritizing them for subsequent incremental model updates. This sample selection strategy not only prevents the model from over-relying on a single dominant modality, but also improves learning efficiency under constrained computational and storage resources.

In summary, although existing research has made progress in sample selection under resource-constrained conditions, most approaches focus solely on the overall importance of samples, overlooking the disparities in information content and discriminative power across different modalities. This oversight can be detrimental in multimodal scenarios: it causes models to gradually favor a single dominant modality during long-term learning, thereby amplifying modality imbalance, weakening cross-modal fusion, and ultimately degrading performance. This challenge is particularly pronounced in incremental learning on edge devices, where limited computational and storage resources make it difficult to simultaneously ensure training efficiency and modality fairness. To address this gap, we propose a modality-centric perspective and introduce MBS. Based on MBS, we design a sample selection strategy that guides the selection process by quantifying the contribution disparities across modalities. This enables more efficient and robust multimodal incremental learning in dynamic, resource-constrained environments.

The main contributions of this work are summarized as follows:

(1)

We introduce the concept and computation of the Modality Balance Score (MBS), which effectively quantifies the contribution of multimodal samples to model convergence during training;

(2)

Based on MBS, we propose a sample selection strategy and demonstrate that it can identify effective samples at earlier stages of training, while exhibiting robustness under larger selection ratios;

(3)

We provide extensive experimental validation, comparing our approach with two other representative sample selection strategies on two multimodal datasets. These results offer useful references for researchers with different preferences in choosing sample selection strategies.

2. Related Work

2.1. Incremental Learning in Resource-Constrained Environments

Deploying deep learning models in resource-constrained edge environments faces multiple challenges, including limited computational capacity, storage space, and energy budgets. Traditional large-scale neural networks (e.g., ResNet, Transformer) struggle to meet real-time and energy-efficiency requirements on edge devices. Against this backdrop, extensive research has focused on lightweight network design and inference optimization, spanning multiple levels from model architecture to system implementation. Liu et al. [1] presented a systematic survey that comprehensively reviewed the progress of lightweight deep learning in resource-constrained environments, covering lightweight network architectures (e.g., Mobile Net, Tiny BERT), model compression techniques (pruning, quantization, distillation), and hardware-aware neural architecture search and compiler optimization mechanisms. Their work emphasized the importance of co-design across algorithms and systems to improve the efficiency of edge intelligence deployment. Meanwhile, Shuvo et al. [2] conducted a survey focusing on inference acceleration on edge devices, systematically summarizing mainstream optimization strategies across the model, compiler, and hardware layers, and highlighting multi-level collaborative optimization as the key pathway to efficient edge inference. To adapt to the computational limitations of edge devices, researchers have proposed various lightweight network architectures. Squeeze Net [3] significantly reduces parameter counts through “fire modules”; Shuffle Net [4] improves information flow efficiency via channel shuffle mechanisms; Efficient Net [5] achieves a balance between accuracy and efficiency using compound scaling strategies. More recently, lightweight Transformer models such as Mobile ViT [6] and Mobile Former [7] have been introduced to support multimodal and sequence modeling tasks. These approaches provide feasible architectural foundations for real-time inference on edge devices.

Beyond architectural design, model compression techniques are equally critical for edge deployment. Han et al. [8] pioneered deep compression through unstructured pruning and weight quantization, though the resulting sparsity offered limited acceleration on general-purpose hardware. Molchanov et al. [9] proposed more refined pruning criteria based on Taylor expansion, while Lee et al. [10] introduced the SNIP method, which efficiently identifies redundant parameters at early training stages. In parallel, quantization methods (e.g., Jacob et al. [11] with 8-bit quantization) and knowledge distillation [12] have been widely adopted to further reduce model complexity. At the hardware level, specialized accelerators such as FPGA, ASIC, and TPU have been extensively employed to enhance inference efficiency [13]. Meanwhile, edge inference paradigms continue to evolve, including fully local inference, edge-server inference, and hierarchical collaborative inference [14]. These approaches provide different trade-offs among latency, energy consumption, and privacy preservation.

In resource-constrained edge environments, model deployment faces multiple challenges such as limited computational power and storage capacity, which have motivated researchers to explore lightweight architectures and efficient inference mechanisms. However, relying solely on architectural optimization remains insufficient to meet the demands of continual learning, especially in scenarios where multimodal streaming data continuously arrive. Incremental learning has gradually emerged as a key pathway to enhance model adaptability and robustness. To support stable model updates in dynamic environments, beyond architectural and system-level optimizations, recent studies have also begun to focus on how data-level selection and pruning can improve training efficiency and generalization in incremental learning scenarios.

Two recent authoritative surveys [15,16] systematically reviewed the development of incremental learning from theoretical, methodological, and application perspectives, emphasizing the central role of memory management and data selection in improving training efficiency, enhancing generalization, and mitigating catastrophic forgetting. Traditional approaches often borrow sample selection strategies from single-modality incremental learning, such as the herding method [17], which selects centroid samples in the feature space to preserve class distribution. However, single-modality representativeness is insufficient to ensure cross-modal consistency. To address this, researchers have proposed joint embedding-based selection method [18], which leverage cross-modal aligned embedding spaces to improve the semantic consistency of replay samples. More recently, researchers have attempted to refine data selection strategies through uncertainty modeling and contrastive learning. For example, Ding et al. [19] proposed Uncertainty-Aware Contrastive Learning (UACL) for semi-supervised classification of multimodal remote sensing images. By distinguishing between high-confidence and low-confidence samples, they adopted “hard” and “soft” contrastive learning strategies, respectively, thereby fully exploiting multimodal data under limited annotation conditions. This idea inspires data selection in incremental learning: beyond representativeness, uncertainty information should also be incorporated to dynamically adjust sample importance. Meanwhile, related research has drawn insights from single-modality data pruning and federated learning scenarios. Paul et al. [20] introduced the “data diet” method, demonstrating that identifying and retaining key samples early in training can significantly reduce dataset size without sacrificing performance. Yang et al. [21] proposed a comprehensive data pruning framework for person re-identification tasks, showcasing the potential of data selection to improve generalization and efficiency. Similarly, Gong et al. [22] investigated online data selection under limited storage in federated learning, proposing dynamic selection strategies to balance storage and performance. These ideas provide important references for memory management in multimodal incremental learning.

In multimodal learning scenarios, Peng et al. [23] proposed the Balanced Multimodal Learning framework, which achieves dynamic balance among modalities through gradient modulation. Although its core contribution lies in fusion strategies, it indirectly inspires sample selection: in incremental learning, priority should be given to retaining samples that promote modality balance and complementarity. Similarly, Fan et al. [24] introduced the Prototypical Modal Rebalance (PMR) method, which dynamically adjusts modality learning speed by incorporating class prototypes, effectively alleviating modality imbalance and offering a new perspective for sample selection in memory management. Recent studies have further expanded the boundaries of data selection. For example, Mahmoud et al. [25] proposed the Sieve method, which leverages image captioning models to generate semantic proxy texts and prunes image–text pairs based on semantic consistency evaluation, significantly improving the quality of multimodal pre-training data. Ye et al. [26] introduced the Fit and Prune strategy during inference in multimodal large language models, employing a training-independent visual token pruning mechanism that substantially reduces computational overhead while maintaining performance. These approaches enrich the technical pathways of sample selection in multimodal learning from perspectives such as semantic alignment, modality redundancy compression, and inference efficiency.

Overall, achieving efficient incremental learning in resource-constrained edge environments faces multiple challenges, ranging from model architecture to data management. Existing research has proposed diverse technical pathways in lightweight architectures, inference optimization, memory management, and sample selection, covering dimensions such as representativeness, difficulty, uncertainty, complementarity, and modality balance. Although these methods have made encouraging progress in mitigating catastrophic forgetting, improving training efficiency, and enhancing generalization, in dynamic tasks where multimodal streaming data continuously arrive, significant disparities remain across modalities in terms of data scale, semantic richness, and noise levels. How to achieve adaptive selection and joint optimization tailored to modality differences remains an urgent and unresolved challenge.

2.2. Data Selection Based on Quantified Sample Attributes

In resource-constrained scenarios, data-level selection and pruning have become important means to improve the training efficiency and generalization capability of incremental learning. As a key component of this research direction, sample selection strategies have gradually evolved from heuristic approaches to score-based mechanisms. The core idea is to quantify critical sample attributes—such as representativeness, uncertainty, and modality contribution—in order to assist in evaluating their value and retention priority during model updates.

For quantifying the importance of training samples, Paul et al. [20] proposed two representative metrics: the GraNd (Gradient Normed) score and the EL2N (Error L2-Norm) score, which have been used as baselines or Crucial References in recent works on continual learning and data pruning [21,22]. The former measures a sample’s contribution to model learning based on the gradient norm of its parameter updates, while the latter evaluates sample difficulty by computing the L2 distance between the predicted distribution and the ground-truth label. Their mathematical definitions are as follows:

(1)

GraNd score:

$$\mathrm{GraNd}\left(x_i,y_i\right)={\parallel {\nabla }_{\theta } \mathcal{l}\left(x_i,y_i;\theta \right)\parallel }_2$$

(1)

where $\mathcal{l}(x_i,y_i;\theta )$ is the cross-entropy loss function. This score reflects the driving force of the samples on gradient updates under the current parameters. The larger the gradient norm, the more significant the impact of the samples on model training.

(2)

EL2N score:

$$\mathrm{EL}2\mathrm{N}\left(x_i,y_i\right)=\parallel p_{\theta }\left(x_i\right)-y_i{\parallel }_2$$

(2)

where $p_{\theta }\left(x_i\right)$ represents the prediction probability distribution of the model for samples $x_i$ under parameter $\theta$, and $y_i$ is the one-hot label vector. The higher the score, the more difficult it is to learn and the higher the importance of the sample.

Both metrics can be computed in the early stages of training and are widely used for data pruning and subset selection. They help reduce training overhead while preserving model performance, playing a crucial role in improving the efficiency and stability of incremental learning.

Furthermore, in multimodal streaming tasks, the disparity in information content and discriminative power across modalities makes it difficult for traditional selection criteria to balance modality fairness and selection efficiency. As a result, modality balance quantification has become one of the key guiding principles for sample selection. In this regard, Fan et al. [24] proposed the Prototypical Modal Rebalance (PMR) method, which systematically models the common issue of modality imbalance in multimodal learning. The core idea is to compute class prototypes within each unimodal representation space and construct a non-parametric classifier to independently assess the learning effectiveness of each modality. Based on the distribution of distances between samples and their corresponding prototypes, the method designs metrics to characterize the learning progress disparity across modalities. Building on this, the authors introduced the Prototypical Cross-Entropy (PCE) loss, which enhances clustering in weaker modalities by minimizing the distance between samples and their class prototypes. In parallel, they proposed Prototypical Entropy Regularization (PER) to suppress overly rapid convergence in dominant modalities, thereby achieving dynamic balance across modalities. This framework provides a general solution for quantifying and regulating modality balance, independent of specific model architectures or fusion strategies. It effectively mitigates performance bottlenecks caused by modality imbalance and improves the robustness and generalization of multimodal learning.

Inspired by the impact of modality imbalance on model performance in multimodal incremental learning, this work further proposes a sample selection strategy based on modality balance quantification, aiming to identify samples with more balanced modality contributions at early training stages.

3. Basic Model

Our work considers a dual-modality (audio-visual) incremental learning paradigm. We further analyze the sample selection requirements for multimodal streaming data under resource-constrained conditions, and formulate an optimization objective that provides a theoretical foundation for the design of subsequent selection strategies.

3.1. Dual-Modality Incremental Learning Paradigm

Consider a dual-modal (audio, video) incremental learning paradigm, which defines a sample reached in the time step $t$ as a binary group $\left(x_t,y_t\right)$:

$$x_t=(\left(x_t^a,\hspace{0.25em}{x}_t^v\right), y_t)$$

(3)

where $x_t^a\in R^a$ represents audio modal characteristics, such as Mel-spectrogram; $x_t^v\in R^v$ represents video modal characteristics, usually image frames; and $y_t\in Y_t$ represents sample label. From this, two types of encoded feature representations can be defined:

$$z_t^a={En}_a\left(x_t^a\right), z_t^v={En}_v\left(x_t^v\right)$$

(4)

As is shown in Figure 2, ${En}_a:R^a\to R^m$ consists of MFCC feature extraction followed by a recurrent neural network (RNN) to capture temporal dynamics in the audio stream, and ${En}_v:R^v\to R^m$ consists of a MobileNet_V2 backbone followed by an RNN to encode visual expressions over time. Their role is to map the features of their respective modes into a unique multimodal feature space. These modality-specific embeddings are then fused to obtain a joint representation:

$$z_t^m=f_{\theta }\left(z_t^a, z_t^v\right)$$

(5)

Here $z_t^m\in R^m$ represents the fused multimodal features. Among them, $f_{\theta }$ can be a variety of different feature fusion methods, including feature summation, feature concatenation, Feature-wise linear Modulation (FiLM), etc. After feature fusion, the classifier $g$ outputs the category probability $P_t$:

$$P_t=softmax\left(g\left(z_t^m\right)\right)$$

(6)

A basic objective function can be defined for incremental learning on audio and video bimodal data samples. The core idea is cross-entropy + knowledge distillation:

$$L_t={\displaystyle \frac{1}{B_t}}\sum _{\left(x,y\right)\in B_t}{L}_{CE}\left(P\left(x\right),y\right)+ {\displaystyle \frac{{\lambda }_{KD}}{M_t}}\sum _{\left(x,y\right)\in M_t}{L}_{KD}\left(P\left(x\right),P_{pre}\left(x\right)\right)$$

(7)

$B_t$ represents the current sample batch, $M_t$ represents a limited sample playback buffer, $L_{CE}$ represents the cross-entropy loss, $L_{KD}$ represents the distillation loss that retains the knowledge of the previous task, $P_{pre}$ represents the model output of the previous stage, and ${\lambda }_{KD}$ is a hyperparameter used to control the knowledge distillation loss.

To support incremental learning, class prototypes for each modality are updated online. For a new sample $(x_t,y_t)$, the prototype of class $y_t$ is updated using an exponential moving average (EMA):

$$c_{y_t}^{\left|m\right|}\leftarrow \alpha c_{y_t}^{\left|m\right|}+(1-\alpha )z_t^{\left|m\right|}$$

(8)

where $\left|m\right|\in \{a,v\}$ denotes the modality and α controls the update momentum. This strategy avoids storing all past samples and ensures stable prototype estimation under streaming data.

3.2. Optimization Objective

Given the large data scale, significant modality-specific disparities, and resource constraints, retaining all raw data indiscriminately during incremental learning would rapidly exhaust computational and storage resources, degrade training efficiency, and increase inference latency. Therefore, our optimization objective focuses on identifying and retaining the most convergence-critical samples as early as possible during edge-side incremental training, under limited computational and storage conditions. This is achieved through an effective sample selection strategy that also ensures balanced modality contributions, thereby improving training efficiency, convergence quality, and robustness across varying selection ratios.

Figure 3 illustrates the basic logic of data selection in the context of multimodal incremental learning. To facilitate our study, we treat the notion of “early training” as an independent pre-training process. This design allows us to flexibly set different numbers of pre-training epochs to observe sample attributes at various stages of early training. Our goal is to identify and retain, through the selection strategy $\mu$, the samples most valuable for model convergence within as few training iterations as possible in the current batch. At the same time, strategy $\mu$ should exhibit strong robustness—namely, even under more stringent resource-constrained conditions (where fewer samples are retained for training), it should still maintain model accuracy at a reasonable level.

Define the minimization goal $J$:

$$min J\left(\mu , \rho , R\right)=L\left(B_t^{\mu }\right)+{\lambda }_1·C_{eff}\left(R\right)+{\lambda }_2·C_{rob}(\mu , \rho )$$

(9)

where $L\left(B_t^{\mu }\right)$ represents the training loss on the subset of samples in the current batch filtered by the policy $\mu (\rho , R)$; $C_{eff}\left(R\right)$ represents the efficiency cost caused by the pre-training rounds, which reflects “whether the target performance can be achieved in as few training rounds as possible”; $C_{rob}(\mu , \rho )$ represents the robustness cost, which characterizes the stability and adaptability of the policy under different screening ratios, especially whether reasonable performance can be maintained under extreme resource constraints (large $\rho$); ${\lambda }_1, {\lambda }_2$ represent the penalty weights. In short, this optimization goal aims to quickly find the most valuable and stable samples to the model in the early stage of training through a reasonable sample screening strategy, with as few training rounds as possible and limited resources, thereby considering convergence speed, efficiency and robustness.

4. Main Method

Starting from the contribution disparities across modalities, we design a modality-balanced sample selection method that identifies and retains samples with more balanced modality contributions during the early stages of training. This approach enhances both the convergence speed and the long-term stability of the model.

4.1. Modality Balance Quantification: MBS

In Equation (3) presented in the previous section, we discussed the fused feature output of the two modalities. This process can be further expressed in detail as:

$$z_t^m=f_{\theta }\left(z_t^a, z_t^v\right)=W·\left(z_t^a; z_t^v\right)+b$$

(10)

This is the output of a linear classifier, where $W$ denotes the weight matrix applied to the concatenated feature representations of the two modalities, and $b$ is the bias term used to maintain output stability. Taking a simple summation strategy as an example (i.e., strategy $\theta$ corresponds to summation), the independent outputs of each modality can be decomposed as follows:

$$f_{\theta }\left(z_t^a, z_t^v\right)=W_a·z_t^a+b_a+W_v·z_t^v+b_v$$

(11)

Accordingly, we can quantify the contribution of each modality using its softmax output, which is commonly referred to as the confidence of sample $x_t$ in a given modality, defined as:

$$\begin{gathered} P_a=softmax\left(g\left(z_t^a\right)\right)=softmax{\left(W_a·z_t^a+b_a\right)}_y, \\ {P}_v=softmax\left(g\left(z_t^v\right)\right)=softmax{\left(W_v·z_t^v+b_v\right)}_y \end{gathered}$$

(12)

According to Equation (11), a sample $x_t=\left(\right(x_t^a,x_t^v),y_t)$ has confidence scores $P_a$ and $P_v$ in the audio and video modalities, respectively. Generally, the relationship between two variables can be characterized either by their difference, which emphasizes absolute variation, or by their ratio, which reflects relative change. In the context of this work, the confidence ratio between the audio and video modalities captures the relative contribution strength across modalities. We define this ratio as the Modality Balance Score (MBS). Under the summation strategy $\theta =\mathrm{sum}$, the MBS of a multimodal sample $x_t$ can be expressed as:

$${MBS}_{x_t}={\displaystyle \frac{P_a}{P_v}}={\displaystyle \frac{softmax{\left(W_a·z_t^a+b_a\right)}_y}{softmax{\left(W_v·z_t^v+b_v\right)}_y}}$$

(13)

4.2. Multimodal Sample Selection Strategy: Based on MBS

Evidently, ${MBS}_{x_t}$ reflects the ratio of contributions from the two modalities of a multimodal sample $x_t$ in the gradient descent computation. When MBS approaches 1, it indicates that the two modalities contribute comparably. Conversely, when $MBS>1$ or $0<MBS<1$, one modality dominates, leading to modality imbalance. Therefore, in principle, sample selection should retain as many samples as possible with MBS values close to 1, while discarding those dominated by a single modality. Specifically, samples with ${MBS}_{x_t}\in \left(\mathrm{0,1}\right]$ and ${MBS}_{x_t}\in (1,+\mathrm{\infty })$ should be preserved in equal proportions.

As mentioned in Section 3.2, to facilitate our study, we treat the notion of “early training” as an independent pre-training process, with the number of pre-training epochs denoted by $R$. Accordingly, we define the batch of samples after $R$ rounds of pre-training as the set:

$$B_R=\left\{x_1,x_2,\dots ,x_n\right\}$$

(14)

Let the selection ratio be $\rho \in \left(\mathrm{0,1}\right]$. Then, the sample set obtained after applying the selection strategy $\mu$ can be expressed as:

$$B_R^{\mu }={\mathrm{T}\mathrm{o}\mathrm{p}}_{\rho }({\mathrm{B}}_R,\hspace{0.25em}\mu \left(x_t\right))$$

(15)

In principle, ${\mathrm{Top}}_{\rho }(B_R,\mu (x_t\left)\right)$ denotes selecting the top $\rho \cdot n$ samples from the set $B_R$ that minimize the function value. The selection strategy is defined as $\mu \left(x_t\right)=\mid \mathrm{MBS}\left(x_t\right)-1\mid$, which we refer to as the modality balance deviation.

$$\begin{gathered} B_R^{\mu <}={\mathrm{T}\mathrm{o}\mathrm{p}}_{{\displaystyle \frac{\rho }{2}}}\left({\mathrm{B}}_R,\left|{MBS}_{x_t}-1\right|\right) \\ {B}_R^{\mu >}={\mathrm{T}\mathrm{o}\mathrm{p}}_{{\displaystyle \frac{\rho }{2}}}\left({\mathrm{B}}_R,\left|{MBS}_{x_t}-1\right|\right) \\ {B}_R^{\mu }=B_R^{\mu <}\cup B_R^{\mu >} \end{gathered}$$

(16)

In practice, since the learning task over the sample set inherently exhibits varying degrees of dependence on different modalities, the median MBS of the samples after $R$ rounds of pre-training is typically not equal to 1. If the theoretical selection strategy were applied directly, it might result in insufficient samples being retained for formal training. Therefore, let the median MBS of the samples after $R$ rounds of pre-training be denoted as $m$. Based on this, the practical selection strategy with respect to the MBS is defined as:

$$\begin{gathered} B_R^{\mu <}={\mathrm{T}\mathrm{o}\mathrm{p}}_{{\displaystyle \frac{\rho }{2}}}\left({\mathrm{B}}_R,\left|{MBS}_{x_t}-m\right|\right) \\ {B}_R^{\mu >}={\mathrm{T}\mathrm{o}\mathrm{p}}_{{\displaystyle \frac{\rho }{2}}}\left({\mathrm{B}}_R,\left|{MBS}_{x_t}-m\right|\right) \\ {B}_R^{\mu }=B_R^{\mu <}\cup B_R^{\mu >} \end{gathered}$$

(17)

At this point, the sample set $B_R^{\mu }$, obtained through the selection strategy $\mu$, can be utilized for subsequent incremental training. This process can be expressed as Algorithm 1.

Algorithm 1: Multimodal Sample Selection based on MBS

Input: Batch of samples $B_R=\left\{x_1,x_2,\dots ,x_n\right\}$ Pretrained encoders $h_a,h_v$ Strategy parameters $\mu =\{W_a,W_v,b_a,b_v\}$ Selection ratio $\rho \in \left(\mathrm{0,1}\right]$ Output: Filtered sample set $B_R^{\mu }$ Steps: 1:  Initialize ${MBS}_{list}=\left[\right]$ 2:  For each sample $x_i\in {\mathcal{B}}_R$ do 3:    $z_i^a=h_a\left(x_i^a\right)$ 4:    $z_i^v=h_v\left(x_i^v\right)$ 5:    $p_i^a$ = $softmax{\left(W_a·z_t^a+b_a\right)}_{y_i}$ 6:    $p_i^v$ = $softmax{\left(W_v·z_t^v+b_v\right)}_{y_i}$ 7:    ${MBS}_i$ = $p_i^a/p_i^v$ 8:    Append ($x_i$, ${MBS}_i$) to ${MBS}_{list}$ 9:  End For 10: Compute $m=median\left(MB{S}_i|\left(x_i,MB{S}_i\right)\in MB{S}_{list}\right)$ 11: Initialize $B_R^{\mu <}=\left[\right],B_R^{\mu >}=\left[\right]$ 12: For each $\left(x_i,MB{S}_i\right)\in MB{S}_{list}$ do 13:    If $0<MB{S}_i\le 1$ then 14:      Append $\left(x_i,\left|MB{S}_i-m\right|\right)$ to $B_R^{\mu <}$ 15:    Else if $MB{S}_i>1$ then 16:      Append $\left(x_i,\left|MB{S}_i-m\right|\right)$ to $B_R^{\mu >}$ 17:    End If 18: End For 19: Sort $B_R^{\mu <}$ and $B_R^{\mu >}$ by deviation $\left|MB{S}_i-m\right|$ in ASC 20: $K=floor(\rho ·n/2)$ 21: Initialize $B_R^{\mu }$ = { } 22: Add top K samples from $B_R^{\mu <}$ to $B_R^{\mu }$ 23: Add top K samples from $B_R^{\mu >}$ to $B_R^{\mu }$ 24: Return $B_R^{\mu }$

4.3. Effectiveness

To preliminarily verify the effectiveness of the sample selection strategy based on modality balance deviation, we conducted 10 rounds of pre-training on the CREMA-D dataset. After pre-training, we computed the EL2N scores, GraNd scores, and MBS for all samples in the training set, with the selection ratio set as $\rho \in \left\{\mathrm{0.1,0.2,0.3}\right\}$.

Under EL2N score ranking, low-score samples that are easier to learn and yield stable predictions were retained; Under GraNd score ranking, high-score samples that exert greater influence on parameter updates were retained; Under MBS ranking, samples at both ends were discarded, with a proportion of $\rho /2$ removed from each side.

Subsequently, 20 rounds of formal training were performed on the sample sets obtained through the above strategies, and the final model accuracies were recorded as shown in

Table 1. Final model accuracy for preliminary test for 10 rounds of pre-training + 20 rounds of formal training. (All tables use bold numbers to represent the maximum value of each column.)

	Ratio	0.1	0.2	0.3
Strategy
EL2N		0.8000	0.8097	0.7870
GraNd		0.8065	0.8076	0.8011
MBS		0.8259	0.8141	0.8065

. The selection strategy based on MBS achieved performance comparable to the other two strategies (details provided in the Appendix A), with the final model accuracy being slightly superior.

This preliminary experiment validates the feasibility and effectiveness of MBS in multimodal sample selection, providing prior support for subsequent extensive experimental evaluations.

5. Evaluation

5.1. Experimental Setup

Our experiments were conducted on two audio-visual multimodal datasets: CREMA-D and AVE. CREMA-D (Crowd-sourced Emotional Multimodal Actors Dataset) [27] is a widely used multimodal dataset for emotion recognition research. It contains 7442 audio-visual clips recorded by 91 actors of diverse genders, ages, and ethnic backgrounds. Each clip features performances of six basic emotions (anger, disgust, fear, happiness, neutral, sadness) as well as an additional emotion, “surprise,” expressed under varying intensities and contexts. The dataset provides both audio and video modalities, supporting cross-modal emotion recognition and fusion studies. AVE (Audio-Visual Event Dataset) [28] is designed for audio-visual event recognition. It consists of more than 4000 ten-second audio-visual clips collected from YouTube, covering 28 categories of everyday events (e.g., “dog barking,” “drum beating,” “street noise,” “crowd cheering”). A distinctive feature of this dataset is that events often exhibit both audio and visual manifestations, but the degree of modality dependence varies significantly. For instance, some events rely primarily on audio signals (e.g., “thunder”), while others depend more on visual information (e.g., “playing musical instruments”).

All experiments in this study were conducted on a high-performance workstation equipped with two NVIDIA RTX A4000 GPUs (16 GB memory each), an Intel Core i9-12900 processor, and 64 GB RAM. The software environment was based on Ubuntu 24.04, with Python 3.9 as the primary programming language. Deep learning experiments were implemented using PyTorch 2.2, combined with CUDA 12.1 and cuDNN 8.9 to fully leverage GPU capabilities.

5.2. Experiment A: Multi-Attribute Observation of the Sample Set

For the CREMA-D and AVE datasets, we first randomly split each dataset into training and test sets with an 8:2 ratio. We then conducted 30 rounds of pre-training and computed the EL2N, GraNd, and MBS values for the training samples involved in this process. A review of the properties described by these three measures is summarized in

Table 2. Characteristics of EL2N, GraNd, and MBS.

Metric	Focus	Sample Attribute Described
EL2N	Learning Difficulty	Indicates whether a sample is easy for the model to learn, and whether it contains noise or lies near the decision boundary
GraNd	Contribution to Parameter Updates	Reflects the extent to which a sample drives the training direction of the model
MBS	Modality Balance	Characterizes whether the contributions of different modalities within a multimodal sample are balanced

.

EL2N score measures the learning difficulty of a sample. A lower EL2N value indicates that the sample is easier for the model to learn, whereas a higher value suggests that the sample is harder to fit, potentially containing noise or lying near the decision boundary; GraNd score quantifies the contribution of a sample to parameter updates. Samples with higher GraNd values generally exert stronger influence on the convergence direction of the model and are therefore more valuable during training; MBS characterizes the balance of contributions between different modalities within a multimodal sample.

Figure 4 and Figure 5 present scatter plots illustrating the evolution of three sample attributes across the CREMA-D and AVE datasets during 30 rounds of pre-training. These attributes become increasingly pronounced as the number of pre-training epochs increases.

Regarding the evolution of EL2N and GraNd, both exhibit a rapid decline in scores during the early training stage (epochs 1–10), indicating that the model quickly learns a subset of “easy samples” and gradually establishes a stratified structure of sample difficulty. At the same time, the GraNd distribution in both datasets demonstrates a long-tail pattern, suggesting that a small number of samples exert significant influence on parameter updates.

However, in the CREMA-D dataset, EL2N scores tend to stabilize in the mid-training stage (after approximately 15 epochs), with the proportion of difficult samples gradually shrinking. The tail of the GraNd distribution also converges, reflecting the limited variation in sample difficulty for emotion recognition tasks. In contrast, the AVE dataset maintains a high degree of dispersion in EL2N scores throughout the entire 30-epoch training process, with difficult samples persisting for longer. Its GraNd distribution exhibits an even longer tail, indicating stronger sample heterogeneity in event recognition tasks and a greater reliance of the model on a small set of critical samples.

The evolution of MBS clearly reveals the modality balance differences between the two datasets. In the CREMA-D dataset, MBS values are generally distributed evenly around 1, with a slight bias toward values greater than 1. This indicates that the audio modality contributes more strongly, though the overall balance remains relatively stable. In contrast, the AVE dataset shows MBS values concentrated within the interval $\left(\mathrm{0,1}\right)$, suggesting that the video modality holds absolute dominance in the sample set.

Moreover, the distribution of CREMA-D gradually converges toward values close to 1 in the mid-training stage, implying that the contributions of audio and video modalities become more balanced and cross-modal synergy is better achieved. By comparison, the AVE distribution remains more dispersed throughout the entire 30 rounds of training, with a substantial proportion of samples still deviating from balanced states. This phenomenon indicates that, relative to emotion recognition tasks, event recognition tasks are more complex, with varying degrees of dependence on visual and auditory information across different events.

Further, we plotted the median ratio $(m_r/m_{r-1})$ of the three scores across the CREMA-D and AVE datasets during 30 rounds of pre-training, as shown in Figure 6. This trend provides a more intuitive illustration of how different sample attributes gradually emerge and stabilize throughout the pre-training process. For the CREMA-D dataset, due to the relative simplicity of the task, all three scores tend to stabilize within the first 10–15 epochs. Among them, MBS converges faster than EL2N and GraNd, indicating that modality balance is more easily established in this dataset. In contrast, the AVE dataset exhibits a longer transition period across all three scores within 30 epochs. The convergence speed of MBS is comparable to that of EL2N and GraNd, reflecting that modality balance in complex event recognition tasks requires more time to manifest.

This observation provides important insights for subsequent sample selection strategies and the choice of selection windows: on CREMA-D, the MBS-based selection strategy can take effect at earlier training stages, quickly identifying and retaining modality-balanced samples, thereby improving training efficiency while maintaining performance. On AVE, however, the stability of MBS emerges later, suggesting that the selection window should be appropriately delayed to ensure that the retained samples genuinely reflect modality balance.

5.3. Experiment B: Model Performance Under Different Selection Strategies

To further validate the generalizability and advantages of the MBS-based selection strategy in formal training, we conducted repeated training experiments based on the results of the 30-round pre-training. Specifically, for both the CREMA-D and AVE datasets, we computed sample scores (EL2N, GraNd, and MBS) at the 5th, 10th, 15th, 20th, 25th, and 30th pre-training epochs. Using these scores, we applied nine different selection ratios $\left(\mathrm{0.1,0.15,0.2,0.25,0.3,0.35,0.4,0.45,0.5}\right)$ to construct filtered sample sets, on which multiple rounds of training were performed to ensure stability and statistical significance of the results. In addition, two control groups were introduced: BASE (no selection, retaining all samples) and RANDOM (randomly selecting samples under the same selection ratios). BASE serves as a reference for the upper bound of model performance, representing the best achievable results when no samples are discarded. RANDOM, on the other hand, is used to evaluate the effectiveness of score-driven selection strategies relative to indiscriminate random selection, ensuring that the observed performance improvements indeed stem from informed sample selection rather than mere changes in sample size.

5.3.1. Model Performance on the CREMA-D Dataset

Figure 7 illustrates the final accuracy of the emotion classification model on the CREMA-D dataset under three selection strategies (EL2N, GraNd, and MBS), applied with different pre-training epochs and selection ratios. Overall, as the number of pre-training epochs increases, the performance of the three score-based selection strategies on CREMA-D gradually shifts from fluctuation to stability. Specifically, in the early training stage (e.g., the 5th and 10th epochs), the results of EL2N-based and GraNd-based selection remain unstable, with model accuracy showing only limited improvement over the RANDOM baseline and, in some cases, even degradation under certain selection ratios. This indicates that the corresponding scores have not yet sufficiently converged, making it difficult to effectively distinguish valuable samples. After the 15th epoch, however, the performance of EL2N and GraNd improves and stabilizes, suggesting that their selection effects become more reliable at this stage. In contrast, the MBS-based strategy demonstrates stable advantages at an earlier stage (around the 10th epoch), significantly outperforming RANDOM under low selection ratios. This suggests that modality balance can serve as a reliable selection criterion even in the early stages of training.

Focusing on several key epochs after stabilization (post-15th epoch), the differences among the three strategies under varying selection ratios become clearer. At low ratios (0.1–0.2), MBS consistently achieves the highest accuracy, in some cases approaching the BASE performance, indicating that it can compress the sample set while maintaining model performance. During this stage, GraNd performs slightly better than EL2N but remains inferior to MBS. EL2N maintains reasonable performance at low ratios but declines more rapidly as the ratio increases. At medium-to-high ratios (0.3–0.4), all three strategies experience performance drops, but MBS exhibits the smallest decline, demonstrating stronger robustness. Even at higher ratios (0.4–0.5), where EL2N and GraNd nearly fail (with GraNd only effective when the model is close to convergence), MBS still manages to control performance loss within a relatively small range.

5.3.2. Model Performance on the AVE Dataset

Figure 8 presents the final accuracy of the emotion classification model on the AVE dataset under three selection strategies (EL2N, GraNd, and MBS), applied with different pre-training epochs and selection ratios. Compared with CREMA-D, the accuracy fluctuations on AVE are noticeably smaller across all three strategies. This indicates that event recognition tasks exhibit greater overall stability in performance, but it also implies that differences among strategies require more fine-grained comparison to be revealed.

From the overall trend, as the number of pre-training epochs increases, the performance of all three strategies gradually shifts from slight early fluctuations to stability. Unlike CREMA-D, the early instability of EL2N and GraNd is less pronounced on AVE; however, their improvements under low selection ratios remain limited, and their performance declines more sharply at higher ratios. In contrast, the MBS-based strategy begins to demonstrate smaller performance fluctuations in the mid-training stage (after the 15th epoch). At medium-to-high selection ratios (0.3–0.4), although all three strategies experience performance drops, MBS exhibits the smallest decline, showing stronger robustness.

5.4. Comprehensive Analysis

In the previous experimental results, we have already observed performance differences among the three strategies on the CREMA-D and AVE datasets. To avoid drawing conclusions limited to a single dataset, this section further provides a unified comparison and quantitative analysis of the three strategies from the perspective of abstract optimization objectives, focusing on two dimensions: robustness and efficiency. By constructing robustness scores and efficiency scores, we can more intuitively reveal the strengths and weaknesses of different strategies under cross-task conditions. For strategy $\mu$, the robustness score is calculated as the weighted sum of wins across different epochs $\sum {s\left(\rho \right)}_{win}^{\mu }, s\left(\rho \right)=\rho$, as shown in

Table 3. Robustness Scores of Different Strategies on the CREMA-D Dataset.

	Epoch	5th	10th	15th	20th	25th	30th
Strategy
RANDOM		1.70	0.95	0.20	0.85	0.70	0
EL2N		0.55	0.625	0	0.15	0.175	0.70
GraNd		0.45	0.10	0	0	0	1.35
MBS		0	1.025	2.50	1.70	1.825	0.65

and

Table 4. Robustness Scores of Different Strategies on the AVE Dataset.

	Epoch	5th	10th	15th	20th	25th	30th
Strategy
RANDOM		0.8	0.40	0.2	0.25	0.25	0.25
EL2N		0.20	0.45	0.65	0.80	0.60	0.25
GraNd		0.45	0.70	0	0.375	0.20	1.55
MBS		1.25	1.15	1.85	1.275	1.65	0.65

. Similarly, the efficiency score is computed as the weighted sum of wins across different selection ratios $\sum {s\left(R\right)}_{win}^{\mu }, s\left(R\right)=R^{rev}/10$, as shown in

Table 5. Efficiency Scores of Different Strategies on the CREMA-D Dataset.

	Ratio	0.1	0.15	0.2	0.25	0.3	0.35	0.4	0.45	0.5
Strategy
RANDOM		0	0	3.75	0	0	6.0	4.5	0	1.0
EL2N		4.0	5.5	2.5	1.75	6.0	0	0	0	0
GraNd		2.5	0	3.0	3.0	0	0	0.5	0.5	0.5
MBS		4.0	5.0	1.25	5.75	4.5	4.5	5.5	10	9.0

and

Table 6. Efficiency Scores of Different Strategies on the AVE Dataset.

	Ratio	0.1	0.15	0.2	0.25	0.3	0.35	0.4	0.45	0.5
Strategy
RANDOM		2.5	0	2.0	6.0	0	0	4.0	0	0
EL2N		0	2.0	4.0	2.5	0.33	1.5	1.5	0	5.0
GraNd		4.5	7.5	3.5	0	0.33	0	0.5	0.5	2.5
MBS		3.5	1.0	1.0	2.0	9.83	9.0	4.5	10.0	3.0

.

By mapping the performance across different datasets to the abstract optimization objective defined in Equation (9) of Section 3.2, and combining the robustness and efficiency results presented in the tables above, we can conduct a comprehensive analysis of the three score-driven sample selection strategies: EL2N, GraNd, and MBS.

EL2N (${\mu }^E$) shows certain limitations in terms of efficiency. As indicated in Table 5 and Table 6, its efficiency scores remain acceptable under low selection ratios, but the advantage quickly diminishes as the ratio increases, suggesting that its efficiency cost $C_{\mathrm{eff}}$ rises significantly at higher ratios. Regarding robustness (Table 3 and Table 4), EL2N exhibits strong sensitivity on both datasets, with performance degrading sharply under high ratios, leading to a higher robustness cost $C_{\mathrm{rob}}({\mu }^E,\rho )$. Therefore, EL2N is more suitable for scenarios with medium-to-low selection ratios and relatively sufficient resources, but less appropriate for extreme compression or cross-task conditions.

GraNd (${\mu }^G$) demonstrates a more stage-dependent efficiency pattern. While its efficiency scores are relatively high at certain ratios (e.g., 0.1–0.2), the overall distribution is scattered, indicating unstable efficiency cost $C_{\mathrm{eff}}$. In terms of robustness, GraNd shows considerable fluctuations on both CREMA-D and AVE, with performance dropping rapidly at higher ratios, resulting in a higher robustness cost $C_{\mathrm{rob}}({\mu }^G,\rho )$. Overall, GraNd can capture critical samples under specific conditions but lacks stability across tasks, making it more suitable as a supplementary strategy rather than a primary one.

MBS (${\mu }^M$) achieves a more balanced performance across both efficiency and robustness. As shown in Table 5 and Table 6, its efficiency scores remain consistently high across multiple ratios, particularly stable in the 0.25–0.4 range, with relatively low efficiency cost $C_{\mathrm{eff}}\left(R\right)$. Table 3 and Table 4 further demonstrate that MBS achieves the highest robustness scores on CREMA-D, and although its advantage emerges later on AVE, its overall fluctuations are the smallest, yielding the lowest robustness cost $C_{\mathrm{rob}}({\mu }^M,\rho )$. Taken together, these results suggest that MBS exhibits stronger generalizability under different task conditions.

In summary, by unifying the performance of the three strategies into the optimization objective:

$$J({\mu }^M,\rho ,R)| < |J({\mu }^E,\rho ,R)| < |J({\mu }^G,\rho ,R)$$

(18)

It is important to emphasize that the three strategies do not exhibit fully homogeneous performance in terms of efficiency and robustness. MBS (${\mu }^M$) demonstrates overall superiority under cross-task conditions, particularly at moderate selection ratios (0.25–0.4), where it effectively balances efficiency and robustness, making it a relatively stable choice. EL2N (${\mu }^E$) retains practical utility at medium-to-low ratios, but its stability depends on a greater number of pre-training epochs. GraNd (${\mu }^G$), while capable of capturing critical samples in specific scenarios, incurs relatively higher efficiency and robustness costs overall. Therefore, the main advantage of MBS lies in its generalizability and stability, though it is not absolutely optimal under all conditions. EL2N and GraNd can still serve as complementary strategies in particular contexts, providing additional flexibility in score-driven sample selection.

5.5. Stress Testing and Ablation Analysis

To further examine the behavioral differences among various sample selection strategies, we conduct an ablation-style analysis under several extreme and rarely encountered conditions, including very high pruning ratios (0.80–0.95) and inadequate pre-training phase. Although such settings are not representative of typical deployment scenarios, they provide a stress test that helps reveal the intrinsic robustness and failure modes of each strategy. The two additional experimental figures (Figure 9) visualize the performance trends across pruning ratios of 0.80–0.95 under pretraining epochs of 15.

Across all conditions, EL2N consistently exhibits the weakest performance, with accuracy degrading sharply as the pruning ratio increases. This behavior is closely tied to its underlying metric: by relying solely on prediction error to quantify sample difficulty, EL2N fails to distinguish modality-specific contributions. When a multimodal sample is informative in one modality but weak in another, EL2N often misjudges its importance, leading to the systematic removal of samples that contain valuable cross-modal information. As pruning becomes more aggressive, this misalignment is amplified, ultimately causing EL2N to collapse.

In contrast, the behaviors of GraNd, MBS, and Random selection display clear dataset-dependent patterns. On CREMA-D, where audio and visual modalities are both of high quality and exhibit strong semantic alignment, the natural balance between modalities reduces the need for explicit modality-aware regulation. The dataset’s well-distributed diversity means that even random selection tends to preserve a sufficient number of informative samples. As a result, GraNd, MBS, and Random show a noticeable degree of performance “resonance,” with GraNd benefiting from stable gradients and MBS maintaining robustness without a pronounced advantage. In such a balanced setting, the inherent structure of the dataset diminishes the differences among selection strategies.

However, when shifting to the more challenging AVE dataset—characterized by substantial modality disparities, higher noise levels, and weaker cross-modal alignment—the distinctions among strategies become much more pronounced. GraNd’s gradient-based importance estimation becomes highly sensitive to the dominant or noisier modality, resulting in unstable and sometimes misleading importance scores. Random selection also becomes unreliable, as the probability of selecting noisy or modality-inconsistent samples increases significantly. In contrast, MBS explicitly evaluates the relative contributions of each modality and effectively filters out samples dominated by a single modality or corrupted by noise. Consequently, MBS demonstrates markedly stronger robustness on AVE, maintaining stable performance even under extreme pruning conditions.

Taken together, these extreme-condition experiments form an implicit ablation study. The results indicate that the performance gains of MBS cannot be attributed to sample diversity—otherwise Random would perform comparably—nor to difficulty or gradient magnitude alone—otherwise EL2N/GraNd would consistently dominate. Instead, the key determinant of robustness is the ability to identify and retain samples with balanced modality contributions. The consistent performance of MBS across both datasets, underscores the central role of modality balance in achieving stable multimodal sample selection under resource-constrained incremental learning.

6. Conclusions

This study investigates the problem of effectively filtering streaming multimodal samples under resource-constrained incremental learning, and proposes as well as systematically validates a sample selection strategy based on the Modality Balance Score (MBS). We first define and compute MBS to quantify the balance of contributions among modalities during training. Based on this definition, we design an MBS-driven selection strategy and conduct comparative analyses against existing EL2N and GraNd strategies. Experiments on two representative multimodal datasets, CREMA-D and AVE, examine the evolution of scores across different pre-training stages and the performance under varying selection ratios, thereby providing a comprehensive evaluation of the effectiveness and generalizability of MBS.

Extensive experimental results reveal that dataset characteristics directly influence the performance of selection strategies. For datasets such as CREMA-D, where modality balance is relatively strong, the MBS-based strategy can take effect at earlier training stages, quickly identifying and retaining modality-balanced samples, thus improving training efficiency while maintaining performance. In contrast, for datasets such as AVE, which exhibit stronger modality imbalance and higher task complexity, the advantages of MBS emerge later. Nevertheless, under medium-to-high selection ratios, MBS still demonstrates smaller performance fluctuations and stronger robustness, indicating its reliability as a selection criterion in complex scenarios.

In resource-constrained environments where data arrive in a streaming fashion and modality differences are pronounced, the MBS-based selection strategy not only prevents the model from over-relying on a single modality but also maintains performance stability under extreme compression conditions. This makes MBS a more practical and valuable strategy for incremental learning, providing solid support for intelligent systems in complex task scenarios. While MBS does not aim to replace difficulty-based selection, it offers consistent improvements in robustness and stability under modality imbalance, which is not addressed by existing strategies. Future work will explore the applicability of MBS to larger-scale and more complex multimodal datasets, and investigate its integration with methods such as dynamic weight allocation and modality completion to achieve more efficient modality rebalancing. In addition, we plan to deploy and validate this strategy in real-world unmanned aerial vehicle (UAV) swarm systems, assessing its real-time capability and scalability, thereby advancing the practical implementation of multimodal incremental learning.