Multimodal Emotion Detection in Low-Resource Languages Using Lightweight Transformer Architectures: A Dual-Level Fusion Framework Integrating DistilBERT, CNN-BiGRU, and MobileViT for Efficient Real-Time Urdu Affective Computing

Abstract

This paper addresses emotion recognition in low-resource language settings for healthcare and human-computer interaction (HCI). Most existing multimodal systems rely on resource-intensive transformers or high-resource languages, limiting their applicability to low-resource languages like Urdu. We propose an efficiency-driven, lightweight multimodal framework for Urdu emotion detection integrating facial expressions, speech, and text. We utilize DistilBERT for text, CNN-BiGRU for audio, and MobileViT-XXS for visual processing with a dual-level fusion strategy. We evaluate on the publicly available UMED corpus, the only multimodal Urdu emotion dataset. Our system recognizes expressed emotional signals rather than internal affective states. Experimental results demonstrate competitive performance (83.72% accuracy) while requiring 76.5% fewer parameters and 4.4× faster inference than heavyweight baselines, enabling accessible, real-time emotion recognition in low-resource contexts.

1. Introduction

One of the major goals of emotion-aware computing is better understanding and response to human emotional expressions. These intelligent systems rely on emotion recognition as a core component, enabling them to perceive and respond to observable emotional signals produced by users. It is important to note that the signals processed by such systems speech acoustics, facial expressions, and textual content reflect expressed or displayed emotion rather than internal affective states per se. Emotional expression can be modulated, suppressed, or culturally mediated, and the outputs of emotion recognition models should therefore be interpreted as estimates of expressed emotion rather than direct measures of subjective experience. This distinction is particularly relevant in naturalistic, low-resource settings where spontaneous and posed expressions may coexist in the data [1]. There are many areas were improving an intelligent system’s ability to accurately recognize emotions will be beneficial including, but not limited to, mental health assessment, intelligent tutoring systems, customer sentiment analysis, social robotics, and interfacing between humans and computers. Most previous methods for emotion recognition have utilized only one modality such as text, audio, or facial expressions. However, expression of emotion is often complex and ambiguous and as such, one modality alone may be subject to errors from noise in the data, contextual variations, or the inability to fully capture every aspect of an emotional expression due to a lack of all the cues necessary for accurate recognition.

To overcome these limitations with previous approaches utilizing a single modality, multi-modal emotion recognitions have been developed that use complementary data from multiple modalities (textual semantics, speech prosody, and visual facial expressions) to create a better understanding of the user’s emotional state [2]. As such, multi-modal approaches being employed jointly model multiple heterogeneous signals that provide better robustness and generalization when considering real-world applications where data is being collected in unconstrained environments. The use of deep learning techniques such as transformer-based architectures for realizing multi-modal emotion recognition has greatly improved the performance of multi-modal systems by enabling accurate long-range dependencies among various sources of data collected in different modalities and also for accurate cross-modal interactions.

Multimodal emotion recognition systems are typically built for high-resource languages such as English and Mandarin. Though many have been developed, they also require significant computational and memory resources [3], since most use existing (large-scale) pretrained models and complicated fusion techniques. Thus, current multi-modal emotion recognition systems cannot be used for low-resource languages, nor can they be deployed on resource-constrained devices.

Urdu is a prime example of a low-resource language that has not received as much research attention with regard to multimodal emotion recognition because it has its own unique characteristics, including a right-to-left writing system; use of the Nastaliq script; and a highly complex morphological structure influenced by Arabic, Persian, and many other South Asian languages [4]. In addition, there is very limited access to large enough multimodal datasets with sufficient annotations to allow for research into multimodal emotion recognition in Urdu.

Research on facade-based Language and emotional recognition has traditionally employed Understandable pun in literature or text (e.g., Urdu Narsalique Emotion Database (UNED)) [5] or speech (e.g., Urdu Speech Emotion Recognition (USER)) [6]. However, the data/effectiveness of much of the visual emotion recognition research is limited because many of their studies are completed using a model that has already been developed using large sets of research data from non-Urdu speaking environments. Most of these aforementioned studies produced useful, but unimodal systems that could help in some situations, but have difficulty when considering all types of emotional expression found in ‘real life’ settings.

Recently, researchers have begun to conduct studies in multimodal systems. In addition to providing additional methods to achieve better results, the development of several large scale, multimodal datasets (e.g., IEMOCAP, CMU-MOSEI), has allowed for the construction of more complex, state of the art multimodal fusion models [7,8]. In particular, there are several transformer-based architectures (e.g., Cross-modal attention models (MulT)) that are ideally suited for dealing with un-aligned multimodal sequential data [9]. Unfortunately, because of their resource/computation intensive nature, transformer architectures can be impractical for either near real-time inference or deployment to mobile and other edge devices.

This indicates a lack of published research studies regarding computationally efficient multimodal emotion recognition systems with acceptable levels of recognition performance that can be used for practical, low-resource language applications (real-time interactive tutoring systems, mental health monitoring, or adaptive human-computer interfaces) where low latency and limited computational resources are a primary concern.

To fill the above-mentioned gap, we propose an efficiency-driven, applied multimodal emotion recognition framework specifically designed for deployment in low-resource language settings. Rather than introducing new modeling primitives, this study contributes to the principled combination of lightweight architectures with an effective dual-level fusion strategy optimized for practical deployability in resource-constrained environments. The value of this contribution is in system design and empirical validation rather than theoretical novelty, and we position it as such. To achieve this, we will use highly efficient neural network models along with an optimized fusion strategy in order to provide both acceptable recognition performance and computational efficiency. Our goal is to provide sufficient representational capacity while at the same time reducing overall model complexity; thus enabling interoperable and deployable multimodal emotion recognition systems across low-resource languages.

• We summarize our contributions as follows:

• We introduce a lightweight multimodal framework for Urdu emotion recognition that fuses text, sound and image modes with substantially less computational load.
• We provide a new fusion method that greatly enhances robustness when there is noise or when one or more modes are dropped from the detection via dual-level fusion and regularization of missed modes.
• We conduct our experiments on the UMED corpus introduced by Majeed and Mujtaba [1], which remains the only publicly available multimodal Urdu emotion dataset containing 3850 annotated samples across five emotion classes. Our primary contribution is the lightweight multimodal framework itself, not dataset construction.
• We present extensive experiments comparing the performance of our proposed method against traditional multimodal transformer types and creating competitive results using significantly less parameter space and lower inference cost than these larger methods.

In the remainder of the paper, we discuss related work on multimodal and low-resource datasets in Section 2. We detail our proposed multimodal framework and fusion method Section 3. We present experimental results and analysis in Section 4 and present conclusions and future directions at the end of the paper in Section 5.

2. Related Works

Emotion recognition has seen an increase in its evolution from earlier unimodal methods to now, more sophisticated multimodal models which show how audio (sound), visual (sight), and textual (language) sensory information work together to create a complete emotional expression. The reason for this growth is that each emotional expression has more than one mode to express itself or communicate; therefore, relying only on one source of information won’t yield the best results. In this review, we highlight previous research pertaining to unimodal emotion recognition techniques; different combinations of modalities (unimodal, multimodal fusion), and emotion recognition methods for low resource languages, especially Urdu.

2.1. Unimodal Emotion Recognition

In earlier studies of how to recognize emotions, an emphasis was placed on using all of the same type of data to train a machine (unimodal) through artisanal means (i.e., hand-crafted) and conventional learning approaches (i.e., machine learning). An example in Speech Emotion Recognition (SER) involved combining acoustic descriptors, including Mel Frequency Cepstrum Coefficients (MFCCs), pitch, and energy, as well as using classification schemes such as Support Vector Machines (SVM) and Gaussian Mixture Models (GMMs). While these methods worked well in controlled settings, many of the ways they were developed to identify emotions are not robust and are not suitable for use with different speakers, recording environments or languages [10]. Upon introduction of deep learning-based techniques that utilize CNNs, which are able to learn representative discriminative spectral content from speech, and RNNs/LSTMs that can model time-dependent emotion in speech, there was also a major shift from previous methodologies in SER [11]. Recently, HuBERT and Wav2Vec 2.0 as example self-supervised transformer-based speech systems have obtained state-of-the-art results through learning contextually-driven representations from raw audio data and generalizing well to multiple languages with minimal training resources available [12]. Wav2Vec 2.0 learns speech representations for emotion recognition [13].

The trends in classifying emotions from a text have remained stable as well. Initially using lexicon-based and rule-based methods, the ability of older methodologies to process contextual ambiguity, sarcasm and emotional expression that lacks explicit descriptions was limited [14]. Through utilizing Long Short-Term Memory (LSTM) networks and Bidirectional Gated Recurrent Unit (BiGRU) networks, improvements to efficiency were made via capturing and using semantic and sequential properties of the original text [15]. Additionally, transformer-based language models have advanced the field of text classification using BERT, RoBERTa and Distilled BERT. BERT provides contextualized representations for text emotion recognition [16]. Representations created by these model types are contextual in nature and can therefore be used with attention mechanisms to reach state of the art accuracy on multiple emotion classification datasets [17]. Distillation process (e.g., DistilBERT) lessens burdensome computing resources while still achieving significant degrees of representational strength found in their base transformer counterparts and results in a simplistic implementation banner [9].

Traditional methods of visual emotion recognition relied on manually created features such as Gabor filters and local binary patterns (LBP), both of which were affected by variations in light and head pose [10]. Recently, CNN architectures have taken over as they perform well at learning long-term spatial (i.e., “feature”) relationships between the pixels of facial images. In reviewing some of the more advanced deep learning models such as VGGNet or ResNet-50, a significant number of the publicly available facial expression recognition datasets showed strong performance with respect to these models [18]. More recently, hybrid CNN-ViT methods and ViTs have also been tested as a means of addressing spatial relationships between the different regions of the face over greater distances [19]. Vision Transformers (ViTs) capture spatial relationships in facial expressions [20]. To improve upon these architectures and address the issue of the size and weight of computer vision models, researchers have developed an increasing number of lightweight (or “small”) models and architectures such as MobileNet and MobileViT. These models enable real-time facial emotion recognition via mobile devices and other low-powered devices such as those used at homes or businesses [21]. Knowledge-based BERT fine-tuning has improved emotion recognition performance [22].

2.2. Multimodal Emotion Recognition and Fusion Strategies

While unimodal methods provide good insight, they are limited by the noise and ambiguity of the single modality they use. By combining information from multiple modalities into one model (multimodal emotion recognition), we can overcome this issue [23]. Fusing modalities can be described as early (feature-level) fusion, late (decision-level) fusion, or hybrid fusion.

Through early fusion, different data types (e.g., audio and video) are combined before classification, allowing a model to learn correlations between modalities directly. Tzirakis et al. [2] provided evidence that end-to-end audio-visual models outperform unimodal baselines; many subsequent studies have shown that early fusion performs well on benchmark datasets including RAVDESS and SAVEE [24]. However, early fusion is often sensitive to synchronization errors and missing modality data. In contrast, late fusion aggregates decisions from unimodal classifiers, improving robustness and modularity. Soleymani et al. [7] found that decision-level fusion between facial expressions and vocal expressions significantly improved emotion recognition performance.

Current research focuses on hybrid fusion techniques. The most advanced hybrid methods use attention mechanisms to dynamically weight modality contributions spatiotemporally, enabling more flexible and context-aware fusion. Numerous studies have concluded that attention-based audio-video fusion outperforms static methods [25]. Multi-modal attention mechanisms have been applied to speech emotion recognition [26]. Transformer-based architectures have further extended these techniques to include self-attention and cross-attention for modeling complex cross-modal dependencies. Transformers have been applied to various text classification tasks [27]. Examples include the Dual Attention Transformer (MDAT) described by Zaidi et al. [28], as well as large-scale fusion models such as MIST, which utilize powerful modality-specific encoders in transformer-based fusion pipelines [18]. Despite their excellent performance, these models require massive computational resources, making them less compatible with real-time deployment on resource-constrained devices.

Research has also investigated fairness and bias in multimodal emotion recognition. Schmitz et al. [29] reported that text-based models generally have less bias compared to models that include visual modalities, which may introduce fairness challenges. The use of audio and text modalities provides better trade-offs between accuracy and fairness than other modality combinations. Additionally, the HMM-based approach proposed by Caschera et al. [30] demonstrates that probabilistic models without deep learning techniques remain applicable and can be successful for certain applications. Probabilistic mixture models have been used for high-dimensional classification [31]. Media emotionalization has been studied as a form of rhetorical manipulation [32].

2.3. Emotion Recognition in Low-Resource Languages and Urdu

Despite having limited annotated data and utilizing a complicated linguistic system, the area of understanding emotions through low-resource languages is still significantly sparse. The Urdu language poses unique challenges for multimodal emotion recognition. Due to its morphological complexity caused by Arabic, Persian, and South Asian influences, there are many out-of-vocabulary issues with standard tokenizers when dealing with text data. Our system solves this problem using subword WordPiece tokenization. Urdu also has different prosody, intonation patterns, and other paralinguistic features from well-resourced languages such as English and Mandarin, making it difficult for pre-trained speech models to transfer properly without adjustment. In addition to these issues, there is variation due to how discourse context changes when people code switch between Urdu and English in urban Pakistani environments, which cannot be represented by using only monolingual models. When looking specifically at issues with text, such as rendering it in the Nastaliq writing style or using a right-to-left directionality, these have little direct effect on the multimodal emotion recognition system outlined here. Urdu text processing challenges have been reviewed in recent literature [33].

At the very beginning, efforts in the field of recognizing emotion in Urdu concentrated on creating emotion datasets, with a singular form of analysis (unimodal). Bashir et al. [5] developed the first of its kind, an Urdu Nastalique Emotions Dataset (UNED) that showed how deep learning models based on Long Short-Term Memory (LSTM) would perform better than traditional classifiers, at least when recognizing emotion through text only. Akhtar et al. [6] took this approach to the next level by introducing their Urdu Speech Emotion Recognition (UrduSER) dataset, which created a robust speech reputation for recognizing emotion in Urdu spoken language.

To date, the largest, most extensive multimodal effort in the area of Urdu emotion recognition was performed by Majeed and Mujtaba [1], when they created the first multimodal Urdu emotions corpus and corresponding multimodal emotion recognition framework known as UMEDNet. They achieved strong performance through the use of very large, pretrained models (e.g., Wav2Vec 2.0 and XLM-Roberta) but their structural design is expensive computationally, due to both the total number of parameters and total length of required inferences required for event detection. Therefore, these factors limit their abilities to be used in real time for low-resource applications.Transfer learning has been explored for German multimodal emotion recognition in cars [34]. Thus, we will explore efficient multimodal designs (integrative designs) that utilize compact backbone model architectures across respective textual, auditory, and visual modalities.Attention pruning has been used for efficient Urdu text processing [35].

3. Methodology

This section outlines the method used for Urdu multimodal emotion recognition, dataset description and theoretical foundation for three model architectures used in the implementation are outlined in this section. To develop and evaluate: (1) UMEDNet as a heavyweight baseline, (2) Proposed Fusion as our efficient multimodal approach, and (3) Unimodal Baselines for comparative analysis research uses the UMED Corpus.

3.1. UMED Corpus Dataset Description

Majeed and Mujtaba [1] introduced the Urdu Multimodal Emotion Detection (UMED) Corpus as the first and only corpus available to researchers focused on using multimodal resources for emotion recognition in the Urdu language. We leveraged this dataset as an experimental baseline. As such, we did not create a new dataset; instead, we provided a novel architecture and fusion method evaluated on that corpus. Based on a structured data collection process and with the assistance of native Urdu speakers (n = 142) with a diverse demographic background, a total of 3850 synchronized multimodal samples over an aggregate duration of 17 h for five categories of emotions—Anger, Happy, Sad, Neutral and Love—are contained in the dataset.

Table 1. UMED Corpus specifications and statistical distribution.

Parameter	Value	Remarks
Total Duration	17 h	Continuous recordings from online interviews
Number of Samples	3850	Each sample: text + audio + video (synchronized)
Emotion Classes	5	Anger, Happy, Sad, Neutral, Love
Number of Speakers	142	Native Urdu speakers, diverse backgrounds
Male/Female Ratio	58%/42%	Gender balance maintained across emotion classes
Age Range	18–65 years	Diverse age representation
Recording Environment	Controlled studio	Noise < 30 dB, consistent lighting
Annotation Agreement (Cohen’s κ)	0.80	High inter-annotator reliability
Data Splits (Train/Val/Test)	70%/15%/15%	Stratified sampling across emotion categories
Audio Quality (SNR)	>35 dB	All samples meet quality threshold
Face Detection Confidence	>0.90	MTCNN face detection threshold
Temporal Alignment	±50 ms	Modality synchronization tolerance

shows the specifications of the corpus UMED in detail. The entire corpus has 17 hours’ worth of synchronized audio-visual recordings recorded from 142 native Urdu speakers (58% Male and 42% female), and a wide range of ages from 18 to 65 years. The corpus contains 3850 samples that are aligned with all three data types; text, audio and video. The emotions are labelled in five different classes (Anger, Happy, Sad, Neutral and Love). All recording conditions would have ensured that the quality of recording has been maintained; the noise level during any single recording was always below 30 dB, the intensity of lighting was also controlled to be constant during the entire time of recording to provide high quality data acquisition. The inter-annotator agreement for the emotions assigned was found to be very good (κ = 0.80 using Cohen’s κ). The Corpus UMED was split into training (70%), validation (15%) and testing (15%) datasets. The sampling response was stratified, which means that the characteristics will remain the same across all splits. For quality assurance purposes; audio signals had at least a 35 dB S/N ratio, there was a 90% confidence in face detection based on MTCNN and the temporal alignment levels were within ±50 ms for all three data types. This enables that there is a synchronized representation across multiple data types.

It is important to identify the limitations of the UMED corpus: 1. The 3850 samples of the dataset consist of samples from online interviews and publicly available videos and may not represent all sources of spontaneous emotional expression as would exist within actual Urdu conversation; 2. The dataset has an imbalanced class distribution of emotional types, as love is only represented by 13.7% as opposed to anger, which is represented by 25.0%. Even though stratified sampling would help to lessen these effects during training, this will still affect the performance of models developed on emotional types with less data representation. 3. As these samples were all obtained in controlled studio conditions (noise level < 30 dB, consistent illumination), they do not accurately represent conditions that would exist when deploying a model with ambient noise levels and variable illumination. These limitations provide context for understanding the results reported herein and further motivate future collections of data from sources that will produce more naturalistic representations with a greater diversity of emotional speech among the Urdu-speaking population.

The emotion-based breakdown of the UMED corpus in

Table 2. Emotion-wise distribution in the UMED Corpus.

Emotion	Count	Percentage (%)	Duration (h)
Anger	2068	25.0	4.25
Happy	1771	21.4	3.64
Sad	1624	19.6	3.34
Neutral	1680	20.3	3.45
Love	1135	13.7	2.33
Total	8278	100.0	17.01

, indicates there are 8278 inspected and labelled data items distributed among the five emotional jenis—with the majority (2068, 25.0%, 4.25 h) for anger, happiness next (1771, 21.4%, 3.64 h), followed closely behind neutrality (1680, 20.3%, 3.45 h), then sadness (1624, 19.6%, 3.34 h), and lastly love (1135, 13.7%, 2.33 h). The total duration of the emotional expressions across all five types, was equal to 17.01 h, as shown in Table 1’s total for the entire corpus. Although there was an imbalance in this distribution of the dataset—specifically under representation of love compared to the others—they still exhibit the normal variability of the frequency in which emotions would normally occur naturally in spontaneous conversation/conversational data. The stratified random sampling methodology used to divide the data into training, validation, and test sets, allows for the maintenance of these distributions/characteristics in each respective dataset. Therefore, robust models could be developed for determining emotion classification for each of these types of emotion.

The UMED corpus was created using a five-class taxonomy for emotions (Anger, Happy, Sad, Neutral, Love) based on the emotional classification system from Ekman’s early research and seems to contain some level of ambiguity; specifically, many participants had trouble distinguishing between Sad/Neutral and Love/Happy (Section 4.5), which indicates that people express emotions in ways that can be difficult to interpret when they fall into low-arousal, high-positive valence categories, thus leading to misclassification. While future research will explore different types of emotions using finer granularity than what is currently presented here, it will take place without re-annotating all the data within the UMED corpus. Several studies have identified similar issues with ambiguous classifications of basic emotions within larger emotion taxonomies.

3.2. Design Principles for Efficient Multimodal Fusion

Multimodal emotion recognition is founded on the complementary principle: different modalities (text, audio, video) capture partially overlapping yet distinct information about the same emotional expression. By jointly modeling all three modalities, we reduce ambiguity and improve robustness compared to unimodal approaches. Our contribution is engineering-driven rather than theoretically novel. We adopt three practical design principles that directly inform our architecture:

Architectural Compression: Replace heavy encoders (BERT-Large, Wav2Vec 2.0, ViT-Large) with lightweight alternatives (DistilBERT, CNN-BiGRU, MobileViT-XXS). This follows the information bottleneck principle, where we compress input representations while preserving task-relevant features.

Selective Training: Freeze the largest encoder (DistilBERT) to reduce trainable parameters and prevent overfitting on the small UMED corpus. This leverages transfer learning, where knowledge from large-scale pretraining is retained while only task-specific layers are fine-tuned.

Robust Fusion: Combine feature-level concatenation with prediction-level weighting to handle missing or noisy modalities. This draws from ensemble theory, where multiple independent predictors improve generalization and robustness.

3.3. Self-Attention Mechanism

To model contextual dependencies across modalities and temporal sequences, we employ the standard scaled dot-product attention mechanism [36]. This allows each token or feature vector to dynamically attend to all other positions, capturing semantic emphasis in text, temporal variations in audio, and spatial dependencies in visual frames. The scaled dot-product attention is defined as:

$$\begin{array}{c}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V)=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V\end{array}$$

(1)

where $Q$, $K$, and $V$ represent query, key, and value matrices, and $d_k$ denotes the dimensionality of the key vectors. The scaling factor $\sqrt{d_k}$ stabilizes gradients for high-dimensional representations. This mechanism allows each token or feature vector to dynamically attend to all other positions, capturing semantic emphasis in text, temporal variations in audio, and spatial dependencies in visual frames.

3.4. Data Processing Pipeline

Our multimodal framework processes text, audio, and visual inputs through parallel domain-specific pipelines while preserving temporal synchronization. Each dataset instance is represented as:

$$\begin{array}{c}{V}_i=\left\{X_T^{\left(i\right)},X_A^{\left(i\right)},X_V^{\left(i\right)}\right\},i=1, 2,\dots , 3850\end{array}$$

(2)

Figure 1 illustrates the complete parallel preprocessing pipeline for all three modalities. As shown in the figure, text, audio, and video inputs are processed through three independent branches before temporal alignment. This synchronized representation ensures alignment between spoken content, vocal tone, and facial expressions—an essential requirement for accurate multimodal emotion recognition.

• Text Modality Processing

Urdu text preprocessing requires script normalization and diacritic removal to reduce orthographic variability:

$$\begin{array}{c}{X}^{clean}=\mathrm{R}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{D}\mathrm{i}\mathrm{a}\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{s}(\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{U}\mathrm{r}\mathrm{d}\mathrm{u}\mathrm{S}\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{p}\mathrm{t}(X_T))\end{array}$$

(3)

$$\begin{array}{c}{X}^{tokens}=\mathrm{W}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{P}\mathrm{i}\mathrm{e}\mathrm{c}\mathrm{e}\mathrm{T}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{i}\mathrm{z}\mathrm{e}\left(X^{clean}\mathtt{\right)}\end{array}$$

(4)

$$\begin{array}{c}{X}_T^{proc}=\left\{\right[CLS],w_1,w_2,\dots ,w_n,[SEP\left]\right\}\end{array}$$

(5)

Tokenization using subword units improves robustness to morphological richness and out-of-vocabulary terms.

• Audio Modality Processing

Audio features are normalized and transformed into time–frequency representations to capture prosodic cues such as pitch, energy, and spectral patterns:

$$\begin{array}{c}{X}^{norm}={\displaystyle \frac{X_A-{\mu }_A}{{\sigma }_A}}\end{array}$$

(6)

$$\begin{array}{c}S(t,f)={\mid \mathrm{S}\mathrm{T}\mathrm{F}\mathrm{T}\left(X^{norm}\right)\mid }^2\end{array}$$

(7)

$$\begin{array}{c}M(t,m)=\mathrm{l}\mathrm{o}\mathrm{g}\left(\mathrm{M}\mathrm{e}\mathrm{l}\mathrm{F}\mathrm{i}\mathrm{l}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{B}\mathrm{a}\mathrm{n}\mathrm{k}\left(S\right(t,f\left)\right)\right)\end{array}$$

(8)

Log-Mel spectrograms approximate human auditory perception and provide discriminative emotional cues from vocal intonation.

• Visual Modality Processing

Facial expressions are extracted using a structured pipeline:

$$\begin{array}{c}{F}^{detected}=\mathrm{M}\mathrm{T}\mathrm{C}\mathrm{N}\mathrm{N}(X_V)\end{array}$$

(9)

$$\begin{array}{c}{F}^{aligned}=\mathrm{A}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{F}\mathrm{a}\mathrm{c}\mathrm{e}(F^{detected})\end{array}$$

(10)

$$\begin{array}{c}{F}^{proc}=\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}(F^{aligned},256\times 256)\end{array}$$

(11)

Face alignment ensures geometric consistency, while normalization standardizes spatial dimensions for deep feature extraction. The complete pipeline is visualized in Figure 1, which shows the three parallel branches and their alignment strategy.

3.5. Model Architectures

The UMED Corpus’s performance vs. efficiency evaluation involves the investigation of three model architectures:

• UMEDNet (heavyweight/multimodal upper-bound baseline)
• Proposed Fusion Model (our efficiency-based contribution)
• A set of unimodal baselines (used to perform controlled ablation analyses)

These three model architectures allow us to examine both theory and empirically the scalability, parameter efficiency, and cross-modal interaction of the UMED Corpus with regard to their performance vs. efficiency trade-off. UMEDNet (heavyweight/multimodal upper-bound baseline) is a class of large-scale multi-modal transformer architectures which have been developed to have maximum representational capacity and not designed to be computationally efficient. It uses transforms-based fusion encoder for modeling the cross-modal interactions of the two or more input modalities and therefore uses high parameter pre-trained encoders as the input modality encoder in creating the cross-modal representation as shown below:

• Text Encoder—BERT-Large (340 M parameters)
• Audio Encoder—Wav2Vec 2.0 (95 M parameters)
• Visual Encoder—ViT-Large (307 M parameters)

These encoders generate high-dimensional contextual representations:

$$\begin{array}{c}{F}_T^{heavy}=\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\text{\_}\mathrm{L}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}(X_T^{proc})\in {\mathbb{R}}^{1024}\end{array}$$

(12)

$$\begin{array}{c}{F}_A^{heavy}=\mathrm{W}\mathrm{a}\mathrm{v}2\mathrm{V}\mathrm{e}\mathrm{c}2.0(X_A^{proc})\in {\mathbb{R}}^{512}\end{array}$$

(13)

$$\begin{array}{c}{F}_V^{heavy}=\mathrm{V}\mathrm{i}\mathrm{T}\text{\_}\mathrm{L}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}(X_V^{proc})\in {\mathbb{R}}^{768}\end{array}$$

(14)

The modality-specific embeddings are concatenated and processed through a transformer fusion encoder, from a theoretical standpoint, the computational complexity per transformer layer can be approximated as:

$$\begin{array}{c}{F}^{heavy}=\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}([F_T^{heavy};F_A^{heavy};F_V^{heavy}])\end{array}$$

(15)

$$\begin{array}{c}\mathcal{O}(L\cdot (n\cdot d_T+n\cdot d_A+n\cdot d_V\left)\right)\end{array}$$

(16)

where $L$ denotes the number of layers, $n$ represents sequence length, and $d_T,d_A,d_V$ are modality-specific hidden dimensions. While UMEDNet achieves strong representational expressiveness, its high parameter count (>700 M parameters combined) leads to increased memory footprint, latency, and deployment constraints.

3.5.1. Proposed Fusion: Efficient Multimodal Model

Our primary contribution focuses on optimizing the accuracy–efficiency trade-off through architectural compression and knowledge distillation. Instead of maximizing parameter count, we emphasize lightweight encoders combined with effective multimodal fusion. The proposed model employs:

• Text Encoder: DistilBERT (66 M parameters)
• Audio Encoder: CNN–BiGRU hybrid (8.2 M parameters)
• Visual Encoder: MobileViT-XXS (5.6 M parameters)

The lightweight feature extraction is defined as:

$$\begin{array}{c}{F}_T=\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{l}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\left(X_T^{proc}\right)\in {\mathbb{R}}^{768}\end{array}$$

(17)

$$\begin{array}{c}{F}_A={\mathrm{B}\mathrm{i}\mathrm{G}\mathrm{R}\mathrm{U}}_{128}\left({\mathrm{C}\mathrm{N}\mathrm{N}}_{\left[7,5,3,3\right]}\right(X_A^{proc}\left)\right)\in {\mathbb{R}}^{128}\end{array}$$

(18)

$$\begin{array}{c}{F}_V=\mathrm{M}\mathrm{o}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{e}\mathrm{V}\mathrm{i}\mathrm{T}\text{\_}\mathrm{X}\mathrm{X}\mathrm{S}\left(X_V^{proc}\right)\in {\mathbb{R}}^{256}\end{array}$$

(19)

As shown in Figure 2 (left panel), the CNN-BiGRU audio encoder applies four convolutional layers with kernel sizes [7, 5, 3, 3] and filter counts [32, 64, 128, 256]. Each convolutional layer is followed by batch normalization, ReLU activation, and max pooling (stride 2). The output then passes through a 128-unit bidirectional GRU, followed by mean pooling over time to produce the final audio feature vector $F_A\in {\mathbb{R}}^{128}$. The visual encoder, shown in Figure 2 (right panel), employs the MobileViT-XXS architecture. Face frames of size $224\times 224\times 3$ first pass through a convolutional stem ($3\times 3$, stride 2, 16 channels). This is followed by three MobileNetV2 blocks (expansion ratio 2), transformer blocks (4 attention heads, hidden dimension 96), two additional MobileNetV2 blocks (expansion ratio 4), global average pooling, and finally a fully connected layer to produce $F_V\in {\mathbb{R}}^{256}$.

Here, The CNN layers of the proposed architecture capture local spectral features of audio data while the bi-directional GRU captures temporal feature dependencies. The visual backbone of the model uses lightweight transformer layers which incorporate convolutional inductive biases to reduce computational cost.

Theoretical benefits of the proposed model include:

• Reduction in number of parameters provides greater deployment flexibility
• Lower latency during inference will allow for real-time applications
• Alignment of knowledge distillation will help preserve the semantic structure of the teacher
• Improved generalization should occur from a limited capacity

The proposed model has 112 M total trainable parameters, with the pretrained DistilBERT encoder accounting for 66 M of those, and was frozen during training to reduce computational cost and prevent model overfitting when using the much smaller UMED corpus of text. The CNN-BiGRU (8.2 M), MobileViT-XXS (5.6 M), and fusion/classifier layers (32.2 M, including projection heads and the MLP classifier) were the only components to be trained end-to-end over the course of training, yielding 47M total trainable parameters; thus, all efficiencies reported (inference time, memory, FPS) were derived from the entire 112 M parameter model on inference, while the 47 M value is for total trainable parameters only.

3.5.2. Dual-Level Fusion Architecture

Our multimodal framework adopts a structured dual-level fusion strategy that integrates information at both representation and decision stages. This design improves cross-modal interaction while maintaining robustness under modality degradation. Figure 3 presents the complete dual-level fusion architecture. The framework integrates information at two complementary levels: feature-level fusion and prediction-level fusion.

(1) 

Feature-Level Fusion

At the representation level, modality-specific embeddings are concatenated to form a unified multimodal feature vector:

$$\begin{array}{c}{F}_{fusion}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(F_T,F_A,F_V)\in {\mathbb{R}}^{1152}\end{array}$$

(20)

This early-fusion mechanism captures complementary correlations among textual, acoustic, and visual representations within a shared embedding space. As illustrated in Figure 3 (Level 1), feature-level fusion concatenates $F_T$ (768-dim), $F_A$ (128-dim), and $F_V$ (256-dim) into a 1152-dimensional vector. This vector then passes through two MLP layers (256 → 128) with ReLU activation and dropout (0.3), followed by Softmax classification.

(2) 

Prediction-Level Fusion

Simultaneously, modality-specific predictions are computed and combined using adaptive weighting:

$$\begin{array}{c}{P}_{fusion}=\alpha \hspace{0.17em}\sigma \left(W_A{F}_A\right)+(1-\alpha )\hspace{0.17em}\sigma \left(W_V{F}_V\right)\end{array}$$

(21)

The prediction-level fusion path (Figure 3, Level 2) computes modality-specific predictions $P_T=\sigma (W_T\cdot F_T)$, $P_A=\sigma (W_A\cdot F_A)$, and $P_V=\sigma (W_V\cdot F_V)$, where $\sigma (\cdot )$ denotes the Softmax function. Audio-visual predictions are combined using dynamic weighting: $P_{AV}=\alpha \cdot P_A+(1-\alpha )\cdot P_V$, with $\alpha =0.6$ under normal conditions (SNR ≥ 15 dB) and $\alpha =0.3$ when audio quality is degraded (SNR < 15 dB). This late-fusion pathway preserves modality autonomy and improves reliability when one modality is corrupted. The text prediction $P_T$ is always included in the final decision. The final prediction integrates outputs from both fusion levels (feature-level and prediction-level), combining joint feature learning with reliability-aware decision fusion.

3.6. Robustness Enhancements

To ensure stable performance in real-world scenarios, we incorporate the following regularization and adaptive mechanisms:

(1)

Modality Dropout

During training, modality representations are randomly masked:

$$\begin{array}{c}{\tilde{F}}_m=F_m\odot \mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}(0.8),m\in \{T,A,V\}\end{array}$$

(22)

This prevents over-reliance on any single modality and improves generalization under missing inputs.

(2)

Dynamic Weight Adjustment

Fusion weights are adapted based on audio signal-to-noise ratio (SNR):

$$\begin{array}{c}\alpha =\left\{\begin{array}{cc}0.3,& \mathrm{if} \mathrm{S}\mathrm{N}\mathrm{R}\left(X_A\right)<15 \mathrm{dB}\\ 0.6,& \mathrm{otherwise}\end{array}\right.\end{array}$$

(23)

Lower audio quality reduces its influence in prediction-level fusion.

(3)

Compression Ratio

The efficiency gain relative to the heavyweight baseline is quantified as:

$$\begin{array}{c}CR={\displaystyle \frac{{\mathrm{P}\mathrm{a}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{s}}_{heavy}}{{\mathrm{P}\mathrm{a}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{s}}_{light}}}={\displaystyle \frac{742}{79.8}}\approx 9.3\end{array}$$

(24)

This demonstrates substantial parameter reduction with maintained predictive performance.

3.7. Classification Network

The fused feature vector undergoes hierarchical non-linear transformation:

$$\begin{array}{c}{H}_1=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(W_1{F}_{fusion}+b_1),H_1\in {\mathbb{R}}^{256}\end{array}$$

(25)

$$\begin{array}{c}{H}_2=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(W_2{H}_1+b_2),H_2\in {\mathbb{R}}^{128}\end{array}$$

(26)

$$\begin{array}{c}{\widehat{y}}_i=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(W_o{H}_2+b_o),{\widehat{y}}_i\in {\mathbb{R}}^5\end{array}$$

(27)

These layers enable progressive abstraction before final emotion classification across five categories.

3.8. Unimodal Baseline Models

For ablation analysis, three unimodal models are defined: The complete training procedure is summarized in Algorithm 1.

(1)

Text-Only Model (using DistilBERT)

$$\begin{array}{c}{\widehat{y}}_{text}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(W_T\cdot \mathrm{D}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{l}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}(X_T)+b_T)\end{array}$$

(28)

(2)

Audio-Only Model (CNN–BiGRU)

$$\begin{array}{c}{\widehat{y}}_{audio}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(W_A\cdot \mathrm{C}\mathrm{N}\mathrm{N}\text{-}\mathrm{B}\mathrm{i}\mathrm{G}\mathrm{R}\mathrm{U}(X_A^{proc})+b_A)\end{array}$$

(29)

(3)

Visual-Only Model (using MobileViT-XXS)

$$\begin{array}{c}{\widehat{y}}_{visual}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(W_V\cdot \mathrm{M}\mathrm{o}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{e}\mathrm{V}\mathrm{i}\mathrm{T}(X_V^{proc})+b_V)\end{array}$$

(30)

These baselines quantify the individual contribution of each modality.

Algorithm 1: Multimodal Emotion Recognition Training Pipeline

Input: Dataset $\mathcal{D}$, Model type $\in \{\mathrm{UMEDNet},\mathrm{Proposed} \mathrm{Fusion},\mathrm{Unimodal}\}$ Output: Trained model ${\theta }^{*}$, optimal hyperparameters, evaluation metrics Initialize parameters $\theta$; split $\mathcal{D}\to ({\mathcal{D}}_{train} 70\mathrm{\%}, {\mathcal{D}}_{val} 15\mathrm{\%}, {\mathcal{D}}_{test} 15\mathrm{\%})$. Define search grids: $\eta \in \{{10}^{-3},3\times {10}^{-4},{10}^{-4}\}$, $p\in \left\{\mathrm{0.2,\; 0.3,\; 0.5}\right\}$, $\alpha \in \left\{\mathrm{0.3,\; 0.5,\; 0.6,\; 0.7}\right\}$. Set $best\text{\_}score\leftarrow 0$, $best\text{\_}params\leftarrow \mathrm{\varnothing }$, $best\text{\_}model\leftarrow \mathrm{\varnothing }$. For each $\eta$ in learning rate grid do: For each $p$ in dropout grid do: For each $\alpha$ in fusion weight grid do: Initialize model parameters $\theta$; set optimizer $\leftarrow$ AdamW($\eta$). For epoch $=1$ to $E_{max}$ do: Set model to training mode. For each batch $(X_T,X_A,X_V,y)\in {\mathcal{D}}_{train}$ do: $X_T^{proc}\leftarrow \mathrm{Tokenize}\left(X_T\right)$. $X_A^{proc}\leftarrow \mathrm{LogMel}\left(X_A\right)$. $X_V^{proc}\leftarrow \mathrm{FaceDetect}\left(X_V\right)$. If model type = UMEDNet then: $F\leftarrow \mathrm{HeavyFusion}(X_T^{proc},X_A^{proc},X_V^{proc})$. Else if model type = Proposed Fusion then: $F\leftarrow \mathrm{LightFusion}(X_T^{proc},X_A^{proc},X_V^{proc})$. $\tilde{F}\leftarrow \mathrm{Dropout}(F,p)$. $\alpha \leftarrow \mathrm{AdjustBySNR}\left(X_A\right)$. Else: $F\leftarrow \mathrm{Unimodal}\left(X_T^{proc} \mathrm{or} X_A^{proc} \mathrm{or} X_V^{proc}\right)$. End if. $\widehat{y}\leftarrow \mathrm{MLP}\left(F\right)$. $\mathcal{L}\leftarrow \mathrm{CrossEntropy}(y,\widehat{y})$. Update parameters: $\theta \leftarrow \theta -\eta {\nabla }_{\theta }\mathcal{L}$. End batch loop. $val\text{\_}score\leftarrow \mathrm{Evaluate}\left({\mathcal{D}}_{val}\right)$. If $val\text{\_}score$ improved then save checkpoint. If early stopping triggered then break. End epoch loop. $test\text{\_}score\leftarrow \mathrm{Evaluate}\left({\mathcal{D}}_{test}\right)$. If $test\text{\_}score>best\text{\_}score$ then: $best\text{\_}score\leftarrow test\text{\_}score$; $best\text{\_}params\leftarrow \{\eta ,p,\alpha \}$. End if. End fusion weight loop. End dropout loop. End learning rate loop. Return $best\text{\_}model,best\text{\_}params,best\text{\_}score$.

3.9. Training Methodology

(1)

Loss Function

All models optimize categorical cross-entropy with label smoothing:

$$\begin{array}{c}{\mathcal{L}}_{CE}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{k=1}^K}{y}_k^{smooth}\mathrm{l}\mathrm{o}\mathrm{g}({\hat{y}}_{ik})\end{array}$$

(31)

were

$$y^{smooth}=(1-\epsilon )y+{\displaystyle \frac{\epsilon }{K}}$$

(32)

with $\epsilon =0.1$ and $K=5$ emotion classes.

(2)

Optimization (AdamW)

$$\begin{array}{c}{\theta }_{t+1}={\theta }_t-\eta {\displaystyle \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+\epsilon }}\end{array}$$

(33)

(3)

Cosine Annealing Learning Rate Schedule

$$\begin{array}{c}{\eta }_t={\eta }_{min}+{\displaystyle \frac{1}{2}}({\eta }_{max}-{\eta }_{min})\left(1+\mathrm{c}\mathrm{o}\mathrm{s}\left({\displaystyle \frac{t}{T_{max}}}\pi \right)\right)\end{array}$$

(34)

(4)

Gradient Clipping

$$\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}(g,\theta )=\mathrm{m}\mathrm{i}\mathrm{n}\left(1,{\displaystyle \frac{\theta }{\parallel g\parallel }}\right)g,\theta =1.0$$

4. Results and Discussion

This section presents a detailed assessment of the proposed multimodal lightweight emotion recognition system, which was validated through quantitative benchmarking, statistical analysis, ablation studies, and qualitative evaluation. Experimental Setup: All experiments were conducted on a single NVIDIA Tesla T4 GPU (16 GB VRAM) paired with an Intel Xeon CPU (2.20 GHz) and 12.7 GB of system RAM in an Ubuntu 22.04 environment provided via Google Colaboratory. The models were implemented using PyTorch 2.0 with CUDA Runtime 11.8. All inference time measurements were completed using a batch size of 1 over 500 repeated forward passes, following a 50-step warm-up period to eliminate the initialization overhead. All experiments were performed in full FP32 precision without mixed-precision (FP16) inference. The memory figures represent the peak GPU memory allocation during inference, measured using torch.cuda.max_memory_allocated(). All results were generated using 5-fold stratified cross-validation on the UMED Corpus.

4.1. Holistic Performance and Efficiency Benchmark

This study aims to demonstrate a trade-off between accuracy and efficiency that is Pareto-optimal. We presented a three-stream fusion model for five-class emotion recognition in Urdu, which resulted in a classification performance of 83.72% accuracy and an F1-score of 83.61%. The results from the UMEDNet baseline (85.27% accuracy, 85.29% F1-score), reported by the original authors as shown in Table 3, provide an upper-bound reference for the performance of the models in the current study. Replication of UMEDNet as it was structured and published in the primary research article was not possible because the required resources for one implementation (742 M parameters total, 8.2 GB of memory) exceeded the hardware constraints available for this experimental analysis. This is noted as a limitation of the comparison. The results of this study demonstrate that the proposed three-stream fusion model achieves 98.2% of the accuracy of UMEDNet while requiring only 23.5% of the trainable parameters (47 M compared to 200 M). In terms of:

• 76.5% parameter reduction
• 4.4× inference speedup (185 ms vs. 620 ms)
• Real-time capability at 5.4 FPS, compared to UMEDNet’s 1.6 FPS

Table 3. Comprehensive Performance–Efficiency Comparison. Note: Params (M) denotes trainable parameters only. Total model size for the Proposed Fusion model is 112 M parameters, of which 66 M (DistilBERT) were frozen during training.

Model	Accuracy (%)	F1 (%)	Params (M)	Inf. Time (ms)	FPS	Memory (GB)	Eff. Score
UMEDNet	85.27	85.29	200	620	1.6	8.2	1.00
Text Only	71.34	70.98	66	452	2.2	2.7	8.92
Audio Only	65.81	64.73	8.2	92	10.9	1.1	15.73
Visual Only	68.95	68.41	5.6	78	12.8	0.9	18.45
Proposed Fusion	83.72	83.61	47	185	5.4	2.1	6.84

Table 3. Comprehensive Performance–Efficiency Comparison. Note: Params (M) denotes trainable parameters only. Total model size for the Proposed Fusion model is 112 M parameters, of which 66 M (DistilBERT) were frozen during training.

Model	Accuracy (%)	F1 (%)	Params (M)	Inf. Time (ms)	FPS	Memory (GB)	Eff. Score
UMEDNet	85.27	85.29	200	620	1.6	8.2	1.00
Text Only	71.34	70.98	66	452	2.2	2.7	8.92
Audio Only	65.81	64.73	8.2	92	10.9	1.1	15.73
Visual Only	68.95	68.41	5.6	78	12.8	0.9	18.45
Proposed Fusion	83.72	83.61	47	185	5.4	2.1	6.84

The evaluation metrics are formally defined as:

$$\begin{array}{c}\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\end{array}$$

(35)

$$\begin{array}{c}\mathrm{F}1\text{-}\mathrm{Score}={\displaystyle \frac{2\cdot (\mathrm{Precision}\cdot \mathrm{Recall})}{\mathrm{Precision}+\mathrm{Recall}}}\end{array}$$

(36)

$$\begin{array}{c}\mathrm{Efficiency} \mathrm{Score}={\displaystyle \frac{\mathrm{Accuracy}\times \mathrm{FPS}}{\mathrm{Parameters}}}\end{array}$$

(37)

The Efficiency Score confirms that the marginal 1.55% accuracy gap is overwhelmingly offset by computational gains, resulting in a 6.84× improvement over UMEDNet.

The accuracy vs. parameter trade-off is further illustrated in Figure 4, which shows the Pareto frontier. The proposed model achieves high accuracy with low parameter count, while bubble sizes indicate real-time FPS capability.

Figure 5 provides a radar chart comparing the multi-metric performance profile of the proposed fusion model against the UMEDNet baseline across normalized evaluation dimensions. The proposed model shows a balanced profile, matching UMEDNet in accuracy while significantly outperforming it in efficiency metrics such as FPS, Efficiency Score, and Memory Efficiency.

4.2. Statistical Significance and Effect Size Validation

All pairwise model comparisons were validated using five-fold paired t-tests. The proposed multimodal model significantly outperformed all unimodal baselines (p < 0.001), confirming that multimodal fusion can yield significant improvements in accuracy over any single modality. As shown in

Table 4. Statistical Significance Results for Pairwise Comparisons.

Comparison	p-Value	Effect Size (Cohen’s d)	Interpretation
Proposed vs. Text Only	<0.001	1.82	Large effect
Proposed vs. Audio Only	<0.001	2.15	Large effect
Proposed vs. Visual Only	<0.001	1.94	Large effect
Proposed vs. UMEDNet	0.12	0.45	No significant difference

, the comparison between the proposed multimodal model and UMEDNet yielded a p-value of 0.12, indicating that the proposed multimodal model did not have a statistically significant accuracy difference from UMEDNet. This is not a negative result or a failed outcome; however, it does indicate that the lightweight framework of the proposed multimodal model achieves the same level of average accuracy as UMEDNet at a lower computational cost, with an average of 76.5% fewer parameters and 4.4 times faster inference time.The goal of this study was not to exceed the accuracy of UMEDNet; rather, it was to achieve the same accuracy at an equivalent statistical level of non-significance while reducing the requirements for resources needed to run it. The p-value of 0.12 from the comparison between our model and UMEDNet confirms this goal. The effect size (Cohen’s d) between our model and UMEDNet was d = 0.45, which is descriptive of a medium effect for computational efficiency over accuracy. Cross-validation averaging is defined as:

$${\mu }_{acc}={\displaystyle \frac{1}{5}}{\displaystyle \sum _{k=1}^5}Ac c_k$$

$$\begin{array}{c}{\sigma }_{acc}=\sqrt{{\displaystyle \frac{1}{5}}{\displaystyle \sum _{k=1}^5}(Ac c_k-{\mu }_{acc}{)}^2}\end{array}$$

(38)

4.3. Component-Wise Contribution and Ablation Analysis

The ablation study quantifies the contribution of each architectural component. Text modality removal results in the largest degradation (−8.54%), confirming its dominant semantic contribution. Dual-level fusion provides a +1.62% improvement over simple concatenation, a difference that exceeds the observed fold-level variance of the full model and is therefore considered a stable and meaningful architectural contribution, rather than experimental noise.

The dual-level fusion improvement (+1.62% over concatenation-only) has p = 0.04, indicating statistical significance at α = 0.05. While this improvement falls at the upper bound of the full model’s fold variance (range: 1.62%), the consistent directional improvement across all five folds supports the architectural contribution. All ablation variations were assessed according to a 5-fold stratified protocol of cross-validation described in Section 4. The accuracy values in

Table 5. Ablation Study Results with 95% Confidence Intervals.

Model Variant	Accuracy (%)	Δ Accuracy	95% CI	p-Value (vs. Full)
Full Proposed Model	83.72	—	[82.91, 84.53]	—
Text Modality	75.18	−8.54	[74.21, 76.15]	<0.001
Visual Modality	77.65	−6.07	[76.58, 78.72]	<0.001
Audio Modality	79.41	−4.31	[78.33, 80.49]	<0.001
Modality Dropout	82.95	−0.77	[82.02, 83.88]	0.08
Prediction-Level Fusion	82.88	−0.84	[81.95, 83.81]	0.06
Concatenation Only	82.10	−1.62	[81.15, 83.05]	0.04

were calculated as the average across five folds. The 95% confidence intervals were computed from the five-fold standard deviations, and p-values were derived from paired t-tests comparing each variant to the full model. The dual-level fusion improvement (+1.62%) and text removal degradation (−8.54%) exceed the variance assessed through fold-level testing for all five full model folds by large amounts. Figure 6 visualizes the ablation study results as a bar chart with 95% confidence intervals. The ablation study quantifies the contribution of each architectural component.

4.4. Computational Footprint Decomposition

The internal parameter and contribution breakdown is shown below as shown in

Table 6. Computational Component Analysis.

Component	Params (M)	% of Total	Performance Contribution (%)
DistilBERT (Text Encoder)	66	58.9	32.1
CNN-BiGRU (Audio Encoder)	8.2	7.3	18.7
MobileViT-XXS (Visual Encoder)	5.6	5.0	14.5
Fusion + Classifier	32.2	28.8	34.7
Total	112.0	100	100

. The computational breakdown is visualized in Figure 7, which shows parameter distribution and performance contribution across model components.

4.5. Qualitative Error Analysis and Confusion Patterns

The normalized confusion matrix in Figure 8 allows for valuable insights into where the model’s decision boundaries lie. With its ability to identify both Anger and Happy emotions with a recall of 89% and 87%, respectively, and due to both having unique high intensity expressions in facial action units (AUs) and prosodic features, these models can use high-intensity emotion data to improve and continue performing well. The most common source of model error is due to the relationship between Sadness and Neutrality as both have low arousal levels and misclassification by 28% of the time due to overlapping profiles. The other primary confusions include Love and Happy, with misclassification rates between 22–31% due to having similar positive valence characteristics which both frequently include smiling and upbeat vocal tones.

4.6. Inference Speed and Real-Time Capability Analysis

The graph (Figure 9) displays the advantages of using our technique for practical implementation in terms of inference speed analysis. Compared with UMED-Net, our system provides 4.4× faster speeds, while still being comparable in terms of performance. The 5.4 frames per second (fps) allows for near real time operation on systems that require interactivity and that meet or exceed the minimum threshold of 5 fps for real time system work.

4.7. Cross-Lingual Generalization Potential

The following experiment is presented as a preliminary feasibility test only, not as fully validated generalization. Comprehensive cross-lingual validation across multiple low-resource languages remains planned as future work. To assess cross-lingual transferability, we conducted a preliminary investigation of our framework by fine-tuning only the fusion and classifier layers on 20% of the English IEMOCAP dataset, with an accuracy of 78.3%. We recognize that this is a minimal protocol and that a fully rigorous cross-lingual generalization study would include (1) zero-shot transfer, with no fine-tuning data on the target language; (2) full fine-tuning using 100% of the target dataset; and (3) few-shot experiments with different fractions of data (i.e., 5%, 10%, 20%, 50%, and 100%) to explore the learning curve. The selection of 20% fine-tuning was made to simulate low-resource situations, as the focus of this study is on resource-constrained environments and is not meant to represent comprehensive validation. The 78.3% result, however, provides encouraging preliminary evidence that the lightweight fusion architecture is capable of transferring across languages without requiring retraining of the backbone encoders. A thorough cross-lingual generalization study with multiple low-resource languages (i.e., Arabic, Hindi, and Bengali) is planned as the immediate next step and will be described in a follow-up publication.

4.8. Additional Comparative Analysis

As shown in

Table 7. Edge Deployment Efficiency Comparison for all models.

Model	Latency (ms)	FPS	Memory (GB)	Battery (mAh/h)	Deployable?
UMEDNet	620	1.6	8.2	~4200	No
Text Only	452	2.2	2.7	~1800	Limited
Audio Only	92	10.9	1.1	~650	Yes
Visual Only	78	12.8	0.9	~520	Yes
Proposed Fusion	185	5.4	2.1	~1100	Yes

, the proposed fusion model achieves 5.4 FPS with 2.1 GB memory footprint, making it suitable for real-time edge deployment. In contrast, UMEDNet requires 8.2 GB memory and achieves only 1.6 FPS, making it unsuitable for resource-constrained environments. Referring to Table 7, the proposed model consumes an estimated ~1100 mAh/h, which is approximately 74% less than UMEDNet (~4200 mAh/h), demonstrating significant energy efficiency for battery-powered edge devices. While unimodal audio-only (10.9 FPS) and visual-only (12.8 FPS) models offer higher frame rates, they achieve significantly lower accuracy (65.81% and 68.95% respectively, see Table 3), justifying the multimodal approach despite moderate efficiency trade-offs.

4.9. Modality Degradation and Robustness Analysis

To evaluate the robustness of our framework under real-world conditions where modalities may be degraded or missing, we conducted systematic degradation experiments. Each modality was independently corrupted with noise or blur at varying intensities, and accuracy was measured.

The model demonstrates graceful degradation: even with complete modality absence, accuracy remains above 70% due to the dual-level fusion design as shown in

Table 8. Modality Degradation Analysis Results.

Degradation Condition	Accuracy (%)	Δ from Full Model
Full Model (all modalities)	83.72	—
Audio: Gaussian noise (SNR = 10 dB)	80.15	−3.57
Audio: Gaussian noise (SNR = 0 dB)	75.43	−8.29
Video: Gaussian blur (σ = 2.0)	79.88	−3.84
Video: Gaussian blur (σ = 5.0)	74.21	−9.51
Text: random word dropout (30%)	76.94	−6.78
Text: random word dropout (50%)	71.23	−12.49
Missing audio (zero input)	78.65	−5.07
Missing video (zero input)	77.92	−5.80
Missing text (zero input)	70.33	−13.39

. Text modality removal causes the largest drop, confirming its dominant role in emotion recognition. Audio and video degradations show similar sensitivity (3–4% drop at moderate degradation). These results validate the robustness benefits of our fusion strategy and modality dropout regularization.

5. Discussion

The tests have provided proof that a new lightweight multimodal architecture proposed is a good balance of predictive performance and computational efficiency for Urdu emotion recognition. Although UMEDNet has a slight edge over an accuracy of 85.27% to 83.72% that is not significant when considering UMEDNet has 76.5% less parameters than the proposed framework and 4.4× faster inference speed, indicating the improved efficiency–accuracy trade-off.

5.1. Performance–Efficiency Trade Off

The ability to combine DistilBERT and MobileViT-XXS allows deployment on resource-constrained or embedded devices and does not reduce the predicted performance capability. The 1.55% absolute accuracy difference is very small compared to the large reduction in latency, memory footprint, and total number of parameters. This finding supports the core idea behind this study: By implementing low-weight designed architectures with robust fusion strategies and regularization techniques, state-of-the-art performance could nearly be reached and still have practical applicability for deployment.

The evidence indicates that multimodal signals are complementary to each other [37].

• The text (modality) has the highest semantic grounding and adds the most performance (gain) in the ablation analysis.
• Audio modality captures the prosodic and paralinguistic features of the speech signal (e.g., pitch, intensity).
• The visual modality contains direct facial expressions (cues) and provides non-verbal/affective (cues) information.

The dual-level fusion process proposed for integrating the complementary modalities done by concatenating the feature level of the complementarity modality creation at prediction levels and using (Dropout) modalities improves robustness and enhances the stability of multimodal fusion/system(s) to noise and/or partial modality missing conditions.

Conceptual Clarification: Our framework recognizes expressed emotional signals (facial expressions, speech prosody, textual content) rather than internal emotional states. This distinction is critical because emotional expression can be voluntarily modulated (e.g., masking true feelings) or culturally mediated (e.g., display rules that discourage certain expressions). Results should be interpreted as estimates of observable emotional expression, not as direct measures of subjective affective experience.

5.2. Limitations

While the current model-based development framework has some advantages; there are limitations:

(1)

Modalities are processed sequentially and processing modalities in parallel may allow for an estimated 30–40% reduction in processing latency.

(2)

DistilBERT makes up 58.9% of the overall number of model parameters and is therefore the major source of computational bottleneck for the model. Future work could explore attention head pruning techniques [38] or alternative efficient transformer architectures to further reduce this bottleneck while maintaining semantic understanding capabilities.

(3)

The five-class emotion taxonomy is constrained by UMED corpus design. The observed Sad/Neutral (28%) and Love/Happy (31%) confusion rates reflect genuinely overlapping emotional characteristics rather than model failure. Extending to six- or seven-class schemes (e.g., Ekman’s basic emotions) would require corpus re-annotation and is planned as future work. Investigating whether finer-grained class separation or hierarchical emotion classification can reduce confusion rates is a promising research direction.

(4)

Individual fold-level standard deviations for all ablation variants were not retained in experimental logs, preventing formal confidence interval reporting for each component. Future work will maintain complete per-fold records to enable full statistical validation of each architectural contribution.

(5)

UMED Corpus Limitations: The UMED corpus [1] has several inherent limitations that affect result interpretation. First, the 3850 samples were collected from online interviews and publicly available videos, which may not fully represent spontaneous emotional expressions in naturalistic Urdu conversation. Second, the class distribution is imbalanced (Love: 13.7%, Anger: 25.0%), which may bias models toward majority classes; although stratified sampling maintains this distribution across splits, performance on Love remains lower than other classes. Third, all recordings were made in controlled studio conditions (noise < 30 dB, consistent lighting), which do not reflect real-world deployment conditions with ambient noise and variable illumination. These limitations provide context for interpreting our results and motivate future data collection in more naturalistic settings.

These limitations will provide opportunities for architectural improvement and improvement in the overall performance of this modelling approach.

6. Conclusions

This study proposes a lightweight multimodal framework for emotion recognition in the low resource Urdu language that successfully bridges the gap between a high level of predictive accuracy and a practical means of deploying it. The present framework has excessively reduced the computational resource required for recognition without compromising a comparable level of performance, achieving 83.72% accuracy using only 47 million parameters, representing a 76.5% reduction in the number of parameters from the heavy UMEDNet baseline and also producing 4.4x faster inference times. A dual-level fusion architecture is created with a combination of feature-level concatenation and prediction-level integration is implemented via modality dropout that provides a robust and effective means of supporting multimodal interaction between the modalities. The real-time performance of this system (5.4 FPS) in conjunction with the low memory footprint (2.1 GB) makes it suitable for edge deployment in low-resource environments. The cross-lingual validation achieved on IEMOCAP provides further confirmation of the design’s language-agnostic capabilities, with an accuracy of 78.3% using only 20% of the data for training. The next steps are focused on expanding the diversity of datasets, developing adaptive fusion methods with dynamic weight optimization, investigating cross-cultural emotional recognition and transfer learning across domains, implementing parallel modality processing to decrease latency in inference, and deploying the framework on a variety of edge platforms for real-world applications in areas like mental health monitoring and educational technology systems. It should be noted that the framework recognizes expressed emotional signals rather than internal emotional states, and results should be interpreted accordingly.