Small Language Models for Speech Emotion Recognition in Text and Audio Modalities

Abstract

Speech emotion recognition has become increasingly important in a wide range of applications, driven by the development of large transformer-based natural language processing models. However, the large size of these architectures limits their usability, which has led to a growing interest in smaller models. In this paper, we evaluate nineteen of the most popular small language models for the text and audio modalities for speech emotion recognition on the IEMOCAP dataset. Based on their cross-validation accuracy, the best architectures were selected to create ensemble models to evaluate the effect of combining audio and text, as well as the effect of incorporating contextual information on model performance. The experiments conducted showed a significant increase in accuracy with the inclusion of contextual information and the combination of modalities. The results obtained were highly competitive, outperforming numerous recent approaches. The proposed ensemble model achieved an accuracy of 82.12% on the IEMOCAP dataset, outperforming several recent approaches. These results demonstrate the effectiveness of ensemble methods for improving speech emotion recognition performance, and highlight the feasibility of training multiple small language models on consumer-grade computers.

1. Introduction

Speech can be defined as the expression of articulated sounds for the purpose of communicating thoughts, ideas, opinions, etc. By analyzing speech, humans are able to interpret the speaker’s intent, which enables effective communication. To understand speech, emotions play a crucial role; tone, rhythm, or word choice are changed according to the emotion the speaker feels or intends to convey. In the last two decades, speech emotion recognition (SER) [1] has gained increasing popularity in several research areas such as human–computer interaction [2], human–robot interaction [3], healthcare [4,5], or education [6,7]. During this period, handcrafted audio features such as energy, duration, harmonic-to-noise ratio, and classical feature extraction methods such as Mel-Frequency Cepstral Coefficients (MFCCs) combined with traditional machine-learning (ML) classifiers such as support vector machines (SVMs) have been successfully used for SER. However, in recent years, deep-learning (DL) methods have been increasingly adopted and gradually outperform traditional methods [8]. Deep-learning, and especially transformer-based architectures, achieve state-of-the-art accuracy on most popular SER datasets [9].

Part of the success of these approaches lies in their general-purpose design. Unlike traditional methods, whose knowledge capability is usually problem-specific, knowledge from a deep model can be easily transferred to perform a different task. This capability allows DL models to learn not only from small, manually labeled SER datasets, but also from much larger datasets focused on a different task, or even from unlabeled data [10].

The year 2018 marked a turning point in NLP, with the presentation of bidirectional encoder representations from transformers (BERT) [11] and generative pre-training (GPT) [12]. These transformer-based models demonstrated a great ability to capture longer-term dependencies in language in various NLP tasks, such as text generation, translation, summarization, question answering, or sentiment analysis. With 110 and 117 million parameters, respectively, the BERT and GPT architectures were originally considered large language models (LLMs). Since then, there has been significant progress in the development of much larger models, with architectures of billions or even hundreds of billions of parameters becoming standard [13]. As a result, models with just over 100 million parameters, previously considered large, are now referred to in the scientific community as small language models (SLMs) [14,15,16].

In general, LLMs report higher performance than SLMs in a wide range of NLP tasks [17]. However, their large size limits the usability of this type of model due to the large amount of computational resources required for both training and inference. On the other hand, SLMs can be fine-tuned relatively quickly on consumer-grade computers. And using techniques such as quantization [18] or low-rank adaptation (LoRA) [19], SLMs can be implemented in low-end devices, greatly increasing the number of potential applications for this type of model. The relatively light weight of SLMs facilitates the assembly of multiple models, which can significantly increase system performance in SER, especially when combining models of different modalities such as audio and text [20].

Several review articles published in recent years highlight the growing number of approaches using language models for SER [9,21,22], mostly models based on BERT for text processing or Wav2Vec2 [23] for audio processing. However, these models are often used as the backbone within more complex architectures, and research often focuses on the proposed architecture, sometimes neglecting the importance of the choice of language model.

In this paper, the performance of some of the most popular SLMs for text and audio modalities on the SER task is evaluated using a public benchmark dataset. All models are evaluated based on their accuracy and computational efficiency, with the aim of determining the most suitable architectures for the proposed task for each modality. Subsequently, the selected models are combined into an ensemble architecture to investigate the effect of combining models of the same modality but from different families on system performance, as well as the effect of combining text and audio modality models. In this paper, the terms context and contextual information are used to refer specifically to the immediately preceding utterance.

The key contributions of this paper are the following:

• The performance of popular fine-tuned SLMs for speaker-independent SER is evaluated on a public benchmark dataset.
• The effect of incorporating contextual information (previous utterance) on model performance is investigated.
• Ensemble architectures for unimodal and bimodal SER are developed and tested.

The rest of the paper is organized as follows. Section 2 provides a brief overview of related works. Section 3 describes the materials and methods used. Section 4 details the results and provides a comparison with other relevant works. Section 5 discusses the findings and outlines the limitations of the study, and Section 6 concludes the paper.

2. Related Work

In this section, an overview of relevant research on SER using SLMs is given. The section is divided into two parts: one presents different types of SLMs tested in this work, while the second examines recent work using language models for SER.

2.1. Architectures of SLMs

Based on their architecture, transformer-based models can be classified into three main categories: encoder-only, decoder-only, and encoder-decoder [9].

2.1.1. Encoder-Only Models

These models generate contextual representations for each token and are capable of understanding the interconnections between all input tokens. They are particularly well suited for natural language understanding tasks, where language features are extracted for subsequent applications such as text classification. However, several experiments show that the accuracy of this type of model is comparable to other models of similar size with encoder-decoder or decoder-only architectures in various text classification tasks such as financial sentiment analysis [24], fake news detection [25], or empathy detection in text [26].

In the text modality, BERT [11] is one of the first encoder-only models, and probably the most representative. BERT is a bidirectional language model designed to take into account the context of the surrounding tokens, both upstream and downstream of a given token. BERT’s architecture consists of multiple encoders arranged sequentially in a feed-forward structure, where the output of each encoder is used as the input of the next encoder. As a result of the great popularity of BERT since its release, several models based on its architecture have been introduced that have demonstrated better performance than the original model in several NLP tasks. The main improvements over the base model are briefly described below:

• RoBERTa [27]: Uses an optimized BERT training procedure by adjusting the hyperparameters, eliminating the next sentence prediction task, and using mini-batches and higher learning rates during training.
• ERNIE 2.0 [28]: Improves language understanding by integrating external knowledge sources during training and using knowledge masking to improve semantic understanding.
• ALBERT v2 [29]: Splits the embedding matrix into two smaller matrices and uses repeated layers split into groups to reduce the number of trainable parameters and increase training speed.
• DeBERTaV3 [30]: Uses disentangled attention, an improved mask decoder, and ELECTRA-style pre-training with gradient disentangled embedding sharing for improved performance over BERT and RoBERTa.

On the other hand, the BERT architecture has also served as a basis for the creation of audio models that apply similar principles of learning contextual representations to acoustic data. For example, Wav2Vec2 masks the speech input in latent space using a convolutional neural network (CNN) and solves a contrastive task defined over a quantization of the latent representations jointly learned in a BERT-like transformer. Similarly, HuBERT [31] uses the same blocks as Wav2Vec2, but introduces an offline clustering operation to provide aligned target labels for a BERT-like prediction. Similarly, WavLM [32] is based on the HuBERT framework with an emphasis on modeling spoken content and preserving speaker identity. This model extends the transformer architecture by incorporating a gated relative position bias, which improves its performance in recognition tasks. Additionally, squeezed and efficient Wav2Vec (SEW) [33] has been proposed as a lightweight alternative to this type of architecture, which modifies the CCN feature extractor and squeezes the input to the transformer for improved efficiency.

2.1.2. Decoder-Only Models

Decoder-only models are characterized by their auto-regressive approach, where each new token is predicted based only on the elements previously generated in the sequence. Although encoder-only architectures have been very successful in text processing, they are not typically used for audio classification. These models are particularly well suited for text generation tasks, but can be used for any NLP task. Within this category, GPT [12] and its variants are the most well-known family of SLMs. Just as BERT inspired the development of several encoder-only models, GPT has inspired the emergence of several notable decoder-only models, including:

• GPT-2 [34]: Second version of OpenAI’s GPT model. The model’s architecture is built by stacking multiple transformer decoder blocks composed of self-attention and linear layers.
• GPT Neo [35]: Trained on a larger dataset, it uses a similar architecture to GPT-2, but differs by using local attention in a different layer with a window size of 256 tokens.
• Pythia [36]: Uses a similar architecture to the previous ones, optimized for scalability. This model features changes from GPT-2, such as the use of rotary embeddings instead of positional embeddings.
• OPT [37]: Uses an architecture similar to GPT, but optimized for efficiency. Some versions of this model achieve accuracy comparable to GPT-3 on various NLP tasks, with a substantially lower carbon footprint during training.
• XLNet [38]: Introduces a permutation approach to an auto-regressive model, allowing it to learn bidirectional context without the need to mask tokens during training, as is done in BERT-like models.

2.1.3. Encoder-Decoder Models

These models use a classical transformer structure, where an encoder converts the input sequence into a fixed-length contextual representation that feeds the decoder, which converts it back into a natural language sequence. Within this category, the T5 [39] and BART [40] models stand out for text processing tasks. Both models use a similar approach and have demonstrated similar performance for text summarization [41,42]. However, a recent study by Rahman et al. [43] showed that BART outperformed T5 in accuracy for depression emotion classification, despite having significantly fewer trainable parameters.

In the field of audio processing, several approaches have been presented, mostly focused on automatic speech recognition. In this type of architecture, the encoder part is used to extract audio features and the decoder part is used to generate text from these features. For audio classification tasks, however, only the encoder part is usually used. Some of the most popular models are SpeechT5 [44], S2T [45], and Whisper [46]. Inspired by the popular T5 model, SpeechT5’s architecture consists of a shared encoder-decoder network and six modality-specific pre/post networks. It accepts both text and audio as input and is capable of generating both text and audio as output. On the other hand, S2T incorporates a convolutional downsampler to reduce the length of speech inputs by 3/4 before feeding them to the encoder. The model was trained with an auto-regressive cross-entropy loss and generates transcriptions or translations in an auto-regressive manner. Finally, Whisper uses a similar architecture to S2T, but it has been trained on a huge audio dataset with transcriptions, which makes this model stand out in transfer learning applications, presenting a high zero-shot performance.

2.2. SLMs for SER

Since the presentation of the BERT model in 2018, a large number of research papers have been published using this architecture or its variants as the backbone for SER from audio transcription, often combined with acoustic processing models to create multimodal systems. For example, a trimodal model was developed by merging text features extracted with RoBERTa, visual features extracted with DenseNet, and audio features extracted with OpenSmile through a long short-term memory (LSTM) and a multimodal attention module [47]. Another proposal fused MFCC features together with BERT embeddings in a recurrent neural network (RNN) with different attention mechanisms [48]. In the same vein, the location of the merging layer effect in a hybrid model combining BERT together with MFCC and other audio features has been explored [49]. The results of this work reflect an increase in the efficiency of the model. However, they point out a small influence of the location of the merging layer on the results.

Similarly, a BERT model was combined with an attention-based gated recurrent unit (GRU) for processing audio features from MFCC and different types of spectrograms [50]. The same approach was later used by replacing the BERT encoder with ALBERT and performing data augmentation using a generative adversarial network (GAN), which significantly improved the accuracy of the model [51]. On the other hand, a multihead attention module was used to fuse MFCC and other audio features with features extracted using a RoBERTa-based model, along with electroencephalogram (EEG) features, to create a multimodal brain-computer fusion network for human intention recognition [52].

In addition, a considerable amount of research has been done on architectures that combine BERT-type encoders for text feature extraction with CNNs for audio feature extraction from spectrograms or MFCCs. A visual geometry group (VGG)-type architecture with MEL spectrograms as input has been used in some works [53,54]. A modified AlexNet was used for feature extraction from audio spectrograms [55], and a similar approach but using three-dimensional spectrograms was also adopted [56]. A customized CNN incorporating RNN layers has also been proposed [57]. Finally, another customized CNN, but followed with a channel-wise global head-pooling, has been introduced [58].

On the other hand, although auto-regressive models are not commonly used in classification tasks, several proposals using this approach for SER have been presented recently. A model based on XLNet with bidirectional GRU and an attention mechanism has been proposed [59]. This model showed superior performance compared to BERT and the standard version of XLNet. Also, a generative framework for sentiment analysis designed to learn generic sentiment knowledge among different sentiment analysis subtasks was introduced [60]. The approach was evaluated on three different backbone architectures: BART, T5, and GPT-2. The results of the study showed that BART showed superior performance compared to the other two models. In addition, a trimodal approach using GPT for text, WaveRNN for audio, and FaceNet for images was used [61]. The results of this work indicate a superior accuracy of the text modality over the other two. However, the highest performance was achieved with the combination of the three modalities. A variational autoencoder combining RoBERTa as an encoder and the BART decoder for emotion recognition in conversation was developed [62]. In this approach, the decoder was used to train the utterance reconstruction model, which was shown to improve emotion recognition performance.

On the other hand, several studies using audio language models as the backbone of different architectures for SER have been published in recent years. For example, the output of Wav2Vec2 has been processed with CNNs, achieving improved accuracy over the base model in emotion recognition [63,64]. Similarly, the HuBERT and WavLM architectures were tested with different output blocks such as multilayer perceptron, CNN, or transformer [65]. Also, Wav2Vec2 features were fused with prosody features using a multi-headed attentional method [66,67]. Finally, a multi-modal autoencoder was proposed to combine features extracted with Wav2Vec2 and BERT and OpenFace [68].

Finally, several approaches that combine text and audio modalities have recently been presented [69]. Audio and text transformer features can be easily fused using attention-based methods to improve the overall SER accuracy. For example, Yi et al. [70] proposed a transformer encoder and dual modal cross-attention. Their approach achieved superior performance over existing models.

3. Materials and Methods

3.1. Dataset

There are dozens of public datasets that can be used for SER. According to a very recent systematic review on automatic speech emotion recognition [71], the most widely used of all available datasets is the Interactive Emotional Dyadic Motion Capture (IEMOCAP) [72]. In addition to its considerable popularity, this dataset has three key features that make it suitable for evaluating the models described in this paper. First, the dataset is divided into 5 sessions, with two different speakers in each session, which allows speaker-independent cross-validation of the models. Second, unlike other datasets that contain only a limited number of utterances spoken with different intonations, this dataset contains thousands of different utterances, allowing predictions based on text alone. Finally, it contains both audio and full transcriptions of these audio utterances, which makes it possible to study the effect of incorporating contextual information on model performance.

The dataset contains 12 h of recordings of two types of scenarios, emotional scripts and improvised scenarios. The dataset is divided into utterances, and each utterance is classified into one of the following emotion categories: neutral, happiness, sadness, anger, surprise, fear, disgust, frustration, excitement, and other. Most work on SER uses only four emotions: happiness, neutral, anger, and sadness. In our study, these four emotions are used to facilitate comparability of results with previous research.

3.2. Language Models

A selection of architectures with a similar number of trainable parameters and computational complexity was made.

Table 1. Number of trainable parameters (millions) and GFLOPs for the baseline architectures used in the study.

Model Name	Version	Modality	# Parameters	GFLOPs
ALBERT	v2-base	text	11.69	8.70
BERT	base	text	109.49	8.50
DeBERTa	v3-base	text	184.43	16.88
ERNIE	v2-base	text	109.49	8.69
RoBERTa	base	text	124.65	8.69
BART	base	text	140.01	9.92
T5	base (encoder-only)	text	109.63	17.72
GPT-2	124m	text	124.44	8.50
GPT-Neo	125m	text	125.20	8.69
OPT	125m	text	125.24	8.69
Pythia	160m	text	123.69	8.50
XLNet	base	text	117.31	10.29
HuBERT	base	audio	94.37	69.61
SEW	small	audio	89.62	25.26
WavLM	base	audio	94.39	69.64
Wav2Vec2	base	audio	94.37	69.61
S2T	ASR-large (encoder-only)	audio	156.81	40.83
SpeechT5	ASR (encoder-only)	audio	94.40	76.49
Whisper	small (encoder-only)	audio	87.01	43.58

lists all the baseline architectures evaluated, along with the number of trainable parameters for each model and the number of giga-floating point operations (GFLOPs) required to process one utterance. Parameter count is a common proxy for model complexity, but it does not always correlate with actual computational cost. GFLOPs provide a more direct measure of the operations required during inference. For instance, ALBERT reduces parameter count through techniques such as cross-layer parameter sharing, yet maintains similar GFLOPs to BERT. This highlights the relevance of GFLOPs when comparing the efficiency of different architectures. A fixed length of 100 tokens for text and 10 s for audio was used for each utterance as approximately 99% of the text sequences and over 90% of the audio sequences fell within these limits. The difference in coverage percentages is justified by the heavier computational demands of audio processing and the unique characteristics of prosody in audio data, which enable the prediction of emotional content even in incomplete dialogue segments. The value of GFLOPs was estimated using the fvcore [73] library developed by Facebook AI Research (https://ai.meta.com/research/, accessed on 2 February 2025). Models listed with (encoder-only) next to their name are encoder-decoder models where only the encoder part is used. In such a case, the values in Table 1 refer to the part of the model used, not the entire architecture.

The baseline models were used as feature extractors. A 0.1 dropout layer was added to the output of the models to prevent overfitting, followed by a linear layer used as a classifier. The combination of different SLMs was also investigated. In this case, the output of the SLM was taken to a linear layer to equalize the output dimensionality of the different models before merging their features. The output features were then fused by concatenation. Figure 1 shows the proposed multimodal ensemble architecture, which represents the combination of four models. The same structure was followed when only two models were combined.

3.3. Experimental Setup

Training and validation of the different models was done by 5-fold cross-validation, where 4 sessions in each fold were used for training and the remaining session was used for validation. All SLMs were initialized with the pre-trained weights available from Hugging Face (https://huggingface.co/, accessed on 12 January 2025). Given that these models are pre-trained, a low learning rate (${10}^{-5}$) was chosen. In addition, a small batch size of 12 was chosen and an early stopping mechanism was applied to prevent overfitting. This mechanism consisted of stopping the training process if there was no improvement in the validation unweighted accuracy (UA) after 3 epochs. Furthermore, a maximum of 20 epochs was set for the training duration. In addition, the loss function was weighted by class weights (inversely proportional to class size) to compensate for the effect of class imbalance of the IEMOCAP dataset.

The audio models were trained using 10-s sequences, with padding or truncation applied as necessary to adjust them to that duration. In the case of Whisper, which is configured to support only 30-s sequences by default, the positional embedding layer was truncated to work with 10-s sequences. This change was made to allow a fair comparison with the other architectures. However, this truncation was not strictly necessary, as Whisper could have been trained by padding the audio inputs with 20 s of silence. An attention mask is applied to prevent the model from attending to these silent sections, which would hypothetically yield similar results to those obtained here, but with a significantly higher computational cost.

All individual and text ensemble models were trained on a workstation with an NVIDIA RTX 2070 Super GPU with 8 GB of VRAM, while the audio ensemble and multimodal models were trained on a workstation with an NVIDIA P100 GPU with 16 GB of VRAM.

The evaluation of the models was performed on the basis of weighted accuracy (WA) and UA, with the latter metric being used to select the best model. Different definitions of WA and UA exist in the literature, making it difficult to compare results. In this paper, one of the most extended definitions is used, where WA is considered as the percentage of correct predictions, while UA is defined as the average of the recall metric for each class:

$$WA={\displaystyle \frac{{\sum }_{i=1}^{K}T{P}_i}{{\sum }_{i=1}^K(T{P}_i+F{N}_i)}}$$

(1)

$$UA={\displaystyle \frac{1}{K}}\sum _{i=1}^K{\displaystyle \frac{T{P}_i}{T{P}_i+F{N}_i}}$$

(2)

where K is the number of classes, $T{P}_i$ represents the true positives for class i, and $F{N}_i$ represents the false negatives for class i.

After training the baseline models, the model with the best validation AU in the text modality was selected for each category. In the audio modality, where no decoder-only model was used, the best encoder-only model and the best encoder-decoder model (using only the encoder part in this experiment) were selected. With the selected models, the effect of adding contextual information was investigated by combining different models or by using two copies of the same model, one for the utterance and one for the contextual information. In addition, the combination of models from different modalities was explored. Based on the results of this experiment, a multimodal model was assembled that included contextual information (see Figure 2). The decision to compare only models of different architectures and modalities was made for two main reasons. First, the number of combinations that would result from testing the 19 base models would be prohibitively large. Second, ensemble models often tend to perform better when combining features that are significantly different from each other, as diversity among classifiers has been shown to be a key factor in ensemble performance [74].

4. Results

4.1. Baseline Architectures

Table 2. Cross-validation results of the baseline models, showing the average accuracy over 5 folds.

Model	Architecture	Modality	WA (%)	UA (%)
ALBERT	encoder-only	text	64.65	64.79
BERT	encoder-only	text	69.61	70.64
DeBERTa	encoder-only	text	69.80	70.60
ERNIE	encoder-only	text	68.11	69.67
RoBERTa	encoder-only	text	68.62	70.55
BART	encoder-decoder	text	69.16	70.23
T5	encoder-decoder	text	66.20	67.52
GPT-2	decoder-only	text	66.43	67.38
GPT-Neo	decoder-only	text	63.73	64.27
OPT	decoder-only	text	68.49	69.73
Pythia	decoder-only	text	64.31	65.97
XLNet	decoder-only	text	67.34	68.17
HuBERT	encoder-only	audio	68.85	66.99
SEW	encoder-only	audio	67.81	64.35
WavLM	encoder-only	audio	69.13	66.81
Wav2Vec2	encoder-only	audio	71.56	68.73
S2T	encoder-decoder	audio	65.30	63.34
SpeechT5	encoder-decoder	audio	67.83	68.07
Whisper	encoder-decoder	audio	72.95	72.82

Note: Bold indicates the highest value per architecture type and modality.

shows the cross-validation results of the baseline architectures. The “Architecture” column refers to the original model architecture. Note, however, that in many encoder-decoder architectures only the encoder part was used. The best text classification results were obtained with the BERT model, closely followed by DeBERTa and RoBERTa. In the encoder-decoder models, BART clearly stands out with 70.23% UA, while in the encoder-only model, OPT stands out with a UA of 69.73%. On the other hand, in the audio modality, Whisper achieves the highest UA, beating the second best model (Wav2Vec2) by more than 4 points and the best text model by more than 2 points.

4.2. Ensemble Models

Based on these results, the best model in each category was selected to investigate the effect of adding contextual information on SER accuracy. For each model, its performance was evaluated using a copy of the same model to process the contextual information, as well as using the other models of the same modality to process the contextual information (see Figure 2).

Table 3. Cross-validation results for unimodal ensemble models using contextual information.

Utterance	Context	Modality	WA (%)	UA (%)
BERT	BERT	text	75.83	78.38
BERT	BART	text	77.25	78.42
BERT	OPT	text	75.48	78.13
BART	BART	text	76.84	79.07
BART	BERT	text	75.47	78.32
BART	OPT	text	75.29	77.38
OPT	OPT	text	77.70	79.15
OPT	BERT	text	76.22	77.85
OPT	BART	text	76.92	78.68
Wav2Vec2	Wav2Vec2	audio	71.72	70.84
Wav2Vec2	Whisper	audio	72.28	71.60
Whisper	Whisper	audio	76.86	76.31
Whisper	Wav2Vec2	audio	75.73	75.17

Note: Bold indicates the highest value per modality.

summarizes the results of the different combinations studied.

In all combinations analyzed, a significant improvement in accuracy was observed over the individual models. However, the combination of different models did not lead to an improvement compared to using the same architecture twice. In the text modality, accuracy improvements of more than 9 points were achieved in both WA and UA. Conversely, in the audio modality, the improvements were more discrete. The use of two Wav2Vec2 models with contextual information only resulted in an increase in UA of about 2 points, while WA remained almost the same. On the other hand, the use of two Whisper models resulted in an increase of almost 4 points in both WA and UA.

In addition, the impact of combining text and audio models was evaluated to determine how the integration of different modalities affects overall performance, as shown in

Table 4. Cross-validation results for multimodal ensemble model without contextual information.

Audio Model	Text Model	WA (%)	UA (%)
Wav2Vec2	BERT	76.93	76.92
Wav2Vec2	BART	75.36	75.99
Wav2Vec2	OPT	76.11	75.79
Whisper	BERT	76.33	75.48
Whisper	BART	76.42	74.62
Whisper	OPT	79.06	78.05

Note: Bold indicates the highest value.

. The best multimodal classification results were obtained with the combination of Whisper and OPT. However, all the combinations studied achieved accuracies higher than those reported by the baseline models (see Table 2).

Considering that Whisper and OPT showed the best performance in both unimodal and multimodal ensemble architectures, these models were selected to build a multimodal ensemble architecture that supports contextual information according to the scheme shown in Figure 1. The ensemble model is composed of four SLMs, namely two Whisper models to process the audio with contextual information, and two OPT models to process the audio transcripts, also with contextual information.

The proposed ensemble model showed a WA of 82.12% and a UA of 83.34%. Figure 3 shows the confusion matrix combined from the 5 folds, where the lowest accuracy corresponds to the neutral class (74.06%) and the highest to the angry class (89.66%). The model showed rapid convergence, reaching its maximum validation accuracy after an average of only 2.2 epochs. In addition, it achieved a WA of 80.11% and a UA of 81.68% after only one epoch. Furthermore, it is important to highlight the stability of the results, WA remained above 80% and UA above 81.5% during the first 4 epochs. Due to the early stopping criterion used, results are only available for the first 4 epochs in the 5 folds. However, in cases where training extended beyond this range, such as the fold that used session 1 as the validation set, where training stopped at epoch 7, both WA and UA metrics remained stable over the 7 epochs, with standard deviations of 1.4% for both metrics.

To better understand the contribution of each component in the final model, we conducted an ablation study by progressively removing individual elements from the full ensemble, see

Table 5. Ablation study of the proposed model. WA (%) and UA (%) after removing each component.

Audio		Text		WA (%)	UA (%)
Utterance	Context	Utterance	Context
✓	✓	✓	✓	82.12	83.34
✓	—	✓	✓	81.84	82.76
✓	✓	✓	—	80.66	81.11
✓	✓	—	—	76.86	76.31
—	—	✓	✓	77.70	79.15
✓	—	✓	–	79.06	78.05

Note: Bold indicates the highest value.

. The results showed that removing the audio context led to a slight decrease in performance (UA dropped from 83.34% to 82.76%), while removing the text context caused a more pronounced decline (UA decreased to 81.11%). Excluding the entire text modality resulted in a significant drop in accuracy (UA fell to 76.31%). Similarly, eliminating both audio context and text context reduced performance considerably. These findings demonstrated that both modalities and their contextual information contributed meaningfully to the model’s effectiveness.

Figure 4 shows the learning curve of the proposed model during the first epoch, using session 5 as the validation set. The model achieved an accuracy of more than 80% after training with less than 150 batches, which represents 50% of the total dataset. Although some instability is observed at the beginning, both accuracy and loss tend to stabilize towards the end of the epoch.

4.3. Comparison with Previous Works

Table 6. Comparison with previous research.

Method	Year	WA (%)	UA (%)
AA-CBGRU [75]	2022	72.8	67.8
TFCNN + SL + Vggish [76]	2022	73.0	—
HD-MFM [77]	2022	72.0	73.4
ATFNN [78]	2023	73.8	64.5
Wav2Vec2 and prosody [66]	2023	74.2	75.6
CHAN [79]	2023	75.0	75.9
MM-EMOR [80]	2023	77.9	77.1
BERT + Wav2Vec2 + CA Aux [81]	2023	78.3	79.6
CCTG-NET [82]	2023	88.4	89.1
SSL-SF-MoE [83]	2024	73.9	72.3
Prosody2Vec + HuBERT Large [84]	2024	72.4	73.3
Vesper (Shiftformer) [85]	2024	73.7	74.3
BERT-RoBERTa BiLSTM [86]	2024	74.0	74.3
TsPCA [87]	2024	74.0	74.6
MAF network [88]	2024	75.7	75.5
MSER [89]	2024	77.2	76.6
Mul-TAB [90]	2024	83.9	81.9
Bi-modal Bi-task ERM [91]	2024	82.3	83.0
SE3D-CARN [92]	2024	84.0	83.0
MVAN [93]	2024	86.9	86.6
AMGCN [94]	2025	74.3	65.1
AFEA-Net [95]	2025	75.1	75.3
DSTM [96]	2025	76.3	77.4
MemoCMT [97]	2025	81.9	81.3
MTLSER [98]	2025	82.6	82.2
Whisper + OPT with context (Ours)	2025	82.1	83.3

Note: Bold indicates the highest value.

provides a comparison of the results of this work with those reported in previous studies. The comparison only includes works that performed a cross-validation similar to ours, which guarantees that the models were evaluated independently of the speaker. This decision was motivated by the fact that the selection of the validation set was found to have a significant impact on accuracy. Some studies validate their results using a single session out of the five into which the dataset is divided, without cross-validation. While this ensures speaker independence, it does not allow for a fair comparison, as the choice of validation session can significantly affect accuracy. In our experiment, we observed that when session 2 was used as the validation set, accuracy was significantly higher than when any other session was used for validation. This variability highlights the importance of using a cross-validation strategy to ensure a more reliable and fair assessment.

As shown in Table 6, while the proposed ensemble architecture outperforms several recent approaches, it falls below the current state-of-the-art. However, it is crucial to consider the methodology used during the training process when interpreting the results. In our approach, very short training without hyperparameter tuning was carried out to avoid overfitting.

In contrast, many of the current approaches use much longer training, sometimes without early stopping criteria, combined with exhaustive hyperparameter search. In this sense, three of the approaches with better WA and UA [82,91,93] performed meticulous hyperparameter fitting. In addition, another outstanding work [90], which like us relied on a pre-trained model ensemble, performed 100 epochs of training and referred their results to the best epoch. Similarly, the other approach with high WA and UA [92] performed a 200 epoch training with a scheduled learning rate decay but without early stopping. While these procedures are valid and common in the literature, they can sometimes produce better results on a given benchmark at the cost of reducing the model’s ability to generalize to new data.

On the other hand, the fast convergence of our proposal is a notable advantage over other recent approaches. Some architectures outperform our model by resorting to data augmentation techniques, without which their accuracy drops by more than 5 points [93]. In contrast, our model is able to generalize after processing only half of the dataset once.

5. Discussion

Speech emotion recognition is a complex task that is usually addressed by implementing specific models for this purpose, usually involving a combination of different feature extraction methods or the use of complex attention mechanisms. The most common approach, as evidenced by the works reviewed in Section 2.2, often employs one or more SLMs as a backbone and their subsequent optimization by means of various strategies such as exhaustive hyperparameter tuning [52], self-labeling [76], use of advanced attention mechanisms [75], or customized loss functions [85].

In this paper, we have proposed a different approach. We fine-tuned 19 different general-purpose SLMs on a limited number of epochs, adding only a dropout layer and a linear layer to perform classification. We deliberately refrained from data augmentation, exhaustive hyperparameter tuning, or any other technique that could improve the accuracy of the analyzed models. Given that the architectures analyzed are typically used as the backbone of more complex models, our experiments aimed to establish a baseline that can serve as a reference for future research, providing valuable information about the architectures and their combinations that yield the best performance with minimal modifications.

The classification results of the baseline models highlight the importance of the choice of architecture for SER, as differences of more than 5 points were obtained between different architectures, despite having a similar number of trainable parameters. Encoder-only models are usually considered the most suitable for classification tasks and are the most widely used in the literature. In our study, these models were the most accurate for text classification. However, BART (encoder-decoder) and OPT (decoder-only) reported accuracies similar to those of the encoder-only architectures. Furthermore, when used in ensemble architectures with contextual information, BART and OPT slightly outperformed BERT.

The “Neutral” class was the most frequently misclassified across both text and audio models, likely because neutral utterances often lack strong prosodic variations or distinctive lexical markers that facilitate the recognition of other emotions. Additionally, in a subject-independent evaluation, the neutral tone of one speaker may resemble the slightly emotive tone of another, increasing classification difficulty. This pattern aligns with prior studies on the same dataset, where neutral is also the worst-classified class [82,83,91,92,95,96,97].

In the case of audio models, although encoder-only models are the most commonly used in the literature, the experiments conducted showed that Whisper, an encoder-decoder model (only the encoder pair was used in this work), produced the best classification results. This is particularly remarkable considering that Whisper is the architecture with the lowest number of trainable parameters and the lowest computational complexity. These results challenge the widely accepted notion that encoder-only models are the best performers in classification tasks, and invite reflection on the potential of other types of architectures.

The observed performance differences across models highlight the significant influence of pre-training data and objectives alongside architectural design. For instance, Whisper outperforms other audio models like Wav2Vec2 despite having fewer parameters and lower complexity, which likely reflects its extensive pre-training on a much larger and more diverse dataset. Similarly, among text models sharing similar architectures, variations in accuracy (e.g., OPT vs. GPT-2) suggest that pre-training corpus size and diversity, as well as the specific tasks used during pre-training, critically affect the models’ capacity to capture relevant emotional cues. Given that large-scale pre-training requires vast computational and data resources often unavailable to most researchers, our analysis considers architecture and pre-training jointly. This approach better reflects practical conditions and the overall impact on SER performance. The importance of dataset size and diversity in pre-training has been highlighted in several studies [39], where increased data scale yields substantial improvements beyond architectural changes.

On the other hand, the assembly of models, either to add contextual information or to combine modalities, was found to significantly increase the accuracy of SER. In this work, the incorporation of contextual information was done by combining two models, sometimes using the same architecture twice, although it could have been done with a single model. This choice was made for two main reasons. First, the use of two models allows the combination of different architectures to evaluate their performance and to make a fair comparison by also training an ensemble model composed of two copies of the same architecture. Second, the number of GFLOPs in inference of an ensemble model built with two copies of a model is practically the same as if a single model were used to process a sequence twice as long.

Finally, the combination of contextual and multimodal information produced the best classification results, outperforming many recent proposals. However, even if it falls below other approaches (e.g., [82,93]), the results obtained are particularly valuable because they were achieved without resorting to complex classifiers or advanced feature fusion techniques. The fact that these results were obtained by simply concatenating the feature vectors and using only a linear layer as the classification head suggests that significantly better results could potentially be obtained if more advanced fusion techniques were employed. For example, in a recent work, a feature fusion module based on multi-head cross-attention with two auxiliary tasks was proposed that significantly increased the accuracy of an ensemble model combining Wav2Vec2 and BERT compared to feature fusion by concatenation [81]. Although our results do not surpass the state-of-the-art, it is important to note that this work is not focused on achieving maximum accuracy, but on the backbone architectures and their possible combination. By employing a minimalist setup with a simple linear classifier, we demonstrate the great potential of these architectures. The simplicity of our approach highlights their adaptability and efficiency, making them ideal candidates for integration into more sophisticated models with advanced classifiers.

It may also be of interest for future work to perform feature fusion at an intermediate point in the SLM, which has been proven in other emotion classification tasks to improve performance without drastically increasing model complexity [99].

In addition, the rapid convergence of the proposed architecture is a major advantage over other approaches, as it allows for rapid retraining to adapt to new data, enabling its implementation in dynamic environments.

Limitations

An important limitation of this study is that the baseline models used were pre-trained on different datasets, which means that the comparison between different models is not completely fair. However, it is important to note that pre-training this type of model requires a large amount of computational resources due to the enormous size of the datasets used. This makes pre-training these architectures unaffordable for many researchers. For this reason, it is common to consider the model and its pre-training as a whole when making comparisons.

Another limitation is the size of the models used, which, although considered small for transformer-based models, still have a significant number of parameters. They are therefore heavier than many of the models used in SER. However, it is important to note that all the ensemble models analyzed can be trained on a single GPU with 16 GB of VRAM, using a batch size of 12 and float 32 precision. It would even be possible to train them on more modest hardware, using a reduced batch size with gradient accumulation, or using float 16 precision.

In addition, the evaluation of the models on a single dataset is another limitation. However, IEMOCAP is probably the most widely used dataset in SER and, unlike other popular datasets, it contains full transcripts of all audio recordings. This makes it easier to assess the impact of adding multimodal contextual information without relying on automatic speech recognition tools, which could alter the results in favor of one of the architectures. In light of the above considerations, we believe that evaluating the models on a single dataset is justified.

6. Conclusions

The aim of this paper was to evaluate the effectiveness of different general-purpose SLMs for the SER task using a consistent methodology, as well as to study the impact of adding contextual information and combining modalities on the performance of the models. The experiments conducted have validated the validity of these types of architectures for the proposed task. The results show a significant increase in accuracy when assembling audio and text models, as well as when assembling models of the same modality to process context. In particular, models such as Whisper and OPT, which have been used much less frequently in the literature than encoder-only models, showed superior performance in this study. This finding challenges the prevailing notion that encoder-only architectures are always the best choice for classification tasks such as SER.

Although the proposed solution is not state-of-the-art, it was not designed to maximize the performance of a model in a specific problem, but rather to demonstrate the potential of SLMs in SER and to provide a useful starting point for future research. The results of this study can be of great benefit to researchers and developers due to its ease of implementation and fast convergence. The evaluated architectures can easily be used as the backbone of more complex models. Future research could explore the use of these architectures together with fusion techniques based on attention mechanisms, advanced loss functions, or novel optimizers, which could potentially significantly improve their performance. Considering that the analyzed models are not specifically designed for emotion recognition, our findings may be useful for future research in other NLP areas beyond SER.