RAMHA: A Hybrid Social Text-Based Transformer with Adapter for Mental Health Emotion Classification

Abstract

Depression, stress, and anxiety are mental health disorders that are increasingly becoming a huge challenge in the digital age; at the same time, it is critical that they are detected early. Social media is a rich and complex source of emotional expressions that requires intelligent systems that can decode subtle psychological states from natural language. This paper presents RAMHA (RoBERTa with Adapter-based Mental Health Analyzer), a hybrid deep learning model that combines RoBERTa, parameter-efficient adapter layers, BiLSTM, and attention mechanisms and is further optimized with focal loss to address the class imbalance problem. When tested on three filtered versions of the GoEmotions dataset, RAMHA shows outstanding results, with a maximum accuracy of 92% in binary classification and 88% in multiclass tasks. A large number of experiments are performed to compare RAMHA with eight standard baseline models, including SVM, LSTM, and BERT. In these experiments, RAMHA is able to consistently outperform the other models in terms of accuracy, precision, recall, and F1-score. Ablation studies further confirm the contributions of the individual components of the architecture, and comparative analysis demonstrates that RAMHA outperforms the best previously reported F1-scores by a substantial margin. The results of our study not only indicate the potential of the adapter-enhanced transformer in emotion-aware mental health screening but also establish a solid basis for its use in clinical and social settings.

1. Introduction

Mental health is a complex subject that significantly impacts the way individuals think, feel, and behave. Psychological health problems encompass more than 200 distinct conditions, among which depression and anxiety disorders—including post-traumatic stress disorder (PTSD) and obsessive–compulsive disorder (OCD) [1]—are common. The International Classification of Diseases (ICD), maintained by the World Health Organization (WHO), provides a standardized system of classifications and codes for all known illnesses and health conditions worldwide. The most recent edition, ICD-11, contains over 55,000 codes [2] covering diseases, symptoms, injuries, and risk factors, and it is widely used for diagnosis, treatment, research, insurance, and healthcare management.

Mental health disorders affect approximately one in seven children and adolescents aged 10–19, accounting for about 15% [3] of the global disease burden in this age group. Among these, depression, anxiety, and behavioral disorders are particularly prominent and constitute major causes of morbidity and disability in adolescents. Globally, it is estimated that around 3.8% of the population suffers from depression, with women and older adults at the highest risk, amounting to roughly 280 million individuals. Depression is especially prevalent among women during pregnancy and postpartum. Furthermore, suicide is the third leading cause of death worldwide, claiming more than 700,000 lives annually, and it ranks fourth among individuals aged 15–29 years [4].

Depression, anxiety, and stress-related disorders are recognized as major public health problems worldwide, particularly in connection with the increased use of online communication and social media. As more individuals—including younger cohorts—share their feelings and experiences online, vast volumes of user-generated text data have become available, offering a rich opportunity for automated mental health assessment via natural language processing (NLP) methods [5]. Without timely treatment or diagnosis, these conditions can severely degrade quality of life and potentially develop into chronic psychological disorders or lead to suicidal tendencies [6].

Online forums and social media platforms have proven to be valuable for mental health research and sentiment analysis [7,8], particularly in monitoring populations [9]. However, a key challenge remains: the lack of reliable methods to classify and assess the severity of mental illnesses based on user-generated text, ideally validated with clinical confirmation [10]. Progress is further constrained by the scarcity of high-quality, large-scale, clinically validated datasets specifically targeting depression severity [11].

Traditional machine learning methods, such as SVMs or Naive Bayes, have also been applied to predict psychological distress from text. However, the bag-of-words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF), or hand-crafted lexicon features have a shallow nature, which has been known to constrain such models. Although they may take the word frequency on the surface level or hypothesis-driven pattern, they cannot take into consideration the emotional connotations of language and the subtleties of context. This limitation is even clearer when it comes to discriminating between sentiment (overall positive, negative, or neutral attitude) and emotion (specific psychological states like anxiety, stress, or depression), which require more detailed representations [12,13]. The emergence of Transformer-based models, notably BERT [14] and RoBERTa [15], has significantly advanced the field by providing context-aware representations of textual units, enhancing robustness for sentiment and intent analysis. Despite these improvements, challenges remain: these models are computationally expensive to fine-tune, often struggle to adapt to new application domains, and frequently act as black boxes, with internal mechanisms that are difficult to interpret, particularly in high-stakes scenarios such as mental health diagnosis [16,17].

Problem Statement

Although NLP has made significant progress, existing models for mental health detection face substantial limitations: limited understanding of subtle context, inability to capture long-range dependencies, high computational costs, and difficulty detecting implicit emotional cues. These limitations hinder the accurate and timely classification of depression, stress, and anxiety in user-generated text.

To address these challenges, we propose a new hybrid deep learning model with enhanced capacity to extract meaningful emotional signals from text while maintaining efficiency, and accuracy. Specifically, we introduce RAMHA (RoBERTa Adapter Mental Health Attention), a hybrid model designed to automatically identify depression, anxiety, and stress from textual data. RAMHA aims to balance semantic performance, and computational efficiency. Its architecture consists of a RoBERTa-large Transformer backbone with adapter-tuning via Pfeiffer Adapter Fusion [16], which enables task-conditioned adaptation without retraining the entire model. This is followed by a BiLSTM layer that captures sequential emotional dependencies often expressed across multiple tokens, as BiLSTMs process [18] text bidirectionally. A token-level attention mechanism highlights emotionally salient cues, supporting interpretable mental health classification [19].

RAMHA distinguishes itself from prior hybrid Transformer–BiLSTM–attention models in several ways: (i) it employs adapter-based fine-tuning for parameter-efficient training, (ii) integrates task-specific attention after BiLSTM for token-level interpretability, (iii) uses focal loss to address class imbalance, and (iv) is evaluated on curated GoEmotions subsets for both binary and ternary classification, achieving consistent improvements over conventional fine-tuning approaches.

The major theoretical and empirical contributions of this work are as follows:

• RAMHA Architecture:We propose a novel multi-stage hybrid model that integrates RoBERTa-large, lightweight adapters, BiLSTM, and attention mechanisms in an interpretable and parameter-efficient framework for mental health classification.
• Emotion-to-Mental Mapping: We perform a psychologically meaningful remapping of an emotion-rich dataset into three diagnostic classes—depression, anxiety, and stress—enhancing real-world applicability.
• Adapter Efficiency: Leveraging Pfeiffer Adapter Fusion, RAMHA fine-tunes task-specific parameters while largely preserving the generalization capabilities of the pretrained language model, with minimal training costs.
• Empirical Verification: RAMHA is evaluated on GoEmotions data and demonstrates superior performance compared to baseline Transformer-based classifiers, particularly in detecting subtle emotional signals indicative of mental health conditions.
• Binary and Ternary Classification: The model addresses both binary classification tasks (e.g., detecting the presence or absence of depression) and ternary classification tasks (e.g., distinguishing between depression, anxiety, and control).

RAMHA combines the semantic richness of pretrained language models, the emotional sensitivity of recurrent attention modules, and the flexibility of adapter-based tuning to provide and a scalable framework for early detection of mental health conditions. This approach is valuable for digital mental health platforms, clinical decision-support systems, and personalized self-care tools, particularly in low-resource or privacy-sensitive settings. Although the current implementation uses an English RoBERTa backbone, RAMHA is language-agnostic by design; replacing the encoder with a multilingual model or training language-specific adapters allows deployment in other languages with minimal retraining.

The paper is structured as follows: Section 2 reviews related work, Section 3 describes the methodology and model design, Section 4 details the dataset and hyperparameters, Section 5 presents results, discussion, and comparative analysis, and Section 6 concludes the study and outlines future directions.

2. Related Work

Early prediction of mental disorders, including depression, anxiety, and stress in textual data, has become one of the most investigated tasks over the last few years [20,21] because more and more people choose to disclose information about their mental health on the Internet. Various machine learning and deep learning methods have been provided to extract and classify the psychological signal in user-generated content, such as social media, forum posts, and clinical transcripts.

2.1. Machine Learning Approaches

Conventional machine learning approaches for mental health prediction have primarily relied on feature-based methods, leveraging lexicons and conventional classifiers such as Support Vector Machines (SVMs), Naive Bayes (NB), and Logistic Regression (LR). These methods utilized manually engineered features, including n-grams, part-of-speech tags, and sentiment scores, to infer psychological states [22,23]. However, such models are limited in capturing the nuanced semantics and contextual relationships inherent in natural language, particularly in emotionally complex or ambiguous text.

Recent studies have explored the capability of machine learning to identify depression in social media users even in the absence of explicit mentions of depressive terms [20]. Both individual classifiers and ensemble methods have demonstrated promising performance across diverse social media datasets, employing comprehensive text preprocessing (e.g., tokenization, stop-word removal, and POS lemmatization) and feature extraction (e.g., bag-of-words and n-grams). These approaches highlight the potential for the generalized keyword-independent detection of depressive states.

Further research has focused on detecting the intensity of depression by leveraging emotions expressed in social media content through emotion theories, natural language processing, and machine learning techniques [12]. Such studies indicate that machines can be trained to recognize subtle grammatical cues, cultural differences, and underlying emotional signals to accurately infer users’ psychological states.

Additionally, investigations such as [24] evaluated multiple machine learning algorithms—including K-Nearest Neighbors (KNN), Multilayer Perceptron (MLP), and SVM—to assess depression, anxiety, and stress levels in recovered COVID-19 patients. The study employed feature selection strategies and hyperparameter optimization, finding SVM to achieve the highest accuracy, while Extra Trees outperformed Recursive Feature Elimination (RFE) in feature selection. These findings provide valuable insights for selecting and optimizing models for early and accurate mental health detection.

Most contemporary research [25,26,27] in depression detection utilizes a wide array of machine learning classifiers, such as SVM, MLP, LR, Decision Trees (DT), NB, Maximum Entropy (ME), KNN, Adaptive Boosting (AB), Random Forest (RF), Gradient Boosting (GB), and Bagging Predictors (BP), in both single and ensemble configurations. These predictive approaches are designed to facilitate early identification of depression and related mental health conditions. A key advancement in recent studies is the analysis of how feature selection impacts detection accuracy. Furthermore, while traditional fMRI (functional Magnetic Resonance Imaging) studies could not predict susceptibility to major depression, recent machine learning applications have leveraged neural biomarkers—such as guilt-selective functional disconnection—to enhance predictive performance.

2.2. Deep Learning Approaches

Following the advent of traditional machine learning models, deep learning approaches gained prominence in emotion classification [28,29,30] and sentiment analysis [31], leveraging architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and particularly BiLSTM networks [32].

For instance, studuies by Kim and Mehmood et al. [33,34] developed deep learning models to assess the mental health of users based on their social media activity, with a focus on communities such as Reddit. These models were capable of accurately categorizing posts associated with specific disorders, including depression, anxiety, bipolar disorder, borderline personality disorder, schizophrenia, and autism, highlighting the potential of AI as a supplementary tool for monitoring mental health through online behavior.

Similarly, Amanat et al. [35] proposed an efficient approach using a Long Short-Term Memory (LSTM) network with two hidden layers, combined with an RNN comprising two dense layers, to predict depression from textual data. This framework demonstrates the utility of leveraging textual semantics for early detection of mental health issues, thereby contributing to preventive interventions for mental illnesses and suicidal tendencies.

In study [19], methodologically analyses 98 studies that have been published on machine learning (ML) methods to detect and predict mental stress and associated mental disorders (MDs) based on PRISMA-ScR guidelines. The results indicate that SVM, neural networks, and random forests demonstrate good performance across all models. Common key predictors include physiological signals, including heart rate and skin response. Also, dimensionality reduction methods are essential in improving the performance of models by preprocessing them well.

The paper [36] suggests a hybrid deep learning model of depression identification using both textual and audio data of the DAIC-WoZ dataset. Text and audio data were extracted using separate CNN models, whereas a hybrid model based on a Bi-LSTM was used to combine both data types and reveal more profound behavioral patterns. The research points out that audio aspects play a more important role in the performance of models than textual ones. Thorough assessments based on precision, recall, F1-score, and confusion matrices prove that the proposed models are effective in detecting depression.

Another study [37] presents AnxPred, a hybrid architecture of CNN and SVM to extract and classify data that improves the prediction of anxiety using academic data. The model was trained on the data gathered among students in 15 universities in Bangladesh using the GAD-7 scale. An explainable AI (LIME) was used to confirm that the model was able to determine important stressors related to anxiety. The results indicate that AnxPred is more efficient than traditional machine learning and deep learning models and provides a solid instrument to screen anxiety early and simplify automated mental health interventions in an academic context.

2.3. Transformer-Based Methods

Transformer-based language changed all of this when models like BERT [14] and RoBERTa [15] appeared, enabling large advances in the performance of text classification tasks [38] due to the use of bidirectional context.

Multiple studies have been conducted with BERT and variants of the model in mental health detection. For an illustration, [39] applied a fine-tuned BERT to suicide risk classification, whereas [40] considered larger pre-trained models to provide depression diagnosis on Reddit posts. Although these models are successful, they need to be fine-tuned in a complete fashion that is less resource-intensive and modular to use in a domain transfer setting.

Study [41] provides an in-depth investigation on detecting the severity of depression in texts in social media (in the case of the study, the platform Reddit) by both classic content-based methods (such as N-grams) and modern context-based methods, such as sentence transformers and transformer-based models. Their experiments have shown that transformer architectures are much more effective than conventional approaches because they are able to capture user language subcontextual clues with high efficiency. In addition to increasing the classification accuracy of the depression state into mild, moderate, and severe, the proposed framework provides a solution to perform scalable and flexible adjustment in the real-life setting of diagnosing mental health and early intervention systems.

The study [21] proposed a Transformer-based model to identify the severity of depression in tweets with standards such as DSM-5 and PHQ-9. They proposed a typology and created a collection of 40,191 tweets expert-labeled as non-depressed or with mild, moderate, or severe depression, each with confidence scores. They discovered powerful baselines with the help of models such as BERT and DistilBERT and stressed the necessity of calibrating models and the robustness of datasets to be used in future studies.

Another study [42] created a number of Transformer-based classifiers, namely BERT, RoBERTa, BERTweet, and MentalBERT to diagnose depression on social media postings. They also built ensemble models and tested them on both Reddit and Twitter datasets as well as cross-platform transfer learning. Findings showed that the ensemble models performed better than individual Transformers, in which model aggregation is evident.

2.4. Adapter-Based Fine-Tuning

With the current challenge, the task of fine-tuning large models is computationally inefficient. Houlsby et al. [40] proposed the concept of adapter-based transfer learning, in which lightweight modules are inserted between Transformer layers and are the only part of the network that is trained with the downstream task. Pfeiffer et al. [16] followed up on this and added adapter fusion, where several domain-specific adapters can be concatenated. Subsequently, in the study [43], there was a publicly released library of pre-trained adapters across tasks called AdapterHub. The existing techniques have been shown to decrease training expenses at the cost of maintaining competitive performance, yet their usage has been restricted in mental health groups.

The paper [44] provided a new corpus of hate speech in the low-resource Tamil language related to caste and migration. A Macro-F1 score of 0.73 was obtained by baseline experiments and additional improvement was obtained using custom loss functions and parameter-efficient fine-tuning methods. A common task was performed in order to benchmark methods, and the dataset was openly published to encourage further work on the problem of reducing hate speech on social media.

2.5. Attention-Based Hybrid Architectures

Transformer encoders have been proven to capture emotional dependencies better when combined with BiLSTM and attention mechanisms and increase interpretability. Examples include [45], who used BERT with BiLSTM to perform sentiment classification tasks, or the addition of attention by Bahdanau et al. [24], performing a sequence-to-sequence task. Such hybrids have begun being studied in recent mental health research, though most are not highly tuning efficient (such as through adapters of some sort) or are otherwise severely limited in achieving fair (or decent) trade-offs between complexity and interpretability.

The paper [17] also presented a widely understandable model to diagnose depression on Twitter with a hierarchical attention-based encoder with feed-forward networks. The model applies metaphorical concept mappings (MCMs) to provide performance and interpretability. In contrast to word-level attention in most of the SOTA approaches, it uses context-level attention to source influential tweets and metaphors. This method accompanies psycholinguistic variables and enhances cognitive knowledge on depression. Experiments indicate that compared to conventional encoders, the model has fewer parameters and gives good, interpretable results. Table 1 presents the comparison of RAMHA with related hybrid models in mental health/emotion.

Table 1. Task-Aligned comparison of RAMHA with related hybrid models in mental health/emotion classification.

Model	Design Highlights	Relation to RAMHA
BERT–BiLSTM for Stress Detection (2024) [46]	Uses BERT embeddings followed by BiLSTM to detect chronic stress in tweets.	Similar pipeline but lacks attention and adapter modules. RAMHA introduces attention and adapter-based fine-tuning for better interpretability and efficiency.
Hybrid Transformer + BiLSTM + Feature Fusion (2022) [47]	Combines BERT/RoBERTa, BiLSTM, and handcrafted linguistic features for multiclass mental health prediction.	Fuse multimodal features but without adapters or focal loss. RAMHA is fully end-to-end and handles class imbalance effectively.
Attention-based CNN–BiLSTM for Depression (2024) [48]	Applies an attention-enhanced CNN–BiLSTM network for depression detection on social media text.	Adds attention but lacks Transformer backbone and adapter efficiency; RAMHA uses a Transformer core with adapter-based tuning.
Opinion-BERT (2025) [49]	Multi-task model using BERT, CNN/BiGRU, and opinion embeddings for sentiment and mental health status classification.	Hybrid structure with local and sequential layers, but no adapter fine-tuning or focal loss. RAMHA enhances efficiency, generalizability, and imbalance handling.
RAMHA (proposed)	RoBERTa-Large with adapter fine-tuning, BiLSTM, attention modules, and focal loss—designed for efficient and robust mental health emotion classification.	Unique integration of adapters, attention, and class-imbalance-aware design.

3. Methodology and Model Design

In this section, we present the methodology and model design utilized in our study to characterize mental health states—specifically depression, anxiety, and stress—based on textual data and associated emotional cues. The proposed framework comprises five key components: data preparation, tokenization, model architecture, training, and evaluation. Each component has been carefully designed to enhance both model performance and interpretability, particularly in the context of emotion-driven mental health analysis. Figure 1 illustrates the overall workflow, while Algorithm 1 provides a detailed step-by-step description of the proposed model.

Algorithm 1 RAMHA: RoBERTa–Adapter–BiLSTM–Attention framework.

Require: Input text x Ensure: Predicted class probabilities p  1: $T\leftarrow \mathrm{RoBERTaTokenizer}\left(x\right)$  2: $E\leftarrow \mathrm{RoBERTa}\left(T\right)$  3: $E^{\prime }\leftarrow \mathrm{Adapter}\left(E\right)$  4: $H\leftarrow \mathrm{BiLSTM}\left(E^{\prime }\right)$  5: for each hidden state $h_i\in H$ do  6:    $s_i\leftarrow \mathrm{score}\left(h_i\right)$  7: end for  8: ${\alpha }_i\leftarrow {\displaystyle \frac{exp\left(s_i\right)}{{\sum }_{j}exp\left(s_j\right)}}$  9: $C\leftarrow {\sum }_i{\alpha }_i{h}_i$ 10: $C^{\prime }\leftarrow \mathrm{Dropout}(C,p)$ 11: $z\leftarrow W{C}^{\prime }+b$ 12: $p\leftarrow \mathrm{Softmax}\left(z\right)$ 13: $FL\left(p_t\right)\leftarrow -{\alpha }_t{(1-p_t)}^{\gamma }log\left(p_t\right)$ 14: return p

Figure 1. RAMHA model training pipeline.

3.1. Data Preparation

This is followed by loading and preparing the GoEmotions dataset, which is a large corpus built by Google consisting of 58k English Reddit submissions labeled on 28 emotion types along with a neutral label. The raw units are pieces of text and their emotion labels.

Figure 2 presents the step-by-step mechanism of the proposed architecture. In order to prepare this dataset to fall under the mental health theme, the following sub-steps have been undertaken:

• Loading and Preprocessing: The data is read in CSV format. Every item of text is preprocessed, i.e., all unnecessary punctuation marks, symbols (i.e., dots, commas, and special characters), HTML tags, and a lot of whitespace are deleted. This makes the input noise-free and optimized to be tokenized and modeled downstream.
• Mapping Emotions to Mental Health categories: A mapping dictionary has been formulated that has defined the original 28 emotion labels using three high-level categories of mental illnesses:

  ‑

  Depression: It contains sadness, grief, loneliness, and disappointment.

  ‑

  Anxiety: It consists of fear, nervousness, worry, and embarrassment.

  ‑

  Stress: Stress includes labels such as annoyance, confusion, nervousness, disappointment, embarrassment, and surprise.

• Choice of Primary Labels: A particular Reddit comment can be associated with various emotions; however, there is an argmax operation where we pick out the most obvious mental health label on a per-sample basis. This reduces the multi-label framework into a multi-class classification problem.
• Label Encoding: Label encoding is used so that the category labels in mental health (depression, anxiety, stress) can be translated into integers (e.g., 0, 1, 2) and are compatible with PyTorch models.
• Data Balancing: To mitigate the problem of class imbalance that exists in the combined datasets, we conducted proportional sampling so that each target class—depression, anxiety, and stress—would have the same number of samples before the data was split.
• Data Splitting: The dataset was divided into training and validation subsets using an 80/20 split. The validation set was employed to monitor model performance during training and to mitigate the risk of overfitting. To ensure fairness and balanced representation of classes, we applied class-balancing techniques along with focal loss, making the validation set a reliable indicator of the model’s generalization performance, effectively serving as a substitute for a separate test set.

Figure 2. Block diagram of step-by-step mechanism of Algorithm 1.

3.2. Tokenization

The RoBERTa-Large tokenizer in the Hugging Face Transformers library is used to convert textual inputs to a numerical representation. RoBERTa uses a BPE (byte pair encoding)-based tokenizer to split each word, explained in the following equation.

$$S=\left\{w_1,w_2,\dots ,w_n\right\}$$

(1)

Tokenizer carries out sub-word tokenization as follows:

$$S=\left\{s{i}_1,s{i}_2,\dots ,s{i}_m\right\}$$

(2)

Final sequence shown in the following equation.

$$T=BPE\left(S\right)=\left\{t_1,t_2,\dots ,t_k\right\}$$

(3)

Here, $K\le maxlength$ and $t\in v$, with V being RoBERTa’s vocabulary of subword tokens.

• Truncation and Padding: Each input sequence is truncated to be equal to the maximum model length or padded to a fixed length to provide consistency.
• Special Tokens: RoBERTa-specific tokens and padding tokens will be used where necessary.
• Token Type and Attention Masks: These are automatically generated to create the difference between tokens and attention-based processing.

Encoded label and the attention masks joined with the tokenized sequences are organized under the PyTorch Dataset and Data Loader objects, ensuring that they are efficiently trained in minibatches.

3.3. Model Architecture

The proposed model will incorporate a form of Transformer representation learning and incorporate sequence modeling with an attention mechanism to learn the dynamics of the emotion in texts specific to mental health. The model architecture is illustrated in Figure 3:

Figure 3. RAMHA model architecture starts with pre-processing of the data and mapping labels; then, it uses RoBERTa as a tokenizer. It uses a RoBERTa encoder with Pfeiffer adapter incorporation, a subsequent BiLSTM and attention mechanism to capture the contextual information, and a softmax layer to create multi-class classification (depression, anxiety, and control) predictions.

3.4. Pretrained Encoder (RoBERTa-Large)

In this paper, we use RoBERTa-Large as the encoder upon which the deep contextualized word embedding becomes generated on raw textual data. RoBERTa is an enhanced form of BERT, which contains 24 layers of Transformers; each layer has a hidden dimension of 1024, 16 attention heads inside, and 4096 feedforward, and has a total of approximately 355 million parameters. In each layer, RoBERTa uses multi-head self-attention to derive the token representations, and these can be expressed as follows:

$$Attention(Q,K,W)=softmax{\displaystyle \frac{Q{K}^t}{{\sqrt{d}}_k}}V$$

(4)

In which Q, K, and V are the query, key, and value matrices formed by the hidden states, and dkt is the dimension of the key vectors. This formulation allows the model to infer long-range dependencies since it pays attention to all tokens present within the sequential order, irrespective of their location. A key question arises as to why RoBERTa is considered the most feasible choice; thus, this is illustrated in Figure 4.

Figure 4. Comparison highlighting RoBERTa as the most feasible choice.

Nevertheless, RoBERTa-Large is computationally feasible under our implementation with adapters, which add a few task-specific parameters and freeze the main Transformer weights. The end result of RoBERTa is given by a set of contextual embeddings to the BiLSTM and attention layers for further processing. These embeddings acquire sequential and emotional dependency to classify mental health.

3.5. Adapter Integration: Pfeiffer Configuration

In order to scale RoBERTa to domain-specific analysis of mental health without sacrificing training scalability, we incorporate adapter modules with the AdapterTransformers library. Adapters are small feedforward neural modules that are inserted into every Transformer layer and can be fine-tuned to a specific task without re-training all 355 million weights of RoBERTa-Large. Specifically, we use the Pfeiffer configuration, a widely adopted adapter setup that introduces a bottleneck architecture after the feed-forward sublayer in each Transformer block. Every adapter is composed of a down-projection, non-linearity, and up-projection layer. The adapter transforms a hidden representation in the following form.

$$Adapter\left(h\right)=h+W_{up}.\sigma (W_{down}.h)$$

(5)

The trainable parameters are those of the adapter (usually <1–3% of the full model), and the RoBERTa parameters are fixed in a frozen state, which also considerably lowers the chance of catastrophic forgetting and results in a more efficient computation. This modular architecture allows the domain adaptation with a low resource footprint and facilitates fast. Adapter integration is illustrated in Figure 5.

Figure 5. Adapter integration: Pfeiffer configuration.

3.6. BiLSTM

To further absorb the temporal dependency and emotional evolution in the text, the contextual embeddings returned by RoBERTa are fed into a Bidirectional Long Short-Term Memory (BiLSTM) network. BiLSTM, unlike unidirectional models, makes use of the forward and backward directions of the sequence, allowing the network to be aware of the past and the future context at the same time, a requirement in the recognition of subtle emotion expression in mental health narratives. The mathematical relationships among the various gates, hidden states, and memory cells for the initial forward direction layer are formally defined as follows, where the hidden state is $a_t$.

$$i_t^a=\sigma (U_i^a{x}_t+W_i^{a}a(t-1)+b_i^a)$$

(6)

$$f_t^a=\sigma (U_f^a{x}_t+W_f^{a}a(t-1)+b_f^a)$$

(7)

$$o_t^a=\sigma (U_o^a{x}_t+W_0^{a}a(t-1)+b_o^a)$$

(8)

$$U_t^a=\tanh (U_u^a{x}_t+W_u^{a}a(t-1)+b_u^a)$$

(9)

$$C_t^a={(i_t^a\ast U_t^a\ast f_t^{a}C(t-1)}^a)$$

(10)

$$a_t=o_t^a\ast tanh{C}_t^a$$

(11)

Formal mathematical expressions for the backward layer and the hidden state $c_t$ are presented below.

$$i_t^c=\sigma (U_i^c{x}_t+W_i^{c}c(t-1)+b_i^c)$$

(12)

$$f_t^c=\sigma (U_f^c{x}_t+W_f^{c}c(t-1)+b_f^c)$$

(13)

$$o_t^c=\sigma (U_o^c{x}_t+W_0^{c}c(t-1)+b_o^c)$$

(14)

$$U_t^c=\tanh (U_u^c{x}_t+W_u^{c}c(t-1)+b_u^c)$$

(15)

$$C_t^c={(i_t^c\ast U_t^c\ast f_t^{c}C(t-1)}^c)$$

(16)

$$c_t=o_t^c\ast \tanh C_t^c$$

(17)

Final output is obtained using the following mathematical equations.

$$\overrightarrow{h_t^{\mathrm{LSTM}}}=\overrightarrow{\mathrm{LSTM}}\left(c_t\right),t\in \{1,\dots ,m\}$$

(18)

$$\overleftarrow{h_t^{\mathrm{LSTM}}}=\overleftarrow{\mathrm{LSTM}}\left(c_t\right),t\in \{m,\dots ,1\}$$

(19)

$$h_t^{\mathrm{LSTM}}=\overrightarrow{h_t^{\mathrm{LSTM}}}\oplus \overleftarrow{h_t^{\mathrm{LSTM}}},W_t=h_t^{\mathrm{LSTM}}$$

(20)

Here, the combined vector encodes information from both directions and goes inside a single model, which allows this model to better understand the contextual information, linguistic polarity, and emotional shifts along the sequence. These enhanced embeddings are then fed into an attention mechanism on a per-token importance weighting.

3.7. Attention Mechanism

In order to increase the model’s focus on the most emotionally informative sections of a sentence, we use an attention mechanism on the BiLSTM outputs. This system attaches learned weights to every token, allowing the model to use words that are emotionally significant towards mental health disorders.

Suppose that the BiLSTM output sequence is $s=[h_1,h_2,\dots \dots h_t]$, where $h_t\in R^{2d}$. For each token t, an attention score is computed as

$${\alpha }_t={\displaystyle \frac{exp\left(V^{\top }{h}^{-}\right)}{{\sum }_{t}exp\left(V^{\top }{h}^{-}\right)}}$$

(21)

$$S_{A_w}=\sum _t{\alpha }_t{h}_t$$

(22)

The symbols h and $h^{-}$ represent the forward and backward hidden states obtained from the Bi-LSTM, while V denotes a trainable parameter.

The term $S_{A_w}$ refers to the average of the attention weights assigned to the LSTM hidden states, as illustrated in Figure 6, where the attention score is calculated using the following equation:

$$\mathrm{score}\left(h_t\right)=v^T\tanh (W{h}_t+b)$$

Figure 6. Attention mechanism adopted from study [50,51].

The attention mechanism is applied to the $h_t^{\mathrm{LSTM}}$ outputs, allowing the model to focus more or less on specific terms within the input comment. This mechanism enables the model to selectively emphasize informative words by updating the feature vector, as demonstrated in Equation (23).

$$u_t^{\mathrm{LSTM}}=\tanh (W_w^{\mathrm{LSTM}}{h}_t^{\mathrm{LSTM}}+b_w^{\mathrm{LSTM}})$$

(23)

$${\alpha }_t^{\mathrm{LSTM}}={\displaystyle \frac{exp\left({\left(u_t^{\mathrm{LSTM}}\right)}^{\top }{u}_w^{\mathrm{LSTM}}\right)}{{\sum }_{t}exp\left({\left(u_t^{\mathrm{LSTM}}\right)}^{\top }{u}_w^{\mathrm{LSTM}}\right)}}$$

(24)

$$S^{\mathrm{LSTM}}=\sum _t{\alpha }_t^{\mathrm{LSTM}}{h}_t^{\mathrm{LSTM}}$$

(25)

Here, $u_t$ denotes the hidden representation of $h_t$, while $u_w$ is a context vector that is initialized and learned during training. The importance of each word is determined by comparing $u_t$ with $u_w$, followed by normalization, as shown in Equation (24). The resulting attention weights ${\alpha }_t$ are then used to compute the sentence representation S through a weighted sum, as defined in Equation (25). The sentence vector S effectively captures the most relevant information from the entire sequence and is subsequently passed on for downstream classification tasks.

3.8. Dropout

In order to reduce overfitting and enhance the model to perform better on unseen data, dropout regularization is performed on the attention output. The dropout randomly disables part of the neurons at training time and thus does not allow co-adaptation of features and favors robustness. In particular, we apply a dropout rate of 30%, i.e., during each forward propagation, 30% of the units in the sentence representation derived by the attention and s are randomly set to zero.

$$s^{\prime }=Dropout(s,p=0.3)$$

(26)

The result of such regularized vector $s^{\prime }$ is then fed into the final classification layer. Dropout is one of the simplest and most effective methods to decrease variance and enhance performance with contrasting mental health manifestation models.

3.9. Classification Head

The last part of the architecture is the classification head that is charged with mapping the high-level representation $s^{\prime }$ to mental health categories. We have a fully connected (linear) layer that is followed by a softmax operation that gives us the probabilities of depression, anxiety, or stress. The calculation is referred to as

$$y^{\wedge }=softmax(W_c{s}^{\prime }+b_c)$$

(27)

Above equation provides the output of the probability distribution with the help of learnable weights and biases, and this layer also allows interpretable multi-class prediction, which is based upon the emotionally salient information retrieved by earlier modules.

The adapter-based design also facilitates parameter-efficient transfer learning, allowing RAMHA to be adapted to other classification tasks, including International Classification of Diseases (ICD)-related clinical coding, by training new task-specific adapters and replacing the classification head.

4. Dataset and Hyper-Parameters

In this study, we now have a strong experimental framework to evaluate emotion classification with Transformer-based architectures. Here, we use the GoEmotions dataset with fine-grained emotional annotations, which is widely accepted. Moreover, our models are tuned with the help of well-chosen hyperparameters in order to improve the performance and guarantee a fair analysis in all experimental settings.

4.1. Dataset

This paper uses the GoEmotions dataset, which is a complete and popular benchmark of emotion classification tasks. The data set has three sub-versions that include GoEmotions1 with 70,000 instances, GoEmotions2 with 70,000 instances, and GoEmotions3 with 70,226 instances. Together, the data set offers a total of about 210,226 labeled examples, which guarantee a strong and varied training and evaluation base. Each instance in the dataset is annotated with one or more of 28 fine-grained emotion categories, which are mentioned in Table 2. The dataset is publicly accessible on Kaggle.com (https://www.kaggle.com/datasets/debarshichanda/goemotions, accessed on 1 August 2025). Figure 7 and Figure 8 illustrated datasets’ configuration and visualization of dataset distributions, respectively.

Table 2. Target classes and their corresponding labels.

Target Class	Label Used
Depression	Sadness, grief, remorse, disgust, disappointment, disapproval
Anxiety	Fear, nervousness, embarrassment, desire
Stress	Annoyance, anger, realization, confusion
Control	joy, love, optimism, gratitude, pride, relief, admiration, amusement, approval, caring, excitement

Figure 7. Configuration of the three datasets used in this study, showing the number of instances, labels, and label names.

Figure 8. Comparative visualization of dataset distributions.

4.2. Computational Environment and Hyper-Parameters

All experiments were conducted on a system equipped with an Intel Core i9-12900K CPU, 64 GB RAM, and an NVIDIA GeForce GTX 1080 Ti GPU with 24 GB VRAM. The operating system was Ubuntu 22.04 LTS. The implementation used Python 3.10, PyTorch 2.1, and Hugging Face Transformers 4.39. Other key libraries included NumPy. To ensure reproducibility, random seeds were fixed at 42 across all experiments.

Input text was preprocessed by lowercasing, punctuation removal, and tokenization using the RoBERTa tokenizer. For model training, hyperparameters were selected based on a grid search over a validation set. Specifically, the learning rate was set to 2 × ${10}^{-5}$, the dropout rate to 0.5, and the focal loss parameter $\gamma$ to 2. These values were chosen to balance convergence speed and generalization, as well as to handle class imbalance in the curated GoEmotions subsets. The model was trained until validation loss stabilized. Furthermore, Table 3 highlights hyperparameter settings for the proposed model.

Table 3. Hyperparameter settings of the proposed model.

Hyperparameter	Value / Setting
Pretrained Model	RoBERTa-Large
Tokenizer	RobertaTokenizer (Max Length: 128)
Learning Rate	2 × ${10}^{-5}$
Weight Decay	0.01
Batch Size	16
Optimizer	AdamW
Dropout Rate	0.5
Learning Rate Scheduler	ReduceLROnPlateau (patience = 2, factor = 0.5)
Loss Function	Focal Loss ($\gamma =2$)
Number of Epochs	10
Attention Hidden Size	512
Maximum Sequence Length	128
Evaluation Metrics	Accuracy, F1-Score

5. Results Discussions

The dataset is challenging due to overlapping labels, context-dependent meanings, and class imbalance. Therefore, attention-based deep learning algorithms are more applicable in order to identify these nuances and isolate classes. The performance of the text classifiers provides the evaluation with eight basic classification models named as LR, SVM, NB (Naive Bayes), LSTM, R-CNN, GRU, Bi-GRU, and BERT (here, SVM, LR, and NB are considered weak learners, whereas the remaining models are considered strong learners). We have three sub-versions of the GoEmotions dataset: GoEmotions-1, GoEmotions-2, and GoEmotions-3. Every dataset contains about 70,000 labeled samples that are divided into 28 categories of emotions. Our task-specific data were used to tune these models to identify text as being depression, stress, or anxiety. The performance of each model was evaluated using accuracy, precision, recall, and F1-score to provide a comprehensive assessment of effectiveness. Table 4 presents a comparative analysis of eight well-known classification models. Among traditional machine learning approaches, SVM and Logistic Regression (LR) achieved accuracies of 0.66 and 0.68, respectively, whereas Naive Bayes performed poorly, with an accuracy of 0.61 and an F1-score of 0.54. In contrast, deep learning models demonstrated superior performance. LSTM reached an accuracy of 0.72 with an F1-score of 0.67, while R-CNN and GRU also showed improved predictive capabilities. Notably, the Bi-GRU model outperformed other RNN-based baselines, achieving the highest accuracy of 0.75 and an F1-score of 0.71.

Table 4. Performance comparison of baseline models on GoEmotions datasets using accuracy, precision, recall, and F1-score metrics.

Model	GoEmotions-1				GoEmotions-2				GoEmotions-3
	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score
SVM	0.66	0.61	0.59	0.59	0.67	0.62	0.60	0.61	0.66	0.61	0.60	0.59
LR	0.67	0.61	0.58	0.59	0.68	0.63	0.61	0.61	0.68	0.63	0.60	0.59
NB	0.61	0.60	0.55	0.54	0.61	0.60	0.54	0.54	0.62	0.61	0.55	0.54
LSTM	0.71	0.68	0.66	0.67	0.71	0.69	0.67	0.67	0.72	0.70	0.68	0.67
R-CNN	0.72	0.69	0.67	0.68	0.70	0.68	0.68	0.68	0.72	0.70	0.67	0.68
GRU	0.73	0.70	0.68	0.69	0.73	0.69	0.68	0.68	0.74	0.71	0.69	0.69
Bi-GRU	0.75	0.72	0.71	0.71	0.74	0.71	0.70	0.69	0.74	0.72	0.70	0.69
BERT	0.78	0.76	0.74	0.75	0.77	0.75	0.74	0.75	0.77	0.74	0.75	0.75

Transformer-based BERT was the most spectacular in terms of overall performance on all datasets and achieved an accuracy of 0.78 on GoEmotions-1 and very high results on GoEmotions-2 and 3 with consistent F1-scores of 0.75. These figures proved that pre-trained language models, such as BERT, particularly in multi-label, fine-grained emotion classification, have a strong generalization capability and representation.

Now here in Table 5 is a comparison of the results of six (6) advanced Transformer-based models, BERT, DistilBERT, ALBERT, XLNet, RoBERTa, RoBERTa+BiLSTM, and our proposed model RAMHA on GoEmotions-1, GoEmotions-2, and GoEmotions-3. Accuracy, precision, recall, and F1-score were used to test these models so that they could be properly assessed. Although commonly used Transformer-based architectures such as BERT, ALBERT, and DistilBERT were found to perform reliably (with accuracies of 0.77–0.79), XLNet and RoBERTa performed even better and reached F1-scores of 0.78.

Table 5. Achieved result on GoEmotions datasets against various models.

Model	GoEmotions-1				GoEmotions-2				GoEmotions-3
	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score
BERT	0.78	0.76	0.74	0.75	0.77	0.75	0.74	0.75	0.77	0.74	0.75	0.75
DistilBERT	0.78	0.75	0.73	0.74	0.78	0.74	0.73	0.74	0.79	0.76	0.75	0.75
ALBERT	0.79	0.76	0.75	0.75	0.78	0.76	0.74	0.74	0.79	0.76	0.75	0.75
XLNET	0.80	0.77	0.76	0.76	0.79	0.77	0.75	0.75	0.81	0.77	0.77	0.77
RoBERTa	0.81	0.79	0.78	0.79	0.80	0.78	0.77	0.77	0.81	0.79	0.78	0.78
RoBERTa+BiLSTM	0.83	0.80	0.80	0.80	0.81	0.78	0.77	0.78	0.83	0.79	0.79	0.80
RAMHA (our Model)	0.88	0.86	0.85	0.85	0.87	0.85	0.84	0.85	0.89	0.87	0.86	0.87

The use of BiLSTM together with RoBERTa further increased the ability of the model to capture sequential dependencies, and there were steady improvements with F1-scores of 0.80. Nevertheless, our suggested hybrid model, RAMHA, was substantially better than all baselines in all data sets. On GoEmotions-1, it scored an accuracy of 0.88 and an F1-score of 0.85, with comparable good results on GoEmotions-2 and GoEmotions-3, with the best accuracy of 0.89 and an F1-score of 0.87.

One notable point here is that GoEmotions-3 consistently performed best across all models in Table 5, including BERT, DistilBERT, ALBERT, XLNet, and our proposed RAMHA. This dataset appears more suitable for depression–stress–anxiety classification as it is better aligned with clinically relevant categories and provides clearer class separability with less semantic overlap. In particular, the narrower and coherent label space reduces noise, making adapter-based fine-tuning more effective. Such performance enhancement in RAMHA can further be explained by its integration of adapter-based tuning, bidirectional temporal feature extraction, attention-driven focus on key emotional cues, and the robustness of focal loss in handling class imbalance.

On the whole, the findings verify that RAMHA does not merely outperform the traditional deep learning and Transformer baselines but also offers a very robust and generalizable fine-grained emotion classification solution to large-scale multi-label data.

To further assess the robustness and versatility of our proposed model, RAMHA, we evaluated its performance on the GoEmotions-1, GoEmotions-2, and GoEmotions-3 datasets across different emotion-to-class mappings. As shown in Table 6, RAMHA consistently outperformed other models, achieving high scores across all evaluation metrics, including accuracy, precision, recall, and F1-score. The model demonstrated strong performance in multi-class classification of depression, stress, and anxiety, attaining a maximum accuracy of 89% and an F1-score of 0.87, highlighting its effectiveness in multi-class emotion-based mental health classification.

Table 6. Performance of RAMHA (proposed model) on GoEmotions datasets across different emotion-to-class mapping combinations, compared against various baseline models.

RAMHA withDiff Combination	GoEmotions-1				GoEmotions-2				GoEmotions-3
	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score
Depression, stress, Anxiety (3 classes)	0.88	0.86	0.85	0.85	0.87	0.85	0.84	0.85	0.89	0.87	0.86	0.87
Depression, stress, control (3 classes)	0.90	0.88	0.87	0.88	0.89	0.88	0.87	0.87	0.90	0.88	0.88	0.88
Depression, stress (2 classes)	0.92	0.89	0.91	0.90	0.91	0.89	0.88	0.90	0.91	0.89	0.90	0.90
Depression, control (2 classes)	0.93	0.91	0.92	0.92	0.92	0.90	0.91	0.91	0.93	0.91	0.92	0.91

Notably, performance improved further under binary or alternative tri-class mappings. For instance, using the depression–stress–control mapping, RAMHA achieved an accuracy of 0.90 and an F1-score of 0.88. In the binary depression vs. control scenario, it attained its highest performance with 93% accuracy, precision, and recall between 0.91–0.92, and an F1-score of 0.92. These results underscore the generalization capability of RAMHA across different emotion groupings, demonstrating its practical applicability for mental health classification. The combination of adapter-based fine-tuning, BiLSTM, attention mechanisms, and focal loss proves effective for both fine-grained and coarse-grained mappings, providing a scalable and precise solution.

As a whole, Table 6 provides a broader perspective on how class similarity and opposition affect model performance by examining different class combinations. When depression, stress, and anxiety are grouped, their overlapping and semantically similar vocabulary makes it harder for the models to distinguish between them, leading to slightly lower scores. In contrast, combinations that include the control class achieve higher results, as control exhibits vocabulary patterns that are more distinct from mental health categories, making separation easier. Furthermore, reducing the task to two-class settings consistently improves performance due to less inter-class competition, with the depression–control pair achieving the highest accuracy because of its clear linguistic contrast.

Compared to full Transformer fine-tuning, RAMHA trains substantially fewer parameters, resulting in lower memory usage and faster convergence, while inference adds only a modest BiLSTM-attention overhead, preserving near-backbone latency.

5.1. Ablation Study

Table 7 presents the ablation study results, evaluating the contribution of each architectural component in the RAMHA model across three GoEmotions datasets: GoEmotions-1, GoEmotions-2, and GoEmotions-3. Metrics reported include accuracy, precision, recall, and F1-score.

Table 7. Ablation study: effect of architectural components on performance.

RAMHA Model Components	GoEmotions-1				GoEmotions-2				GoEmotions-3
	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score
RoBERTa	0.81	0.79	0.78	0.79	0.80	0.78	0.77	0.77	0.81	0.79	0.78	0.78
RoBERTa +Adapter (Pfeiffer)	0.82	0.80	0.79	0.80	0.80	0.79	0.78	0.79	0.82	0.80	0.79	0.80
RoBERTa+BiLSTM	0.83	0.80	0.80	0.80	0.81	0.78	0.77	0.78	0.83	0.79	0.79	0.80
RoBERTa +Adapter (Pfeiffer) + BiLSTM	0.84	0.82	0.81	0.82	0.83	0.80	0.80	0.80	0.83	0.81	0.80	0.81
RoBERTa +Adapter (Pfeiffer) + BiLSTM + Attention	0.85	0.83	0.81	0.83	0.84	0.82	0.81	0.82	0.85	0.83	0.82	0.83
RAMHA (our Model)	0.88	0.86	0.85	0.85	0.87	0.85	0.84	0.85	0.89	0.87	0.86	0.87

Using only RoBERTa establishes a baseline, achieving accuracies between 0.80–0.81 and F1-scores of 0.77–0.79, demonstrating the effectiveness of contextual embeddings but revealing limitations in sequential modeling and interpretability. Incorporating Pfeiffer Adapters allows task-specific adaptation without retraining the full model, slightly improving performance (accuracy: 0.80–0.82, F1-score: 0.77–0.80).

Adding a BiLSTM layer captures sequential dependencies, yielding minor gains over RoBERTa alone (accuracy: 0.81–0.83, F1-score: 0.78–0.80). The combination of Adapters + BiLSTM integrates task-adaptive embeddings with sequential modeling, further improving all metrics (accuracy: 0.83–0.84, F1-score: 0.80–0.82).

Introducing Attention focuses the model on emotionally salient tokens, enhancing performance consistently across datasets (accuracy: 0.84–0.85, F1-score: 0.82–0.83). Finally, the complete RAMHA model, which combines RoBERTa, adapters, BiLSTM, attention, and focal loss, achieves the highest performance (accuracy: 0.87–0.89, F1-score: 0.85–0.87), demonstrating the complementary contributions of all components.

Overall, the incremental improvements confirm that RAMHA’s hybrid architecture effectively captures complex emotional signals in text, offering a robust and interpretable framework for multi-class mental health classification.

5.2. Comparative Study

Table 8 presents a comparative analysis of recent studies that employed the GoEmotions dataset and other emotion-related corpora. Earlier approaches, such as [52], implemented classical fine-tuning methods, including SBERT with WordNet and KEA-ELECTRA, achieving relatively low F1-scores of 49.00 and 49.60, respectively. Subsequent models, such as BERT and its variants with Masked Language Modeling (MLM) and Curriculum Prompting Design (CPD), as explored in [53,54], yielded modest improvements, raising the F1-score to 51.25–52.34. Further gains were reported in [55], where Seq2Emo, combined with concatenated datasets (SemEval18 + GoEmotions), achieved an F1-score of 59.57, highlighting the advantages of multi-source learning.

Table 8. Comparative study of existing methods based on F1-score.

Study	Year	Dataset	Method	F1-Score
[52]	2022	EmpatheticDialogues, GoEmotions, Affect in Tweets	SBERT FT+WordNet	49.00
[52]	2021	EmpatheticDialogues, GoEmotions, Affect in Tweets	KEA-ELECTRA	49.60
[53]	2024	GoEmotions	BERT	52.00
[54]	2021	GoEmotions	BERT+MLM	51.25
[54]	2021	GoEmotions	MLM+BERT+CDP	51.96
[54]	2021	GoEmotions	BERT+CPD	52.34
[55]	2021	SemEval18+GoEmotions	BERT	58.49
[55]	2021	SemEval18+GoEmotions	Seq2Emo	59.57
[56]	2024	GoEmotions	CNN+BERT+RoBERTa	84.58
RAMHA (Our Proposed model)	2025	GoEmotions	RoBERTa+adapter+BiLSTM	87.00

The CNN+BERT+RoBERTa ensemble presented in [56] demonstrated a substantial performance increase, achieving an F1-score of 84.58, indicating the potential of hybrid Transformer architectures. Building on this trend, our proposed RAMHA model—which integrates RoBERTa, adapter-based fine-tuning, BiLSTM, attention mechanisms, and focal loss—achieves a new state-of-the-art performance with an F1-score of 88.00. Moreover, RAMHA with different class mappings, such as the binary classification of depression versus control, reached a 92% F1-score and 93% accuracy.

The high F1-score of RAMHA can be attributed to a combination of dataset balancing strategies and loss optimization. Given that the original GoEmotions dataset was imbalanced across mental health categories, we applied class-balancing techniques to ensure equitable representation of depression, anxiety, and stress samples. Additionally, integrating focal loss mitigated bias toward over-represented classes and emphasized challenging-to-classify examples, which are prevalent in under-represented categories. RAMHA effectively balances deep contextual embeddings with sequential emotional dynamics, a synergy that baseline models, such as RoBERTa-only or BiLSTM-only, could not achieve, either lacking refined contextual features or failing to capture bidirectional sequential dependencies. This combination of advanced architecture and imbalance handling enabled RAMHA to detect subtle patterns across all classes, resulting in the highest F1-scores reported in Table 8.

Figure 9 (parts a,c,e) depict the classification accuracy over 15 epochs for GoEmotions-1, GoEmotions-2, and GoEmotions-3, respectively, in a three-class system comprising depression, anxiety, and stress. These graphs demonstrate consistent convergence and steadily increasing accuracy, highlighting the robustness of our model. Correspondingly, Figure 9 (parts b,d,f), shows the training and validation loss curves for the same three-class configurations.

Figure 9. Graph of accuracy, training loss, and validation loss of proposed model RAMHA.

Figure 9 part g,h, presents an alternative three-class configuration (depression, stress, and control) without anxiety, where the model maintains high performance with well-aligned loss curves. Finally, Figure 9 part i,j, illustrate the model’s performance in a binary classification scenario (depression vs. stress), demonstrating high precision and smooth loss reduction. These results collectively indicate the efficiency and adaptability of RAMHA, even when applied to simplified or alternative mappings of emotional categories.

Figure 10 (parts a–c), depict the confusion matrices for GoEmotions-1, GoEmotions-2, and GoEmotions-3, respectively, mapped to three classes: depression, anxiety, and stress. These matrices highlight the model’s ability to differentiate closely related emotional states, where high diagonal values indicate a high true positive rate and low inter-class confusion.

Figure 10. Confusion matrices of proposed model RAMHA.

Figure 10 (part d), shows the adjusted three-class mapping (depression, stress, and control), where the exclusion of anxiety leads to clearer separability among classes, demonstrating enhanced differentiation. Figure 10 (parts e,f), examine binary classification scenarios: depression vs. stress and depression vs. control. In these cases, class boundaries are even more pronounced with minimal misclassification, underscoring the model’s robust and discriminative performance in simpler settings. Overall, the confusion matrices confirm that RAMHA achieves strong discriminative capability across both complex and reduced emotional label spaces.

Potential limitations include domain shift when applying RAMHA to different social platforms, languages, or clinical modalities. Ethical considerations include the risk of misclassification and privacy concerns; deployment should be accompanied by human oversight and compliance with data protection regulations.

6. Conclusions and Future Work

This paper introduced RAMHA (RoBERTa with Adapter-based Mental Health Analyzer), a hybrid deep learning framework designed for early detection of mental health disorders through social media text. By integrating RoBERTa with adapter layers, BiLSTM, attention mechanisms, and focal loss, RAMHA demonstrated the ability to capture nuanced emotional expressions while effectively addressing class imbalance. Experimental evaluations on three filtered versions of the GoEmotions dataset showed that RAMHA consistently outperforms traditional machine learning models (SVM, LSTM) and Transformer-based baselines (BERT, RoBERTa), achieving up to 92% accuracy in binary classification and 88% accuracy in multiclass classification. Here, the dataset GoEmotions-3 consistently demonstrated superior performance across all models, reflecting its closer alignment with clinically relevant categories and clearer class boundaries. The differences in performance by the groupings of classes point (Table 6) to the importance of semantic overlap as compared to distinct vocabulary patterns. The simplified or contrasting class arrangements enhanced accuracy even more, and the role of dataset design should not be underestimated. The ablation studies further confirmed the contribution of each architectural component, while comparative analysis highlighted that RAMHA surpasses the best previously reported F1 scores by a significant margin.

Although the results are encouraging, there are several avenues for future research. First, validation of RAMHA on larger and more diverse datasets, including multilingual and cross-cultural corpora, would enhance its generalizability. Second, in the future, the RAMHA model can be extended to incorporate acoustic and image modalities for mental health classification. In summary, RAMHA not only advances the state of the art in emotion-based mental health detection but also lays the groundwork for future developments in AI-driven mental healthcare systems.