Bidirectional Cross-Modal Attention with Adaptive Thresholding for Multimodal Depression Detection Across the EATD-Corpus and DAIC-WOZ Datasets

Abstract

Background: Early and accurate detection of depression from multimodal data is a critical yet challenging task. Many existing models either rely on a single modality or exhibit limited robustness across datasets, while publicly available datasets are often small and highly imbalanced. Methods: In this study, we propose a robust multimodal fusion framework that leverages bidirectional cross-modal attention to effectively integrate audio and text features, allowing the model to learn complementary information from both modalities. To address class imbalance and limited data, we employ SMOTE-based oversampling. Our model further incorporates validation-based adaptive thresholding and probability ensembling to enhance decision robustness and generalization. Experiments are conducted on two benchmark datasets, EATD-Corpus (Chinese; 162 participants) and DAIC-WOZ (English; 189 usable interviews), and ablation experiments compare cross-modal attention settings. Results: On EATD-Corpus, our fusion model achieves strong performance (F1: 0.820, Recall: 0.934, Precision: 0.762), achieving higher mean performance than the text-only and audio-only baselines evaluated under the same participant-level five-fold protocol. On DAIC-WOZ, the framework remains competitive (F1: 0.836, Recall: 0.925, Precision: 0.775). The ablation results confirm that bidirectional cross-modal interaction yields the most reliable trade-off between Precision and Recall. Conclusions: This paper proposes a deep learning framework for depression screening that jointly leverages audio and text modalities, demonstrating robust performance on small, class-imbalanced datasets and architectural generalizability across two corpora with different languages and interview settings.

1. Introduction

Depression is a global mental health issue affecting approximately 280 million people worldwide. It often occurs with other psychiatric conditions, particularly anxiety disorders, and may be accompanied by appetite disturbance, psychomotor changes, fatigue, and cognitive impairment. A severe condition may lead to suicide [1,2,3]. Early identification of depression is therefore of considerable clinical importance.

Currently, depression assessment commonly begins with questionnaire-based screening and is followed by formal clinical interviews conducted by trained professionals. Widely used instruments include the Personal Health Questionnaire (PHQ) [4], the Hamilton Rating Scale for Depression (HAMD) [5], and the Self-Rating Depression Scale (SDS) [6]. These tools assess depressive symptoms through structured questions, and clinicians also consider contextual responses and behavioral signals during interviews. However, conventional assessment remains time-consuming and depends heavily on clinical resources and expert availability. As a result, many recent studies have explored automatic depression detection using publicly available interview-based datasets [7], with the goal of supporting early screening, monitoring, and remote mental health care.

Existing studies can be broadly categorized into unimodal and multimodal approaches. Unimodal methods typically rely on a single source of information, such as text transcripts or speech signals. Although text-based approaches can capture semantic and contextual information, they often fail to represent non-verbal cues such as vocal expression, prosody, and speaking dynamics, which are also important for psychological assessment. Similarly, audio-only systems may capture affective and paralinguistic characteristics but cannot fully exploit the semantic content conveyed in language. These limitations have motivated increasing interest in multimodal methods that integrate complementary sources of information [8]. Among multimodal approaches, audio–text fusion has received growing attention because it jointly models what a speaker says and how it is expressed. Research has shown that when both audio and text are combined to detect depression, the performance of the model improves, especially the F1 score, compared to using a single modality alone [9,10,11,12].

Nevertheless, several challenges remain: (1) publicly available datasets are typically small and exhibit significant class imbalance, which can bias models toward the majority class and hinder model performance; (2) many existing models lack robust calibration and ensembling strategies, resulting in unstable decision boundaries; and (3) prior work often overlooks the importance of rigorous validation and threshold tuning, which are critical for reliable deployment in real-world settings.

To address these problems, we propose a multimodal fusion framework that integrates text and audio representations through bidirectional cross-modal attention followed by an MLP classifier. In addition, to improve robustness in imbalanced data and stabilize the final decision boundary, we adopt imbalance-aware optimization, validation-based adaptive thresholding, and checkpoint probability ensembling, with all model selection procedures conducted strictly on validation data. We further conduct ablation experiments comparing text-conditioned-on-audio attention, audio-conditioned-on-text attention, and bidirectional cross-modal attention, together with a fixed-threshold 0.5 comparator for the bidirectional model. The proposed model is evaluated on Emotional Audio–Textual Depression Corpus (EATD-Corpus) [12], and the architectural generalizability of the same framework is examined through independent, corpus-specific training and evaluation on the Distress Analysis Interview Corpus–Wizard of Oz (DAIC-WOZ) dataset [13].

The specific methodological contributions of this study are threefold. First, in contrast to conventional concatenation-based or one-directional audio–text fusion approaches [9,14,15], the proposed mechanism performs bidirectional cross-modal conditioning: text representations query audio representations to derive audio-aware textual features, while audio representations query text representations to derive text-aware acoustic features. The two direction-specific outputs are then concatenated for joint classification, allowing complementary information from both conditioning directions to be retained. Second, the fusion model integrates validation-based adaptive thresholding with top-k checkpoint probability ensembling to establish a validation-based adaptive threshold instead of relying solely on the conventional fixed threshold of 0.5. Third, controlled ablation experiments compare both unidirectional variants and the bidirectional mechanism, together with validation-based adaptive thresholding and fixed 0.5 decision strategies for the bidirectional model, while independent evaluations on the Chinese EATD-Corpus and English DAIC-WOZ datasets examine the applicability of the same methodological framework across distinct corpus settings.

The remainder of this paper is organized as follows. Section 2 reviews the related work on unimodal and multimodal depression detection. Section 3 presents the materials and methods, including datasets, text and audio representation learning and fusion architecture setup. Section 4 reports and discusses the unimodal, fusion, and cross-corpus architectural evaluation results. Section 5 concludes the paper.

2. Related Work

2.1. Text-Based Unimodal Automated Depression Detection

Recent advances in natural language processing (NLP) have enabled the development of automated depression-detection systems based solely on textual data. Text features are commonly extracted using Word2vec, a static model that converts words into numerical feature vector embeddings [16]. Dynamic pretrained models such as bidirectional encoder representation from transformers (BERT) [17] and embeddings from language models (ELMo) [18] have improved the ability to capture contextual word meanings. BERT is pretrained on a large text corpus using a bidirectional transformer structure, conditioning on both left and right contexts in all layers. Improved variants include RoBERTa, an optimized BERT model trained on longer sequences with larger batches [19], and DistilBERT, a lighter and more efficient version of BERT developed by HuggingFace [20]. The ELMo pretrained model employs a deep bidirectional long short-term memory (BiLSTM) architecture to learn more contextual aspects in the top layer and syntax in the lower layer. Word embeddings and pretrained models are widely used in sentiment analysis and depression detection with text modality. For example, Alghamdi et al. introduced the ArabDep corpus from an Arabic psychological forum and applied lexicon-based and machine learning approaches using features such as Word2vec embeddings and n-grams with classifiers including Random Forest and Support Vector Machine (SVM), achieving promising results with a Precision of 0.79 and an accuracy above 80% [21]. Zhang et al. proposed a multi-instance learning framework for clinical interview transcripts, integrating two pretrained language models (MT5 and RoBERTa) with improved aggregation strategies; evaluated on the DAIC-WOZ and E-DAIC datasets, the model achieved strong performances, with F1 scores of 0.88 and 0.86, respectively [22]. In addition, ref. [9] employed ELMo for text feature extraction on the DAIC-WOZ dataset and achieved the best F1 score compared with BERT-768 and BERT-1024. Nevertheless, depression detection using text modality itself has limited capacity to capture sentiment, which is one of the key pieces of information in depression detection.

2.2. Audio-Based Unimodal Automated Depression Detection

Voice is a reliable biomarker for depression [23]. Mel-Frequency Cepstral Coefficients (MFCC) and Collaborative Voice Analysis repository (COVAREP) are the commonly used tools to extract voice features [24,25]. Recent advancements have introduced pretrained models such as wav2vec2.0, which employs self-supervised learning on large unlabeled datasets to create generalizable representations, enabling feature extraction from raw audio, thereby reducing dependence on handcrafted features and improving generalizability [26]. For instance, Huang et al. employed wav2vec2.0 to extract audio features from the DAIC-WOZ audio dataset. By fine-tuning the training model hyperparameters through systematic experimentation, the model achieved excellent classification performance, highlighting the potential of a pretrained audio model to support clinical depression screening [27]. Lin et al. compared three commonly used audio feature representations, MFCC, COVAREP, and Mel spectrograms, in a classification experiment. Mel spectrograms achieved the highest F1 score of 0.81 and were adopted for feature extraction. A convolutional neural network (CNN) was then applied for depression detection, evaluated on the DAIC WOZ (English) and AViD Corpus (German) datasets. The model achieved strong results, with an F1 score of 0.81 on DAIC-WOZ, outperforming baseline models, and attained MAE of 9.3 and RMSE of 11.55 on the Audio–Visual Depressive Language Corpus (AViD-Corpus) German dataset. These findings demonstrate the generalizability of Mel spectrogram features across languages [9]. Ma et al. introduced DepAudioNet, a deep audio model combining CNN and LSTM layers with Mel-scale filter bank features and random sampling to address data imbalance; evaluated on DAIC WOZ, the system outperformed the SVM baselines and single CNNs, establishing a benchmark for audio-based depression detection [28].

2.3. Multimodal Automated Depression Detection

As a single modality lacks information from one side, recent research on multimodal data has become popular. Xu et al. proposed a multimodal fusion model for automatic depression detection, integrating wav2vec 2.0 for audio feature extraction and BERT for textual representation. It applied varying sizes of convolutional kernels to capture features of both modalities and applies adaptive average pooling to align and compress features into a shared semantic space before concatenating both feature representations; it then passes to a BiLSTM network to model sequential dependencies, outputting the depression classification. This early fusion method was evaluated on the Chinese Multimodal Depression Corpus (CMDC) and the English DAIC-WOZ dataset. The model achieved an F1 score of 0.9708 on CMDC and 0.9666 on DAIC-WOZ, demonstrating robustness across languages and datasets [29]. Lin et al. proposed a multimodal fusion approach for audio and text data. Audio signals are represented as Mel spectrograms and processed by a one-dimensional convolutional neural network (1D CNN), while textual features are extracted using ELMo embeddings and modeled with a BiLSTM network; the outputs of the fully connected layers from both modalities are fused at the decision level. The fusion network was evaluated on the DAIC-WOZ dataset; the model achieved an F1 score of 0.85 for binary classification, and on the AViD-Corpus, it attained a mean absolute error (MAE) of 3.75 for regression, demonstrating the effectiveness of late fusion across tasks and datasets [9]. In [14], the proposed fusion model first processes the audio data and text data separately. For the audio data, it is preprocessed into mel spectrograms and passed through a CNN-BiLSTM model to extract features. The text data are tokenized and then passed through a pretrained BERT-CNN model. By concatenating the two modality features at the end as late fusion, the depression result is produced. The paper confirms that multimodal analysis improves the accuracy compared to using a single modality.

In [30], an adaptive late fusion model was introduced that assigns modality weights based on relative performance. Compared with the standard late fusion, which averages predictions across individual modalities, assigning higher weights to the modality with the higher concordance correlation coefficient (CCC) improves robustness and performance. This model achieves the highest CCC score of 0.7333 on the AVEC 2019 DDS dataset.

Iyortsuun et al. proposed an innovative multimodal attention mechanism for depression detection. Text and audio features are first encoded and fed into a BiLSTM layer, followed by an Additive Cross-Modal Attention network that computes attention weights, reflecting the relative importance of each modality for the final prediction. On the DAIC-WOZ dataset, the model achieved an F1 score of 0.80 and a Precision of 0.76, while evaluation on the EATD-Corpus yielded an F1 score of 0.70 and a Precision of 0.65, demonstrating generalizability across datasets. By excluding interviewer questions and focusing solely on participant responses, the study highlights the superiority of multimodal approaches over unimodal baselines. The authors suggest that future work could address data imbalance using the Synthetic Minority Over sampling Technique (SMOTE) and explore real-world applications to support therapists and psychological professionals without reliance on preset questionnaires [15].

Beyond audio and text modalities, visual and neurophysiological signals have also been investigated for automated depression detection. For example, Valstar et al. introduced the AVEC 2014 benchmark using audio–visual recordings from the AViD-Corpus for depression severity prediction, incorporating facial expressions and vocal cues as indicators of depressive behavior [31]. In addition, Cai et al. presented the MODMA dataset, which contains high-density 128-channel EEG, wearable three-channel EEG, and speech recordings from clinically diagnosed patients with depression and matched healthy controls [32]. The authors further investigated feature-level fusion of complementary EEG representations for depression recognition [33]. These studies indicate that visual and neurophysiological signals can provide complementary information for multimodal depression analysis. Nevertheless, the present study focuses on audio and text because aligned speech and transcript data are available in both EATD-Corpus and DAIC-WOZ, thereby enabling independent, corpus-specific evaluation of the same architectural framework.

Threshold selection has been shown to critically influence classifier performance, particularly in imbalanced settings. For example, Zou et al. proposed a family-specific threshold-tuning strategy that balanced Precision and Recall by adapting thresholds to each family. Their results demonstrated that uniform thresholds (e.g., 0.5) yield poor performance, while adaptive thresholds significantly improve F1 values [34]. Hancock et al. compared threshold optimization techniques for Extreme Gradient Boosting (XGBoost), Random Forest, and logistic regression on the Kaggle Credit Card Fraud dataset. By selecting thresholds that maximize metrics, such as Precision, Recall, or F-measure, and by testing constraints that balance detection rates, they demonstrated that optimized thresholds consistently outperform the default 0.5 [35]. Beyond thresholding, ensemble methods have been explored to stabilize predictions. Checkpoint ensembles average outputs from multiple high-performing epochs, capturing diverse local optima in the parameter space while reducing overfitting and training time [36].

Table 1 organizes representative studies according to their modality representations, fusion strategies, evaluation settings, and reported performance. Existing approaches include direct or decision-level concatenation [9,14], adaptive weighting of modality outputs [30], Additive Cross-Modal Attention [15], and early feature fusion followed by sequential modeling [29].

Table 1. Summary of representative audio–text depression-detection studies and comparison with the proposed framework.

Study	Modalities	Dataset(s)	Feature Representation/ Encoder	Fusion Methodology	Evaluation Setting and Reported Performance
Lin et al. [9]	Audio, text	DAIC-WOZ; AViD	Mel spectrogram–1D CNN; ELMo–BiLSTM	Decision-level late fusion of modality-specific outputs	Corpus-specific evaluation; DAIC-WOZ F1 = 0.85; AViD MAE = 3.75
Park and Moon [14]	Audio, text	Depression interview data	Mel spectrogram–CNN-BiLSTM; BERT–CNN	Comparison of early and late fusion; early fusion achieved better performance	Corpus-specific evaluation; multimodal models outperformed unimodal models
Sun et al. [30]	Audio, video	AVEC 2019 DDS	Modality-specific audio and visual representations	Adaptive weighted late fusion based on modality performance	Corpus-specific evaluation; CCC = 0.7333
Iyortsuun et al. [15]	Audio, text	DAIC-WOZ; EATD-Corpus	Audio and text encoders followed by BiLSTM	Additive Cross-Modal Attention	Corpus-specific evaluation; F1 = 0.80 on DAIC-WOZ and 0.70 on EATD-Corpus
Xu et al. [29]	Audio, text	CMDC; DAIC-WOZ	wav2vec 2.0; BERT; multi-kernel convolution	Early feature fusion with adaptive pooling and BiLSTM	Corpus-specific evaluation; F1 = 0.9708 on CMDC and 0.9666 on DAIC-WOZ
Proposed framework	Audio, text	EATD-Corpus; DAIC-WOZ	MFCC–BiGRU; BERT–BiLSTM	Bidirectional cross-modal attention; adaptive thresholding	Independent participant-level five-fold cross-validation on each corpus; F1 = $0.820\pm 0.108$ on EATD-Corpus and $0.836\pm 0.108$ on DAIC-WOZ

Note: Evaluation protocols, preprocessing procedures, label definitions, and data partitions differ among the cited studies; therefore, reported values provide methodological context rather than a controlled head-to-head comparison. Details not available in the present manuscript should be consulted in the original publications.

3. Materials and Methods

This study introduces an effective, high-performing fusion approach for depression detection that leverages both audio and text data. Initially, unimodal models are trained separately for each modality to obtain specialized checkpoints, which are then used to initialize the corresponding audio and text encoders within the fusion network. The fusion network integrates these modality-specific encoders and employs bidirectional cross-modal attention; the output is then passed to a multilayer perceptron (MLP) classifier to obtain logits. These logits are subsequently converted to probabilities, and validation-based adaptive thresholding is applied to produce the final binary classification. Figure 1 illustrates the overview of the proposed multimodal fusion architecture. This section details the dataset, the construction of unimodal models, and the design of the fusion model.

Figure 1. Overview of the proposed multimodal fusion architecture.

3.1. Dataset

The Emotional Audio–Textual Depression Corpus (EATD-Corpus) is a publicly available Chinese Depression dataset published by Tongji University in 2022, designed to support automatic depression-detection research. A total of 162 students from Tongji University volunteered to complete three randomly selected questions in an interview. The volunteers completed the interview through an app. A virtual interviewer asked three random questions, and the interviewees recorded their responses through audio and uploaded them online. Following the interview, the participants were required to complete an SDS questionnaire, consisting of 20 items, which is commonly used as a screening measure for depressed individuals in clinical settings [6]. In this study, an SDS score greater than or equal to 53 is used as a threshold to classify depression. A total of 30 participants were classified as depressed and 132 were classified as non-depressed. The audio was preprocessed to eliminate mute audio, audio shorter than 1 s, and silent segments at the beginning and end of the recording. The audio was then transcribed into Chinese text and manually checked and corrected by the publisher.

3.2. Text Representation Model Construction

Each participant contributed three transcript files, corresponding to responses to negative, neutral, and positive prompts from the virtual interviewer, along with a label file containing the Self-Rating Depression Scale (SDS) score. Depression labels were binarized using an SDS threshold of 53: scores greater than or equal to 53 were labeled as depressed (1), and scores less than 53 were labeled as non-depressed (0). The text feature extraction module converts the transcripts into fixed-size participant-level semantic tensors. Strict alignment for participants with audio features is enforced, ensuring that each text sample corresponds exactly to the same individual and fold partition, as in audio and fusion training.

We implemented a 5-fold stratified cross-validation procedure on the EATD Corpus (N = 162). In each fold, participants were partitioned into training, validation, and test subsets while preserving the overall class distribution of non-depressed (≈81.5%) and depressed (≈18.5%) individuals. The training sets comprised 103–104 participants, the validation sets comprised 26 participants, and the test sets comprised 32–33 participants, with proportions consistently maintained across folds. The 5-fold cross-validation splits are depicted in Figure 2.

Figure 2. The 5-fold cross-validation splits.

We utilized BERT to extract contextualized text embeddings. Specifically, the “bert-base-chinese” variant from the Hugging Face Transformers library was employed (https://huggingface.co/bert-base-chinese, accessed on 19 July 2026). For each participant, three transcripts were tokenized with truncation and padding, encoded using the transformer, and aggregated via attention-mask-aware mean pooling to produce one embedding per utterance. The resulting three utterance embeddings were stacked to form a participant tensor of size $[3,768]$, yielding a dataset tensor of $[162,3,768]$ ($162\mathrm{participants}\times 3\mathrm{utterances}\times 768\mathrm{features}$). To ensure multimodal consistency, participant order and identifiers were explicitly validated against the audio metadata, and fold index files were synchronized with the audio processing pipeline. The feature extraction process outputs participant features, binary labels, raw scores, and metadata, thereby providing reproducible and alignment-safe inputs for subsequent unimodal and cross-modal depression-detection models.

The text training model functions as a text encoder and provides initialization checkpoints for the fusion model. For benchmarking, its standalone classification performance is also evaluated. Each participant’s three utterance embeddings, obtained from the text feature extraction module, are input as a tensor of shape $[3,768]$. The model architecture comprises a bidirectional long short-term memory (BiLSTM) network followed by an attention mechanism, enabling the capture of both sequential dependencies and the relative importance of each utterance. The three utterance embeddings for each participant are sequentially processed by the BiLSTM, which encodes contextual information across utterances. The hidden states produced by the BiLSTM are aggregated using a trainable attention layer that assigns higher weights to more informative utterances. The resulting participant-level representation is then passed through fully connected layers to yield a binary depression prediction. The architecture of the text model is illustrated in Figure 3.

Figure 3. Text-based bidirectional long short-term memory (BiLSTM) model.

3.3. Audio Representation Model Construction

We selected MFCCs as the primary audio features for depression detection, as they capture spectral changes in speech associated with psychomotor retardation, tighter vocal tract control, and reduced articulation flexibility in depressed patients [37]. Moreover, selecting MFCC features over heavy transformer-based embeddings (e.g., wav2vec 2.0) is grounded in computational efficiency.

Each participant’s interview audio recordings were converted into structured, participant-level acoustic feature tensors, strictly aligned with the text features and fold partitions described previously. For each utterance, 40 MFCC coefficients were extracted per frame using a fast Fourier transform size of 1024 samples and a hop length of 512 samples. First-order delta and second-order delta (delta-delta) matrices were then computed from the MFCC matrix to capture temporal dynamics. For each of the MFCC, delta, and delta-delta matrices, the mean and standard deviation were calculated across the temporal frame dimension, yielding six 40-dimensional vectors. Concatenating these vectors along the feature dimension produced a fixed-length 240-dimensional feature vector ($6\times 40$) for each utterance. The three utterance-level embeddings were stacked to form a participant tensor of shape $[3,240]$, and all participant tensors were aggregated into a dataset tensor of shape $[N,3,240]$, where $N=162$ in the EATD-Corpus. GRUs, comprising part of a gated recurrent neural network, effectively capture long-range dependencies in sequential audio features such as MFCCs. GRUs can achieve accuracy close to LSTMs while reducing training time [38], which is favorable for improving the efficiency of the depression detection task. To model participant-level acoustic patterns, we employed a bidirectional GRU over the three utterance-level MFCC-based embeddings, treating each participant as a short sequence of responses.

We propose a BiGRU architecture for audio-based depression detection training, consisting of two bidirectional GRU layers with 128 hidden units each, followed by an attention layer that aggregates information across the three utterances. Five-fold stratified cross-validation was performed at the participant level, with strict alignment to the text and fusion folds. Similar to the text training, the audio training functioned as an audio encoder and provided initialization checkpoints for the downstream fusion model. The audio-based model architecture is illustrated in Figure 4.

Figure 4. Audio-based bidirectional gated recurrent unit (BiGRU) model.

3.4. Fusion Model Construction

We propose a bidirectional cross-modal attention fusion architecture for multimodal depression detection, wherein both audio and text modalities are mutually conditioned to enhance representation learning. Following the fold partition in the unimodal work, for each participant, paired text and audio data are loaded into a fusion network comprising a text encoder and an audio encoder. Both encoders are initialized with weights from their respective unimodal pretrained checkpoints, ensuring that the model leverages modality-specialized representations shaped by prior unimodal learning rather than random initialization. Unlike approaches that freeze feature extractors, our encoders are fine-tuned end-to-end: all parameters, including those initialized from unimodal checkpoints, are updated during fusion training to optimize for the multimodal objective.

3.4.1. Class Imbalance Mitigation

To mitigate class imbalance, SMOTE [39] is applied to fixed-length participant-level feature representations rather than to raw, variable-length audio or text sequences. For participant i, the three BERT utterance embeddings, ${\mathbf{T}}_i\in {\mathbb{R}}^{3\times 768}$, and three MFCC-based utterance vectors, ${\mathbf{A}}_i\in {\mathbb{R}}^{3\times 240}$, are flattened and concatenated as

$${\mathbf{x}}_i=[\mathrm{vec}\left({\mathbf{T}}_i\right);\mathrm{vec}\left({\mathbf{A}}_i\right)]\in {\mathbb{R}}^{3024}.$$

(1)

Consequently, SMOTE receives a conventional two-dimensional matrix, ${\mathbf{X}}_{\mathrm{train}}\in {\mathbb{R}}^{N_{\mathrm{train}}\times 3024}$, in which each row represents one participant and each column has a fixed feature position. A synthetic minority representation is generated by

$${\tilde{\mathbf{x}}}_i={\mathbf{x}}_i+\lambda ({\mathbf{x}}_j-{\mathbf{x}}_i),\lambda \in [0,1],$$

(2)

where ${\mathbf{x}}_j$ is a minority-class nearest neighbor. Because text and audio coordinates are concatenated before interpolation, the same participant pair and interpolation coefficient are used across both modality blocks. This preserves audio–text pairing within each generated joint vector and avoids independently synthesizing mismatched modalities. After resampling, each 3024-dimensional vector is separated and reshaped to the original text $(3,768)$ and audio $(3,240)$ dimensions before fusion training. Participant-level training, validation, and test partitions are established before oversampling, and SMOTE is fitted and applied exclusively to the training subset of each fold; validation and test participants therefore do not contribute to synthetic sample generation.

Class imbalance is further addressed during optimization through focal loss and dynamically calculated class weights. Validation-based adaptive thresholding then provides a final operating-point adjustment. Within each fold, candidate thresholds from 0.10 to 0.90 in increments of 0.01 are evaluated using validation probabilities. After training, the three checkpoints with the highest validation F1 scores are retained, and their validation probabilities are averaged. The threshold maximizing F1 on these averaged validation probabilities is selected as the validation-based adaptive threshold and then fixed when classifying the corresponding averaged test probabilities. SMOTE fitting and class-weight calculation use training data only; validation data are used for checkpoint ranking and threshold selection, while test labels and probabilities are not used in any model-selection step.

3.4.2. Bidirectional Cross-Modal Fusion and Classification

Modality-specific embedding vectors are fused via a bidirectional cross-modal attention mechanism with two branches. In Branch A, text features acted as queries, while audio features are linearly projected into the text feature space served as keys and values, producing audio-aware text representations. In Branch B, audio features acted as queries with text features as keys and values, yielding text aware audio representations. Attention operated in a shared 128-dimensional space with 8 heads (16 dimensions per head), using scaled dot product attention with dropout (default = $0.5$). The same multihead attention module and learned parameters are reused for both reciprocal query–key–value assignments. Outputs from both branches are concatenated into a fused feature vector. This fused vector is passed to a multilayer perceptron (MLP) consisting of a linear layer, ReLU activation, dropout, and a final linear layer. Softmax is used to obtain positive-class probability for thresholding. During training, both encoder and MLP weights are updated by backpropagation, with per epoch checkpoints saved.

Figure 5 provides a detailed block-level view of the modality encoders, shared-space projection, reciprocal query–key–value assignments, fusion operation, and final classification path.

Figure 5. Detailed block diagram of the fusion model and bidirectional cross-modal attention mechanism. The text and audio encoders each produce a 128-dimensional participant-level representation. The audio representation undergoes a learned $128\to 128$ projection into the common attention space. The same eight-head attention module is applied twice with reciprocal assignments: in Branch A, text queries projected audio to produce audio-aware text features; in Branch B, projected audio queries text to produce text-aware audio features. The two 128-dimensional outputs are concatenated into a 256-dimensional vector and passed through the fusion MLP. Positive-class probabilities from the top three validation-ranked checkpoints are averaged before application of the validation-based adaptive threshold.

For mathematical clarity, let ${\mathbf{h}}_i^T\in {\mathbb{R}}^{128}$ and ${\mathbf{h}}_i^A\in {\mathbb{R}}^{128}$ denote the participant-level text and audio representations produced by the BiLSTM and BiGRU encoders, respectively. Let ${\mathbf{a}}_i={\mathbf{W}}_P{\mathbf{h}}_i^A+{\mathbf{b}}_P\in {\mathbb{R}}^{128}$, with ${\mathbf{W}}_P\in {\mathbb{R}}^{128\times 128}$, denote the audio representation projected into the common attention space. Scaled dot-product attention is defined as

$$\mathrm{Att}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_h}}}\right)\mathbf{V},$$

(3)

where $d_h$ is the dimension of each attention head and $\mathrm{MHA}(·)$ denotes its eight-head extension. The two directional attention outputs and their fusion are therefore expressed as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{h}}}_i^{T\leftarrow A}& =\mathrm{MHA}({\mathbf{h}}_i^T,{\mathbf{a}}_i,{\mathbf{a}}_i),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{h}}}_i^{A\leftarrow T}& =\mathrm{MHA}({\mathbf{a}}_i,{\mathbf{h}}_i^T,{\mathbf{h}}_i^T),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathbf{z}}_i& =[{\tilde{\mathbf{h}}}_i^{T\leftarrow A};{\tilde{\mathbf{h}}}_i^{A\leftarrow T}].\hfill \end{array}$$

(6)

The fused representation, ${\mathbf{z}}_i$, is passed through a two-layer MLP classifier comprising a linear layer, ReLU activation, dropout, and a final linear layer that produces two class logits. A softmax function converts these logits into probabilities for the non-depressed and depressed classes. The positive-class probability is subsequently used for validation-based adaptive thresholding.

At each epoch of a fold, the validation-based adaptive threshold and model checkpoints are recorded. Training continues until early stopping is triggered. After training each fold, the top-k checkpoints ($k=3$), ranked by validation $F1$, are selected to ensemble validation probabilities. A final validation-based adaptive threshold is then selected using only the ensemble validation probabilities and subsequently applied to the ensemble test probabilities for final binary prediction. The model performance is evaluated after all 5 folds are completed. The overall validation-based adaptive thresholding framework is illustrated in Figure 6.

Figure 6. The overall validation-based adaptive thresholding framework.

To analyze the impact of cross-modal attention, we conducted an ablation study on the fusion network by assessing two variants—one with audio-to-text attention and one with text-to-audio attention—while keeping the remainder of the architecture unchanged.

3.5. Evaluation Metrics

The evaluation metrics used in this study are Precision, Recall, and F1 score, calculated as follows:

$$\begin{array}{cc}\hfill \mathrm{Precision}& ={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill \mathrm{Recall}& ={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill F_1& =2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}\hfill \end{array}$$

(9)

Here, TP and TN denote correctly classified depressed and non-depressed participants, respectively, whereas FP denotes non-depressed participants incorrectly classified as depressed and FN denotes depressed participants incorrectly classified as non-depressed.

4. Results and Discussion

This section presents the performance of both unimodal and fusion models. Their results are compared against recent baselines in the literature. For all models, the mean values of F1 score, Recall, and Precision from stratified 5-fold cross-validation are reported. An ablation experiment is also included to quantify the impact of different cross-modal attention settings. To examine architectural generalizability, the same fusion architecture and methodological pipeline are independently trained and evaluated on DAIC-WOZ using corpus-specific preprocessing. This constitutes a cross-corpus architectural evaluation rather than direct transfer of an EATD-Corpus-trained model. Because external baselines are taken from prior studies that may use different preprocessing pipelines, data splits, and evaluation protocols, cross-study comparisons are interpreted as indicative rather than strictly controlled.

4.1. Text Model Performance

The text-based depression-detection model was implemented as a BiLSTM network with attention, trained on participant-level embeddings derived from the pretrained “bert-base-chinese” model. Evaluation was performed using five-fold participant-level cross-validation. Across the five cross-validation folds, the model achieved an F1 score of $0.590\pm 0.005$, reflecting moderate consistency in balancing Precision and Recall. Recall was high at $0.850\pm 0.034$, indicating strong sensitivity but with noticeable variability across folds. Precision remained low at $0.456\pm 0.001$, suggesting stable yet consistently poor performance in avoiding false positives. The five-fold checkpoints were retained for downstream fusion model initialization. Table 2 compares the proposed model’s performance with recent studies on similar tasks. While the proposed model’s F1 score 0.590 is comparable to prior work, its Recall 0.850 is notably higher, supporting its suitability for screening applications where sensitivity is critical. Table 3 outlines the text-based BiLSTM model configuration.

Table 2. Comparison of the proposed text-based bidirectional long short-term memory (BiLSTM) model with recent studies.

Method	F1 Score	Recall	Precision
T. Al Hanai et al. [10]	0.57	0.63	0.53
N. K. Iyortsuun et al. [15]	0.66	0.58	0.79
Y. Shen et al. [12]	0.65	0.66	0.65
Proposed BiLSTM model	$0.590\pm 0.005$	$0.850\pm 0.034$	$0.456\pm 0.001$

Note: The proposed BiLSTM model is reported as mean ± standard deviation across five folds. The results of the comparison methods are collected from the original papers.

Table 3. Text-based BiLSTM model configuration.

Component	Configuration/Value
BiLSTM	Hidden units: 128; Layers: 2
Attention Layer	Single trainable attention over sequence outputs
Dropout	0.5
Fully Connected Layers	FC1: 256 → 128; ReLU; Dropout(0.5); FC2: 128 → 2 (Softmax)
Loss Function	Cross-Entropy Loss
Optimizer	AdamW
Learning Rate	$1.0\times {10}^{-5}$
Batch Size	8
Early Stopping	Patience = 10 (on validation F1)

4.2. Audio Model Performance

The audio-based depression-detection model was implemented as a BiGRU with attention and evaluated using five-fold participant-level cross-validation. The audio-based model demonstrated stronger overall performance, with a mean F1 score of $0.662\pm 0.046$, a Recall of $0.781\pm 0.010$, and Precision of $0.612\pm 0.056$, as compared to the text model. The audio model achieved a better balance between Precision and Recall, with Recall remaining robust and Precision moderately improved. The audio performance exhibits higher variance as compared with the text model, suggesting that the audio carries complementary information but is less consistent on its own; this provides justification for selecting text as the common-space bidirectional cross-modal attention. The five-fold checkpoints are saved for downstream fusion model use as audio encoder initialization. Table 4 compares the proposed model’s performance with those in recent studies. The proposed model achieves competitive F1 and Recall. Table 5 outlines the audio-based BiGRU model configuration.

Table 4. Comparison of the proposed audio-based BiGRU model with recent studies.

Method	F1 Score	Recall	Precision
T. Al Hanai et al. [10]	0.49	0.56	0.44
N. K. Iyortsuun et al. [15]	0.65	0.60	0.70
Y. Shen et al. [12]	0.66	0.78	0.57
Proposed BiGRU model	$0.662\pm 0.046$	$0.781\pm 0.010$	$0.612\pm 0.056$

Note: The proposed BiGRU model is reported as mean ± standard deviation across five folds. The results of the comparison methods are collected from the original papers.

Table 5. Audio-based bidirectional gated recurrent unit (BiGRU) model configuration.

Component	Configuration/Value
BiGRU	Hidden units: 128; Layers: 2
Attention Layer	Single trainable attention over sequence outputs
Dropout	0.6
Fully Connected Layers	FC1: 256 → 128; ReLU; Dropout(0.6); FC2: 128 → 2 (Softmax)
Loss Function	Cross-Entropy Loss
Optimizer	AdamW
Learning Rate	$1.0\times {10}^{-4}$
Batch Size	8
Early Stopping	Patience = 10 (on validation F1)

4.3. Fusion Model Performance

Participant-level five-fold cross-validation is strictly maintained, with train, validation, test splits preserved within each fold. For each fold, we computed Precision, Recall, and F1 score from the test predictions, and then summarized the results as mean plus/minus standard deviation across the five folds. The model achieved a mean Precision of $0.762\pm 0.195$, mean Recall of $0.934\pm 0.078$, and mean F1 score of $0.820\pm 0.108$, demonstrating competitive performance compared with recent studies. The metrics are listed in Table 6. Relative to the unimodal models in this study, fusion shows a higher F1, Recall, and Precision because bidirectional cross-modal attention jointly models text and audio and learns interactions between modalities in both directions. The larger standard deviations are consistent with a more complex fusion model that is more sensitive to fold composition, class imbalance, and validation-based adaptive thresholding. Figure 7 visualizes this comparison across text, audio, and fusion models. The fusion model configuration is summarized in Table 7.

Table 6. Comparison of the proposed fusion model performance with recent studies.

Method	F1 Score	Recall	Precision
T. Al Hanai et al. [10]	0.57	0.67	0.49
N. K. Iyortsuun et al. [15]	0.70	0.70	0.65
Y. Shen et al. [12]	0.71	0.84	0.62
Proposed fusion model	$0.820\pm 0.108$	$0.934\pm 0.078$	$0.762\pm 0.195$

Note: The proposed fusion model is reported as mean ± standard deviation across five folds. The results of the comparison methods are collected from the original papers.

Figure 7. Comparison of text, audio, and fusion model performance in terms of F1 score, Recall, and Precision (mean ± SD).

Table 7. Fusion architecture and training configuration.

Component	Configuration/Value
Fusion Architecture	BiLSTM (Text); BiGRU (Audio); Bidirectional Cross-Modal Attention; MLP
Text Encoder	BiLSTM
Audio Encoder	BiGRU
Attention Mechanism	Multihead Attention (8 heads)
Fusion MLP	FC1: 256 → 256; ReLU; Dropout(0.5); FC2: 256 → 2
Normalization	LayerNorm on audio and text input
Loss Function	Focal Loss
Optimizer	Adam
Batch Size	4
Early Stopping	Yes
Thresholding	Validation-based adaptive thresholding (F1 objective)
Ensemble	Top-3 validation probability ensemble
Data Augmentation	SMOTE (training set only, per fold)
End-to-End Fine-Tuning	Yes (all parameters trainable)

Fold-specific test results and validation-based adaptive thresholds are reported in Supplementary Table S1. Fold 1 selected the lowest threshold (0.10) and achieved perfect Recall (1.000), but its Precision (0.485) and F1 score (0.653) were the lowest among the five folds, indicating a sensitivity-oriented operating point with comparatively more false-positive predictions. All five validation subsets contained 21 non-depressed and 5 depressed participants, so unusual class prevalence does not explain the Fold 1 result. Instead, the boundary value suggests fold-specific probability-separation and threshold-selection instability, which may be amplified by the small validation subset. Accordingly, the fold-specific thresholds should be interpreted as validation-derived model operating points rather than clinically meaningful cutoffs. Fold 2 also achieved Recall of 1.000; however, its higher threshold of 0.62 improved Precision to 0.630 and F1 to 0.773, showing better control of false positives while retaining all positive cases. Folds 3–5 selected similar thresholds (0.64–0.68) and exhibited a more balanced relationship between Precision and Recall, with Precision ranging from 0.864 to 0.929, Recall from 0.813 to 0.950, and F1 from 0.867 to 0.905. Overall, Recall remained consistently high, whereas Precision and F1 were more sensitive to fold composition and the validation-based adaptive threshold.

4.4. Ablation Study

To analyze the impact of cross-modal attention, we evaluated fusion strategies using unidirectional cross-modal attention. For the single-modality cross-modal attention variants, either the text modality was conditioned on audio features or the audio modality was conditioned on text features. Unidirectional cross-modal attention with text conditioned on audio resulted in the highest Recall, $0.988\pm 0.026$, but a marked decrease in Precision, $0.556\pm 0.107$, yielding the lowest F1 score, $0.707\pm 0.089$, and indicating a propensity for over-prediction of the positive class. Conversely, conditioning audio on text improved the balance between Precision and Recall (F1: $0.731\pm 0.141$, Recall: $0.781\pm 0.250$, Precision: $0.804\pm 0.253$), suggesting that textual cues more effectively guide salient audio representations. As a decision-threshold comparator, the bidirectional model was also evaluated using the conventional fixed threshold of 0.5, without validation-based adaptive thresholding. This variant achieved an F1 score of $0.659\pm 0.229$, Recall of $0.779\pm 0.091$, and Precision of $0.602\pm 0.272$. The proposed bidirectional cross-modal attention model achieved the highest overall performance (F1: $0.820\pm 0.108$, Recall: $0.934\pm 0.078$, Precision: $0.762\pm 0.195$), providing a modest but consistent improvement over the strongest unidirectional variant. Relative to the fixed-threshold variant, validation-based adaptive thresholding increased the mean F1 by 0.161, the Recall by 0.155, and the Precision by 0.160. The fixed-threshold result also exhibited greater fold-to-fold variability in F1 and Precision, whereas the validation-based adaptive thresholding strategy provided a descriptively more consistent balance between sensitivity and positive predictive performance. This comparison evaluates the complete decision strategies because checkpoint ranking is based on validation F1 under the corresponding threshold procedure. Figure 8 visualizes the experiment’s results, and the numerical results are summarized in Table 8.

Figure 8. Ablation study comparison of unidirectional and bidirectional cross-modal attention, including fixed 0.5 and validation-based adaptive thresholds for the bidirectional model.

Table 8. Ablation experimental results.

Method	F1 Score	Recall	Precision
Cross-modal attention using text conditioned on audio	$0.707\pm 0.089$	$0.988\pm 0.026$	$0.556\pm 0.107$
Cross-modal attention using audio conditioned on text	$0.731\pm 0.141$	$0.781\pm 0.250$	$0.804\pm 0.253$
Bidirectional Cross-Modal Attention with fixed threshold 0.5	$0.659\pm 0.229$	$0.779\pm 0.091$	$0.602\pm 0.272$
Proposed Bidirectional Cross-Modal Attention	$0.820\pm 0.108$	$0.934\pm 0.078$	$0.762\pm 0.195$

We also conducted a controlled no-SMOTE ablation while retaining the same folds, random seeds, architecture, encoder initialization, loss and class-weighting procedure, optimization, checkpoint ensembling, and validation-based adaptive threshold selection. Without SMOTE, the model achieved an F1 of $0.275\pm 0.120$, a Recall of $0.950\pm 0.112$, and a Precision of $0.164\pm 0.079$, compared with $0.820\pm 0.108$, $0.934\pm 0.078$, and $0.762\pm 0.195$, respectively, with SMOTE (Supplementary Table S2). Thus, removing SMOTE retained high Recall but substantially increased false-positive predictions. This result supports the empirical utility of SMOTE within the evaluated pipeline, but it does not establish that interpolated representations are semantically or clinically realistic.

4.5. Cross-Corpus Architectural Evaluation on DAIC-WOZ

The DAIC-WOZ depression database is a public depression dataset hosted by the USC Institute for Creative Technologies, comprising participants from the public. The participants first completed a set of questionnaires alone on a computer and then went through an interview conducted by an animated virtual interviewer, Ellie, who was controlled by a human interviewer in another room. All interviewers were fluent English speakers, and the interviews were conducted in English. There are a total of 193 interviews in the Wizard of Oz session, all recorded by a camera and a high-quality close-talking microphone, and transcribed in CSV format, with start time, end time, and speaker for each question-and-answer conversation. Of the participants, 60 were identified as distressed, and 133 were not identified to be in distress. After excluding four unavailable depressed participant interviews, the evaluated sample comprised 189 participants (133 non-depressed and 56 depressed). Participants from the official training, development, and test partitions were pooled and repartitioned using participant-level stratified five-fold cross-validation. Due to privacy constraints, it can be distributed only upon request and only to academics and other non-profit researchers.

In line with our research framework, both audio and text data were used for the fusion method. Unimodal training was conducted to obtain encoder checkpoints for initializing the fusion network, with a BiLSTM model for text and a BiGRU model for audio. The fusion network employed the same cross-modal attention mechanism, followed by an MLP classifier, validation-based adaptive thresholding, and end-to-end training. Model performance was evaluated using five-fold cross-validation, consistent with the EATD-Corpus protocol.

Across the five stratified folds, the training, validation, and held-out test subsets contained 120–121, 31, and 37–38 participants, respectively, with non-depressed/depressed counts of 84–85/35–36, 22/9, and 26–27/11–12. The complete fold-specific distributions are reported in Supplementary Table S3. The evaluated DAIC-WOZ sample had a higher proportion of depressed participants than EATD-Corpus (29.6% versus 18.5%). Accordingly, performance is reported separately for each corpus to facilitate a comparison.

The model achieves an F1 score of $0.836\pm 0.108$, a Recall of $0.925\pm 0.106$, and a Precision of $0.775\pm 0.133$. Its competitive performance on DAIC-WOZ, following independent corpus-specific training, supports the applicability of the proposed architecture across two different corpora. Table 9 compares the proposed model’s performance with recent studies.

Table 9. The evaluation performance of the proposed model on DAIC-WOZ.

Method	F1 Score	Recall	Precision
T. Al Hanai et al. [10]	0.77	0.71	0.83
N. K. Iyortsuun et al. [15]	0.82	0.86	0.79
Y. Shen et al. [12]	0.85	0.92	0.79
Proposed fusion model	$0.836\pm 0.108$	$0.925\pm 0.106$	$0.775\pm 0.133$

Note: The proposed fusion model is reported as mean ± standard deviation across five folds. The results of the comparison methods are collected from the original papers.

Figure 9 provides a compact visual summary of the cross-corpus architectural evaluation. The plot shows that the proposed fusion model maintains consistently strong Recall and Precision on both EATD-Corpus and DAIC-WOZ, with only a modest change in F1 score across the two datasets, supporting consistent applicability of the independently trained architecture across different languages and corpus settings.

Figure 9. Cross-corpus architectural evaluation of the proposed fusion model on EATD-Corpus and DAIC-WOZ.

4.6. Limitations

The findings reveal several limitations. First, fusion performance varied substantially across folds, with F1 scores ranging from 0.653 to 0.905 and Precision ranging from 0.485 to 0.929. Fold 1 required a threshold of 0.10 and traded low Precision for perfect Recall, whereas Folds 3–5 produced more balanced results. Because the EATD-Corpus contains only 30 depressed participants and each validation subset contains five positive cases, a small number of predictions can materially affect validation F1 and threshold selection. The reported mean performance therefore does not establish a stable clinical operating point.

Second, the no-SMOTE ablation retained high Recall ($0.950\pm 0.112$) but reduced Precision to $0.164\pm 0.079$ and F1 to $0.275\pm 0.120$. Thus, the reported Precision–Recall balance depends strongly on synthetic oversampling within the evaluated pipeline. However, interpolation between minority participant representations does not establish that the generated samples represent clinically realistic variation.

Third, validation-based adaptive thresholding outperformed the fixed 0.5 comparator, but more generally, fold-specific adaptive thresholds may be unstable when estimated from small validation subsets with limited probability separation.

5. Conclusions

The experimental results show that, on EATD-Corpus, the proposed bidirectional fusion model achieved an F1 score of $0.820\pm 0.108$, compared with $0.590\pm 0.005$ for the text-only model and $0.662\pm 0.046$ for the audio-only model. The ablation study further showed higher F1 for bidirectional fusion than for text-conditioned-on-audio ($0.707\pm 0.089$) and audio-conditioned-on-text ($0.731\pm 0.141$) attention, supporting the contribution of bidirectional cross-modal conditioning under the evaluated protocol. Validation-based adaptive thresholds were selected independently within each fold and applied to the corresponding test predictions, thereby defining the operating points used for the reported results without using test labels. Compared with the bidirectional fixed-threshold control (F1: $0.659\pm 0.229$; Recall: $0.779\pm 0.091$; Precision: $0.602\pm 0.272$), the validation-based adaptive thresholding strategy achieved higher mean F1, Recall, and Precision and lower fold-to-fold variability in F1 and Precision. When the same architectural and training framework was trained independently on DAIC-WOZ, it achieved an F1 score of $0.836\pm 0.108$, a Recall of $0.925\pm 0.106$, and a Precision of $0.775\pm 0.133$. Collectively, these findings support the applicability of the framework under corpus-specific training in two datasets.