Evaluating the Performance of eGeMAPS Features in Detecting Depression Using Resampling Methods

Abstract

This paper investigates how well eGeMAPS features can be used to classify depression from a patient’s speech audio samples through the use of statistical resampling methods. We use permutation tests to evaluate, with high confidence, whether eGeMAPS features and the speaker’s depression status are dependent. We use bootstrap confidence intervals to test, with high confidence, whether eGeMAPS features are able to better discriminate depression in male speakers than in female speakers. Lastly, we compare the detection power of different subsets of the eGeMAPS features. We use an open-source dataset of depressed and non-depressed speakers (E-DAIC), an open-source audio feature extractor (eGeMAPS), and open-source machine learning classifiers (WEKA) to enable replication of results and establish a baseline for future studies.

1. Introduction

Major depression is one of the most prevalent mental disorders in the United States [1], underscoring the urgent need for effective approaches in understanding and treating this debilitating condition. However, it could take a clinician several meetings with a patient to diagnose them with major depression as they familiarize themselves with the patient’s behaviors. If a tool existed to aid in the detection of depression, clinicians could reference it to help make their decision.

One way to aid in the detection of depression is to use machine learning to analyze a patient’s speech. The general approach is to observe the acoustic parameters of the patient’s voice and notify the clinician if certain features were in ranges that indicate depression. As described in Literature Review Section, previous researchers have considered using audio features and machine learning to detect depressed patients; however, the results are inconclusive, often reporting conflicting classification performance. Also, analyses are often based on training datasets that are not large enough, raising the question of whether classification models were overfitting the training set. Furthermore, often the results are based on proprietary datasets or use classification tools with a high number of hyperparameters, making it difficult to replicate the results.

The goal of our paper is to further evaluate the detection performance of audio features in detecting depression while using open-source datasets and tools to enable replication of results. We evaluate the depression detection performance of several machine learning methods using (1) an open database (E-DAIC) of depressed and non-depressed speakers; (2) the open-source OpenSMILE tool; (3) a well-defined set of audio features (eGeMAPS), which are readily available in the OpenSMILE distribution; and (4) the open-source WEKA machine learning tool. To facilitate the replication of our results and given the limited availability of training data, we focus on traditional machine learning methods as opposed to deep learning methods. Details about E-DAIC, OpenSMILE, eGeMAPS, and WEKA are described in Section 2.1, Section 2.2, Preparation of Audio Files from Speakers Section, and Section 2.3, respectively.

We start by addressing a fundamental question that has not been properly addressed before: can we state, with high confidence, that audio features and depression status are dependent on each other? We answer this question by using resampling methods. More precisely, we use permutation tests to test the hypothesis.

${\displaystyle H_0^{\left(I\right)}}$

: eGeMAPS audio features of speakers are independent of their depression status.

Rejecting ${\displaystyle H_0^{\left(I\right)}}$ provides evidence that, with high confidence, eGeMAPS features may be correlated with depression. While permutation tests have been considered in the past to evaluate the performance of machine learning algorithms [2], this is the first study that applies permutation tests to evaluate the efficacy of audio features in detecting depression. We further use bootstrapping to provide confidence intervals for the probability of decision errors, to compare the discrimination power of eGeMAPS features in detecting depression in both male and female speakers, and to compare the discrimination power of various subsets of eGeMAPS features.

The main contributions of this paper are as follows:

• We show that, at least in the speaker set from E-DAIC, hypothesis ${\displaystyle H_0^{\left(I\right)}}$ can be rejected with 95% confidence in many instances, meaning a strong indication that eGeMAPS audio features and speakers’ depression status are in general dependent.
• Our results further reinforce previous results obtained in the literature by providing 95% bootstrap confidence intervals for the probability of decision errors when using eGeMAPS features.
• We show that eGeMAPS features are able to better classify depression in male speakers than in female speakers.
• We show that eGeMAPS temporal features provide higher discrimination power when detecting depression in females, while eGeMAPS energy features provide higher discrimination power when detecting depression in males.

Literature Review

Depression detection from speech is a well-established concept, as individuals suffering from major depression exhibit speech characteristics described as dull, monotone, monoloud, and lifeless [3]. These characteristics manifest in lowered pitch, reduced pitch range, slower speaking rates, and increased articulation errors [4]. Various acoustic features have been employed to detect these speech alterations, including fundamental frequency (F0), formants, jitter, shimmer, intensity, and speech rate [3,5,6]. While some studies incorporate multimodal approaches that integrate audio, video, and textual features for depression detection, this paper focuses exclusively on audio-based analysis.

Several studies have investigated depression detection using traditional machine learning approaches. One study explored monolingual and multilingual depression detection by analyzing vowel-based features across three languages: German, Hungarian, and Italian. A Support Vector Machine (SVM) classifier was employed for both binary and multi-class classification tasks, with results indicating a 26% improvement in classification performance when accounting for gender differences [7]. Another study used Gaussian Mixture Models to model depressed and neutral speech, finding that the first formant (F1) was the most effective single feature, despite prior research suggesting that the second formant (F2) may better capture emotional and cognitive influences linked to depression [8]. Additionally, research on speech landmark features examined acoustic event transitions such as vocal cord vibrations, nasality, frication, and speech bursts. These features, converted into n-gram representations and used with an SVM classifier, achieved an F1 score of 0.86 on the E-DAIC dataset [9].

Although deep learning methods generally require larger datasets than traditional learning methods [10], some studies have employed convolutional neural networks and Long Short-Term Memory networks to classify depression. The authors in [11,12] reported moderate detection performance, with F1 scores ranging between 0.52 and 0.63. Another investigation [13] utilized a deep convolutional neural network model with over 4.5 million parameters, reporting balanced accuracy scores ranging between 0.54 and 0.60. Using a proprietary dataset containing over 84 h of speech data from 4748 participants, the authors in [14] compared various machine learning techniques, including deep learning models, and reported unweighted average recall (UAR) values ranging between 0.56 and 0.63. These studies emphasize the difficulty of accurately detecting depression using only audio-based features. On the other hand, using Long Short-Term Memory networks, the authors in [15] used a new corpus of 118 native Italian speakers and reported F1 scores of 0.83.

The findings across these studies present both promising and inconsistent outcomes. While some research reports high classification accuracy [7,9,15,16], others highlight limitations in audio-based depression detection [11,13,14]. Furthermore, gender has been identified as a key factor influencing classification performance [5,7,8,11,16]. Acoustic features such as prosodic and glottal characteristics, F0, formants, Mel-Frequency Cepstral Coefficients (MFCCs), shimmer, and jitter have consistently proven effective in detecting depressed speech [3,5,6,17,18,19]. Several studies have also raised concerns regarding the small sample sizes used in many investigations, which may affect the generalizability of their results [16,17].

The E-DAIC dataset is widely used in depression detection research [4,9,11,12,13]; however, reported classification performance is inconclusive. While some have reported strong classification performance [9], others have reported significantly lower classification performance metrics on this dataset [11,12,13].

OpenSMILE is a widely used tool for extracting speech features, and the included eGeMAPS feature set has been frequently applied in depression detection. eGeMAPS includes low-level descriptors such as shimmer, jitter, and formants, which have been identified as relevant indicators of depressive speech patterns. However, studies show inconsistent effectiveness, with some demonstrating strong classification performance [16], while others highlight limitations in predictive capabilities in both traditional and deep machine learning models [13,14].

Although the studies reviewed do not explicitly mention WEKA, many of their approaches could be implemented using this framework, as it provides the same machine learning algorithms mentioned above. WEKA offers a comprehensive collection of classification, regression, and feature selection methods, making it a valuable tool for reproducible research in depression detection.

2. Materials and Methods

We evaluate the depression detection performance of several machine learning methods using the E-DAIC dataset of depressed and non-depressed speakers, the OpenSMILE tool, and the WEKA machine learning tool. In this section, we provide more details about the dataset and tools and how they were used.

2.1. The E-DAIC Dataset

The Extended Distress Analysis Interview Corpus (E-DAIC) is a dataset created by the University of Southern California containing 275 speakers. These speakers are divided into training, development, and test sets. The training set contains 163 speakers, the development set contains 56 speakers, and the test set contains the remaining 56 speakers. E-DAIC has been used as a benchmark in many evaluation efforts [4].

When creating the dataset, each speaker was assigned an identifier and was interviewed by a virtual assistant. The interview questions aimed at diagnosing the speaker for psychological conditions such as depression. For this, the designers of E-DAIC used the Eight-Item Patient Health Questionnaire (PHQ-8) [20], which is a commonly used tool to indicate the level of depression in a subject. Depending on the answers given, each speaker was given a score ranging from 0 to 24. From this score, E-DAIC labels the speaker as depressed or non-depressed: speakers with scores 10 or below are considered non-depressed; otherwise, the speaker is considered to be depressed. More details about E-DAIC can be found in [21,22].

We highlight that our research uses the E-DAIC dataset by considering the whole set of 275 speakers to build various training and test subsets. This was done to better create balanced sets and avoid bias in the training of the classifier [12]; i.e., to control the number of depressed and non-depressed speakers as well as the number of male and female speakers during different classification trials. Another motivation for this was to enable the creation of several balanced sets using different E-DAIC IDs each time.

Preparation of Audio Files from Speakers

For each speaker, the E-DAIC database contains a single audio WAV file. The WAV file contains audio from both the interviewer and the speaker during a back-and-forth interview process. Thus, before extracting audio features from the WAV file, it is important to remove the audio from the interviewer. For this, we used E-DAIC’s transcript files, which indicate the time periods containing the speaker’s audio samples. Using such transcripts, we extracted the audio samples from the WAV file from only the periods of time in which the speaker was speaking.

For each time period with a speaker’s speech, we partitioned it into nonoverlapping segments of equal size $\left(T_{UL}\right)$. Time periods shorter than $T_{UL}$ and any remaining portions of the time period that did not fit into a full segment of $T_{UL}$ duration were discarded.

2.2. OpenSMILE

OpenSMILE is a free and open-source software designed by researchers from the Technical University of Munich [23]. OpenSMILE extracts various features from an audio stream and it has been used by many researchers in classification tasks involving audio features [5]. In fact, it has been used in two of the Audio/Visual Emotion Challenges (AVEC2017 and AVEC2019) [4,24].

OpenSMILE works by first creating a configuration file specifying the desired audio features to be extracted from a given input audio file. OpenSMILE offers several built-in configuration files. In this paper, we consider the eGeMAPS configuration file and feature set specified, present in OpenSMILE distribution under the name ‘eGeMAPSv02’.

The eGeMAPS Features Set

In order to standardize the set of OpenSMILE parameters to better assess studies that extract key elements of vocal features from speakers, the Geneva Emotion Research Group created the Extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [25].

The number of features present in eGeMAPS is 88. These features cover energy, temporal, frequency, and spectral characteristics and aim at extracting the changes in speech during emotional expression. The Geneva Emotion Research Group selected those features based on its reported efficacy, theoretical importance, and historical usage [25]. The 88 features are obtained from statistical functionals of 25 low-level descriptors (LLDs). To produce the functionals, LLDs features are first smoothed using a 3-frame moving average filter. We highlight that only voiced regions are used to smooth functionals related to pitch, jitter, and shimmer. Additional functionals such as percentiles and slope statistics are also calculated within voiced regions for loudness and pitch. The only exception are loudness functionals, which are calculated for all regions. Certain functionals, such as arithmetic means and variation coefficients, are extracted from both voiced and unvoiced segments. Arithmetic means and variation coefficients are also extracted from features related to formant bandwidths; however, such functionals only consider features from voiced regions. Further features and functionals included in eGeMAPS are the spectral flux and MFCC 1–4 coefficients’ arithmetic mean and variation coefficient in voiced regions; and the spectral flux arithmetic mean in unvoiced regions. Detailed descriptions of all the eGeMAPS features can be found in [25].

Each segment of $T_{UL}$ duration is treated with OpenSMILE using the eGeMAPSv02 configuration file, which produces an ARFF output file for posterior processing by WEKA. It should be mentioned that OpenSMILE may output features of 0.0 value for some attributes; for instance, when voiced segments are not found in a segment. To avoid such segments impacting the analysis, any audio segment producing an ARFF output file with 10 or more consecutive features of 0.0value was removed from the analysis.

2.3. WEKA

The Waikato Environment for Knowledge Analysis (WEKA) is a free software tool developed by researchers at the University of Waikato in New Zealand. WEKA offers a large collection of machine learning algorithms for data mining and classification tasks [26]. It has been used by researchers and practitioners to preprocess and visualize data and to evaluate the effectiveness of machine learning tools. The version of WEKA used in this research is 3.8.6. The classifiers that we focus on in this study are: Support Vector Machines (SVM), Decision Trees, and Naive Bayes.

For the SVM classifier, WEKA implements the Sequential Minimal Optimization (SMO) algorithm described in [27]. In our results, we considered various values for the complexity constant (C). All other WEKA SMO parameters were set at their default level, including standard normalization and Polynomial kernel with exponent 1.0 (WEKA command for $C=0.1$: weka.classifiers.functions.SMO -C 0.1 -L 0.001 -P 1.0E-12 -N 1 -V -1 -W 1 -K “weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 250007” -calibrator “weka.classifiers.functions.Logistic -R 1.0E-8 -M -1 -num-decimal-places 4”.). We highlight that no internal cross-validation was used. This is important because a speaker may have multiple audio segments producing multiple (features/label) entries in the training set, and any internal cross-validation would bias the classifier.

For the Naive Bayes classifier, we used the “Flexible Naive Bayes” classifier proposed by [28], which is implemented in WEKA. This variant uses kernels to estimate the distributions for each feature (WEKA command: weka.classifiers.bayes.NaiveBayes -K.).

For the Decision Tree classifier, we used WEKA’s J.48 classifier, which is an implementation of the C4.5 Decision Tree algorithm proposed in [29]. In our results, we considered various confidence thresholds for prunning (C) and various minimum numbers of instances per leaf (M). All other WEKA J.48 parameters were set at their default level (WEKA command for $C=0.25$ and $M=15$: weka.classifiers.trees.J48 -C 0.25 -M 15.).

2.4. Resampling Methods

In this paper, we use two resampling methods to evaluate eGeMAPS effectiveness in classifying depression: permutation tests and bootstrapping.

2.4.1. Permutation Tests

Let ${\displaystyle {\left\{x_b\right\}}_{b=1}^B}$ be realizations of B i.i.d. random variables ${\displaystyle {\left\{X_b\right\}}_{b=1}^B}$; let ${\displaystyle {\left\{y_b\right\}}_{b=1}^B}$ be realizations of B i.i.d. random variables ${\displaystyle {\left\{Y_b\right\}}_{b=1}^B}$; and assume one observes $1/B{\displaystyle {\sum }_{b=1}^B}{y}_b>1/B{\displaystyle {\sum }_{b=1}^B}{x}_b$, which could suggest that ${\displaystyle {\left\{Y_b\right\}}_{b=1}^B}$ has a different distribution than ${\displaystyle {\left\{X_b\right\}}_{b=1}^B}$.

The permutation test can be used to formally test the hypothesis $H_0:$${\displaystyle {\left\{X_b\right\}}_{b=1}^B}$ and ${\displaystyle {\left\{Y_b\right\}}_{b=1}^B}$ form a set of $2B$ i.i.d. random variables. For this, let ${\mathbf{z}}_0:=[x_1,\dots ,x_B,y_1,\dots ,y_B]$ and compute a statistic from the last B values of ${\mathbf{z}}_0$, which covers only the realizations of ${\displaystyle {\left\{Y_b\right\}}_{b=1}^B}$; for instance, let $S_0:=1/B{\displaystyle {\sum }_{b=B+1}^{2B}}{z}_b$. Subsequently, compute the same statistic from multiple $N_{perm}$ random permutations of ${\mathbf{z}}_0$. Let ${\mathbf{z}}_n$ be the $nth$ permutation, and let $S_n:=1/B{\displaystyle {\sum }_{b=B+1}^{2B}}{\mathbf{z}}_n$.

The p-value for testing $H_0$ is then estimated with

$$\widehat{p}:={\displaystyle \frac{1+{\displaystyle {\sum }_{n=1}^{N_{perm}}}\mathbf{1}\{S_n>S_0\}}{1+N_{perm}}},$$

(1)

where $\mathbf{1}\left\{statement\right\}=1$ when $statement$ is true and $\mathbf{1}\left\{statement\right\}=0$ otherwise, and small $\widehat{p}$ provides evidence to reject $H_0$.

If $x_b$ and $y_b$ represent statistics obtained from the same subjects that undergo different experiments, then one can increase the power of the permutation test by considering permutation in pairs: Let $S_0:=1/B{\displaystyle {\sum }_{b=1}^B}{y}_b$ as before and create $N_{perm}$ random permutations by randomly switching between $x_b$ and $y_b$ for each b. More precisely, for the $nth$ random permutation, let ${\mathbf{q}}^{\left(n\right)}:=[{\displaystyle q_1^{\left(n\right)}},\dots ,{\displaystyle q_B^{\left(n\right)}}]$ where ${\displaystyle q_b^{\left(n\right)}}$ equals $x_b$ or $y_b$ with probability 0.5, let $S_n:=1/B{\displaystyle {\sum }_{b=1}^B}{\displaystyle q_b^{\left(n\right)}}$, and estimate the p-value using (1).

2.4.2. Permutation Tests for Evaluating Machine Learning Classifiers

In this paper, we use permutation tests to evaluate how well eGeMAPS features can detect depression using the methodology described by Ojala and Garriga in [2]. Ojala and Garriga propose to evaluate whether a machine learning classifier is able to find structure in the data by permuting the labels in the data.

More precisely, letting $\mathsf{B}:={\displaystyle {\left\{({\mathbf{X}}_k,y_k)\right\}}_{k=1}^K}$ be a set of training data, where $y_k$ is the label of each feature vector ${\mathbf{X}}_k$, Ojala and Garriga evaluate a classifier f using the leave-one-out cross-validation error $P_e(f,\mathsf{B})=(1/K){\displaystyle {\sum }_{k=1}^K}\mathbf{1}\{f_{\{D-\left\{({\mathbf{X}}_k,y_k)\right\}\}}\left({\mathbf{X}}_k\right)\ne y_k\}$, where $f_{\{D-\left\{({\mathbf{X}}_k,y_k)\right\}\}}$ represents the classifier trained with the training set without $({\mathbf{X}}_k,y_k)$.

Ojala and Garriga then propose to test the hypothesis that the features ${\displaystyle {\left\{{\mathbf{X}}_k\right\}}_{k=1}^K}$ are independent of the labels ${\displaystyle {\left\{y_k\right\}}_{k=1}^K}$ using the permutation test: Let ${\displaystyle {\left\{{\tilde{\mathsf{B}}}_n\right\}}_{n=1}^{N_{perm}}}$ represent $N_{perm}$ training sets, each obtained by randomly permuting the labels ${\displaystyle {\left\{y_k\right\}}_{k=1}^K}$; and compute the p-value

$$\widehat{p}:={\displaystyle \frac{1+{\displaystyle {\sum }_{n=1}^{N_{perm}}}\mathbf{1}\{P_e(f,{\tilde{\mathsf{B}}}_n)\le P_e(f,\mathsf{B})\}}{1+N_{perm}}},$$

(2)

and low values for $\widehat{p}$ indicates evidence that the features ${\displaystyle {\left\{{\mathbf{X}}_k\right\}}_{k=1}^K}$ and labels ${\displaystyle {\left\{y_k\right\}}_{k=1}^K}$ are dependent variables and the classifier is able to explore such dependence.

We apply Ojala’s and Garriga’s method to evaluate whether eGeMAPS features are able to detect depression in speakers, as described in detail in the next section.

2.4.3. Bootstrap Confidence Intervals

Let ${\displaystyle {\left\{z_b\right\}}_{b=1}^B}$ be realizations of B i.i.d. random variables ${\displaystyle {\left\{Z_b\right\}}_{b=1}^B}$ for which one desires to estimate $E\left[Z_b\right]$.

The bootstrap method allows one to estimate a 95% confidence interval for $E\left[Z_b\right]$ as follows [30] (p. 8): Let ${\mathcal{Z}}_0:=[z_1,\dots ,z_B]$ be the array containing the B realizations obtained in the experiment. Let ${\mathcal{Z}}_n:=[z_{n,1},\dots ,z_{n,B}]$ be a B-dimensional array obtained by resampling from ${\mathcal{Z}}_0$ with replacement. More precisely, the value for $z_{n,b}$ for each b is chosen from among the B values in ${\mathcal{Z}}_0$ following an equiprobable distribution. Compute the average of ${\mathcal{Z}}_n$; i.e., let ${\displaystyle {\mu }_n^{\left(Z\right)}}:=(1/B){\displaystyle {\sum }_{b=1}^B}{z}_{n,b}$. After repeating the resampling process $N_{perm}$ times, one builds a 95% confidence interval for $E\left[Z_b\right]$ by first sorting the ${\displaystyle {\left\{{\displaystyle {\mu }_n^{\left(Z\right)}}\right\}}_{n=1}^{N_{perm}}}$ values and then using the values at the 2.5% and 97.5% percentiles for the confidence interval. The confidence interval is not exact; however, the method is known to perform good estimates when $B>100$ [30] (p. 19).

2.4.4. Hypothesis Testing Using Bootstrap

The bootstrap method can also be used to test the hypothesis that the expected values of two populations are equal or not [31] (p. 104): let ${\displaystyle {\left\{z_b\right\}}_{b=1}^B}$ be realizations as above; let ${\displaystyle {\left\{w_b\right\}}_{b=1}^B}$ be realizations of B i.i.d. random variables ${\displaystyle {\left\{W_b\right\}}_{b=1}^B}$; let ${\mathcal{W}}_0:=[w_1,\dots ,w_B]$; build ${\displaystyle {\left\{{\mathcal{W}}_n\right\}}_{n=1}^{N_{perm}}}$; and compute their averages ${\displaystyle {\left\{{\displaystyle {\mu }_n^{\left(W\right)}}\right\}}_{n=1}^{N_{perm}}}$ as above. To test the hypothesis $H_0:E\left[Z_b\right]=E\left[W_b\right]$, one builds a 95% bootstrap confidence interval for the parameter $E\left[Z_b\right]-E\left[W_b\right]$ using the values from ${\displaystyle {\{{\displaystyle {\mu }_n^{\left(Z\right)}}-{\displaystyle {\mu }_n^{\left(W\right)}}\}}_{n=1}^{N_{perm}}}$. If the confidence interval does not contain 0, then $H_0$ is rejected with 95% confidence.

2.5. Testing Independence of Features and Depression Labels

In this section, we use the permutation test to test the hypothesis ${\displaystyle H_0^{\left(I\right)}}$ specified in Section 1; i.e., to test whether eGeMAPs features of speakers are independent of their depression status. We will test ${\displaystyle H_0^{\left(I\right)}}$ for male and female speakers separately.

To test ${\displaystyle H_0^{\left(I\right)}}$ for speakers of the selected gender, we used the following procedure:

2.5.1. Step 1: Preprocessing

We extracted audio segments from all speakers of the selected gender in the E-DAIC database, as described in Preparation of Audio Files from Speakers Section.

For a given utterance length duration $T_{UL}$, let ${\displaystyle N_{seg}^{\left(s\right)}}$ be the number of segments with duration $T_{UL}$ for a speaker s.

For a given minimum number of segments $N_{seg,\min }$, we discarded any speaker with ${\displaystyle N_{seg}^{\left(s\right)}}<N_{seg,\min }$. As described in Section 3.1, we experimented with different values for $T_{UL}$.

We used OpenSMILE to extract eGeMAPS audio features for each of the ${\displaystyle N_{seg}^{\left(s\right)}}$ segments of each remaining speaker s, as described in Section 2.2. For most of the results of this paper, all 88 eGeMAPS features were extracted; however, in Section 3.3, we consider different subsets of features.

2.5.2. Step 2: Build Random Balanced Training Sets

Using the eGeMAPS features, we built B balanced training sets ${\displaystyle {\left\{{\mathsf{B}}^{\left(b\right)}\right\}}_{b=1}^B}$. By a balanced set, we mean a set with an equal number of segments from depressed and non-depressed speakers. More precisely, let ${\mathsf{D}}_{\mathsf{all}}$ and ${\mathsf{N}}_{\mathsf{all}}$ respectively represent the set of depressed and non-depressed speakers of the selected gender; and let $\left|S\right|$ represent the number of elements in a set S. We note that ${\mathsf{D}}_{\mathsf{all}}$ and ${\mathsf{N}}_{\mathsf{all}}$ contain all speakers of E-DAIC; i.e., all speakers of E-DAIC’s training, development, and testing sets from the selected gender were included in ${\mathsf{D}}_{\mathsf{all}}$ and ${\mathsf{N}}_{\mathsf{all}}$.

Since $|{\mathsf{D}}_{\mathsf{all}}|<|{\mathsf{N}}_{\mathsf{all}}|$ in the E-DAIC dataset, we build each balanced set ${\mathsf{B}}^{\left(b\right)}$ of training data as follows:

• Include all $|{\mathsf{D}}_{\mathsf{all}}|$ speakers in ${\mathsf{B}}^{\left(b\right)}$.
• Randomly choose $|{\mathsf{D}}_{\mathsf{all}}|$ speakers from the non-depressed set ${\mathsf{N}}_{\mathsf{all}}$ and include them in ${\mathsf{B}}^{\left(b\right)}$. Each non-depressed speaker is equally likely to be chosen but any non-depressed speaker is chosen at most once.
• Let $\mathsf{D}$ and $\mathsf{N}$ be the subsets of speakers from ${\mathsf{D}}_{\mathsf{all}}$ and ${\mathsf{N}}_{\mathsf{all}}$ included in ${\mathsf{B}}^{\left(b\right)}$. For each speaker $s\in \mathsf{D}\cup \mathsf{N}$, randomly choose $N_{seg,\min }$ segments out of the ${\displaystyle N_{seg}^{\left(s\right)}}$ segments from such a speaker and include their corresponding feature vectors and truth labels into ${\mathsf{B}}^{\left(b\right)}$. Each segment of speaker s is equally likely to be chosen but any segment is chosen at most once.

To facilitate the explanation of our procedure, we enumerate each segment chosen and each speaker in each of $\mathsf{D}$ and $\mathsf{N}$ and let ${\displaystyle {\left\{{\displaystyle {\{{\displaystyle {\mathbf{X}}_{(i,j,\mathsf{D})}^{\left(b\right)}},{\displaystyle y_{(i,j,\mathsf{D})}^{\left(b\right)}}\}}_{j=1}^{N_{seg,\min }}}\right\}}_{i=1}^{\left|\mathsf{D}\right|}}$ and ${\displaystyle {\left\{{\displaystyle {\{{\displaystyle {\mathbf{X}}_{(i,j,\mathsf{N})}^{\left(b\right)}},{\displaystyle y_{(i,j,\mathsf{N})}^{\left(b\right)}}\}}_{j=1}^{N_{seg,\min }}}\right\}}_{i=1}^{\left|\mathsf{D}\right|}}$ represent the feature vectors ($\mathbf{X}$) and the truth labels (y) from the $jth$ chosen segment of the $ith$ speaker from each subset ($\mathsf{D}$ or $\mathsf{N}$) in the training set ${\mathsf{B}}^{\left(b\right)}$. To avoid creating new notation for the depressed and nondepressed classes, we shall abuse the notation and use $\mathsf{D}$ and $\mathsf{N}$ to also represent the classes of each feature vector; i.e., ${\displaystyle y_{(i,j,\mathsf{D})}^{\left(b\right)}}=\mathsf{D}$ and ${\displaystyle y_{(i,j,\mathsf{N})}^{\left(b\right)}}=\mathsf{N}$.

We also define

$$\begin{array}{ccc}\hfill {\displaystyle {\mathsf{B}}_{i,:,\mathsf{D}}^{\left(b\right)}}& :=& {\displaystyle {\{{\displaystyle {\mathbf{X}}_{(i,j,\mathsf{D})}^{\left(b\right)}},{\displaystyle y_{(i,j,\mathsf{D})}^{\left(b\right)}}\}}_{j=1}^{N_{seg,\min }}}\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill {\displaystyle {\mathsf{B}}_{i,:,\mathsf{N}}^{\left(b\right)}}& :=& {\displaystyle {\{{\displaystyle {\mathbf{X}}_{(i,j,\mathsf{N})}^{\left(b\right)}},{\displaystyle y_{(i,j,\mathsf{N})}^{\left(b\right)}}\}}_{j=1}^{N_{seg,\min }}}\hfill \end{array}$$

(4)

to represent the set of all feature vectors and their truth labels from the $ith$ speaker from each class; and the balanced set ${\mathsf{B}}^{\left(b\right)}$ is given by

$${\mathsf{B}}^{\left(b\right)}:={\displaystyle \bigcup _{i=1}^{\left|\mathsf{D}\right|}}{\displaystyle {\mathsf{B}}_{i,:,\mathsf{D}}^{\left(b\right)}}\cup {\displaystyle {\mathsf{B}}_{i,:,\mathsf{N}}^{\left(b\right)}}$$

(5)

At the end of this process, the set ${\mathsf{B}}^{\left(b\right)}$ will contain $2·\left|\mathsf{D}\right|·N_{seg,\min }$ feature vectors and their corresponding truth labels.

We note that all ${\displaystyle {\left\{{\mathsf{B}}^{\left(b\right)}\right\}}_{b=1}^B}$ contain all speakers from $\mathsf{D}$; however, each ${\mathsf{B}}^{\left(b\right)}$ contains random $N_{seg,\min }$ segments of each depressed speaker.

2.5.3. Step 3: Training and Testing a Classifier Using ${\mathsf{B}}^{\left(b\right)}$

Let $f:{\mathrm{I}\mathrm{R}}^{88}\to \{\mathsf{N},\mathsf{D}\}$ be a classifier (SVM, trees, NaiveBayes) with a chosen set of hyperparameters to decide the depression status of an input sample from its 88 eGeMAPS features. For each training set ${\mathsf{B}}^{\left(b\right)}$, we used WEKA to train f and estimate its error performance using samples outside the set used to learn f.

Similar to [2], we evaluated the performance of f using the cross-validation error; however, we did not use leave-one-out cross-validation; instead, we used a $\left|\mathsf{D}\right|$-fold cross-validation: In each fold, we removed from ${\mathsf{B}}^{\left(b\right)}$ the $2N_{seg,\min }$ feature vectors corresponding to one pair of depressed/non-depressed speakers and used such feature vectors to test the function learned with the remaining feature vectors in ${\mathsf{B}}^{\left(b\right)}$. More precisely, letting $f_{\{{\mathsf{B}}^{\left(b\right)}-({\displaystyle {\mathsf{B}}_{i,:,\mathsf{D}}^{\left(b\right)}}\cup {\displaystyle {\mathsf{B}}_{i,:,\mathsf{N}}^{\left(b\right)}})\}}$ represent the classifier learned from the feature vectors and labels of ${\mathsf{B}}^{\left(b\right)}$ excluding all the feature vectors and labels from the $ith$ depressed speaker and from the $ith$ non-depressed speaker, we define

$$\begin{array}{c}{P}_{e,i}(f,{\mathsf{B}}^{\left(b\right)}):={\displaystyle \frac{1}{2·N_{seg,\min }}}·\hfill \\ \left({\displaystyle \sum _{j=1}^{N_{seg,\min }}}\mathbf{1}\left\{f_{\{{\mathsf{B}}^{\left(b\right)}-({\displaystyle {\mathsf{B}}_{i,:,\mathsf{D}}^{\left(b\right)}}\cup {\displaystyle {\mathsf{B}}_{i,:,\mathsf{N}}^{\left(b\right)}})\}}\left({\displaystyle {\mathbf{X}}_{(i,j,\mathsf{D})}^{\left(b\right)}}\right)\ne \mathsf{D}\right\}+\mathbf{1}\left\{f_{\{{\mathsf{B}}^{\left(b\right)}-({\displaystyle {\mathsf{B}}_{i,:,\mathsf{D}}^{\left(b\right)}}\cup {\displaystyle {\mathsf{B}}_{i,:,\mathsf{N}}^{\left(b\right)}})\}}\left({\displaystyle {\mathbf{X}}_{(i,j,\mathsf{N})}^{\left(b\right)}}\right)\ne \mathsf{N}\right\}\right).\hfill \end{array}$$

(6)

and we evaluate the error performance of the classification algorithm f for the training set ${\mathsf{B}}^{\left(b\right)}$ with

$$P_e(f,{\mathsf{B}}^{\left(b\right)}):={\displaystyle \frac{1}{\left|\mathsf{D}\right|}}{\displaystyle \sum _{i=1}^{\left|\mathsf{D}\right|}}{P}_{e,i}(f,{\mathsf{B}}^{\left(b\right)}).$$

(7)

We highlight that the reason for using feature vectors from multiple segments when building ${\mathsf{B}}^{\left(b\right)}$ is to enable better training of the classifier. The reason for removing all feature vectors from a speaker that is used in the cross-validation fold is to avoid biasing the classifier with any feature vector from the speaker. Lastly, the reason for adding both a depressed and a non-depressed speaker in each fold is to ensure that both training and evaluation are done with balanced sets, avoiding the biasing of the classifier towards either class. These steps are necessary because of the relatively limited number of depressed speakers in the E-DAIC feature set.

From ${\displaystyle {\left\{P_e(f,{\mathsf{B}}^{\left(b\right)})\right\}}_{b=1}^B}$, we compute

$$\begin{array}{ccc}\hfill \overline{P_e}& :=& {\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{P}_e(f,{\mathsf{B}}^{\left(b\right)}).\hfill \end{array}$$

(8)

Step 3 is repeated for various sets of hyperparameters; and the hyperparameter set providing the smallest $\overline{P_e}$ is selected. Such a $\overline{P_e}$ is then used in Step 5 to conclude the test of ${\displaystyle H_0^{\left(I\right)}}$.

2.5.4. Step 4: Permutation of Labels

In this step, we randomly permute the labels in each ${\mathsf{B}}^{\left(b\right)}$. The rationale for this step is the following: If ${\displaystyle H_0^{\left(I\right)}}$ were true; i.e., if eGeMAPS features were indeed independent of the depression status of speakers, then permuting the labels in ${\mathsf{B}}^{\left(b\right)}$ would not change the distribution of features and, therefore, would not change the distribution of $P_e(f,{\mathsf{B}}^{\left(b\right)})$.

To precisely define this step, let $\mathsf{A}:=\{1,\dots ,|\mathsf{D}\left|\right\}\times \{1,\dots ,N_{seg,\min }\}\times \{\mathsf{D},\mathsf{N}\}$ represent the set of triplets $(i,j,k)$ representing the index of the speaker, its segment number, and the class of each segment in the training set. Let $\pi$ be a random permutation of the elements in $\mathsf{A}$; i.e., $\pi :\mathsf{A}\to \mathsf{A}$ is any bijective function from $\mathsf{A}$ to itself; and let $\pi {(i,j,k)}_l$ represent the $lth$ coordinate of $\pi (i,j,k)$. For instance, if $\pi (3,9,\mathsf{D})=(7,1,\mathsf{N})$, then $\pi {(3,9,\mathsf{D})}_1=7$, $\pi {(3,9,\mathsf{D})}_2=1,$ and $\pi {(3,9,\mathsf{D})}_3=\mathsf{N}$.

Using a random $\pi$, we permute the labels ${\displaystyle y_{(i,j,\mathsf{D})}^{\left(b\right)}}$ and ${\displaystyle y_{(i,j,\mathsf{N})}^{\left(b\right)}}$ among all speakers, segments, and classes. More precisely, let ${\displaystyle {\tilde{y}}_{(i,j,k)}^{\left(b\right)}}={\displaystyle y_{\pi (i,j,k)}^{\left(b\right)}}$, which means that the label for the $jth$ segment of the $ith$ speaker in the depressed class (${\displaystyle y_{(i,j,\mathsf{D})}^{\left(b\right)}}$) is replaced by the label from the segment $\pi {(i,j,\mathsf{D})}_2$ of the speaker $\pi {(i,j,\mathsf{D})}_1$ of the class $\pi {(i,j,\mathsf{D})}_3$, which may be different than $\mathsf{D}$. This means that the permutation $\pi$ may cause ${\displaystyle {\tilde{y}}_{(i,j,\mathsf{D})}^{\left(b\right)}}=\mathsf{N}$ or ${\displaystyle {\tilde{y}}_{(i,j,\mathsf{N})}^{\left(b\right)}}=\mathsf{D}$.

Define

$$\begin{array}{ccc}\hfill {\displaystyle {\tilde{\mathsf{B}}}_{i,:,\mathsf{D}}^{\left(b\right)}}& :=& {\displaystyle {\{{\displaystyle {\mathbf{X}}_{(i,j,\mathsf{D})}^{\left(b\right)}},{\displaystyle {\tilde{y}}_{(i,j,\mathsf{D})}^{\left(b\right)}}\}}_{j=1}^{N_{seg,\min }}}\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill {\displaystyle {\tilde{\mathsf{B}}}_{i,:,\mathsf{N}}^{\left(b\right)}}& :=& {\displaystyle {\{{\displaystyle {\mathbf{X}}_{(i,j,\mathsf{N})}^{\left(b\right)}},{\displaystyle {\tilde{y}}_{(i,j,\mathsf{N})}^{\left(b\right)}}\}}_{j=1}^{N_{seg,\min }}},\hfill \end{array}$$

(10)

which represent the set of all feature vectors from the $ith$ speaker with labels permuted as described above; and define the permuted training set ${\tilde{\mathsf{B}}}^{\left(b\right)}$ as

$${\tilde{\mathsf{B}}}^{\left(b\right)}:={\displaystyle \bigcup _{i=1}^{\left|\mathsf{D}\right|}}{\displaystyle {\tilde{\mathsf{B}}}_{i,:,\mathsf{D}}^{\left(b\right)}}\cup {\displaystyle {\tilde{\mathsf{B}}}_{i,:,\mathsf{N}}^{\left(b\right)}}$$

(11)

Using the permuted training set ${\tilde{\mathsf{B}}}^{\left(b\right)}$, we compute the error performance of the classification algorithm f using $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$ and $P_{e,i}(f,{\tilde{\mathsf{B}}}^{(b)})$ of (7) and (6). We highlight that ${\mathsf{B}}^{(b)}$, ${\displaystyle {\mathsf{B}}_{i,:,\mathsf{D}}^{(b)}}$, and ${\displaystyle {\mathsf{B}}_{i,:,\mathsf{N}}^{(b)}}$ are replaced with ${\tilde{\mathsf{B}}}^{(b)}$, ${\displaystyle {\tilde{\mathsf{B}}}_{i,:,\mathsf{D}}^{(b)}}$, and ${\displaystyle {\tilde{\mathsf{B}}}_{i,:,\mathsf{N}}^{(b)}}$ in (7) and (6) for this computation.

The random permutation of labels and computation of $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$ is performed B times, one time for each ${\mathsf{B}}^{(b)}$.

2.5.5. Step 5: Permutation Test of ${\displaystyle H_0^{(I)}}$

If ${\displaystyle H_0^{(I)}}$ were true; i.e., if eGeMAPS features were indeed independent of the depression status of speakers, then for each b, ${\displaystyle {\mathbf{X}}_{(i,j,\mathsf{D})}^{(b)}}$ and ${\displaystyle {\mathbf{X}}_{(i,j,\mathsf{N})}^{(b)}}$ would be i.i.d. vectors, permuting the labels would not impact their distribution, and the estimates $P_e(f,{\mathsf{B}}^{(b)})$ and $P_e(f,{\tilde{\mathsf{B}}}^{(b)}))$ would also be i.i.d.

This allows us to test ${\displaystyle H_0^{(I)}}$ using a paired-permutation test as follows: From steps 3 and 4, we obtain B pairs of probability of error estimates: ${\displaystyle {\{(P_e(f,{\mathsf{B}}^{(b)}),P_e(f,{\tilde{\mathsf{B}}}^{(b)}))\}}_{b=1}^B}$. Using these pairs, we generate $N_{perm}$ B-dimensional vectors ${\displaystyle {\left\{{\mathbf{q}}^{(n)}\right\}}_{n=1}^{N_{perm}}}$ by randomly choosing either $P_e(f,{\mathsf{B}}^{(b)})$ or $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$. More precisely, for each n, each entry in ${\mathbf{q}}^{(n)}:=[{\displaystyle q_1^{(n)}},\dots ,{\displaystyle q_B^{(n)}}]$ is defined from B i.i.d. realizations $w_b$ of a Bernoulli(0.5) random variable: If $w_b=0$, assign $P_e(f,{\mathsf{B}}^{(b)})$ to ${\displaystyle q_b^{(n)}}$; otherwise, assign $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$ to ${\displaystyle q_b^{(n)}}$.

Let

$$\begin{array}{ccc}\hfill {\overline{q}}^{\left(n\right)}& :=& {\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{\displaystyle q_b^{\left(n\right)}};\hfill \end{array}$$

(12)

i.e., ${\overline{q}}^{\left(n\right)}$ contains the average of the randomly chosen probabilities of error in the $nth$ permutation. Note that, if ${\displaystyle H_0^{\left(I\right)}}$ were true, then the expected value of ${\overline{q}}^{\left(n\right)}$ would be the same as the expected value of $\overline{P_e}$ computed at (8) without any permutation of labels.

Lastly, we obtain an estimate for the p-value of ${\displaystyle H_0^{\left(I\right)}}$ with:

$${\widehat{p}}^{\left(I\right)}={\displaystyle \frac{1+{\displaystyle {\sum }_{n=1}^{N_{perm}}}\mathbf{1}\{{\overline{q}}^{\left(n\right)}\le \overline{P_e}\}}{1+N_{perm}}},$$

(13)

and low values of ${\widehat{p}}^{\left(I\right)}$ provide evidence to reject ${\displaystyle H_0^{\left(I\right)}}$. For instance, if the p-value of ${\displaystyle H_0^{\left(I\right)}}$ is less than $0.05$, then we are justified in rejecting ${\displaystyle H_0^{\left(I\right)}}$ and conclude that, with high confidence, eGeMAPS features are not independent of the depression status of speakers of the selected gender.

3. Results and Discussion

3.1. Testing Independence of Features and Depression Labels

We performed the Steps 1 through 5 of Section 2.5 to test ${\displaystyle H_0^{\left(I\right)}}$ for female and male speakers of the E-DAIC set for various values of utterance length ($T_{UL}$).

For Step 1, we considered $N_{seg,\min }=30$; i.e., speakers needed at least 30 segments of $T_{UL}$ duration to be considered in the test. This resulted in a different number of speakers in $\mathsf{D}$ for each $T_{UL}$. For $T_{UL}=8$ s, $|{\mathsf{D}}_{female}|=18$ and $|{\mathsf{D}}_{male}|=15$. For $T_{UL}=6$ s, $|{\mathsf{D}}_{female}|=21$ and $|{\mathsf{D}}_{male}|=19$. For $T_{UL}=4$ s, $|{\mathsf{D}}_{female}|=26$ and $|{\mathsf{D}}_{male}|=24$. For $T_{UL}=2$ s, $|{\mathsf{D}}_{female}|=30$ and $|{\mathsf{D}}_{male}|=33$.

For Step 2, we considered a total of $B=300$ random balanced sets.

For Step 3, we considered three classifier algorithms: SVM, Naive Bayes, and Decision Trees (J.48). To select the set of hyperparameters for each classifier, we used 100 of the 300 random balanced sets to compute $\overline{P_e}$ for various hyperparameter sets. For the J.48 algorithm, we considered $C\in \{0.1,0.25,0.5\}$ and $M\in \{2,5,10,15\}$ for the prunning coefficient and minimum number of instances per leaf. The hyperparameters that provided smallest $\overline{P_e}$ were: for female speakers, $C=0.1$ and $M=15$; and for male speakers, $C=0.1$ and $M=5$. It should be mentioned that the maximum variation in the $\overline{P_e}$ among all hyperparameters considered was less than 0.05. For SVM, we observed a wider variation. We considered $C\in \{{10}^{-5},{10}^{-4},{10}^{-3},0.01,0.1,1.0,10.0\}$ for the complexity constant. For both male and female speakers, higher values ($C=1$ and $C=10.0$) provided the smaller $\overline{P_e}$, with $C=10.0$ providing the smallest. Since the value $C=10.0$ provided excess running time and the value $C=1.0$ resulted in less than $0.001$ more $\overline{P_e}$, well within the standard deviation, we used $C=1.0$ for the permutation tests (We note that the Naive Bayes implementation of WEKA does not offer hyperparameter selection).

Table 1. Average ($\overline{P_e}$) and standard deviation of the probability of classification error among the $B=300$ random balanced sets $({\displaystyle {\{P_e(f,{\mathsf{B}}^{(b)})\}}_{b=1}^B})$ for female and male speakers of the E-DAIC set for various $T_{UL}$ and various classification algorithms.

${\mathit{T}}_{\mathit{UL}}$	2 s	4 s	6 s	8 s
female
SVM	$0.494\pm 0.033$	$0.463\pm 0.038$	$0.451\pm 0.043$	$0.464\pm 0.051$
Bayes	$0.526\pm 0.025$	$0.520\pm 0.030$	$0.511\pm 0.036$	$0.513\pm 0.044$
Tree	$0.519\pm 0.024$	$0.510\pm 0.032$	$0.500\pm 0.035$	$0.500\pm 0.042$
male
SVM	$0.471\pm 0.039$	$0.439\pm 0.054$	$0.466\pm 0.060$	$0.411\pm 0.062$
Bayes	$0.504\pm 0.028$	$0.506\pm 0.041$	$0.491\pm 0.049$	$0.481\pm 0.063$
Tree	$0.484\pm 0.024$	$0.467\pm 0.037$	$0.452\pm 0.052$	$0.431\pm 0.057$

shows the average ($\overline{P_e}$) and the standard deviation of ${\displaystyle {\left\{P_e(f,{\mathsf{B}}^{\left(b\right)})\right\}}_{b=1}^B}$ for the final set of hyperparameter sets in each of the classifier algorithms for various $T_{UL}$. We note that the results for the SVM classifier are in agreement with previous results in the literature, which reported $\overline{P_e}\in [0.35,0.45]$ [14]. We also note that, if we were to solely rely on $\overline{P_e}$ among the random training sets, we would not be able to reject ${\displaystyle H_0^{\left(I\right)}}$ since a typical 95% confidence interval obtained by multiplying the standard deviation by 1.96 would include $0.50$, which is the error probability of a random guess classifier. Even in the case of male speakers, as indicated in Figure 1, the 95% confidence interval for the average $P_e(f,\mathsf{B})$ would also include $0.50$, showing weak evidence for rejecting ${\displaystyle H_0^{\left(I\right)}}$. This shows the nontriviality of the permutation test proposed.

To properly test ${\displaystyle H_0^{\left(I\right)}}$, we performed Steps 4 and 5 of Section 2.5 and the permutation test results are shown in

Table 2. Results of ${\displaystyle H_0^{\left(I\right)}}$ tests (average and standard deviation of ${\widehat{p}}^{\left(I\right)}$) for female and male speakers of the E-DAIC set for various $T_{UL}$ and various classification algorithms.

${\mathit{T}}_{\mathit{UL}}$	2 s	4 s	6 s	8 s
female
SVM	$0.21\pm 0.20$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.01\pm 0.01$
Bayes	$1.00\pm 0.00$	$0.99\pm 0.03$	$0.87\pm 0.12$	$0.86\pm 0.16$
Tree	$0.18\pm 0.19$	$0.10\pm 0.15$	$0.03\pm 0.08$	$0.04\pm 0.04$
male
SVM	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.03\pm 0.06$	$0.00\pm 0.00$
Bayes	$0.65\pm 0.29$	$0.73\pm 0.19$	$0.24\pm 0.31$	$0.08\pm 0.06$
Tree	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$

. For Step 5, we considered $N_{perm}=10,000$ and, to obtain confidence intervals for the p-value estimates ${\widehat{p}}^{\left(I\right)}$, we partitioned the 300 balanced sets into 10 sets of 30 different random balanced sets. The values shown in Table 2 correspond to the average and standard deviation of the 10 p-value estimates obtained.

Since Table 2 shows average p-values below 0.05 for at least 1 classifier when $T_{UL}\ge 4$ s, we can reject ${\displaystyle H_0^{\left(I\right)}}$ for both female and male speakers, and conclude that there is strong evidence that eGeMAPS features are dependent on the depression status of E-DAIC speakers. We note that the high p-values obtained for the Naive Bayes classifier do not contradict this conclusion: such high p-values only indicate that this specific classifier is not able to explore the dependence between eGeMAPS features and the depression status of E-DAIC speakers.

The histograms of $P_e(f,{\mathsf{B}}^{(b)})$ and $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$ shown in Figure 1 support this conclusion. These histograms illustrate the distribution of $P_e(f,{\mathsf{B}}^{(b)})$ and $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$ for all training sets (300 in total) built under $T_{UL}=8$ s in each of the classifiers. As expected, when randomly shufflying the depressed and non-depressed labels, $P_e(f,{\tilde{\mathsf{B}}}^{(b)})$ is concentrated around 0.50. When considering the actual labels, both the SVM and Tree classifiers were able to change the distribution of $P_e(f,{\mathsf{B}}^{(b)})$ and reduce its mean below 0.50.

To further support this conclusion, we computed the 95% confidence intervals for $E\left[\overline{P_e}\right]$ using the bootstrap resampling method described in Section 2.4.3. Such intervals are shown in

Table 3. 95% Bootstrap confidence intervals for the expected probability of classification error $(E\left[\overline{P_e}\right])$ for female and male speakers of the E-DAIC set for various $T_{UL}$ and various classification algorithms.

${\mathit{T}}_{\mathit{UL}}$	2 s	4 s	6 s	8 s
female
SVM	$[0.490,0.497]$	$[0.459,0.467]$	$[0.446,0.456]$	$[0.459,0.471]$
Bayes	$[0.523,0.529]$	$[0.517,0.524]$	$[0.507,0.515]$	$[0.508,0.518]$
Tree	$[0.516,0.521]$	$[0.507,0.514]$	$[0.497,0.504]$	$[0.496,0.505]$
male
SVM	$[0.466,0.475]$	$[0.432,0.445]$	$[0.459,0.472]$	$[0.404,0.418]$
Bayes	$[0.500,0.507]$	$[0.501,0.510]$	$[0.485,0.496]$	$[0.474,0.488]$
Tree	$[0.482,0.487]$	$[0.462,0.471]$	$[0.447,0.458]$	$[0.424,0.437]$

. By comparing these intervals with the ${\widehat{p}}^{(I)}$ estimates of Table 2, we can see that bootstrap intervals farther from 0.50 result in low values of ${\widehat{p}}^{(I)}$. It is also possible to note that the intervals move further away from 0.50 as $T_{UL}$ increases, confirming that longer values of $T_{UL}$ provide better detection performance.

3.2. eGeMAPS Dependence with Speaker Gender

The results of Table 1, Table 2 and Table 3 suggest that eGeMAPS is more effective in detecting depression in male than female speakers of the E-DAIC set. In this section, we formally test the hypothesis

${\displaystyle H_0^{(G)}}$ 

: eGeMAPS discrimination power does not depend on the speaker gender.

• using the bootstrap method described in Section 2.4.3.

To test ${\displaystyle H_0^{(G)}}$, let ${\displaystyle {\left\{{\displaystyle {\mathsf{B}}_F^{(b)}}\right\}}_{b=1}^B}$ be a sequence of training sets randomly built from female speakers following Steps 1 and 2 of Section 3.1; and, for a given classifier algorithm f, let $P_e(f,{\displaystyle {\mathsf{B}}_F^{(b)}})$ be the error performance obtained by f when trained on ${\displaystyle {\mathsf{B}}_F^{(b)}}$. Likewise, let ${\displaystyle {\left\{{\displaystyle {\mathsf{B}}_M^{(b)}}\right\}}_{b=1}^B}$ and ${\displaystyle {\left\{P_e(f,{\displaystyle {\mathsf{B}}_M^{(b)}})\right\}}_{b=1}^B}$ respectively be training sets randomly built from male speakers and their corresponding error performance obtained by the same classification algorithm f.

Let ${\mathcal{Z}}_0:=[P_e(f,{\displaystyle {\mathsf{B}}_M^{(1)}}),\dots ,P_e(f,{\displaystyle {\mathsf{B}}_M^{(B)}})]$ and ${\mathcal{W}}_0:=[P_e(f,{\displaystyle {\mathsf{B}}_F^{(1)}}),\dots ,P_e(f,{\displaystyle {\mathsf{B}}_F^{(B)}})]$.

As described in Section 2.4.3, build $N_{perm}$ resamples with replacement of ${\mathcal{Z}}_0$ and ${\mathcal{W}}_0$ and compute the averages of each resample, obtaining ${\displaystyle {\left\{{\displaystyle {\mu }_n^{(Z)}}\right\}}_{n=1}^{N_{perm}}}$ and ${\displaystyle {\left\{{\displaystyle {\mu }_n^{(W)}}\right\}}_{n=1}^{N_{perm}}}$. Use the 2.5% and 97.5%-percentiles of ${\displaystyle {\{{\displaystyle {\mu }_n^{(W)}}-{\displaystyle {\mu }_n^{(Z)}}\}}_{n=1}^{N_{perm}}}$ to build a 95% confidence interval. If the confidence interval contains only positive values, then one has strong evidence to reject $H_0$ and conclude that eGeMAPS features are better able to detect depression in males than in females.

Using Steps 1 and 2 of Section 2.5, we tested ${\displaystyle H_0^{(G)}}$ for various utterance durations ($T_{UL}$) and various classifier algorithms. As in Section 3.1, we evaluated the error performance of SVM, Naive Bayes, and Decision Trees with the same hyperparameters as in Section 3.1. All results were obtained considering $N_{seg,\min }=30$ and $B=300$; i.e., we considered 300 random sets of depressed/non-depressed speakers, each with a random selection of 30 segments from each speaker.

The 95% confidence intervals shown in

Table 4. Results of testing ${\displaystyle H_0^{\left(G\right)}}$ for various classifiers and $T_{UL}$ values.

${\mathit{T}}_{\mathit{UL}}$	2 s	4 s	6 s	8 s
SVM	$[0.017,0.029]$	$[0.017,0.033]$	$[-0.023,-0.007]$	$[0.044,0.063]$
Bayes	$[0.018,0.027]$	$[0.009,0.020]$	$[0.013,0.027]$	$[0.023,0.041]$
Tree	$[0.030,0.038]$	$[0.038,0.049]$	$[0.041,0.055]$	$[0.061,0.078]$

provide strong evidence to reject ${\displaystyle H_0^{(G)}}$ and conclude that there is strong evidence that eGeMAPS features are able to discriminate depression in male speakers better than in female speakers, particularly at high $T_{UL}$ values and when using SVM or Tree (J.48) classifier.

We note that the SVM classifier with $T_{UL}=6$ s showed a confidence interval with negative values, suggesting an opposite conclusion; i.e., better discrimination performance in female speakers, while the Tree (J.48) and Naive Bayes Classifier showed better discrimination performance in male speakers. This observation was confirmed even when considering a different random set of 6 s utterances for the 21 female and 19 male speakers. This result illustrates that the performance of algorithms can be highly dependent on the subset of speakers chosen for comparison. This result also illustrates that discrimination performance can also vary widely between classifiers.

Nevertheless, the positive confidence intervals in all other $T_{UL}$ and classifiers suggest that better discrimination performance is expected in male speakers.

3.3. Detection Power of Subsets of eGeMAPS Features

In this section, we compare the depression detection performance of different subsets of eGeMAPS features by building bootstrap confidence intervals.

For this, we start by choosing 2 subsets of eGeMAPS features. For the first subset, let ${\displaystyle {\{{\displaystyle {\mathsf{B}}_1^{(b)}}\}}_{b=1}^B}$ be a sequence of training sets randomly built from speakers of a given gender following the Steps 1 and 2 of Section 3.1 and considering features of the first subset of features; and, for a given classifier algorithm f, let $P_e(f,{\displaystyle {\mathsf{B}}_1^{(b)}})$ be the error performance obtained by f when trained on ${\displaystyle {\mathsf{B}}_1^{(b)}}$. For the second subset, let ${\displaystyle {\{{\displaystyle {\mathsf{B}}_2^{(b)}}\}}_{b=1}^B}$ be a sequence of training sets obtained from the same set of sepakers as in the first subset; however, considering only features of the second subset. Let ${\displaystyle {\{P_e(f,{\displaystyle {\mathsf{B}}_2^{(b)}})\}}_{b=1}^B}$ be the corresponding error performance.

Let $\overline{P_{e,1}}$ and $\overline{P_{e,2}}$ respectively represent the average probability of error obtained with each feature set. Since these averages are estimates of the true expected probability of error, we cannot conclude that the first subset provides better detection performance if $\overline{P_{e,1}}<\overline{P_{e,2}}$ unless the 95% confidence intervals are nonoverlapping. We can however use bootstrap to build a 95% confidence interval for $\overline{P_{e,1}}-\overline{P_{e,2}}$. If such an interval is in ${\mathrm{I}\mathrm{R}}^{+}$, then we can conclude that the second subset offers lower probability of decision error.

To build the 95% bootstrap confidence interval, let ${\mathcal{Z}}_0:=[P_e(f,{\displaystyle {\mathsf{B}}_1^{(1)}}),\dots ,P_e(f,{\displaystyle {\mathsf{B}}_1^{(B)}})]$ and ${\mathcal{W}}_0:=[P_e(f,{\displaystyle {\mathsf{B}}_2^{(1)}}),\dots ,P_e(f,{\displaystyle {\mathsf{B}}_2^{(B)}})]$. As described in Section 2.4.3, build $N_{perm}$ resamples with replacement of ${\mathcal{Z}}_0$ and ${\mathcal{W}}_0$ and compute the averages of each resample, obtaining ${\displaystyle {\{{\displaystyle {\mu }_n^{(Z)}}\}}_{n=1}^{N_{perm}}}$ and ${\displaystyle {\{{\displaystyle {\mu }_n^{(W)}}\}}_{n=1}^{N_{perm}}}$. The 2.5% and 97.5%-percentiles of ${\displaystyle {\{{\displaystyle {\mu }_n^{(W)}}-{\displaystyle {\mu }_n^{(Z)}}\}}_{n=1}^{N_{perm}}}$ are then used to build a 95% confidence interval.

Using the procedure above, we built 95% bootstrap confidence intervals for the difference in error performance ($\overline{P_e}$) of the following subsets of eGeMAPS features (see [25] for more details):

• All eGeMAPS features ($\mathrm{all}$): subset with all 88 features defined in [25];
• Temporal features ($\mathrm{temp}$): (6 features) rate of loudness peaks, mean length and standard deviation of continuously voiced regions and of unvoiced regions, number of continuous voiced regions per second (Features 82–87 produced by the configuration file eGeMAPSv02.conf available in the OpenSMILE website.).
• eGeMAPS features excluding temporal features (${\mathrm{temp}}^c$).
• Frequency features ($\mathrm{freq}$): (24 features) mean and coefficient of variation in pitch, jitter, center frequency and bandwidth of the first, second, and third formants; and the 20th, 50th, and 80th percentiles of pitch, the 20th–80th percentile of pitch, and the mean and the standard deviation of the rising/falling slopes of pitch (Features 1–10, 31, 32, 41–44, 47–50, 53–56 produced by eGeMAPSv02.conf.).
• eGeMAPS features excluding frequency features (${\mathrm{freq}}^c$).
• Energy features ($\mathrm{ener}$): (15 features) mean and coefficient of variation in shimmer, loudness, harmonic-to-noise ratio; equivalent sound level; and the 20th, 50th, and 80th percentiles of loudness, the 20th–80th percentile of loudness, and the mean and the standard deviation of the rising/falling slopes of loudness (Features 11–20, 33-36, and 88 produced by eGeMAPSv02.conf.).
• eGeMAPS features excluding energy features (${\mathrm{ener}}^c$).
• Spectral features ($\mathrm{spec}$): (43 features) mean and coefficient of variation in alpha ratio (voiced and unvoiced segments), Hammarberg index (voiced and unvoiced segments), spectral slope from 0 to 500 Hz, spectral slope from 500 to 1500 Hz; relative energy of the first, second, and third formants; ratio of the energy of the spectral harmonic peak between a formant’s center frequency and the pitch frequency for the first, second, and third formants; ratio between the energy of the first harmonic to the second and between the energy of the first to the third harmonics; MFCCs 1 through 4, and spectral flux (Features 21–30, 37-40, 45, 46, 51, 52, and 57–81 produced by eGeMAPSv02.conf.).
• eGeMAPS features excluding spectral features (${\mathrm{spec}}^c$).

For each pair of feature subsets being compared, we performed the Steps 1 and 2 of Section 2.5 for each feature set and for each gender considering $T_{UL}=8$ s and $N_{seg,\min }=30$; i.e., speakers needed at least 30 segments of 8 s duration. From $B=300$ random speaker sets, we obtained ${\displaystyle {\{P_e(f,{\displaystyle {\mathsf{B}}_1^{(b)}})\}}_{b=1}^B}$ and $P_e(f,{\displaystyle {\mathsf{B}}_2^{(b)}}){\displaystyle {)\}}_{b=1}^B}$.

Figure 2 shows the 95% bootstrap confidence intervals for $\overline{P_{e,1}}-\overline{P_{e,2}}$ for various pairs of feature subsets and considering the SVM and the Decision Tree classifiers. For each classifier and feature subset, we selected the best set of hyperparameters by using 100 random balanced sets and selecting the set of hyperparameters providing the highest $\overline{P_e}$, as described in Section 3.1.

Although both the SVM and Decision Tree classifiers provided similar results when testing the hypotheses ${\displaystyle H_0^{(I)}}$ and ${\displaystyle H_0^{(G)}}$, $\overline{P_{e,1}}-\overline{P_{e,2}}$ varied significantly between these two classifiers. For instance, while the 95% C.I. for $\overline{P_{e,\mathrm{ener}}}-\overline{P_{e,\mathrm{spec}}}$ was negative for SVM, it was positive for Decision Tree. Such differences suggest that SVM and Decision Trees are building significantly different decision regions.

The results of $\overline{P_{e,\mathrm{all}}}-\overline{P_{e,{\mathrm{ener}}^c}}$ and $\overline{P_{e,\mathrm{all}}}-\overline{P_{e,{\mathrm{spec}}^c}}$ reinforce this conclusion, while SVM is able to better classify depression in males when including energy features, Decision Trees is able to better classify depression in males when excluding energy features; and while SVM is able to better classify depression in females when including spectral features, Decision Trees is able to better classify depression in females when excluding spectral features.

Regarding $\overline{P_{e,\mathrm{all}}}-\overline{P_{e,{\mathrm{freq}}^c}}$, the 95% C.I. for females suggest that frequency features provide additional, albeit small, discrimination power to both SVM and Decision Tree classifiers.

Regarding $\overline{P_{e,\mathrm{all}}}-\overline{P_{e,{\mathrm{temp}}^c}}$, its 95% C.I. centered at 0 in both males and females suggest they are not providing additional discrimination power to either SVM or Decision Tree classifiers.

Focusing on feature classes in isolation, it is interesting to observe that, for detecting depression in females, the Trees classifier was able to significantly exploit temporal-only features when detecting depression in females, but the conclusion was the opposite for males. SVM also showed significantly worse detection performance for temporal features in males. It is also interesting to observe that the Trees classifier was able to exploit spectral-only features in males, providing lower $\overline{P_e}$ when compared against all the other types of features.

4. Conclusions

Using resampling methods, we showed in this paper that eGeMAPS features contain information that can assist in the detection of depression in speakers. Although several previous studies addressed this question before, they often lack confidence intervals or statistical guarantees for their conclusions. Using a permutation test, we showed that eGeMAPS features and speakers’ depression status are dependent on each other.

Using bootstrap methods, we were also able to show that eGeMAPS features are able to better discriminate depression in male speakers than female speakers. Although previous studies have addressed this question before, this is the first study that addressed the question providing statistical guarantees.

We further evaluated the performance of different subsets of eGeMAPS features in their power to discriminate depression. This is important because machine learning algorithms are known to be sensitive to feature configuration and eGeMAPS was designed to be used not only for depression detection, but also for emotion recognition and other mental health diseases.

It is important to mention that, similar to other studies, our results are restricted to eGeMAPS features, to the E-DAIC dataset, and to the classifiers used. Different feature sets, different depression datasets, or different classifiers (e.g., deep neural networks) may offer different detection performance. However, by using standard open-source datasets, open-source audio features, and open-source machine learning tools, our results can be replicated and be used as a baseline for future studies.

As avenues for future research, we highlight the need for additional and larger annotated datasets that are made open to all researchers. Larger datasets would not only increase the confidence level of the analysis, but also facilitate the training of deep learning methods. It would also be interesting to consider how detection performance is sensitive to the population sampled in the datasets, the culture and language of speakers, and the task performed by speakers. We also highlight that some of our observations need further study from a physiological mechanism perspective. For instance, it would be important to investigate the underlying causes for eGeMAPS temporal features providing a higher discrimination power in females and for eGeMAPS energy features providing a higher discrimination power in males.

Further research is also needed to make depression detection from speech a practical approach. Clearly, the detection performance (probability of error) of speech-based depression detection is far from desirable: the average probability of error was greater than 0.40 (in agreement with previous studies) and had large standard deviations, highlighting the difficulty in detecting depression from speech-only features. One avenue to improve the detection performance is to collect features from additional signals, such as videos or textual information, and build classifiers that fuse such features along with audio features to detect depression [4,19]. Another avenue is to develop classifiers tailored to detect depression episodes on a specific patient [32]; i.e., instead of designing a classifier to detect depression in any speaker, the classifier would focus on a particular patient. Such a classifier would train on audio samples of the patient during both non-depressed and depressed times. Such a classifier could then be used to determine whether treatment is reducing the rate of depression episodes in the patient.