A Depression Recognition Method Based on the Alteration of Video Temporal Angle Features

Abstract

In recent years, significant progress has been made in the auxiliary diagnosis system for depression. However, most of the research has focused on combining features from multiple modes to enhance classification accuracy. This approach results in increased space-time overhead and feature synchronization problems. To address this issue, this paper presents a single-modal framework for detecting depression based on changes in facial expressions. Firstly, we propose a robust method for extracting angle features from facial landmarks. Theoretical evidence is provided to demonstrate the translation and rotation invariance of these features. Additionally, we introduce a flip correction method to mitigate angle deviations caused by head flips. The proposed method not only preserves the spatial topological relationship of facial landmarks, but also maintains the temporal correlation between frames preceding and following the facial landmarks. Finally, the GhostNet network is employed for depression detection, and the effectiveness of various modal data is compared. In the depression binary classification task using the DAIC-WOZ dataset, our proposed framework significantly improves the classification performance, achieving an F1 value of 0.80 for depression detection. Experimental results demonstrate that our method outperforms other existing depression detection models based on a single modality.

1. Introduction

Depression is a mental disorder characterized by a persistent low mood, typically accompanied by diminished self-esteem, impaired concentration, excessive guilt or self-deprecation, a sense of hopelessness regarding the future, sleep disturbances, and feelings of fatigue or reduced energy [1]. Furthermore, depression exacerbates the risk of suicide among affected individuals, imposing a substantial burden and exerting significant societal impacts [2]. Moreover, a more concerning issue is the ongoing increase in the prevalence of this disease, attributed to various factors, including societal pressures [3].

At present, the diagnosis of depression primarily relies on clinical interviews conducted by doctors and utilizes standardized rating scales, such as the Hamilton Depression Rating Scale (HDRS) [4]. Additionally, self-assessment tools like the Self-Rating Depression Scale (SDS) [5], the Patient Health Questionnaire (PHQ) [6], and the Baker Depression Inventory (BDI) [7] are commonly employed. However, these methods can be influenced by external factors, including the clinician’s expertise and the patient’s tendency to conceal symptoms. Consequently, an accurate automatic diagnosis system [8,9,10] can offer patients effective daily prevention and auxiliary assessment, enabling the early detection of depression and reducing medical costs and societal impact [11].

After the Audio–Visual Emotion Challenge and Workshop (AVEC) [12], numerous scholars have embarked on studying the utilization of text, audio, and visual modalities for depression prediction. According to related studies, facial expressions, vocal features, and semantic features play distinct roles in conveying emotional information [13,14,15]. Facial expressions have shown high reliability and consistency, accounting for 55% of the emotional information. Voice characteristics convey 38% of emotional information, but they are susceptible to interference from distractions such as noise, accents, and background music. In comparison, semantic features, which are emotional expressions based on text analysis, convey only 7% of the emotional information. Due to the influence of language habits, spoken language, abbreviations, and other factors, the processing of semantic features is relatively complex. Considering these factors, in the context of depression recognition algorithms, feature extraction and classification recognition based on video modality data prove to be highly effective. This approach holds the potential for further promotion and application in medical equipment.

In this paper, we present a framework based on visual temporal modality for depression detection. We extract features from patients’ faces in spatial and temporal dimensions using facial landmarks. To address the issue of recognition capability in single-modal depression detection models, we design a feature extraction method that exhibits a strong robustness and low dimensionality. Finally, we utilize the DAIC-WOZ [12] dataset to construct and validate the efficacy of our proposed approach.

The main contributions of this paper are as follows:

(1)

This paper introduces a novel feature extraction method based on facial angle. The method produces facial features that possess translation invariance and rotation invariance. Additionally, it counteracts the interference caused by patients’ head and limb movements through flipping correction, resulting in highly robust extracted features.

(2)

A new model is developed by combining GhostNet with multi-layer perceptron (MLP) modules and video process headers. The model is fine-tuned with respect to the features extracted using the proposed method in this paper. This novel model offers a valuable addition to the depression classification task.

(3)

Extensive experiments are conducted on the DAIC-WOZ dataset to validate the feasibility of the proposed method. The results obtained outperform those of similar methods, establishing the efficacy of the proposed approach.

2. Related Works

Facial expressions and facial movements have a significant impact on nonverbal communication [16,17,18,19]. They reflect an individual’s mental and emotional state and are less influenced by personal subjectivity. Conversely, individuals with depression exhibit distinct facial expressions and movements, typically characterized by frowning, drooping eyes, downturned corners of the mouth, and diminished smiles [20,21]. Analytical modelling based on facial expressions and movements proves to be highly effective in the field of depression recognition.

In the study of facial features, many scholars utilize the Facial Action Coding System (FACS) as the foundation for constructing features. Cohn et al. [22] employed the Active Appearance Model (AAM) to establish 68 facial landmark points, which were subsequently utilized in the classification process. Features such as distance, angle, and area between these landmark points were extracted for analysis. Mclntyre et al. [23] defined facial activity units (AUs) as distinct regions and examined the facial expressions of patients with depression using both original AUs and the defined AU regions. They obtained facial activity patterns based on these analyses. Hamm et al. [24] developed an automated FACS system that calculated the frequency of individual AUs and combined AUs in videos. This system was applied to analyze facial expressions in depressed patients.

Yang et al. [25] utilized a hidden Markov information fusion method to extract the cross-correlation of AU features in a time information sequence, demonstrating its advantages and effectiveness in detecting emotional disorders. Gupta et al. [26] chose structural features, including head movement, eyes opening and closing, the average distance between the upper and lower eyelids, and the distance from the mouth to specific landmarks. Nasir et al. [27] examined facial landmarks as a time series, investigating displacement, velocity, and acceleration as motion-related features, and attained the highest performance indicators in the AVEC2016 competition.

Pampouchidou et al. [17] encoded facial landmark features into grayscale images, generating the Landmark Motion History Image (LMHI) of facial landmarks. White pixels represent the most recent motion, while the darkest grey represents the earliest motion. Grayscale image extraction was performed using Local Binary Patterns (LBPs) and Histogram of Oriented Gradients (HOGs) features. Wang et al. [28] proposed Sampling Sliced Long Short-Term Memory Multiple Instance Learning (SS-LSTM-MIL), a multi-instance learning method that utilizes two-dimensional facial key points as input. This approach employs LSTM and pooling techniques for detecting facial depression in specific video segments. Sun et al. [29] presented the Robust Adversarial Adaptation (SRADDA) method, which addresses domain adaptation across varying scales, label spaces, and depression severity assessments in deep evaluation networks.

The previous studies demonstrate that these facial features rely on a human clinical understanding of depression. These features are derived by quantifying geometric properties like distance, angle, and area to analyze the facial changes exhibited by individuals with depression. Dynamic information regarding features is represented through the calculation of speed and acceleration for each motion frame. Nevertheless, this method of feature extraction leads to the loss of both the topological structure information of the face and additional features that go beyond clinical understanding.

This paper presents a framework for depression detection based on visual temporal modality. Facial landmarks are utilized to extract features from patients’ faces in both spatial and temporal dimensions. A feature extraction method is then designed to classify depression, addressing the limitation of single-modal depression detection models. The method exhibits strong robustness and low dimensionality.

3. Proposed Framework

The framework is comprised of three main components: feature extraction and training unit acquisition, GhostNet neural classification network, and aggregation classification. As shown in Figure 1. Each of these components will be introduced in detail in Section 3 and Section 4, respectively.

This article establishes two concepts regarding angles for the ease of subsequent discussions:

Definition 1.

Target angle: In the target plane, the angle between two straight lines is called “angle” for short.

Definition 2.

Target image angle: In the image plane, the angle between two straight lines is called “image angle” for short.

3.1. Facial Landmark Detection

Facial landmark detection is a computer vision technique [30] that automatically detects the locations of specific points (e.g., eyes, nose, mouth) in facial images. These specific points are commonly referred to as facial landmarks or face landmarks.

This paper utilizes the facial landmark detection method proposed by Tadas et al. [29,30]. The method establishes 68 landmark points on the human face, comprising 17 points for face shape, 10 points for eyelashes, 12 points for the nose, and 18 points for the mouth. These 68 landmark points provide a more intuitive outline of the human face model. Figure 2 illustrates the impact of facial landmark detection.

3.2. Angle Selection Method

The fundamental principle of feature selection is to accurately capture the angle of expression change. This paper selects the following angles as the features representing facial expression changes, based on this principle.

The face is analyzed using the key point detection method described in Section 3.1, resulting in the extraction of 68 marker point positions. The 68 landmarks are assigned numbers ranging from 1 to 68. The numbering sequence is provided in

Table 1. Angle of facial landmarker.

Sequence Number	Coordinate Definition	Meaning
${\mathrm{A}}_1$	Ang(18,20,22)	Curvature of the left eyelash
${\mathrm{A}}_2$	Ang(23,25,27)	Curvature of the right eyelash
${\mathrm{A}}_3$	Ang(20,31,25)	Grain lift 1
${\mathrm{A}}_4$	Ang(20,28,25)	Grain lift 2
${\mathrm{A}}_5$	Ang(38,37,42)	Degree of left eye corner opening
${\mathrm{A}}_6$	Ang(37,20,40)	Degree of left eye squinting
${\mathrm{A}}_7$	Ang(47,46,45)	Degree of right eye corner opening
${\mathrm{A}}_8$	Ang(43,25,46)	Degree of right eye squinting
${\mathrm{A}}_9$	Ang(49,03,37)	Degree of left cheek contraction
${\mathrm{A}}_{10}$	Ang(55,15,46)	Degree of right cheek contraction
${\mathrm{A}}_{11}$	Ang(51,49,59)	Degree of left outer lip opening
${\mathrm{A}}_{12}$	Ang(62,61,68)	Degree of left inner lip opening
${\mathrm{A}}_{13}$	Ang(53,55,57)	Degree of right outer lip opening
${\mathrm{A}}_{14}$	Ang(64,65,66)	Degree of right inner lip opening
${\mathrm{A}}_{15}$	Ang(49,34,55)	Degree of mouth opening
${\mathrm{A}}_{16}$	Ang(28,02,31)	Nose wrinkle 1
${\mathrm{A}}_{17}$	Ang(28,16,31)	Nose wrinkle 2
${\mathrm{A}}_{18}$	Ang(Le,31,Re)	Upside down
${\mathrm{A}}_{19}$	Ang(01,30,03)	Right deflection
${\mathrm{A}}_{20}$	Ang(17,30,15)	Left deflection

.

In feature ${\mathrm{A}}_{18}$, Le = mean(37~42) represents the centroid position of the left eye, while Re = mean(43~48) represents the centroid position of the right eye. Here, mean(∙) denotes the function for calculating the average value.

Each feature carries a distinct significance and can partially depict the variations in facial expressions. For instance, when individuals smile, their eyes contract into a line, while they widen when experiencing fear. The selected features ${\mathrm{A}}_5$ to ${\mathrm{A}}_8$ in this study are capable of reflecting these changes in facial expressions.

3.3. Robustness Analysis of Angle Features

Facial expressions primarily manifest through variations in angle information. For instance, a smile is characterized by an upward movement of the corners of the mouth, while intense crying involves a downward movement of the corners of the mouth. In this paper, we extract features ${\mathrm{A}}_{11}$ to ${\mathrm{A}}_{14}$ to precisely capture these expression changes. Human head movement can be decomposed into six components: horizontal translation (x-axis), vertical translation (y-axis), depth translation (z-axis), yaw rotation (left/right tilt), pitch rotation (head up/down), and roll rotation (left/right head). It is important to investigate whether these six actions have an impact on our angle features and if the features extracted in this study effectively withstand the influence of these actions, thus demonstrating their robustness. The subsequent section presents an analysis and discussion on the robustness of these features.

3.3.1. Rotation Invariance

The target undergoes rotation along the z-axis, as depicted in Figure 3. Patients frequently tilt their heads to the left and right during the recording of the interrogation video. Consequently, the tilting of the patient’s head to the left and right can be considered facial rotation.

Theorem 1.

Let ∆ABC be an image on the target surface π after camera imaging, and let ∆A′B′C′ be its corresponding image. When the target rotates along the z-axis, the angles ∠A′B′C′, ∠B′A′C′, and ∠B′C′A′ remain unchanged. This property is referred to as the rotation invariance theorem.

Proof of Theorem 1.

The rotation of the target around the z-axis is equivalent to the rotation of the camera around the optical axis. Figure 3a is the imaging model when the camera rotates around the optical axis, and Figure 3b is the imaging model when the target rotates around the z-axis. By establishing the camera space coordinate system $OXYZ$, as shown in Figure 3a, let $A_1{B}_1$ be the image of target $AB$ in image plane ${\pi }_1$. When the camera rotates to shoot, although the image $A_1{B}_1$ of the target $AB$ will change relative to the image coordinates, the length |$A_1{B}_1$| of $A_1{B}_1$ will not change. This is shown in the vignette in Figure 3a. Thus, the following conclusions can be drawn: in the process of target rotation, the length of the image obtained by obliquely shooting the target at any angle will not change.

Let the image of target $ABC$ be $A^{\prime }{B}^{\prime }{C}^{\prime }$; when the target point $ABC$ rotates around the z-axis, a new target $A_1{B}_1{C}_1$ is formed, and correspondingly, there is a new corresponding image $A_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$ on the image plane. According to the theory of target rotation around the z-axis, there is the following Formula (1):

$$\left\{\begin{array}{c}\left|A_1{B}_1\right|=\left|A_1^{\prime }{B}_1^{\prime }\right|\\ \left|A_1{C}_1\right|=\left|A_1^{\prime }{C}_1^{\prime }\right|\\ \left|C_1{B}_1\right|=\left|C_1^{\prime }{B}_1^{\prime }\right|\end{array}\right.$$

(1)

That means $\mathsf{∆}{A}^{\prime }{B}^{\prime }{C}^{\prime }\cong \mathsf{∆}{A}_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$. Therefore, the image angles $\angle A^{\prime }{B}^{\prime }{C}^{\prime }$, $\angle B^{\prime }{A}^{\prime }{C}^{\prime }$, $\angle B^{\prime }{C}^{\prime }{A}^{\prime }$ will not change with the change of the target rotation. □

3.3.2. Translation Invariance

Translation involves the parallel displacement of a target point along the X, Y, or Z-axes within the OXYZ plane. In the context of a patient’s medical examination, it refers to bodily movements such as flexion, extension, lateral shifting, as well as anterior and posterior shifting.

Theorem 2.

Let the image of any $\mathsf{∆}ABC$ in the target surface $\pi$ after the camera imaging be $\mathsf{∆}{A}^{\prime }{B}^{\prime }{C}^{\prime }$; when the target $\mathsf{∆}ABC$ translates arbitrarily along the x-axis, y-axis, or z-axis, the image angles $\angle A^{\prime }{B}^{\prime }{C}^{\prime }$, $\angle B^{\prime }{A}^{\prime }{C}^{\prime }$, $\angle B^{\prime }{C}^{\prime }{A}^{\prime }$ do not follow the target changes in position. Hence, this property is referred to as the translation invariance theory.

Proof of Theorem 2.

Establish the coordinate system shown in Figure 4, set the image of triangle $ABC$ in image plane ${\pi }_1$ as $\mathsf{∆}{A}^{\prime }{B}^{\prime }{C}^{\prime }$, move triangle $ABC$ to $A_1{B}_1{C}_1$ along the y-axis, and the image of triangle $A_1{B}_1{C}_1$ is $A_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$.

According to the principle of pinhole imaging, triangle $ABC$ is similar to triangle $A^{\prime }{B}^{\prime }{C}^{\prime }$. The following Formula (2) represents this relationship:

$$\mathsf{∆}ABC\approx \mathsf{∆}{A}^{\prime }{B}^{\prime }{C}^{\prime }$$

(2)

When triangle $ABC$ moves to $A_1{B}_1{C}_1$ along the y-axis, triangle $ABC$ will not be deformed due to parallel movement, so there is a congruent relationship between triangle $ABC$ and triangle $A_1{B}_1{C}_1$. There is the following Formula (3):

$$\mathsf{∆}ABC\cong \mathsf{∆}{A}_1{B}_1{C}_1$$

(3)

Triangle $A_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$ is the image of triangle $A_1{B}_1{C}_1$, and so, there is the following Formula (4):

$$\mathsf{∆}{A}_1{B}_1{C}_1\approx \mathsf{∆}{A}_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$$

(4)

Then, it can be concluded that triangle $A^{\prime }{B}^{\prime }{C}^{\prime }$ and triangle $A_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$ have a similar relationship. There is the following Formula (5):

$$\mathsf{∆}{A}^{\prime }{B}^{\prime }{C}^{\prime }\approx \mathsf{∆}{A}_1^{\prime }{B}_1^{\prime }{C}_1^{\prime }$$

(5)

Thus, the image angles $\angle A^{\prime }{B}^{\prime }{C}^{\prime }$, $\angle B^{\prime }{A}^{\prime }{C}^{\prime }$, and $\angle B^{\prime }{C}^{\prime }{A}^{\prime }$ do not change as the target moves along the y-axis. Similarly, when the target moves along the x-axis and z-axis, the image angle will not change either. □

3.4. Flip Correction

Human head movements commonly involve shaking the head laterally (left and right deflection) as well as tilting it upward and downward (referred to as pitch movement). The lateral deflection (roll) and pitch movement have a significant impact on the angular properties, consequently influencing facial expression recognition. Angle correction is essential to address changes in angular properties caused by target flipping. To achieve accurate flipping correction, it is crucial to comprehend the relationship between the flipping angle, target angle, and image angle in the two given scenarios.

3.4.1. When the Target Is Flipped Horizontally, There Exists a Relationship among the Target Angle, Image Angle, and Flip Angle

The target’s left and right deflection, known as “Roll”, indicates rotation around the y-axis or a parallel straight line. Deflection of the target surface in the left and right directions causes a change in the image angle.

First, establish a space coordinate system $XFY$ as shown in Figure 5, where the point $F$ is the optical center of the camera. Assuming that $AB$ is parallel to the y-axis, and the image of $AB$ is $A^{\prime }{B}^{\prime }$, in the image plane ${\pi }^{\prime }$, establish the $u{O}^{\prime }v$ plane coordinate system, where $u$ and $v$ represent the horizontal axis and the vertical axis, respectively. Obviously, the $A^{\prime }{B}^{\prime }$ and $v$ axes are parallel. Suppose the intersection point of the target and the main optical axis of the camera is $O$, and a plane ${\pi }^{\prime }$ parallel to the image plane ${\pi }_1$ is made through point $O$, connect $FA$ and extend ${\pi }^{\prime }$ to intersect $A_1$, connect $FB$ and extend ${\pi }^{\prime }$ to intersect $B_1$. $A^{\prime }{B}^{\prime }$ intersects $u$ axis at $D^{\prime }$, connects $D^{\prime }F$ and extends, and intersects $AB$, $A_1{B}_1$ at $D$, $D_1$, respectively. Connect $OD$ and $O{D}_1$. In plane $XFZ$, a straight line parallel to the z-axis passing through point $D$ intersects $O{D}_1$ at point $D_2$, and a straight line drawn through point $D$ along $O{D}_1$ intersects the z-axis at $D_3$.

Because ${\pi }^{\prime }$ is parallel to ${\pi }_1$, there is the following Formula (6):

$$\angle A^{\prime }{O}^{\prime }{B}^{\prime }=\angle A_{1}O{B}_1$$

(6)

Let $\mathrm{\angle }{A}_{1}O{B}_1=\varphi ,\mathrm{\angle }{A}_{1}O{D}_1={\varphi }_1,\mathrm{\angle }{B}_{1}O{D}_1={\varphi }_2$, $\mathrm{\angle }AOB=\theta ,\mathrm{\angle }AOD={\theta }_1,\mathrm{\angle }BOD={\theta }_2$. and ${\varphi }_1+{\varphi }_2=\varphi$, ${\theta }_1+{\theta }_2=\theta$. Set the distance between point $O$ and the optical center of the camera as $m$, that is, $\left|FO\right|=m$. Also, set the distance from point $O$ to line segment $AB$ as $d$, that is, $\left|OD\right|=d$. Suppose the length of line segment $AB$ is $c$, the length of $AD$ is $c_1$, and the length of $BD$ is $c_2$, then there is $c_1+c_2=c$. Let $\alpha$ be the angle between plane $\mathsf{∆}OAB$ and plane ${\pi }_1$. Since $D{D}_2\perp O{D}_1$, then there is the following Formula (7):

$$\left\{\begin{array}{c}\left|D{D}_2\right|=d\times \mathit{sin}\alpha \\ \left|D{D}_3\right|=d\times \mathit{cos}\alpha \end{array}\right.$$

(7)

$\because$: $\mathsf{∆}DF{D}_3\approx \mathsf{∆}{D}_{1}FO,\therefore$:

$$\left|O{D}_1\right|=\frac{m\times d\times \mathit{cos}\alpha }{m-d\times \mathit{sin}\alpha }$$

(8)

$\because$: $\mathsf{∆}DFA\approx \mathsf{∆}{D}_{1}F{A}_1,\therefore$:

$$\left|A_1{D}_1\right|=\frac{c_1\times m}{m-d\times \sin \alpha }$$

(9)

In $\mathsf{∆}{D}_{1}O{A}_1$, there is the following Formula (10):

$$\tan {\varphi }_1=\frac{\left|A_1{D}_1\right|}{\left|O{D}_1\right|}=\frac{c_1}{d\times \mathit{cos}\alpha }$$

(10)

$\because$: $\mathit{tan}{\theta }_1=\frac{c_1}{d},\therefore$:

$$\mathit{tan}{\varphi }_1=\frac{\mathit{tan}{\theta }_1}{\mathit{cos}\alpha }$$

(11)

In the same way, there is the following Formula (12):

$$\mathit{tan}{\varphi }_2=\frac{\mathit{tan}{\theta }_2}{\mathit{cos}\alpha }$$

(12)

To sum up, the relationship between the angle in the target image (referred to as the image angle) $\angle A^{\prime }{O}^{\prime }{B}^{\prime }$, the target angle $\angle AOB$, and the deflection angle $\alpha$ can be obtained:

$$\left\{\begin{array}{c}\varphi ={\varphi }_1+{\varphi }_2\\ \begin{array}{c}\mathit{tan}\varphi =\frac{\mathit{tan}{\theta }_1}{\mathit{cos}\alpha }\\ \mathit{tan}{\varphi }_2=\frac{\mathit{tan}{\theta }_2}{\mathit{cos}\alpha }\\ \theta ={\theta }_1+{\theta }_2\end{array}\end{array}\right., \alpha \in \left[-\pi /2,\pi /2\right]$$

(13)

It can be obtained from Formula (13) that when the target $\mathsf{∆}AOB$ moves around the point $O$ parallel to the y-axis, its image angle $\varphi$ has no relevance with the distance $m$ between the target point $O$ and the optical center of the camera. The relationship between $\varphi$, $\theta$, and $\alpha$ is shown in Figure 6.

It can be determined from Figure 6 that when the face is not deflected, that is, $\alpha =0$, the target angle $\theta$ and image angle $\varphi$ will be equal in value. And when the face is deflected, $\alpha \ne 0$. The target angle $\theta$ will not be the same as the image angle $\varphi$, and the image angle $\varphi$ will definitely be greater than the target angle $\theta$. The flip action then disturbs the angular characteristics. To mitigate this interference, the approach involves capturing a frontal shot of the face, extracting relevant values at specific angles that remain unaffected by facial expression changes, and subsequently applying corrections to other face frames using Formula (13). Detailed information on obtaining a front face photo can be found in Section 3.5.1.

3.4.2. When the Target Is Flipped Vertically, There Exists a Relationship among the Target Angle, Image Angle, and Flip Angle

The target flips up and down (pitch), which means that the target rotates around the x-axis or a line parallel to the x-axis. When the target flips up and down, the image angle will change.

Due to the symmetry of camera imaging, there is symmetry between left and right deflections and up and down flips; thus, simply swap the x-axis and y-axis in Figure 5 to obtain the relationship between the target angle, image angle, and flip angle when flipping up and down.

Let the flip angle be $\beta$, and, according to the symmetry, the relationship between them can be obtained:

$$\left\{\begin{array}{c}\varphi ={\varphi }_1+{\varphi }_2\\ \begin{array}{c}\mathit{tan}{\varphi }_1=\frac{\mathit{tan}{\theta }_1}{\mathit{cos}\beta }\\ \mathit{tan}{\varphi }_2=\frac{\mathit{tan}{\theta }_2}{\mathit{cos}\beta }\\ \theta ={\theta }_1+{\theta }_2\end{array}\end{array}\right., \beta \in \left[-\pi /2,\pi /2\right]$$

(14)

In Formula (14), $\varphi$ and $\theta$ have the same meaning as Formula (19).

3.4.3. Deflection/Flip Correction Method

From Formulas (13) and (14), it can be seen that if the target angle can be known in advance, then the deflection/flip angle can be solved. In this paper, ${\mathrm{A}}_{18}$ is used for flip correction, and either ${\mathrm{A}}_{19}$ or ${\mathrm{A}}_{20}$ is used for deflection correction.

For angle ${\mathrm{A}}_{18}\left(\mathrm{A}\mathrm{n}\mathrm{g}\right(\mathrm{L}\mathrm{e},31,\mathrm{R}\mathrm{e}\left)\right)$, point $\mathrm{L}\mathrm{e}$ is the center point of the left eye, point 31 is the tip of the nose, and point $\mathrm{R}\mathrm{e}$ is the center point of the right eye. These three points will not move with the change of expression, which means that the angle information formed by these three points does not become larger or smaller with facial expressions, but does increase or decrease in size accordingly with flipping up and down. Therefore, flip correction can be performed by means of this one feature.

Suppose there is a frame of frontal image $I$, then let ${Ang}_I\left(Le,31,Re\right)=\theta$, and then $\theta$ is calculated by Formula (15):

$$\theta =\angle A^{\prime }{C}^{\prime }{B}^{\prime }={\mathit{cos}}^{-1}\left(\frac{{\left|B^{\prime }{C}^{\prime }\right|}^2+{\left|A^{\prime }{C}^{\prime }\right|}^2-{\left|A^{\prime }{B}^{\prime }\right|}^2}{2\left|B^{\prime }{C}^{\prime }\right|\left|A^{\prime }{C}^{\prime }\right|}\right)$$

(15)

Another frame of image $I_1$ is set, so ${\mathrm{A}\mathrm{n}\mathrm{g}}_{{\mathrm{I}}_1}\left(\mathrm{L}\mathrm{e},31,\mathrm{R}\mathrm{e}\right)=\varphi$ can be calculated by Formula (15), Similarly, the flip angle $\beta$ of the image $I_1$ relative to the reference frame $I$ can be calculated by Formula (14).

When the flip angle $\beta$ is obtained, the error caused by the flip of some features can be corrected. It is advisable to set the angle to be corrected as $A_i^{UD}$, then $A_i^{UD}$ is calculated by Formula (16):

$$A_i^{UD}={\mathit{tan}}^{-1}\left(\mathit{cos}\beta \times \mathit{tan}\left(A_i\right)\right)$$

(16)

$A_i$ in Formula (16) is the image angle, and the image angle can be obtained by Formula (15).

In the same way, angle ${\mathrm{A}}_{19}$ or ${\mathrm{A}}_{20}$ can be used to correct the left and right deflection of some angles that need deflection:

$$A_i^{LR}={\mathit{tan}}^{-1}\left(\mathit{cos}\alpha \times \mathit{tan}\left(A_i\right)\right)$$

(17)

3.5. Feature Extraction and Training Unit

Figure 1 illustrates the initial stage of the framework, which involves feature extraction and training unit acquisition. Feature extraction in the first stage is divided into two steps: (1) the selection of a reference frame and (2) the extraction of angle features using difference calculation. The acquisition of training units, i.e., multiple instances of a video, is accomplished using the active Windows mechanism. Subsequently, the feature extraction process is described.

3.5.1. Determine the Reference Frame

The reference frame image must have the patient’s face plane parallel to the image plane, that is, the frontal photo of the patient’s face. Although it may not be possible to find a frame with the face plane parallel to the image plane during the patient’s consultation process, it is necessary to identify the frame in the diagnosis video that is closest to the parallel image plane. In this scenario, we simply identify the frame image closest to the image plane and use it as the reference frame image.

From formulas (13) and (14), it can be known that when the head does not deflect and flip, that is, $\alpha =\beta =0$, the face plane and the image plane are parallel, and the image angle $\varphi$ is exactly equal to the target angle $\theta$. When there is a deflect and flip situation, the image angles ${\mathrm{A}}_{18}$, ${\mathrm{A}}_{19}$, and ${\mathrm{A}}_{20}$ will decrease as well. Therefore, only one frame of image needs to be found so that Formula (18) holds:

$$I_j=\underset{I_j\left(j=1,2,\cdots ,n\right)}{\max }\left({\mathrm{A}}_{18}+{\mathrm{A}}_{19}+{\mathrm{A}}_{20}\right)$$

(18)

When a reference frame satisfying Formula (18) is found, use Formula (18) to calculate all $A_1~A_{20}$ in Table 1, denote it as $A_{ref,1}~A_{ref,20}$, and save it. Among them, the three angles of ${\mathrm{A}}_{18}$, ${\mathrm{A}}_{19}$, and ${\mathrm{A}}_{20}$ will be used as hyperparameters in the flip correction of the feature extraction process (Figure 7); n is the number of video frames.

3.5.2. Difference Extraction of Angle Features

In order to accurately describe changes in expression and mitigate misjudgment caused by deflection and flip actions, the angle feature extraction scheme aims to minimize their influence. To achieve this objective, Figure 7 illustrates the designed process for angle feature extraction in this paper.

In Figure 7, the “Determine the reference frame” step yields certain correction parameters, which are subsequently utilized for flip correction in the feature extraction process, as depicted in step 1 below. And in Figure 7, The “feature extraction” step must execute the following specific operations:

(1)

Flip correction: Use Formulas (16) and (17) to correct the deflection and flip of the angle of this frame, and obtain $A_i^{\prime }=0.5\times \left(A_i^{\mathrm{U}\mathrm{D}}+A_i^{\mathrm{L}\mathrm{R}}\right),i=1,2,\cdots ,20$.

(2)

Feature calculation: Calculate the expression difference between the two frames before and after; then, the feature $F_{j,i}$ is calculated by:

$$F_{j,i}=A_{j,i}^{\prime }-A_{j-1,i}^{\prime }, i=1,2,\cdots ,20, j=1,2,\cdots ,n-1$$

(19)

n in Formula (19) is the total number of frames that can detect expressions.

Following the above feature extraction process, we obtain the original input data needed for the experiment from the video. The subsequent step involves partitioning the original feature data into training units.

3.5.3. Training Unit Acquisition

A training unit refers to the extraction of several instances from a single piece of datum, where each instance serves as a representative of that particular datum. For instance, in the case of experimental data obtained from a subject without mental illness, the majority of instances are likely to be non-mental illness in nature. In this paper, a sliding window is employed for reference, enabling the extraction of multiple instances on a larger scale. This technique transforms one sample into multiple instances of the same scale. Regarding the size of the sliding window, we specify the parameters in Section 4.3. of the experimental dataset introduction.

4. Experimental Studies

4.1. GhostNet

GhostNet [31] is a lightweight neural network with several key features. Firstly, it demonstrates a high efficiency when compared to other neural network structures like ResNet [32] and MobileNetV2 [33]. GhostNet achieves this through faster processing, fewer parameters, and reduced calculations. Secondly, GhostNet utilizes group convolution technology, dividing the convolution kernel into smaller groups. This approach reduces parameter size and calculation requirements for each convolution layer, resulting in an improved operational efficiency. Thirdly, GhostNet incorporates a channel attention mechanism. This involves adding a channel attention module after each convolutional layer to dynamically adjust the weight of each channel. This adjustment enhances the network’s learning capability. Lastly, GhostNet introduces a feature reorganization mechanism. This involves rearranging the feature maps generated by different convolutional layers to further enhance training efficiency and network accuracy.

The Ghost Module (GM) [31] in Figure 8 consists of three steps, namely conventional convolution, Ghost generation, and feature map splicing.

(a)

First, by providing the input data $X\in {\mathbb{R}}^{c\times h\times w}$, use the convolution operation to obtain the intrinsic feature map $Y^{\prime }\in {\mathbb{R}}^{h^{\prime }\times w^{\prime }\times m}$. Where $c$ is the number of input channels, $h$ and $w$ are the height and width of the input data.

$$Y^{\prime }=X\ast f^{\prime }$$

(20)

where $f^{\prime }\in {\mathbb{R}}^{c\times k\times k\times m}$ is the convolution kernel, $k\times k$ is the size of the convolution kernel, and the bias term is omitted.

(b)

In order to obtain the required Ghost feature map, each feature map $y_i^{\prime }$ of $Y^{\prime }$ is used to generate a Ghost feature map with operation ${\Phi }_{i,j}$.

$$y_{ij}={\Phi }_{i,j}\left(y_i^{\prime }\right), \forall i=1,...,m,j=1,...,s,$$

(21)

(c)

The intrinsic feature map obtained in the first step and the Ghost feature map obtained in the second step are spliced (identity connection) to obtain the output of the Ghost module.

Inspired by the work presented in [31,34,35,36,37], we made changes to GhostNet based on the following factors: (1) In our video classification task, the initial input channels vary from traditional images. For images, the initial channel number is 3, representing the RGB channels. However, in this paper, the initial channel number corresponds to the total number of each feature angle designed in Section 3.1. (2) While largely preserving the original GhostNet structure, this section selectively modifies the second part of GhostNet’s structure, specifically the GhostNet Bottleneck, with the objective of employing GhostNet as a bottleneck graph to extract the network’s backbone. (3) The fourth part of GhostNet, which comprises an FC neural classification network, has been removed and replaced with a 4-class MLP [32] structure for classification purposes. The revised network structure is depicted in Figure 8, and the video processing head is used to shape the unit to fit the GhostNet input.

In addition, we have included

Table 2. Parameters of GhostNet network.

Input	Operator	#Exp	#Out	SE	Stride
642 × 20	Conv2d 3 × 3	-	84	-	2
322 × 84	G-bneck	120	84	1	1
322 × 84	G-bneck	240	84	0	2
162 × 84	G-bneck	200	84	0	1
162 × 84	G-bneck	184	84	0	1
162 × 84	G-bneck	184	84	0	1
162 × 84	G-bneck	480	112	1	1
162 × 112	G-bneck	672	112	1	1
162 × 112	G-bneck	672	160	1	2
82 × 160	G-bneck	960	160	0	1
82 × 160	G-bneck	960	160	1	1
82 × 160	G-bneck	960	160	0	1
82 × 160	G-bneck	960	160	1	1
82 × 160	Conv2d 1 × 1	-	960	-	1
12 × 960	AvgPool 8 × 8	-	-	-	-

, which presents data to facilitate the understanding of the modifications made in GhostNet. The current GhostNet network consists of approximately 2.7 M parameters, and the flops (floating-point operations) are around 1.0 G. The specific parameters in the network structure depicted in Figure 8 are listed in Table 2.

Table 2 presents the following designations: G-bneck refers to the GhostBottleneck module, #exp represents the expansion distance, #out signifies the number of output channels, and SE indicates the usage of the SE module.

4.2. Aggregate Classification

In the PHQ-8 scale [38], the scoring is as follows: 0–4 points indicate no depression, 5–9 points suggest mild depression, 10–14 points indicate moderate depression, and a score of 15 or higher indicates moderate to severe depression.

The four types of pseudo-labels (1, 2, 3, and 4) correspond to different PHQ-8 scores, denoted as ${\mathrm{P}\mathrm{H}\mathrm{Q}}_{\mathrm{s}}$. The general depression consultation data will only provide tags corresponding to the entire duration of the long video. To transform the depression classification task of long videos into the score classification task of segmented short time series units, a new label must be created for each unit.

Based on depression-related research findings, the short-sequence units corresponding to pseudo-label 4 have been found to be more significant than those corresponding to pseudo-labels 1, 2, and 3 in assessing depression. However, the frequency of facial changes corresponding to label 4 is lower throughout the entire video. Thus, we developed a score aggregation function for adjusting the weights of various pseudo-labels during the aggregation process.

Assuming that video $V$ has $l$ short time series units, the model output label of the $i\mathrm{t}\mathrm{h}(0<i\le l)$ short time series unit is denoted as $v_n$. All short sequential units can be expressed as $V=\left\{v_1,v_2,v_3,...,v_l\right\}$. Then, the expression of the score aggregation function is as follows:

$${\widehat{s}}_V=\frac{1}{l}\sum _{i=1}^{i=l}\left\{\begin{array}{c}{\rho }_1\times v_i (if v_i=1)\\ {\rho }_2\times v_i (if v_i=2)\\ {\rho }_3\times v_i (if v_i=3)\\ {\rho }_4\times v_i (if v_i=4)\end{array}\right.$$

(22)

For the weight parameters ${\rho }_1$, ${\rho }_2$, ${\rho }_3$, and ${\rho }_4$, we make selections via the differential evolution algorithm [39]. Finally, we determine whether the corresponding patient suffers from depression by judging whether the aggregated score $\widehat{s}$ is greater than the set threshold.

4.3. Experimental Dataset

The DAIC-WOZ dataset [12] is mainly video, audio, and questionnaire survey data collected to diagnose mental disorders such as depression and anxiety. This dataset is referenced in the Depression Recognition Task Module in the 2014 and 2016 AVEC competitions. The interviews are conducted through situational dialogues between the virtual interviewer, Ellie, and the subjects to elicit corresponding emotional responses and record the characteristics in real-time. According to official standards, the entire dataset exposes 189 sample data and is divided into 107 training sets, 35 validation sets, and 47 test sets. Since no official public label indicates whether the subjects in the test set are depressed or not, this paper employs only 142 sample data from the combined training and test sets to ensure the accuracy and validity of the experimental results. Moreover, to safeguard privacy, the DAIC-WOZ dataset does not provide the original video data but instead offers the 2D coordinate data of facial landmarks extracted by OpenFace [30]. Therefore, the experiment in this paper directly utilizes the provided coordinate data of facial landmarks, eliminating the need for facial landmark extraction steps. The detailed classification of depression or non-depression in the sample data is shown in

Table 3. This table presents the sample distribution of the DAIC-WOZ dataset [12].

Datasets	Non-Depression	Depression
Training set	77	30
Test set	23	12

.

In this paper, the DAIC-WOZ dataset was utilized, and a sliding window approach was implemented to sample the original data. The window size was set to 120 s, with an overlap size defined as 1/6 of the window size. The original data consisted of video samples at a rate of 30 Hz. During the data cleaning process, a filter was applied to exclude video frames with a face confidence lower than 0.7 from the original video. This filtering process aimed to mitigate the influence of significant face deflection or flipping on the experimental results.

4.4. Experimental

4.4.1. Evaluation

In this paper, the GhostNet network is chosen as the feature extraction network and modified to suit the input. An MLP [36] is employed for four-class classification. The original video data consist of 20 feature angles across 3600 frames, which are reshaped into a 60 × 60 × 20 input format. The learning rate plays a crucial role in optimizing neural networks. For this experiment, the Adam optimizer [40] is utilized. The initial learning rate is set to 0.001 and reduced to 0.1 times the original rate at the 200th and 400th epochs, respectively. $\mathrm{S}\mathrm{o}\mathrm{f}{\mathrm{t}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ is used during training to calculate the loss, mitigating the impact of hard labels on network training. $\mathrm{S}\mathrm{o}\mathrm{f}{\mathrm{t}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ is calculated as follows:

$$\mathrm{S}\mathrm{o}\mathrm{f}{\mathrm{t}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}=\left\{\begin{array}{cc}\left[\mathrm{1,0}\right],\hfill & {PHQ}_s\le 5\\ \left[\left(10-{PHQ}_s\right)/10,0\right],\hfill & 6\le {PHQ}_s<9\hfill \\ \left[0,\left(PH{Q}_s-9\right)/10\right],\hfill & 9\le {PHQ}_s<14\\ \left[\mathrm{0,1}\right],\hfill & PH{Q}_s\ge 14\end{array}\right.$$

(23)

In this paper, precision rate (P), recall rate (R), and F1 value (F-score) are used as the evaluation indicators of sentiment polarity classification. The calculation formula is as follows:

$$\mathrm{P}=\mathrm{T}\mathrm{P}/\left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}\right)$$

(24)

$$\mathrm{R}=\mathrm{T}\mathrm{P}/\left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}\right)$$

(25)

$$\mathrm{F}1=\mathrm{ }(2\times \mathrm{P}\times \mathrm{R})/(\mathrm{P}+\mathrm{R})$$

(26)

4.4.2. Experimental Results Comparison

The methods employed in this paper involve extracting one-dimensional or multidimensional modal features from the patient’s face, voice, and text data. These extracted features are then used for training and verification on various neural networks.

Table 4. Comparison of the impact of video modality versus non-video modality data.

Methods	M	P	R	F1
Manoret et al., 2021 [41]	A	0.64	0.91	0.75
Othmani et al., 2022 [14]	A	0.51	0.50	0.50
Rejaibi et al., 2022 [42]	A	0.69	0.35	0.46
Sun et al., 2019 [13]	T	0.67	0.50	0.57
Dinkel et al., 2020 [43]	T	0.85	0.83	0.84
Arioz et al., 2022 [44]	A + T	0.53	0.57	0.53
Lam et al., 2019 [45]	A + T	0.60	0.75	0.67
Ours	V	0.77	0.83	0.80

Note: M = Modality in the table refers to the modality type. V = Visual refers to the visual modality data. A = Audio refers to the audio modality data. T = Text refers to the text modality data. + refers to the corresponding mixed multimodal.

provides a comprehensive comparison between our developed unimodal depression algorithm based on facial features and previous studies utilizing different modalities. The designed features demonstrate commendable performance in terms of precision rate, recall rate, and F1 score, achieving values of 0.77, 0.83, and 0.80, respectively.

Manoret et al. [41] obtained the result of a mixed model of multiple neural networks, including CNN and LSTM, with a recall rate of 0.91. However, the precision rate of this algorithm is only 0.64, resulting in a final F1 of 0.75, which is 0.05 lower than the one proposed algorithm in this paper. Additionally, the F1 scores of Othmani [14] and Rejaibi et al. [42] in the audio single-mode design algorithm are 0.30 and 0.34 lower, respectively, compared to the performance achieved by the features designed in this paper.

Text is considered unimodal due to its characteristic of containing precise information. The F1 score of the text single-modal feature proposed by Sun et al. [13] is 0.23 lower compared to this paper. Dinkel et al. [39] and other researchers effectively utilize text features by employing the BGRU neural network for processing text information after performing word embedding (Word2Vec) on the text data, resulting in an F1 value that is 0.04 higher compared to ours. Additionally, when comparing with the multimodal detection algorithms proposed by Arioz [44] and Lam et al. [45], it is evident that the algorithm presented in this paper outperforms them in terms of precision, recall, and F1 scores when utilizing mixed modes.

To objectively evaluate the algorithm proposed in this paper, we compare the video modality used in this paper with other algorithms, including video modality. The comparison results are presented in

Table 5. Comparison between the impact of the video modality itself and the data incorporating the video modality.

Methods	M	P	R	F1
Wang et al., 2020 [28]	V	0.81	0.75	0.78
Song et al., 2018 [46]	V	0.64	0.58	0.61
Wei et al., 2022 [47]	V + A + T	0.89	0.57	0.70
Haque et al., 2018 [48]	V + A + T	0.71	0.83	0.77
Yang et al., 2021 [49]	V + A + T	0.71	0.55	0.63
Ours	V	0.77	0.83	0.80

.

The results presented in Table 5 demonstrate that the features proposed in this study yield remarkable improvements in algorithm performance. Wei et al. [47] achieved a precision rate of 0.89 in their three-modal (visual + audio + text) fusion, surpassing the pure visual mode in this study by 0.12. However, the recall rate was adversely affected, leading to an F1 value 0.1 lower compared to this paper. Haque et al. [49], utilizing the Causal Convolutional Network (C-CNN), achieved a similar recall rate to this study in multimodal analysis. However, their F1 value was 0.03 lower compared to this paper. Conversely, both Wang [28] and Song [46] utilized visual modality data in their studies. In general, the algorithm proposed in this study excels at extracting features from the pure visual modality, leading to superior outcomes.

In the DAIC-WOZ dataset [12], we are provided with several types in terms of visual modal data, including head Pose, eye annotation direction Gaze, and 2D face68 key points.

Table 6. Comparison of the effects of the vision itself.

Feature	M	P	R	F1
Gaze	V	0.85	0.50	0.63
Pose	V	0.72	0.66	0.69
2D landmarks	V	0.69	0.75	0.72
2D Landmarks + Pose	V	0.73	0.91	0.81
2D Landmarks + Gaze	V	0.61	0.91	0.73
Ours Angle	V	0.77	0.83	0.80

presents our method and the results of Guo et al. [50] using the time–space expanded convolutional network (TDCN) and feature attention module (FWA) in different video modalities. Among all the visual modalities. The effectiveness of our method is second only to the approach that utilizes the mixed visual information of facial key points and head pose simultaneously. The F1 values are only 0.1 lower than those achieved by that approach.

As shown in Table 6, when using Gaze data alone, they obtained an F1 value of 0.63, whereas our method achieved a 0.17 higher F1 value. For the experiment using Pose data alone, their F1 value was 0.69, which is 0.11 lower than ours. In the case of 2D Landmark data, they obtained an F1 value of 0.72, which is 0.08 lower than our F1 value. When utilizing both 2D Landmark and Gaze data, our method outperformed theirs by 0.07 in F1 values. While using 2D Landmark and Pose simultaneously, our method achieved an F1 value only 0.01 lower than theirs, but the difference in recall rate and precision rate for our method is only 0.06, whereas theirs shows a difference of 0.18. These results indicate that although our method is slightly lower in F1 value, it exhibits a greater robustness and improved clarity and conciseness.

Our comparative analysis reveals that the features we designed exhibit a superior expressive capability. Additionally, this verification highlights the enhanced robustness of our specifically designed angular features compared to general visual unimodal data.

To determine the optimal window size for capturing angle features effectively, we performed three sets of ablation experiments, as presented in

Table 7. Ablation experiment for optimal sliding window.

Window_Size	P	R	F1
180 s	0.53	0.75	0.62
120 s	0.77	0.83	0.80
60 s	0.56	0.75	0.64

. Three indicators are considered for determining the window size. The optimal result is achieved when using a sliding window of 120 s. Therefore, we selected a sliding window size of 120 s for the experiment.

In order to assess the model optimization [51,52,53], this paper utilizes the ROC scatter plot [54]. As depicted in Figure 9, a total of five models were tested on the DAIC-WOZ [12] dataset: (1) the experiment using the native structure of GhostNet with a sliding window of 3600, referred to as GhostNet; (2) the experiment using the native structure of GhostNet combined with MLP at a sliding window of 3600, referred to as GhostNet+MLP; and (3) the modified GhostNet plus MLP with sliding windows of 5400, 3600, and 1800, referred to as GhostNet + Win1, GhostNet+Best, and GhostNet + Win2, respectively.

According to the nature of the ROC scatter diagram, a model located in the upper-left quadrant is considered better, and the larger the area of the circle representing the model, the higher its optimization level. Based on this, the model with a sliding window of 3600, namely GhostNet+Best, performs the best and serves as the final experimental result of this paper. Both the GhostNet + Win2 and GhostNet + Win1 models are inferior to the GhostNet+MLP model due to the sliding window being set too large and too small, respectively.

5. Conclusions

This paper introduces facial angle features as a novel replacement for the original visual modality data. The theoretical proof is provided to demonstrate that these angle features exhibit translation invariance in the front–back, left–right, and up–down directions, as well as rotation invariance. This illustrates the effectiveness of the proposed method in countering translation and rotation attacks on the head. To address head-up and head-down flips, as well as left and right deflection attacks, this paper presents a corresponding countermeasure scheme that effectively mitigates these attacks. This further demonstrates the robustness of the proposed features.

For classifier selection, this paper presents a GhostNet deep neural network classifier. The use of hard labels (0, 1) for loss training calculations on the network is abandoned in favor of soft-labels. Three sets of comparative experiments are conducted to evaluate the proposed method’s detection performance against other single modes. The experimental results demonstrate the superiority of the proposed method. Furthermore, compared to other multimodal methods, the proposed method outperforms them in certain detection indicators, highlighting its superior feature extraction performance. Additionally, three sets of ablation experiments are performed to investigate the impact of sliding window size. The results verify that a window size of 120 s yields the best effect, achieving an F1-score of 0.80, which surpasses most other single-mode models or mixed modes.

In our future work, we plan to accomplish the following three tasks: (1) construct a larger dataset for depression; (2) investigate more robust feature angles to enhance the performance of our handcrafted features; and (3) explore the fusion of angle features from multiple modalities, including textual speech, with our visual modal data.