Learning Better Representations for Audio-Visual Emotion Recognition with Common Information

Abstract

Audio-visual emotion recognition aims to distinguish human emotional states by integrating the audio and visual data acquired in the expression of emotions. It is crucial for facilitating the affect-related human-machine interaction system by enabling machines to intelligently respond to human emotions. One challenge of this problem is how to efficiently extract feature representations from audio and visual modalities. Although progresses have been made by previous works, most of them ignore common information between audio and visual data during the feature learning process, which may limit the performance since these two modalities are highly correlated in terms of their emotional information. To address this issue, we propose a deep learning approach in order to efficiently utilize common information for audio-visual emotion recognition by correlation analysis. Specifically, we design an audio network and a visual network to extract the feature representations from audio and visual data respectively, and then employ a fusion network to combine the extracted features for emotion prediction. These neural networks are trained by a joint loss, combining: (i) the correlation loss based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation, which extracts common information between audio data, visual data, and the corresponding emotion labels, and (ii) the classification loss, which extracts discriminative information from each modality for emotion prediction. We further generalize our architecture to the semi-supervised learning scenario. The experimental results on the eNTERFACE’05 dataset, BAUM-1s dataset, and RAVDESS dataset show that common information can significantly enhance the stability of features learned from different modalities, and improve the emotion recognition performance.

1. Introduction

Emotion recognition is an important component in affect-related human-machine interaction systems [1,2], as emotion can provide implicit feedback about human experience and conditions that are not easily captured by the explicit input. Audio-visual emotion recognition is a common type of emotion recognition [3,4]. The comprehensive overview can be found in the surveys [5,6,7,8,9,10]. Recent works have successfully applied it for many areas, such as disease diagnosis [11,12], affective tutoring system [13,14], marketing [15,16], and entertainment [17,18]. One challenge of audio-visual emotion recognition is how to extract feature representations with an acceptable size from audio and visual data that are effective for emotion recognition. A number of previous works [19,20,21,22,23,24,25,26,27] have been proposed to tackle this challenge. Although progresses have been made by previous works, they usually suffer from the following two limitations.

First and foremost, these conventional strategies usually cannot efficiently utilize common information between different modalities by correlation analysis. For example, in [19,20,21], the common information is captured by combining the features that are learned from each modality into a feature vector. This technique often fails to exploit the complex dependencies and interactions between different modalities. Motivated by this concern, approaches [22,23,24] that are based on canonical correlation analysis (CCA) and some variant methods, such as kernel probabilistic CCA [25], sparse local discriminative CCA [26], and low-rank representation [27], are proposed. Although these methods have made some performance improvements, they may suffer from numerical issues [28,29]. This instability arises in that they need to use the inverse of the empirical covariance matrix, which easily become singular over some mini-batches. As a workaround, such methods often limit the feature dimensionality to be relatively small in order to ensure stability, which is undesirable for the complex audio-visual emotion recognition task.

Besides, some previous works use heuristic features [22,23,24,25,27] or the features learned from shallow neural network structures [19,20,21,26] for emotion recognition. For example, the widely used audio heuristic features include prosody features and voice quality features [9], while typical visual heuristic features include Gabor features and HOG-TOP features [30]. Recently, convolutional neural networks (CNNs) have become popular for extracting audio and visual features for emotion recognition [19,20,21]. However, due to the high-dimensional emotional data, the learned representations using such methods with shallow structures are not expressive enough to predict emotions, which may lead to limited performance.

To address the above two problems, we propose an deep learning framework for audio-visual emotion recognition by efficiently utilizing common information between audio data, visual data, and the corresponding emotion labels. Figure 1 presents the structure of our system, which satisfies: (i) the highly non-linear correlation of the feature representations among audio data, visual data, and the corresponding labels should be fully analyzed in order to capture the common information, and (ii) the learned audio and visual features should have enough expressiveness to classify the emotions. By considering these two goals together, we can learn the feature representations that are fully discriminative for the emotion recognition task.

Specifically, we design an audio network and a visual network to learn the feature representations from audio and visual data, respectively, and then adopt a fusion network to combine the extracted audio and visual features for emotion prediction. Our neural network is trained by a joint loss function which is the linear combination of correlation loss and classification loss. The correlation loss is used to extract common information between audio data, visual data, and the corresponding emotion labels. We adopt common information here to describe the effectiveness of combining emotional information from different modalities by correlation analysis. It is implemented by a deep learning version of Hirschfeld-Gebelein-Rényi (HGR) maximal correlation [31,32,33], a well-known measure of dependence, to learn the maximally correlated feature representations of different modalities. The classification loss is used in order to extract discriminative information from each modality for emotion prediction. Further, we generalize our framework to the semi-supervised learning scenario. We conduct experiments on three public audio-visual datasets: eNTERFACE’05 [34], BAUM-1s [35], and RAVDESS [36]. The results demonstrate that, by capturing common information with HGR maximal correlation, our deep learning approach can significantly enhance the stability of features that are learned from different modalities and improve the emotion recognition performance.

To summarize, our main contributions are as follows:

• We design a deep learning framework to learn the discriminative feature representations from the audio and visual data for emotion recognition.
• We propose a correlation loss that is based on HGR maximal correlation to capture common information between audio data, visual data, and the corresponding emotion labels.
• We generalize our framework to the semi-supervised learning scenario with common information.
• We conduct experiments on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets to demonstrate the effectiveness of our system.

To the best of our knowledge, our method is the first work to utilize HGR maximal correlation in order to extract common information between audio data, visual data, and the corresponding emotion labels for audio-visual emotion recognition. The remainder of this paper is organized, as follows. In Section 2, we describe the related works. In Section 3, we explain HGR maximal correlation. In Section 4, we introduce our approach in detail. Subsequently, we perform extensive experiments on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets in Section 5. Finally, we draw conclusions and point out some future works in Section 6.

2. Related Works

Audio-visual emotion recognition is related to multimodal learning and feature extraction with deep learning. In this section, we review these two works.

2.1. Multimodal Learning

In the real world, human emotion expression has the inherent multimodality characteristic [38,39]. Multimodal learning [40,41], also referred to as multiview learning [42], is proposed to build models that can better process and relate information from multiple modalities in order to efficiently use the multimodal data similar to the emotional data [43]. One important research topic is to extract the information between different modalities to achieve better predictions, which is consistent with the common information in our paper.

The straightforward way is to concatenate the feature representations that were learned from each modality into a feature vector. Although this strategy is widely used, it often ignores the complex dependencies and interactions between different features of each modality and they may result in unsatisfactory performance [44]. Some CCA-based approaches are further proposed in order to capture the complex correlation between different modalities. It can extract the linearly maximally correlated feature mappings of two random variables. Kernel CCA [45,46] and deep CCA [47] are proposed in order to generalize CCA to non-linear setting using the kernel method and the deep neural network (DNN) respectively. Subsequently, Wang et al. [48] proposed the deep canonically correlated autoencoder (DCCAE) by combining the canonical correlation loss and the reconstruction loss of autoencoders.

HGR maximal correlation is another important measure of dependence [29]. It can extract the maximally non-linear correlated features of different modalities, showing higher efficiency than CCA. Recently, ref [28,49] implemented the HGR maximal correlation with deep learning. HGR maximal correlation and the deep learning version have become widely used [29,50,51,52]. For example, ref [52] adopted HGR maximal correlation to extract the correlation between different modalities and uses the hand-crafted features as the input for audio-visual emotion recognition while [29] considered the HGR maximal correlation in autoencoder architectures for multimodal emotion recognition.

Although HGR maximal correlation is used in the DNN methods [29,52] to compute the common information between the input audio and visual data, the emotional label information is not considered in the correlation loss, so it cannot guarantee that the learned features have sufficient discrimination ability for the emotion recognition task. Besides, the stability of common information in the deep learning models is not investigated in these methods. Here, we will sufficiently study the effectiveness of common information for emotion recognition.

2.2. Feature Extraction with Deep Learning

One key challenge in audio-visual emotion recognition is feature extraction [4,53], which decides what types of features with acceptable sizes are learned for emotion recognition. Since the success of deep learning, DNN has been gradually used to extract features and achieves better performance than traditional heuristic methods [54,55]. Here, we focus on DNN approaches for emotion recognition, which can be divided into two categories according to the type of input data: raw data-based and hand-crafted features-based.

For audio emotion recognition, some works use hand-crafted features as the input to DNN models. For example, Ma et al. [56] proposed a multi-task attention-based DNN model and feed the hand-crafted features into the model. Some works directly feed the raw data to the DNN models. For example, Chen et al. [20] designed a network with several convolutional layers to extract audio features. Tzirakis et al. [57] proposed a new CNN with Long Short-Term Memory (LSTM) for end-to-end audio emotion recognition. Fu et al. [26] used a sparse autoencoder in order to obtain the hidden features of audio data. Dai et al. [58] presented an approach to learn discriminative features from the audio data by integrating center loss in the deep learning model for audio emotion recognition.

For visual emotion recognition, most works directly send visual data to the DNN models. For example, Mollahosseini et al. [59] proposed a network that consisted of two convolutional layers, each of which was followed by max-pooling and four Inception layers. The Inception layers increased the depth and width of the network while keeping the computational budget constant. Jain et al. [60] proposed a hybrid convolution-recurrent neural network. It consists of convolution layers followed by a recurrent neural network, which can consider the temporal dependencies that exist in the facial images. Hickson et al. [61] presented an algorithm to automatically infer facial expressions by analyzing only a partially occluded face while the user is engaged in the virtual reality experience. Zhang et al. [4] employed a 3D CNN pre-trained on large-scale video classification tasks to capture the feature representations in visual data.

Besides, the representation of the input data also has a significant impact on the feature extraction process. For example, Li et al. [62] divided the visual emotion recognition methods into two categories according to the representation: static-based and dynamic-based. In static-based methods [59,63,64], two-dimensional (2D) networks are used to extract spatial information from the single facial expression image, whereas dynamic-based methods [65,66,67] use 3D networks to capture the spatial and temporal information from the facial expression sequence. Although 3D networks contain more information than 2D networks, it is difficult to choose the appropriate 3D networks that can effectively improve the emotion recognition performance. Besides, the large scale of 3D networks may lead to cumbersome training processes. Analogously, in terms of audio emotion recognition, some works [29,54] directly send the time-domain audio data into the DNN models, while other works [58,68,69] first convert the audio data into spectrum representations that are similar to the RGB images, and then feed the spectrums into deep learning models. The latter approach is considered to be more effective [69].

Recently, some popular deep learning models have been widely used, such as VGGNet [70], GoogLeNet [71], and ResNet [37]. When compared with VGGNet and GoogLeNet, ResNet has some advantages. For example, ResNet is shown to have better performance than VGGNet and GoogLeNet using a residual learning framework to ease the training of networks that are substantially deeper than those used previously [37]. Besides, ResNet has been successfully applied to audio emotion recognition [72,73,74,75] and visual emotion recognition [76,77,78,79,80]. These factors inspire us to use ResNet as the backbone of our whole network for audio-visual emotion recognition.

3. Preliminary

HGR maximal correlation can be regarded as a generalization of Pearson’s correlation [81]. For joint distributed random variables X and Y with ranges $\mathcal{X}$ and $\mathcal{Y}$, HGR maximal correlation with k features is defined, as shown in Equation (1):

$${\rho }^{\left(k\right)}(X,Y)=\underset{\begin{array}{c}\mathbf{f}:\mathcal{X}\to {\mathbb{R}}^k,\mathbb{E}\left[\mathbf{f}\right]=\mathbf{0},\mathrm{cov}\left(\mathbf{f}\right)=\mathbf{I}\\ \mathbf{g}:\mathcal{Y}\to {\mathbb{R}}^k,\mathbb{E}\left[\mathbf{g}\right]=\mathbf{0},\mathrm{cov}\left(\mathbf{g}\right)=\mathbf{I}\end{array}}{sup}\mathbb{E}\left[\mathbf{f}{\left(X\right)}^{\mathrm{T}}\mathbf{g}\left(Y\right)\right]$$

(1)

where the supremum is taken from all Borel measurable functions. Besides, $0\le {\rho }^{\left(k\right)}(X,Y)\le 1$, and ${\rho }^{\left(k\right)}(X,Y)=1$ represents that X is independent of Y. HGR maximal correlation can help us to extract the non-linear feature representations, $\mathbf{f}$ and $\mathbf{g}$. From the perspective of information theory, $\mathbf{f}$ learned from X has the maximum information towards some aspects of Y and vice versa, which can be used to extract the common information that is shared in X and Y [28,82].

In [28,49], based on HGR maximal correlation, an objective function that can be directly used for deep learning is proposed, as shown in Equation (2):

$$\begin{array}{cc}\hfill \underset{\mathbf{f},\mathbf{g}}{\max }& \mathbb{E}\left[{\mathbf{f}}^{\mathrm{T}}\left(X\right)\mathbf{g}\left(Y\right)\right]-\frac{1}{2}tr\left(\mathrm{cov}\left(\mathbf{f}\left(X\right)\right)\mathrm{cov}\left(\mathbf{g}\left(Y\right)\right)\right)\hfill \end{array}$$

(2)

where $\mathrm{cov}\left(\mathbf{f}\right(X\left)\right)$ and $\mathrm{cov}\left(\mathbf{g}\right(Y\left)\right)$ represent the covariance matrix of $\mathbf{f}\left(X\right)$ and $\mathbf{g}\left(Y\right)$, respectively. $\mathrm{tr}(·)$ represents the matrix trace operator. It is shown in [28] that Equation (2) can implement HGR maximal correlation equivalently. Inspired by this, we can design the loss function of our system based on HGR maximal correlation in order to extract the common information between different modalities.

HGR maximal correlation is appealing to multimodal learning. On the one hand, it can extract the maximally non-linear correlated features of different modalities, but the Pearson’s correlation cannot. On the other hand, it has strong efficiency in deep learning frameworks. These factors inspire us to integrate HGR maximal correlation into the deep learning model for audio-visual emotion recognition.

4. Methodology

Our goal is to efficiently learn the feature representations from audio and visual data in order to improve the emotion recognition performance with common information. To achieve the goal, we propose the system, as shown in Figure 1. In the following, we first formalize the audio-visual emotion recognition problem and, then, we present how the whole model is trained.

4.1. Problem Formulation

Here, we present the specific definition of audio-visual emotion recognition. Suppose that, in the training stage, we are given the samples $\left\{({\mathbf{x}}^{\left(i\right)},{\mathbf{y}}^{\left(i\right)},z^{\left(i\right)})\mid {\mathbf{x}}^{\left(i\right)}\in {\mathbb{R}}^{D_{\mathbf{x}}},{\mathbf{y}}^{\left(i\right)}\in {\mathbb{R}}^{D_{\mathbf{y}}},z^{\left(i\right)}\in \mathcal{Z}=\{1,2,\cdots ,|\mathcal{Z}\left|\right\},i=1,\cdots ,m\right\}$, where ${\mathbf{x}}^{\left(i\right)}$, ${\mathbf{y}}^{\left(i\right)}$, and $z^{\left(i\right)}$, respectively, represent the visual data, audio data, and the emotion category label, such as anger, happiness, and sadness. Their corresponding random variables are denoted as X, Y, and Z. Subsequently, we use $\mathbf{f}\left(X\right)={\left[f_1\left(X\right),f_2\left(X\right),\cdots ,f_k\left(X\right)\right]}^{\mathrm{T}}$, $\mathbf{g}\left(Y\right)={\left[g_1\left(Y\right),g_2\left(Y\right),\cdots ,g_k\left(Y\right)\right]}^{\mathrm{T}}$, and $\mathbf{h}\left(Z\right)={\left[h_1\left(Z\right),h_2\left(Z\right),\cdots ,h_k\left(Z\right)\right]}^{\mathrm{T}}$ to represent the k-dimensional feature functions of X, Y, and Z, respectively. We capture the HGR maximal correlation between $\mathbf{f}$, $\mathbf{g}$, and $\mathbf{h}$, as shown in Figure 1, in order to learn the audio and visual features with common information to predict the corresponding emotions.

4.2. Model Learning

To jointly extract emotional features from audio and visual data, we propose the full loss function of the whole network, which is a linear combination of classification loss and correlation loss, defined as Equation (3):

$$\begin{array}{c}\hfill L=L_{clf}+\alpha L_{corr}\end{array}$$

(3)

The classification loss, $L_{clf}$, measures the classification performance. The correlation loss, $L_{corr}$, measures the dependencies between audio data, visual data, and the corresponding emotion labels. The parameter $\alpha$ is the weight coefficient. By considering the correlation loss in the training process, our model can extract the non-linear correlated feature representations with common information in order to improve the performance of audio-visual emotion recognition.

When designing the correlation loss, most of previous works [29,52] only compute the correlation between audio data and visual data, but ignore the effect of label information, which may lead to the learned features not directly related to emotion prediction. It is significant to incorporate emotional label information into the correlation loss in order to enhance the discrimination ability of the learned feature representations. Therefore, we introduce a new form of correlation loss based on HGR maximal correlation, $L_{corr}$, as shown in Equation (4).

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{corr}(X,Y,Z)& =-\mathbb{E}\left[{\mathbf{f}}^{\mathrm{T}}\left(X\right)\mathbf{g}\left(Y\right)\right]+\frac{1}{2}\mathrm{tr}\left(\mathrm{cov}\left(\mathbf{f}\left(X\right)\right)\mathrm{cov}\left(\mathbf{g}\left(Y\right)\right)\right)\hfill \\ & -\mathbb{E}\left[{\mathbf{f}}^{\mathrm{T}}\left(X\right)\mathbf{h}\left(Z\right)\right]+\frac{1}{2}\mathrm{tr}\left(\mathrm{cov}\left(\mathbf{f}\left(X\right)\right)\mathrm{cov}\left(\mathbf{h}\left(Z\right)\right)\right)\hfill \\ & -\mathbb{E}\left[{\mathbf{g}}^{\mathrm{T}}\left(Y\right)\mathbf{h}\left(Z\right)\right]+\frac{1}{2}\mathrm{tr}\left(\mathrm{cov}\left(\mathbf{g}\left(Y\right)\right)\mathrm{cov}\left(\mathbf{h}\left(Z\right)\right)\right)\hfill \end{array}\end{array}$$

(4)

where the first, second, and third rows of $L_{corr}$ compute the HGR maximal correlation between $\mathbf{f}$ and $\mathbf{g}$, $\mathbf{f}$ and $\mathbf{h}$, $\mathbf{g}$ and $\mathbf{h}$, respectively, which can be regarded as learning common information between audio data, visual data, and the corresponding emotion labels. By extracting the correlation among different modalities, $L_{corr}$ can ensure that our model has sufficient discrimination ability for emotion recognition. We can think the first row of $L_{corr}$ as a simple version and call it $L_{corr\_simple}$. It only considers the HGR maximal correlation between the audio feature $\mathbf{f}$ and visual feature $\mathbf{g}$, as shown in Equation (5).

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{corr\_simple}(X,Y)& =-\mathbb{E}\left[{\mathbf{f}}^{\mathrm{T}}\left(X\right)\mathbf{g}\left(Y\right)\right]+\frac{1}{2}\mathrm{tr}\left(\mathrm{cov}\left(\mathbf{f}\left(X\right)\right)\mathrm{cov}\left(\mathbf{g}\left(Y\right)\right)\right)\hfill \end{array}\end{array}$$

(5)

Following [29], we use cross-entropy loss as the classification loss for emotion classification, as shown in Equation (6):

$$\begin{array}{c}\hfill L_{clf}(X,Y,Z)=-\mathbb{E}[log{P}_{Z|XY}]\end{array}$$

(6)

where

$$P_{Z=j|XY}(X,Y,Z)=\frac{exp\left({\mathbf{\Phi }}^{\mathrm{T}}\left(\mathbf{f}\left(X\right),\mathbf{g}\left(Y\right)\right){\mathit{\theta }}_j\right)}{{\sum }_{i=1}^{\left|\mathcal{Z}\right|}exp\left({\mathbf{\Phi }}^{\mathrm{T}}\left(\mathbf{f}\left(X\right),\mathbf{g}\left(Y\right)\right){\mathit{\theta }}_i\right)}$$

(7)

where $j=1,\cdots ,\left|\mathcal{Z}\right|$, ${\mathit{\theta }}_j$ represents the j-th term of the weights in the last layer of the fusion network, $\mathbf{\Phi }$ represents the feature function of fully connected layers in the fusion network. Additionally, ${\mathbf{\Phi }}^{\mathrm{T}}\left(\mathbf{f}\left(X\right),\mathbf{g}\left(Y\right)\right)$ represents the function, ${\mathbf{\Phi }}^{\mathrm{T}}$, of the concatenation of $\mathbf{f}\left(X\right)$ and $\mathbf{g}\left(Y\right).$ In Section 5.3.3, we will conduct experiments to consider different forms of $\mathbf{\Phi }$ to test the stability of common information in our framework.

We can further consider the semi-supervised learning scenario [83,84], where labeled data may be expensive or time-consuming to obtain. Semi-supervised learning can help us use labeled data and unlabeled data for better emotion recognition. Suppose that, in the training process, we have labeled data $(X_l,Y_l,Z_l)$ and unlabeled data $(X_u,Y_u)$, where $X_l$, $Y_l$, and $Z_l$, respectively, represent the labeled audio modality, labeled visual modality, and the corresponding emotional labels, and $X_u$ and $Y_u$ represent the unlabeled audio and visual modalities. We use Equation (4) in order to compute the correlation loss $L_{corr}$ with the labeled data and Equation (5) to compute the correlation loss $L_{corr\_simple}$ with the unlabeled data. The sum of correlation losses on these two parts is defined as $L_{corr\_semi}$, as shown in Equation (8):

$$\begin{array}{c}\hfill L_{corr\_semi}(X_l,Y_l,Z_l,X_u,Y_u)=L_{corr}(X_l,Y_l,Z_l)+L_{corr\_simple}(X_u,Y_u)\end{array}$$

(8)

When the training data are 100% labeled, $L_{corr\_semi}$ will become $L_{corr}$. Besides, we use the labeled data $(X_l,Y_l,Z_l)$ to compute the classification loss $L_{clf}$. Subsequently, we combine $L_{corr\_semi}$ and $L_{clf}$ in the form of Equation (3) to jointly train our network for semi-supervised learning. In this way, our model can make full use of labeled data and unlabeled data for emotion recognition.

5. Experiments

In this section, we evaluate our system in the following aspects: (i) to show our approach achieves higher performance than the previous works, (ii) to show our approach can enhance the stability of features learned from audio and visual data for emotion recognition, and (iii) to show that our approach can be easily generalized to the semi-supervised learning scenario.

5.1. Datasets

We perform experiments on three audio-visual emotional datasets to evaluate the effectiveness of our approach, including eNTERFACE’05, BAUM-1s, and RAVDESS, because they are available to the research community and widely used in audio-visual emotion recognition.

The eNTERFACE’05 dataset [34] has 1287 English video samples from 42 subjects coming from 14 different nationalities. Each subject are first told to listen to six different situations, with each of them eliciting one of the following emotions: anger, disgust, fear, happiness, sadness, and surprise. They then react to each of the situations and two human experts judged whether the reaction expressed the emotion in an unambiguous way. Happiness has 213 samples, and each of the other five emotions has 216 samples. The frame rate is 25 frames per second. The audio sample rate is 48,000 Hz. In all the samples, the shortest duration is 1.12 s and the longest duration is 106.92 s. 95% samples have the duration more than 1.56 s.

The BAUM-1s dataset [35] is a spontaneous audio-visual Turkish database that contains expressions of affective as well as mental states. It has 1134 video samples from 31 subjects. The subjects are first shown a sequence of images and short video clips, which are not only meticulously fashioned, but also timed to evoke a set of emotions and mental states. Subsequently, they express their feelings and ideas about the images and video clips they have watched in their own words, without using predetermined scripts. The subjects are not guided in any way regarding how to perform the emotion. The database contains recordings reflecting the six basic emotions (anger, disgust, fear, happiness, sadness, and surprise) as well as boredom and contempt. The database also contains several mental states, namely unsure, thinking, concentrating, and bothered. Following [4], our work focuses on recognizing the six basic emotions, which have 521 video clips. To be specific, anger, disgust, fear, happiness, sadness, and surprise have 56, 80, 37, 173, 134, and 41 samples, respectively. The frame rate is 29.97 frames per second. The audio sample rate is 48,000 Hz. In all the samples, the shortest duration is 0.43 s and longest duration is 29.2 s. 95% samples have the duration more than 1.03 s.

The RAVDESS dataset [36] is a validated multimodal database of emotional speech and song. It is gender-balanced consisting of 24 professional actors (12 female, 12 male), vocalizing lexically-matched statements in a neutral North American accent. Emotional expressions are elicited while using techniques the actors are trained in, including method acting or emotional memory techniques. Here, we consider the speech video clips, which are recorded in audio-visual format with 1440 samples. It includes calm, happy, sad, angry, fearful, surprise, disgust, and neutral expressions. Each expression is produced at two levels of emotional intensity. Ratings are provided by 247 individuals who are characteristic of untrained research participants from North America. Analogously, we only consider recognizing the six basic emotions as the eNTERFACE’05 and BAUM-1s datasets, each of which has 192 samples. The frame rate is 29.97 frames per second. The audio sample rate is 48,000 Hz. In all samples, the shortest duration is 2.99 s and longest duration is 5.31 s. 95% samples have the duration more than 3.24 s.

In Figure 2, we show some cropped facial images in order to illustrate the visual information of eNTERFACE’05, BAUM-1s, and RAVDESS datasets. We can also further provide the audio information of each dataset. However, the spectrogram of each sample is visually indistinguishable. Here, we statistically average the spectrograms of different samples that belong to the same class, as shown in Figure 3. It can be found that both visual information and audio information can differentiate emotions on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets.

5.2. Implementation Details

5.2.1. Data Preprocessing

The audio and visual data need to be extracted from emotional video samples, which always vary in time duration. We consider splitting each emotional video sample into several segments with the same length and extract audio data and visual data from them. Some previous works [4,29] use a moving window with a fixed length to segment the video samples. Additionally, the moving step of the window should also be carefully determined. Because the number of segments for each video sample depends on the time length of the corresponding video sample and the time lengths for different video samples are very different, the distribution of segmented dataset may be different from that of the original video dataset. Because of this phenomenon, such an approach may affect the final performance of audio-visual emotion recognition.

Here, we propose to randomly obtain a segment from a given video sample with a window and repeat this operation 30 times. Therefore, we can obtain 30 segments belonging to the same video sample. The label of the given video sample is used as the labels for these 30 segments. This technique can effectively ensure the distribution of the segmented dataset is consistent with that of the original dataset, and it also has the data augmentation effect. The best window size for emotion recognition is still unclear and it is reported in [85] that a segment longer than 0.25 s includes sufficient emotional information. Inspired by this, we set the window length to 0.5 s. The experimental results show that this length is suitable for emotion recognition.

Each segment usually contains several consecutive frames, which express the same emotion in a very similar way. Additionally, in [86], it is said that at the start and end of the emotional videos, that the subject usually prepares to express the emotion from the neutral state or return to the neutral state after the emotion is shown. These factors motivate us to choose the central frame in each segment as the key frame and take all key frames as the visual data. This way can effectively make the visual data contain rich emotional information for emotion classification and avoid redundancy. Subsequently, we use the MTCNN (multi-task cascaded convolutional network), as proposed in [87], to detect the human face from visual data with the squared size $160\times 160\times 3$. After the preprocessing, we feed the visual data into the visual network.

In addition, we extract the speech signals from all segments. Subsequently, we extract the log Mel-spectrogram from the speech signal for each segment as audio data. For each speech signal, we adopt 94 Mel-filter banks in order to obtain the log Mel-spectrogram with a 40 ms hanning window and a 10ms overlapping, resulting in the representation with size 94 × 94. Afterwards, we convert the representation into three channels by copying the original Mel-spectrogram to each channel. Finally, we send the audio data into the audio network.

5.2.2. Network Architecture

We use ResNet-50 [37] as the backbone architectures of the audio network and visual network. The fully connected layer before the softmax layer of ResNet-50 has 512 units, denoted as feature functions $\mathbf{f}$ and $\mathbf{g}$ in the visual network and audio network, respectively. Additionally, the weights of the two networks are initialized by copying the parameters of trained on the ImageNet dataset [88]. Inspired by [89], we first convert the emotion labels into one-hot form and then use a fully connected layer with 512 units as the label network to obtain the feature function $\mathbf{h}$. We then concatenate $\mathbf{f}$ and $\mathbf{g}$ into the fusion network for emotion prediction. The fusion network has several fully connected layers. The last layer of the fusion network is the softmax layer. Each fully connected layer is followed by the ReLU function. These fully connected layers before the softmax layer in the fusion network correspond to the feature function $\mathbf{\Phi }$. Because common information in different layers of our deep learning model may have different performance, we consider different forms of the feature function, $\mathbf{\Phi }$. Specifically, we respectively make the fusion network has one, two, and four fully connected layers before the softmax layer, as shown in Figure 4. We will report the performance of different settings in Section 5.3.3. In this way, we can test the stability of common information in our deep learning model.

In [4,90], it is said that the pre-trained strategy can effectively enhance the expressiveness of the learned features. Accordingly, here, we firstly train audio and visual network separately with the cross-entropy loss function. In this way, the networks can contain sufficient discriminative information from each modality. Subsequently, we use their weights as the initial weights of our whole model for joint training.

After the emotion probabilities of each segment are predicted, the average results across all segments belonging to the same video samples are used in order to predict the video-level emotion labels.

5.2.3. Experimental Settings

In our experiments, 70% samples are used as training data, 15% samples are used as validation data, and the remaining 15% samples are used as test data. The segments that belong to the same original video sample are assigned together as the training data or validation data or test data. Each experiment is run five times and the average recognition accuracy (%) is reported as the final result. We train our model while using the Adam [91] optimizer with the learning rate of 0.001. The batch size is set to 90. The number of epochs is set to 100. Pytorch [92] is used to implement our proposed model. We run the experiments on a NVIDIA TITAN V GPU card.

5.3. Experimental Results

In this section, we first show the performance of unimodal emotion recognition, show the performance of audio-visual emotion recognition, give the stability analysis of common information, and finally describe how our approach can be used for semi-supervised learning.

5.3.1. Unimodal Performance

To show the effectiveness of the learned audio and visual feature representations, we report the performance of our method in audio emotion recognition and visual emotion recognition, respectively, as shown in

Table 1. The performance comparison of audio emotion recognition with previous works on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets.

Dataset	Method	Accuracy (%)
eNTERFACE’05	Chen et al. [20], DNN	66.3
	Ma et al. [29], DNN	58.95
	Ours	74.33
BAUM-1s	Ours	47.09
RAVDESS	Holmström et al. [93], Logistic Model Tree	70
	Singh et al., [94], SVM	64.15
	Ours	75.61

and

Table 2. The performance comparison of visual emotion recognition with previous works on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets.

Dataset	Method	Accuracy (%)
eNTERFACE’05	Yan et al. [95], Sparse Representation	76.23
	Chen et al. [20], DNN	61.7
	Ours	80.52
BAUM-1s	Ours	64.05
RAVDESS	He et al. [96], DNN	79.74
	Ours	95.49

, respectively. In these two scenarios, we only use the corresponding audio network and visual network to extract the feature representations for emotion prediction. At the same time, we compare our work with previous works, which are all based on the original datasets. Some of them use hand-crafted features and some use DNN-based features.

From Table 1, we can see that our learned audio features are more discriminative for emotion recognition than the hand-crafted features [93,94] on the RAVDESS dataset and the features learned from DNN models with shallow structure [20,29] on the eNTERFACE’05 dataset. These results show that our network can effectively learn the audio features for emotion recognition. It is also worth noting that audio emotion recognition performs better on the RAVDESS dataset and eNTERFACE’05 dataset than on the BAUM-1s dataset, which indicate that the first two datasets may contain more emotional information.

From Table 2, we can also observe that our visual network performs better than the methods [20,95,96]. On the eNTERFACE’05 dataset, it can be found that our method achieves higher emotion recognition accuracy than the method [95] with hand-crafted features as input and the method [20] with raw data as input. On the RAVDESS dataset, our learned visual features yield better performance than the method [96] while using shallow DNN models with raw data as input. These results also show the visual features learned from our network have more discriminative power than previous works.

5.3.2. Multimodal Performance

We report the performance of audio-visual emotion recognition on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets in order to further demonstrate the effectiveness of our system. In the meanwhile, we make a comparison with previous works, which also conduct experiments on these original datasets.

Table 3. The performance comparison of audio-visual emotion recognition with previous works on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets.

Dataset	Method	Accuracy (%)
eNTERFACE’05	Štruc et al. [22], CCA	71
	Poria et al. [97], SVM	85.23
	Seng et al. [98], PCA + LDA + Rules	86.67
	Yan et al. [95], Sparse Kernel Reduced-Rank Regression	87.46
	Chen et al. [20], Attention Mechanism	79.2
	Ma et al. [29], HGR Maximal Correlation + Autoencoder	85.43
	Hossain et al. [21], Extreme Learning Machine + SVM	86.4
	Wang et al. [27], Low-rank Representation	86.98
	Ours	88.66
BAUM-1s	Wang et al. [27], Low-rank Representation	60.05
	Ours	67.59
RAVDESS	Ghaleb et al. [99], Incremental Learning	67.70
	Mansouri et al. [100], Spiking Neural Networks	83.60
	Ours	97.57

summarizes the results. To implement our architecture, we set $\alpha$ equal to 1 and make the fusion network have two fully connected layers on the eNTERFACE’05 dataset, set $\alpha$ equal to 1 and make the fusion network have one fully connected layer on the BAUM-1s dataset, set $\alpha$ equal to 0.1 and make the fusion network have one fully connected layer on the RAVDESS dataset.

From Table 3, we can find that our method is competitive with the compared works. To be specific, on the eNTERFACE’05 dataset, the performance of our method is at least 1% higher than that of the previous methods [20,21,22,27,29,95,97,98]. On the BAUM-1s dataset, we improve the performance of the method [27] from 60.05% to 67.59%. On the RAVDESS dataset, our method performs much better than the methods [99,100] by more than 13%. It is worth noting that the method [22] combines the audio and visual features that are based on CCA and the method [27] utilizes a similar approach with low-rank representation. Our method performs better than these two methods, which show that the correlation loss we propose can learn the discriminative feature representations more effectively. Besides, most of the above methods are based on DNN models, while our method achieves better performance, which shows that the features learned from our deep learning model has more powerful expressiveness. To sum up, our deep learning method can efficiently extract the discriminative feature representations with common information in order to achieve the highest accuracies among all the compared methods.

Figure 5 shows the classification confusion matrices using our method on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets. It can be found that, on the eNTERFACE’05 dataset, “fear” and “surprise” are harder to be recognized when compared to other emotions. This indicates audio-visual cues of these two emotions contain less emotional information. On the BAUM-1s dataset, “happiness” achieves the highest recognition accuracy among the six emotions. However, “anger” and “fear” have lower recognition accuracies. This may be due to the small number of samples of “anger” and “fear” on the BAUM-1s dataset, which results in our model being unable to fully learn the features of “anger” and “fear” for emotion classification. On the RAVDESS dataset, both “anger” and “happiness” can be identified with 100% accuracy. Similar to the eNTERFACE’05 dataset, “fear” and “surprise” are more difficult to be recognized.

In addition to the classification confusion matrices, we show some misclassification examples on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets, as shown in Figure 6. For example, on the eNTERFACE’05 dataset, “disgust” is misclassified as “anger”, “fear” is misclassified as “sadness”. On the BAUM-1s dataset, “anger” is misclassified as “disgust”, “disgust” is misclassified as “sadness”. On the RAVDESS dataset, “disgust” is misclassified as “fear” and “fear” is misclassified as “sadness”. This indicates that similar emotions on these three datasets may be difficult to distinguish.

5.3.3. Stability Analysis of Common Information

From Figure 1 and Figure 4, we can see that our proposed correlation loss can make the input features of the fusion network, $\mathbf{f}$ and $\mathbf{g}$, maximally correlated. The more fully connected layers in the fusion network, the farther common information is from the softmax output, which may lead to different emotion recognition performance. Besides, we know that $\alpha$ determines how the correlation loss works during the training process. Different $\alpha$ values will lead to different operating mechanisms of common information. Motivated by these two factors, we investigate the stability of common information in our deep learning model.

We do the following study with different settings on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets, as shown in

Table 4. The performance of audio-visual emotion recognition with different settings on the eNTERFACE’05 dataset.

Method		Setting
		One	Two	Four
Baseline		87.11	84.54	84.54
$\alpha =0.01$	$L_{corr\_simple}$	87.53	88.35	86.60
	$L_{corr}$	88.14	87.32	86.80
$\alpha =0.1$	$L_{corr\_simple}$	86.39	87.84	87.84
	$L_{corr}$	87.63	87.22	86.19
$\alpha =1$	$L_{corr\_simple}$	86.60	85.36	85.46
	$L_{corr}$	87.42	88.66	88.14
$\alpha =10$	$L_{corr\_simple}$	87.73	87.11	85.98
	$L_{corr}$	88.14	86.29	88.25

,

Table 5. The performance of audio-visual emotion recognition with different settings on the BAUM-1s dataset.

Method		Setting
		One	Two	Four
Baseline		64.56	64.05	64.30
$\alpha =0.01$	$L_{corr\_simple}$	65.32	66.08	65.57
	$L_{corr}$	66.58	64.30	65.32
$\alpha =0.1$	$L_{corr\_simple}$	63.54	65.06	64.05
	$L_{corr}$	65.06	65.57	66.58
$\alpha =1$	$L_{corr\_simple}$	66.33	63.80	64.30
	$L_{corr}$	67.59	65.32	64.56
$\alpha =10$	$L_{corr\_simple}$	64.81	65.82	66.33
	$L_{corr}$	67.34	65.82	65.32

and

Table 6. The performance of audio-visual emotion recognition with different settings on the RAVDESS dataset.

Method		Setting
		One	Two	Four
Baseline		95.83	96.07	96.53
$\alpha =0.01$	$L_{corr\_simple}$	97.46	97.11	96.30
	$L_{corr}$	96.53	97.23	96.76
$\alpha =0.1$	$L_{corr\_simple}$	97.23	97.11	97.46
	$L_{corr}$	97.57	97.23	96.88
$\alpha =1$	$L_{corr\_simple}$	97.34	96.88	96.88
	$L_{corr}$	97.23	96.99	97.11
$\alpha =10$	$L_{corr\_simple}$	96.30	96.42	97.46
	$L_{corr}$	97.11	96.65	96.99

. We, respectively, make the fusion network has one, two and four fully connected layers before the softmax layer, which is shown in the setting column. The baseline in the method column means that we only use classification loss for training. We compare the performance of two forms of correlation loss, one is $L_{corr}$, the other is $L_{corr\_simple}$, which only computes the HGR maximal correlation between the features $\mathbf{f}$ and $\mathbf{g}$. We set $\alpha$ to 0.01, 0.1, 1, and 10, respectively, to implement audio-visual emotion recognition to find how the common information works. For each setting, we report the performance of $L_{corr\_simple}$ method and $L_{corr}$ method, respectively.

We have the following observations on the three datasets: (1) the accuracy of baseline methods is higher than that of audio network and visual network, which indicates that baseline methods can combine the information of audio data and visual data to some extent to improve the emotion recognition performance. (2) When compared with the baseline method, $L_{corr}$ can significantly improve the emotion recognition performance for different weight coefficients and different fully connected layers in the fusion network. This shows that the correlation loss we proposed, $L_{corr}$, is very stable in ing audio-visual emotion recognition. (3) It should be noted that the $L_{corr\_simple}$ method can also improve emotion recognition performance, but it is weaker than $L_{corr}$ method. For example, on the eNTERFACE’05 dataset, when the fusion network has one fully connected layer, $L_{corr\_simple}$ methods with the weight coefficient of 0.1 perform worse than the baseline method. On the RAVDESS dataset, when the fusion network has four fully connected layers, $L_{corr\_simple}$ method with the weight coefficient of 0.01 also performs worse than the baseline method. (4) On the eNTERFACE’05 dataset, the highest accuracy of 88.66% is achieved when we use $L_{corr}$ and set $\alpha =1$ with two fully connected layers in the fusion network. On the BAUM-1s dataset, the highest accuracy 67.59% is achieved when we use $L_{corr}$ and set $\alpha =1$ with one fully connected layer in the fusion network. On the RAVDESS dataset, the highest accuracy 97.57% is achieved when we use $L_{corr}$ and set $\alpha$ as 0.1 with one fully connected layer in the fusion network. These show that our method can benefit from common information with appropriate network settings. (5) When we set the weight coefficient to 0.1 or 1 and make the fusion network has one or two fully connected layers, our model performs better on all three datasets than models with other settings.

5.3.4. Robustness Analysis on Missing Modality

In our analysis above, we assume that audio and visual data are available during the testing stage. However, when generalizing the trained models to the real-world, we may encounter modality missing scenarios, which requires a good fusion model should perform well, even if missing modality occurs during the testing process. Motivated by this, we conduct the following study on the eNTERFACE’05 dataset in order to verify the robustness of our model in the testing process.

In our architecture, we concatenate the feature $\mathbf{f}$ of visual data and the feature $\mathbf{g}$ of audio data into the fusion network for emotion classification. In order to represent the modality missing problem during testing stage, we set the feature of one modality to $\mathbf{0}$ in order to indicate that this modality is missing, and then concatenate it with the feature of another modality to predict emotion. It should be noted that features of another modality is obtained by using the model we have trained. We consider three scenarios: only audio data are missing, only visual data are missing, and audio and visual data are missing. In each scenario, we, respectively, set 20%, 50%, and 80% of test data as missing. For the third scenario, the audio and visual data are missing at half of the missing rate, respectively. For example, the missing rate of 20% means that audio and visual data are missing with 10%, respectively. We compare the performance of three methods, $L_{corr}$, $L_{corr\_simple}$ and baseline. They have the same network structure, with two fully connected layers in the fusion network. For the $L_{corr}$ and $L_{corr\_simple}$ methods, we set $\alpha =1$. The baseline method means that we only use classification loss for training. For each setting, we report the emotion recognition accuracy, which is shown in Figure 7.

From Figure 7, we have the following observations: (1) as the missing rate increases, the emotion recognition accuracies of $L_{corr}$, $L_{corr\_simple}$ and baseline methods decrease. (2) In the three data missing scenarios, the downward tendency of $L_{corr\_simple}$ method with the increase of missing rate is basically consistent with that of $L_{corr}$ method, but the performance of the $L_{corr\_simple}$ method is always lower than that of the $L_{corr}$ method. (3) When compared with the $L_{corr}$ method, the performance of the baseline method decreases faster with the increase of missing rate. The gap between $L_{corr}$ method and baseline method will become more apparent as the missing rate increases, especially in the scenario where only audio data are missing. This shows that the $L_{corr}$ method is more robust to deal with missing modality problem during the testing stage.

5.3.5. Semi-Supervised Audio-Visual Emotion Recognition

Semi-supervised learning has been used in many tasks when labeled data are scarce or difficult to obtain. It assumes that, during the training process, we have some labeled data and some unlabeled data. By using these two parts of data, semi-supervised learning can help us to perform better classification than supervised learning using only labeled data. Here, we show that our method can be easily adapted to semi-supervised audio-visual emotion recognition task. We conduct experiments on the RAVDESS dataset. Specifically, we mask the labels of some training data to indicate that they are unlabeled data. For the labeled data, audio modality, visual modality, and the corresponding emotion labels are available. For the unlabeled data, only audio modality and visual modality are available. We, respectively, set 20%, 50%, 80%, and 100% of the training data as labeled data. For each semi-supervised scenario, we compare the performance of different methods. The average accuracy of each method is reported in

Table 7. The performance of semi-supervised audio-visual emotion recognition on the RAVDESS dataset.

Method		Percentage of Labels
		20%	50%	80%	100%
Audio		42.54	63.01	71.56	75.61
Visual		65.43	84.16	91.79	95.49
Baseline		67.40	88.44	93.29	96.07
$\alpha =0.01$	$L_{corr\_simple}$	68.90	88.79	94.22	97.11
	$L_{corr\_semi}$	69.94	89.36	94.68	97.23
$\alpha =0.1$	$L_{corr\_simple}$	69.60	88.79	94.80	97.11
	$L_{corr\_semi}$	70.64	89.71	94.68	97.23
$\alpha =1$	$L_{corr\_simple}$	69.02	87.63	94.34	96.88
	$L_{corr\_semi}$	69.13	90.29	95.26	96.99
$\alpha =10$	$L_{corr\_simple}$	67.28	88.09	95.72	96.42
	$L_{corr\_semi}$	69.60	89.48	95.72	96.65

.

In the method column of Table 7, audio means that only the audio modality in the labeled data is trained for audio emotion recognition. Visual means that only the visual modality in the labeled data is trained for visual emotion recognition. Baseline means that audio and visual modalities in the labeled data are trained by our whole network with only classification loss for audio-visual emotion recognition. $L_{corr\_simple}$ means that whether labeled data or unlabeled data, we use the correlation loss $L_{corr\_simple}$ to compute the HGR maximal correlation between audio and visual modalities. In addition, for labeled data, we use Equation (6) in order to compute the classification loss $L_{clf}$. $L_{corr\_semi}$ means that we use Equation (8) to compute the correlation loss $L_{corr\_semi}$. Additionally, similar to $L_{corr\_simple}$ method, the classification loss $L_{clf}$ is also computed using labeled data. It is worth noting that, for the $L_{corr\_simple}$ method and $L_{corr\_semi}$ method, classification loss and correlation loss are combined in the form of Equation (3) with the weight coefficient $\alpha$ to train our network together. Here, we make the fusion network have two fully connected layers to implement the baseline, $L_{corr\_simple}$ and $L_{corr\_semi}$ methods for audio-visual emotion recognition. By comparing with the baseline method that only uses labeled data, we can find that our proposed method $L_{corr\_semi}$ can effectively combine unlabeled and labeled data for audio-visual emotion recognition.

From Table 7, we have the following observations: (1) the performance of baseline method is better than that of audio method and visual method, which shows the importance of combining information of different modalities to improve the emotion recognition accuracy. (2) $L_{corr\_semi}$ method achieves the highest accuracy among all methods, and it shows that it can significantly improve the performance of audio-visual emotion recognition for different weight coefficients and different percentages of labels. (3) $L_{corr\_simple}$ method can also contribute to improving the performance of audio-visual emotion recognition, but it is weaker than $L_{corr\_semi}$ method, especially when the labels are insufficient. To sum up, our method can effectively improve the performance of audio-visual emotion recognition, showing its potential for semi-supervised learning.

6. Conclusions

In this paper, we propose an efficient deep learning approach to exploit common information between audio data, visual data, and the corresponding emotion labels for emotion recognition on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets. To be specific, we design an audio network and a visual network to learn the feature representations from audio data and visual data, respectively, and then use a fusion network to combine the audio and visual features for emotional recognition. The full loss function of our whole neural network is a linear combination of correlation loss and classification loss. The former is used to extract common information between audio data, visual data, and the corresponding emotion labels with HGR maximal correlation. The latter is used to extract discriminative information from different modalities. We further generalize our framework to the semi-supervised learning scenario. The experimental results demonstrate that by combining the common information with HGR maximal correlation, our deep learning approach can significantly enhance the stability of features that are learned from different modalities, and improve the emotion recognition performance.

In the future, we will investigate the performance of our method for emotion recognition in more datasets. It is worth noting that, in the real world, both audio data and visual data may be noisy. We will further consider audio-visual emotion recognition in the noisy environment. Besides, in addition to audio and visual data, physiological signals [101,102,103] and text data [104] are important modalities for characterizing human emotions. Therefore, we will consider combining the information of these modalities for multimodal emotion recognition.