HELPFuL: Human Emotion Label Prediction Based on Fuzzy Learning for Realizing Artificial Intelligent in IoT

Abstract

Human emotion label prediction is crucial to Artificial Intelligent in the Internet of Things (IoT). Facial expression recognition is the main technique to predict human emotion labels. Existing facial expression recognition methods do not consider the compound emotion and the fuzziness of emotion labels. Fuzzy learning is a mathematical tool for dealing with fuzziness and uncertainty information. The advantage of using fuzzy learning for human emotion recognition is that multiple fuzzy sentiment labels can be processed simultaneously. This paper proposes a fuzzy learning-based expression recognition method for human emotion label prediction. First, a fuzzy label distribution system is constructed using fuzzy sets for representing facial expressions. Then, two fuzzy label distribution prediction methods based on fuzzy rough sets are proposed to solve the compound emotion prediction. The probability that a sample is likely and definitely belongs to an emotion is obtained by calculating the upper and lower approximations. Experiments show the proposed algorithm not only performs well on human emotion label prediction but can also be used for other label distribution prediction tasks. The proposed method is more accurate and more general than other methods. The improvement of the method on the effect of emotion recognition extends the application scope of artificial intelligence in IoT.

1. Introduction

Human emotion label prediction can play a critical role in realizing artificial intelligence (AI) in the Internet of Things (IoT) domain [1,2,3]. IoT refers to a network of physical devices, vehicles, home appliances, and other objects that are embedded with sensors, software, and network connectivity to exchange data with other devices and systems over the Internet [4,5,6,7]. Facial expression recognition is one of the main AI technologies for human emotion label prediction [8,9]. The integration of facial expression recognition into IoT systems can enhance the intelligence of IoT and make it more responsive to human needs and emotions [10,11,12].

There are some possible applications of facial expression recognition in IoT [13], as shown in Figure 1, including: 1. Smart homes: The system could use facial expression recognition technology to detect the mood of the occupants and adjust the lighting, temperature, and music to create a more relaxing environment; 2. Healthcare: If a patient is feeling anxious, the system could provide calming music or guided meditation to help them relax; 3. Marketing: A retailer could use the system to track which products or promotions generate the most positive emotional responses from customers; 4. Automotive: The system could then provide alerts or take other actions to help keep the driver alert and focused on the road; 5. Education: The system could help teachers identify when students are struggling or disengaged and adjust their teaching strategies accordingly.

Facial expression recognition can be challenging due to the complexity of human emotions, which involve a combination of basic emotions or are difficult to label with a single emotion label [14]. For example, a person may feel both surprise and happy at the same time. Furthermore, the compound emotion may be ambiguous; that is, the degree of emotions also varies. Thus, the main challenges are feature learning, compound emotion expression, and ambiguous emotion expression.

Many studies have been conducted to improve expression recognition methods in terms of feature learning. Lopes et al. [15] apply CNN to extract only expression-specific features from a face image. Li et al. [16] introduce both static images and dynamic image sequences of facial expression feature learning based on deep neural networks and discuss these methods’ advantages and limitations. Wang et al. [17] embed a varied number of region features produced by CNN into a compact fixed-length representation. Rouast et al. [18] found that deep neural networks are used for multiple modal data feature learning, including spatial feature, temporal feature, and joint feature representations. However, the above approach does not consider more complex emotion recognition tasks, such as compound and ambiguous emotion.

Fuzzy learning is an effective mathematical tool to deal with ambiguous problems [19]. Zadeh proposes that the imprecision inherent in natural language is, by nature, possible rather than probabilistic. Fuzzy rough set is an effective mathematical tool that combines the advantages of rough and fuzzy sets [20]. Rough sets provide a sound model for handling uncertainty in classification [21]. They have not only been used in models [22,23,24], but also in several application such as in [25,26,27]. In fuzzy rough sets, a family of fuzzy sets is used to approximate a fuzzy class. The differences between fuzzy upper and lower approximations describe the inconsistency of the instances in fuzzy classification [28]. For dealing with diverse types of the attributes, researchers proposed different extensions of fuzzy rough sets [29,30,31]. To alleviate the shortcoming of fuzzy rough sets being sensitive to noisy samples, robust fuzzy rough sets were developed [32]. Chai et al. proposed a new intuitionistic fuzzy rough set approach [33]. Quran et al. proposed a neutrosophic soft rough set to solve uncertainty [34]. However, the extended models and algorithms only focus on data mapping of the conditional attribute.

In fuzzy learning, emotions can be represented as fuzzy sets that capture the uncertainty and ambiguity of emotion. The membership function of the fuzzy set represents the degree to which an emotion belongs to a particular category [35]. For example, the membership function of a fuzzy set representing a complex emotion such as “bittersweet” may assign a high degree of membership to both the happiness and sadness categories. However, most facial expression recognition methods are based on deep learning [36,37], and few studies have applied fuzzy learning to facial expression recognition methods.

In this paper, a facial expression recognition method based on fuzzy learning for human emotion label prediction is proposed to address the above problems. The contributions of this paper are summarized as follows: 1. constructing a fuzzy label distribution system using fuzzy sets for representing human emotions; 2. proposing two fuzzy label distribution prediction methods based on fuzzy rough sets to solve the compound fuzzy human emotion prediction problem; 3. the proposed method can also be used to solve single-label, multi-label, and probability-based label distribution prediction problems through degradation and transformation.

2. Fuzzy Sets for Label Distribution

2.1. Fuzzy Set

Natural language is full of uncertainties, and when we express fuzziness of words in natural language, such as young, old, good-looking, small, or big, we tend to use fuzzy sets for expression. Unlike a classical set, a fuzzy set has no definite elements and can only be expressed by means of a membership function. Therefore, the definition of a fuzzy set is also often written as follows. Let us assume that FS $=<U,C,D>$ is a fuzzy set, where instance set $U=\{x_1,x_2,\cdots x_N\}$, feature set $C=\{b_1,b_2,\cdots b_M\}$, and label set $D=\{d_1,d_2,\cdots ,d_L\}$. $d_j\left(x_i\right)\in [0,1]$ is regarded as the fuzzy membership degree of $x_i$ in $d_j$. All the labels in the label set are used to describe the instance; in other words, the label set is completed.

For an unseen instance $x^{\prime }$, the task of fuzzy Learning for Label Distribution is to predict the fuzzy membership degree function $d_j\left(x^{\prime }\right)$, where $(d_j\in D)$. The fuzzy membership degree of all labels is from the complete set $D\left(x^{\prime }\right)=\{d_1\left(x^{\prime }\right),d_2\left(x^{\prime }\right),\cdots ,d_L\left(x^{\prime }\right)\}$.

The fuzzy label distribution system can be reduced to the multi-label system or single-label system when the arbitrary label in the label set satisfies $d_j\left(x\right)\in \{0,1\}$. The label of an instance is defined as $(d_j\left(x\right)=\left\{\begin{array}{c}0,x\notin d_j\\ 1,x\in d_j\end{array}\right.)$.

2.2. Fuzzy Rough Sets (FRS)

Let U be a finite set of instances and x, y, z ∈ U, and equivalence relation R satisfies the following conditions $R(x,x)=1$, $R(x,y)=R(y,x)$, and $R(x,y)=1, R(y,z)=1\Rightarrow R(x,z)=1$. U is partitioned into a family of equivalence classes ${\left[x\right]}_R$ by an equivalence relation R.

Given $X\subseteq U$, the upper and lower approximations of X are defined as $\underline{R}X=\left\{{\left[x\right]}_R\right|{\left[x\right]}_R\subseteq X\}$ and $\overline{R}X=\left\{{\left[x\right]}_R\right|{\left[x\right]}_R\cap X\ne \varnothing \}$, respectively. Rough sets are a pair of upper and lower approximations that approach as close as possible to X. $BN{D}_{R}X=\overline{R}X-\underline{R}X$ is called the boundary of X.

The extension of rough sets is to equip U with fuzzy equivalence relation R, which satisfies the conditions $R(x,x)=1$, $R(x,y)=R(y,x)$, and $min\left(R\right(x,y)$, $R(y,z))\le R(x,z)$. A family of fuzzy classes makes a fuzzy partition of U.

Let R be a fuzzy equivalence relation and X be a fuzzy set on U, then the lower and upper fuzzy approximations of X are defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\underline{R}X\left(x\right)=\underset{y\in U}{inf}max(1-R(x,y),X(y\left)\right),\hfill \\ \overline{R}X\left(x\right)=\underset{y\in U}{sup}min\left(R\right(x,y),X(y\left)\right).\hfill \end{array}\end{array}$$

(1)

Fuzzy rough sets are a pair of fuzzy upper and lower approximations that approach as close as possible to X.

Given a fuzzy approximation space $IS=<U,R>$, X is a crisp subset of U, and R is a fuzzy relation. The lower and upper fuzzy approximations are defined as

$$\begin{array}{c}\hfill \begin{array}{c}\underline{R}X\left(x\right)=\underset{y\notin X}{inf}1-\left(R\right(x,y),\hfill \\ \overline{R}X\left(x\right)=\underset{y\in X}{sup}R(x,y).\hfill \end{array}\end{array}$$

(2)

2.3. Advantages of Fuzzy Label Distribution Learning

Label distribution learning (LDL) [38] has been widely studied to deal with the overall label distribution. However, LDL does not fit some applications of the fuzzy concept, which is subjective and has no clear boundaries, such as complex emotions. Each instance may be associated with multiple emotions simultaneously. Each emotion is a fuzzy concept, is subjective, and has no clear boundaries. Fuzzy label distribution learning (FLDL) is a more appropriate model than probability distribution to describe the distribution of all labels of the instance.

The two examples above describe the difference between them: (1) There is a video emotion recognition task shown in Figure 2, where the label set contains four basic emotions (angry, worried, sad, and happy). The value of each label is represented by a fuzzy membership degree; the higher the label value, the larger the membership degree. From the series of facial images in the video, we can see that the man has a complexity of emotions that contain several basic emotions simultaneously. He looks like he is very angry, not very worried, rather sad, and not happy. Therefore, the fuzzy membership degrees of these labels are 0.8, 0.2, 0.7, and 0.05, respectively. (2) There is a nature image classification task shown in Figure 3, where the label set contains four basic labels (sky, cloud, building, and water). The value of each label is proportional to the entire description of the instance. The natural scene image is formed by the sky, mostly cloudy, with some buildings, and lots of water. The proportions of these labels are 0.25, 0.45, 0.2, and 0.1, respectively. These labels sum up to 1.

The task of FLDL can be transformed into the tasks of SLL and MLL. In SLL, the label of an instance corresponds to the maximum fuzzy membership degree. The predicted process of SLL can be formulated as follows:

$$D\left(x^{\prime }\right)\leftarrow \underset{d_j}{arg}max{\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L.$$

(3)

In MLL, the label of an instance is the subset where the fuzzy membership degrees of instance to these labels are larger than $\lambda$. The predicted process of MLL can be formulated as follows:

$$D\left(x^{\prime }\right)\leftarrow \underset{d_j}{arg}{\left\{d_j\right|d_j\left(x^{\prime }\right)\ge \lambda \}}_{j=1}^L.$$

(4)

Examples of prediction labels of SLL, MLL, and FLDL are shown in Figure 4, where the triangle represents the corresponding label associated with an instance and the dotted line represents the constant $\lambda$. Fuzzy membership degrees of an instance to all labels are computed. In SLL, the third label is the maximum fuzzy membership degree. In MLL, the first and third labels are larger than $\lambda$. SLL and MLL can be regarded as special cases of FLDL. Therefore, FLDL is considered the more general model.

The output space of FLDL is larger than SLL and MLL. There are L possible outputs for SLL, and $2^L-1$ possible outputs for MLL. In FLDL, each label satisfies $d_j\in [0,1]$. Therefore, there are infinite possible outputs for FLDL.

3. Human Emotion Label Prediction Based on Fuzzy Learning (HELPFuL)

In this section, we first describe our algorithm in detail, and then we demonstrate the algorithm with an example.

3.1. Algorithm

In this subsection, we propose two FLDL algorithms based on fuzzy rough sets. The first one is the un-weighted algorithm belonging to a lazy learning approach, and the other is the weighted algorithm; the weights are learned from the training data.

Given a fuzzy label distribution system FMLS $= <U,C,D>$, the fuzzy upper and lower approximations of a test instance $x^{\prime }$ to the fuzzy label $d_j\in D$ are defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\overline{R}{d}_j\left(x^{\prime }\right)=\underset{y\in U,x^{\prime }\notin U}{sup}min(R(x^{\prime },y),d_j\left(y\right)),\hfill \\ \underline{R}{d}_j\left(x^{\prime }\right)=\underset{y\in U,x^{\prime }\notin U}{inf}max(1-R(x^{\prime },y),d_j\left(y\right)).\hfill \end{array}\end{array}$$

(5)

The fuzzy upper approximation of the fuzzy label represents the possibility of the instance belonging to this label. The fuzzy lower approximation of the fuzzy label represents the necessity of the instance belonging to this label. Fuzzy rough sets are regarded as the measures of possibility and necessity. In modeling the uncertainty of a judgment, it is natural to expect that the boundary between an event and its contrary is not too rigid. In order to characterize the uncertainty, both possibility and necessity are required. Therefore, we use fuzzy upper and lower approximations to confirm the label of an instance.

Given a fuzzy label distribution system FMLS $=<U,C,D>$, for the test instance $x^{\prime }$, the fuzzy membership degree of $x^{\prime }$ to label $d_j\in D$ is defined as follows:

$$d_j\left(x^{\prime }\right)=u_j\ast \overline{R}{d}_j\left(x^{\prime }\right)+l_j\ast \underline{R}{d}_j\left(x^{\prime }\right),$$

(6)

where $u_j$ and $l_j$ are the weights of the fuzzy upper and lower approximations of $d_j$, and these weights satisfy $u_j\ge 0$, $l_j\ge 0$, $u_j+l_j=1$.

There are three special methods in assigning a label to an instance: pessimism assignment method, optimistic assignment method, and mean assignment method.

1. The optimistic assignment method $D_{upper}\left(x^{\prime }\right)={\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L$: If $u_j=1$ and $l_j=0$, then we have $d_j\left(x^{\prime }\right)=\overline{R}{d}_j\left(x^{\prime }\right)$. The fuzzy upper approximation is regarded as the fuzzy membership degree of $x^{\prime }$ to $d_j$.

2. The pessimism assignment method $D_{lower}\left(x^{\prime }\right)={\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L$: If $u_j=0$ and $l_j=1$, then we have $d_j\left(x^{\prime }\right)=\underline{R}{d}_j\left(x^{\prime }\right)$. The fuzzy lower approximation is regarded as the fuzzy membership degree of $x^{\prime }$ to $d_j$.

3. The mean assignment method $D_{mean}\left(x^{\prime }\right)={\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L$: If $u_j=\frac{1}{2}$ and $l_j=\frac{1}{2}$, then we have $d_j\left(x^{\prime }\right)=\frac{1}{2}(\overline{R}{d}_j\left(x^{\prime }\right)+\underline{R}{d}_j\left(x^{\prime }\right))$. The mean of the fuzzy upper and lower approximation is regarded as the fuzzy membership degree of $x^{\prime }$ to $d_j$.

The optimistic assignment method may lead to overestimation of the fuzzy membership degree. The pessimism assignment method easily results in the underestimation of the fuzzy membership degree. The mean assignment method can avoid the estimation problems mentioned above. However, a simple average method can easily ignore the important information included in the training data. Therefore, we propose the weighted average assignment method by learning the weights of the fuzzy upper and lower approximations of each label.

The label set of $x^{\prime }$, assigned by the weighted average method, is denoted as $D_{weight}\left(x^{\prime }\right)={\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L$, where $d_j\left(x^{\prime }\right)=u_j\ast \overline{R}{d}_j\left(x^{\prime }\right)+l_j\ast \underline{R}{d}_j\left(x^{\prime }\right)$. The fuzzy upper and lower approximation of each training instance to each label are computed. The label of each training instance is known. The weights $u_j$ and $l_j$ are learned by the least squares method. The optimization objective is defined as follows:

$$\begin{array}{c}min{\sum }_{i=1}^n(d\left(x_i\right)-u_j\overline{R}d\left(x_i\right)-l_j\underline{R}d\left(x_i\right){)}^2\hfill \\ s.t.u_j\ge 0;\hfill \\ l_j\ge 0;\hfill \\ u_j+l_j=1.\hfill \end{array}$$

(7)

We now formulate the algorithm of FLDL-based fuzzy rough sets (FFRS), as shown in Algorithms 1 and 2.

Algorithm 1: Learning weights of upper and lower approximations

Input: Fuzzy label distribution system $U=\{x_1,x_2,\cdots ,x_N\}$, label set D for $\forall d_j\in D$ do end Output: $u={\left[u_j\right]}_{j=1}^L$ and $l={\left[l_j\right]}_{j=1}^L$

Algorithm 2: Fuzzy-rough-based Fuzzy label distribution classification

Input: Fuzzy label distribution system $U=\{x_1,x_2,\cdots ,x_N\}$, label set D, test    instance $x^{\prime }$, $u={\left[u_j\right]}_{j=1}^L$, and $l={\left[l_j\right]}_{j=1}^L$ for $\forall d_j\in D$ do end Output: label set $D\left(x^{\prime }\right)={\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L$

3.2. Example

Example 1. We apply an FLDL task in which we encounter four instances. The detailed information is shown in

Table 1. Fuzzy label distribution dataset.

U	C	${\mathit{d}}_1$	${\mathit{d}}_2$	${\mathit{d}}_3$
$x_1$	0.39	0.64	0.31	0.72
$x_2$	0.18	0.35	0.74	0.23
$x_3$	0.35	0.76	0.42	0.55
$x_4$	0.09	0.41	0.65	0.19

, where C is the condition attribute and $D=d_1,d_2,d_3$ is a complete label set.

We consider the Gaussian kernel function to compute the fuzzy equivalence relation between instances. The kernel parameter is set to 0.2. Given the Fuzzy label distribution training dataset and a test instance $x^{\prime }$, the value of condition attribute $C\left(x^{\prime }\right)=0.65$, and the label set $D\left(x^{\prime }\right)$ of $x^{\prime }$ is computed by FLDL-FRS.

First, the fuzzy equivalence relation between the training and test data is computed by the Gaussian kernel function.

$$\begin{array}{c}\hfill \begin{array}{c}R(x^{\prime },x_1)=0.7132,1-R(x^{\prime },x_1)=0.2868;\hfill \\ R(x^{\prime },x_2)=0.3314,1-R(x^{\prime },x_2)=0.6686;\hfill \\ R(x^{\prime },x_3)=0.6376,1-R(x^{\prime },x_3)=0.3624;\hfill \\ R(x^{\prime },x_4)=0.2085,1-R(x^{\prime },x_4)=0.7915.\hfill \end{array}\end{array}$$

Then, the fuzzy upper and lower approximations are computed by the Formula (5).

$$\begin{array}{c}\hfill \begin{array}{c}\overline{R}{d}_1\left(x^{\prime }\right)=\underset{x_i\in U}{sup}min(R(x^{\prime },x_i),d_1\left(x_i\right))=\{0.64,0.3314,0.6376,0.2085\}=0.64;\hfill \\ \overline{R}{d}_2\left(x^{\prime }\right)=\underset{x_i\in U}{sup}min(R(x^{\prime },x_i),d_2\left(x_i\right))=\{0.31,0.3314,0.42,0.2085\}=0.42;\hfill \\ \overline{R}{d}_3\left(x^{\prime }\right)=\underset{x_i\in U}{sup}min(R(x^{\prime },x_i),d_3\left(x_i\right))=\{0.7132,0.23,0.55,0.19\}=0.7132;\hfill \\ \underline{R}{d}_1\left(x^{\prime }\right)=\underset{x_i\in U}{inf}max(1-R(x^{\prime },x_i),d_1\left(x_i\right))=\{0.64,0.6686,0.76,0.7915\}=0.64;\hfill \\ \underline{R}{d}_2\left(x^{\prime }\right)=\underset{x_i\in U}{inf}max(1-R(x^{\prime },x_i),d_2\left(x_i\right))=\{0.31,0.74,0.42,0.7915\}=0.31;\hfill \\ \underline{R}{d}_3\left(x^{\prime }\right)=\underset{x_i\in U}{inf}max(1-R(x^{\prime },x_i),d_3\left(x_i\right))=\{0.72,0.6686,0.55,0.7915\}=0.55.\hfill \end{array}\end{array}$$

Finally, the label set of $x^{\prime }$ can be presented by three methods.

$$\begin{array}{c}\hfill \begin{array}{c}{D}_{upper}\left(x^{\prime }\right)=\{0.64,0.42,0.7132\};\hfill \\ D_{lower}\left(x^{\prime }\right)=\{0.64,0.31,0.55\};\hfill \\ D_{mean}\left(x^{\prime }\right)=\{0.64,0.365,0.6316\};\hfill \end{array}\end{array}$$

Given the Gaussian kernel as follows:

$$\begin{array}{c}\hfill \left[\begin{array}{cccc}1& 0.8021& 0.9992& 0.6376\\ 0.8021& 1& 0.8655& 0.9603\\ 0.9992& 0.8655& 1& 0.7132\\ 0.6376& 0.9603& 0.7132& 1\end{array}\right]\end{array}$$

The fuzzy upper and lower approximations and real label of each training instance are used to learn the weights.

$$\begin{array}{c}\hfill \begin{array}{c}{u}_1=0.8489,u_2=0.5543,u_3=0.4908;\hfill \\ l_1=0.1503,l_2=0.4457,l_3=0.5092.\hfill \end{array}\end{array}$$

The fuzzy upper and lower approximations of the test instance $x^{\prime }$ are computed by Formula (5). We obtain the label set of $x^{\prime }$ by the weighted average assignment method.

$$\begin{array}{c}\hfill D_{weight}\left(x^{\prime }\right)={\left\{d_j\left(x^{\prime }\right)\right\}}_{j=1}^L=\{0.64,0.3710,0.6301\};\end{array}$$

The real label is denoted as Original, and the four methods are denoted as FFRS-u, FFRS-l, FFRS-m, and FFRS-w. We can see that the top and bottom lines denote the upper and lower approximations, respectively. FFRS-m and FFRS-w, which are two lines in the middle, are close to the real label.

4. Experiments

In this section, we report some experiments to demonstrate the effectiveness of the proposed method over several fuzzy label distribution classification datasets.

4.1. Datasets

The CHEAVD dataset (http://www.chineseldc.org/emotion.html (accessed on 1 March 2023)), published by the National Laboratory of Pattern Recognition Institute of Automation at the Chinese Academy of Sciences, is a Chinese natural emotional visual-audio database. This dataset is a multi-modal fuzzy label distribution dataset. It contains 1981 samples, each of which contains eight emotions (happy, angry, surprised, disgusted, neutral, worried, anxious, sad). Each sample is described by 18 attributes. The first 10 descriptors are audio attributes and another eight are visual. Each video is associated with eight memberships of different emotions. We recognize the emotions of the videos according to the audio and image information. Some examples of the video faces in the dataset are shown in Figure 5.

Estimations of our algorithms are applied to fifteen label distribution classification datasets, including biological data, image data, and text data. These datasets are available in the Library of Label Distribution Learning (http://cse.seu.edu.cn/PersonalPage/xgeng/LDL/index.htm (accessed on 1 March 2023)). Brief introductions of these datasets are listed in

Table 2. Database information.

No.	Dataset	Instances	Features	Labels
1	Yeast-alpha	2465	24	18
2	Yeast-cdc	2465	24	15
3	Yeast-elu	2465	24	14
4	Yeast-diau	2465	24	7
5	Yeast-heat	2465	24	6
6	Yeast-spo	2465	24	6
7	Yeast-cold	2465	24	4
8	Yeast-dtt	2465	24	4
9	Yeast-spo5	2465	24	3
10	Yeast-spoem	2465	24	2
11	Human Gene	30,542	36	68
12	Natural Scene	2000	294	9
13	SJAFFE	213	243	6
14	SBU_3DFE	2500	243	6
15	Movie	7755	1869	5

.

The thirteenth and fourteenth datasets are two widely used facial expression image databases. The JAFFE database [39] contains 213 grayscale images and the bigger database BU-3DFE [40] contains 2500 images. In both databases, the instances are represented by the 243-dimensional LBP features [41]. Each facial expression image is associated with six basic emotions, i.e., happy, sad, surprised, afraid, angry, and disgusted. Sixty persons were employed to score each image on a five-level scale [42]. The average score of each emotion was used to represent the intensity of the emotion. As for the dataset SJAFFE (Scored JAFFE), each score indicates the degree of the corresponding emotion. Similarly, each image of the Scored BU-3DFE was scored by 23 persons in the same way as JAFFE. Other datasets were used for the LDL task.

4.2. Evaluation Measures for FLDL

There are many ways to measure the distance between possibility distributions, which in turn can be used to determine the distance between label sets. The measures for the distance, which reflected the difference between the label sets, were contrary to that for the similarity. The higher the distance between the label sets, the lower their similarity. Given two label sets $D=\{d_1,d_2,\cdots ,d_L\}$ and $\tilde{D}=\{{\tilde{d}}_1,{\tilde{d}}_2,\cdots ,{\tilde{d}}_L\}$ in FLDL, there are several measures for the distance.

In particular, Chebyshev distance (Cheb) and Clark distance (Clark) are the distance measurements of the probability distribution, and the smaller their value, the better the performance. Cosine coefficient (Cosine) and intersection similarity (Intersection) are the similarity measurements; the larger their results, the better the performance [43]. In this experiment, we use a 10-fold cross-validation method on each algorithm.

4.3. Experimental Setup

We design two experiments to verify the effectiveness of the proposed methods. One is on the fuzzy label distribution dataset, and the other is on the label distribution datasets. The proposed methods can not only deal with the fuzzy label distribution task but also with the label distribution task when the output of our algorithm is normalized to satisfy the probability distribution.

In the multi-modal Fuzzy label distribution dataset, as the kernel matrices are generated first, we only compared three un-weighted methods (FFRS-u, FFRS-l, FFRS-m) and one weighted method (FFRS-w) with the AA-kNN algorithm. The RBF kernel was used to compute the kernel matrices for audio features with different parameters. The hyper-parameter of the Gaussian function was set as the number of features. With regard to the visual attributes, by extracting the features from each video frame, we could record each video clip as a set of feature vectors. Based on the feature vectors, to describe the video, we introduce three types of models: covariance matrix, Gaussian distribution, and linear subspace on visual attributes. Subsequently, the Riemannian kernel was used to compute the kernel matrices and then they were normalized to a unit diagonal. Four evaluation measures based on the possibility distribution were used to verify the effectiveness of the FLDL algorithms, which are introduced in Section 4.2.

In label distribution datasets, we compare the proposed algorithm FFRS-w with four LDL algorithms, i.e., PT-SVM, PT-Bayes, AA-kNN, and AA-BP. The results are shown in Table 2. Four criteria, listed in

Table 3. Evaluation measures of LDL.

	Name	Formula
1	Hamming distance	$Dis(D,\tilde{D})=\frac{1}{L}{\displaystyle \sum _{l=1}^L}|d_j-{\tilde{d}}_j|$
2	Euclid distance	$Dis(D,\tilde{D})={\left(\frac{1}{L}{\displaystyle \sum _{l=1}^L}{(d_j-{\tilde{d}}_j)}^2\right)}^{\frac{1}{2}}$
3	Lambert distance	$Dis(D,\tilde{D})=\frac{1}{L}{\displaystyle \sum _{i=1}^L}\frac{|d_j-{\tilde{d}}_j|}{d_j+{\tilde{d}}_j}$
4	Fuzzy absolute distance	$Dis(D,\tilde{D})=\frac{{\displaystyle \sum _{l=1}^L}|d_j-{\tilde{d}}_j|}{{\displaystyle \sum _{l=1}^L}(d_j+{\tilde{d}}_j)}$

, were used to verify the effectiveness of LDL algorithms.

4.4. Results and Discussion

The results of four evaluation measures on the multi-modality FLDL dataset are shown in Figure 6. The four evaluation measures for FLDL, listed in

Table 4. Evaluation measures of FLDL.

	Name	Formula
1	Cheb	$Dis(D,\tilde{D})=\underset{1\le l\le L}{max}|d_l-{\tilde{d}}_l|$
2	Clark	$Dis(D,\tilde{D})=\sqrt{{\displaystyle \sum _{l=1}^L}\frac{{(d_l-{\tilde{d}}_l)}^2}{{(d_l+{\tilde{d}}_l)}^2}}$
3	Cosine	$Sim(D,\tilde{D})=\frac{{\displaystyle \sum _{l=1}^L}{d}_l{\tilde{d}}_l}{\sqrt{{\displaystyle \sum _{l=1}^L}{d_l}^2}\sqrt{{\displaystyle \sum _{l=1}^L}{{\tilde{d}}_l}^2}}$
4	Intersection	$Sim(D,\tilde{D})={\displaystyle \sum _{l=1}^L}min(d_l,{\tilde{d}}_l)$

are the distance measurements of the possibility distribution, and the smaller their value, the better the performance.

The results indicate that FFRS-w exhibited better performance than all other algorithms. FFRS-u and FFRS-l exhibited the worst performances of all four evaluation measures. The results were consistent with the previous analysis. FFRS-u and FFRS-l may lead to overestimation and underestimation. FFRS-m and AA-kNN were similar; they avoided the estimation problems by averaging the outputs. However, the average method can easily ignore the important information included in the data. Therefore, FFRS-w exhibited the best performance.

The mean and standard deviation of results are shown from

Table 5. Results (mean ± std (rank)) of the Chebyshev distance (the smaller the better).

No.	PT-Bayes	PT-SVM	AA-BP	AA-kNN	FFRS-w
1	0.174 ± 0.011	0.017 ± 0.002	0.036 ± 0.003	0.015 ± 0.001	0.014 ± 0.001
2	0.172 ± 0.016	0.020 ± 0.003	0.037 ± 0.002	0.018 ± 0.001	0.016 ± 0.001
3	0.166 ± 0.014	0.019 ± 0.001	0.036 ± 0.002	0.018 ± 0.001	0.016 ± 0.001
4	0.167 ± 0.007	0.046 ± 0.004	0.048 ± 0.003	0.039 ± 0.001	0.038 ± 0.002
5	0.176 ± 0.018	0.046 ± 0.001	0.052 ± 0.003	0.045 ± 0.001	0.043 ± 0.001
6	0.178 ± 0.009	0.065 ± 0.006	0.067 ± 0.005	0.064 ± 0.002	0.059 ± 0.004
7	0.177 ± 0.011	0.057 ± 0.003	0.057 ± 0.003	0.055 ± 0.002	0.052 ± 0.002
8	0.177 ± 0.010	0.040 ± 0.001	0.043 ± 0.002	0.039 ± 0.001	0.037 ± 0.001
9	0.211 ± 0.011	0.093 ± 0.006	0.094 ± 0.006	0.096 ± 0.005	0.093 ± 0.006
10	0.190 ± 0.017	0.091 ± 0.005	0.089 ± 0.005	0.093 ± 0.004	0.088 ± 0.005
11	0.195 ± 0.085	0.054 ± 0.004	0.059 ± 0.004	0.065 ± 0.005	0.053 ± 0.004
12	0.407 ± 0.027	0.414 ± 0.036	0.335 ± 0.016	0.374 ± 0.013	0.350 ± 0.015
13	0.121 ± 0.016	0.127 ± 0.017	0.130 ± 0.017	0.114 ± 0.017	0.120 ± 0.016
14	0.116 ± 0.004	0.119 ± 0.006	0.113 ± 0.005	0.103 ± 0.003	0.135 ± 0.005
15	0.199 ± 0.009	0.213 ± 0.039	0.157 ± 0.013	0.154 ± 0.005	0.129 ± 0.008

,

Table 6. Results (mean ± std (rank)) of the Clark distance (the smaller the better).

No.	PT-Bayes	PT-SVM	AA-BP	AA-kNN	FFRS-w
1	6.382 ± 0.305	0.921 ± 0.107	2.352 ± 0.173	0.758 ± 0.040	0.685 ± 0.047
2	4.987 ± 0.222	0.785 ± 0.084	1.718 ± 0.110	0.717 ± 0.041	0.649 ± 0.042
3	4.461 ± 0.282	0.691 ± 0.047	1.488 ± 0.098	0.644 ± 0.016	0.587 ± 0.019
4	1.744 ± 0.071	0.528 ± 0.031	0.568 ± 0.033	0.455 ± 0.011	0.439 ± 0.020
5	1.415 ± 0.102	0.396 ± 0.016	0.459 ± 0.031	0.392 ± 0.010	0.371 ± 0.008
6	1.473 ± 0.069	0.565 ± 0.049	0.599 ± 0.043	0.559 ± 0.024	0.515 ± 0.033
7	0.845 ± 0.059	0.267 ± 0.014	0.268 ± 0.015	0.260 ± 0.013	0.243 ± 0.009
8	0.846 ± 0.051	0.186 ± 0.008	0.204 ± 0.012	0.182 ± 0.007	0.172 ± 0.006
9	0.681 ± 0.038	0.287 ± 0.019	0.291 ± 0.018	0.297 ± 0.016	0.285 ± 0.019
10	0.424 ± 0.038	0.187 ± 0.011	0.184 ± 0.011	0.191 ± 0.008	0.183 ± 0.010
11	34.24 ± 3.634	14.63 ± 0.647	22.79 ± 1.841	16.28 ± 0.818	14.51 ± 0.650
12	7.149 ± 0.109	7.208 ± 0.205	6.767 ± 0.095	3.044 ± 0.137	6.663 ± 0.113
13	0.904 ± 0.086	0.935 ± 0.074	1.046 ± 0.124	0.843 ± 0.113	0.889 ± 0.085
14	1.116 ± 0.020	1.147 ± 0.064	1.051 ± 0.064	0.841 ± 0.014	0.900 ± 0.041
15	1.547 ± 0.075	1.537 ± 0.216	1.269 ± 0.089	1.276 ± 0.046	1.120 ± 0.050

,

Table 7. Results (mean ± std (rank)) of the Cosine coefficient (the larger the better).

No.	PT-Bayes	PT-SVM	AA-BP	AA-kNN	FFRS-w
1	0.743 ± 0.015	0.991 ± 0.002	0.949 ± 0.006	0.994 ± 0.001	0.995 ± 0.001
2	0.766 ± 0.017	0.991 ± 0.002	0.960 ± 0.004	0.992 ± 0.001	0.993 ± 0.001
3	0.780 ± 0.018	0.992 ± 0.001	0.965 ± 0.004	0.993 ± 0.001	0.994 ± 0.001
4	0.856 ± 0.007	0.982 ± 0.002	0.979 ± 0.002	0.986 ± 0.001	0.988 ± 0.001
5	0.866 ± 0.015	0.986 ± 0.001	0.981 ± 0.003	0.986 ± 0.001	0.988 ± 0.001
6	0.859 ± 0.008	0.971 ± 0.005	0.969 ± 0.004	0.972 ± 0.002	0.977 ± 0.003
7	0.898 ± 0.008	0.986 ± 0.001	0.986 ± 0.002	0.987 ± 0.001	0.988 ± 0.001
8	0.897 ± 0.008	0.993 ± 0.001	0.991 ± 0.001	0.993 ± 0.001	0.994 ± 0.001
9	0.893 ± 0.008	0.973 ± 0.003	0.973 ± 0.003	0.970 ± 0.003	0.974 ± 0.003
10	0.914 ± 0.011	0.976 ± 0.003	0.978 ± 0.002	0.975 ± 0.002	0.978 ± 0.002
11	0.456 ± 0.089	0.832 ± 0.011	0.726 ± 0.026	0.766 ± 0.020	0.834 ± 0.011
12	0.559 ± 0.014	0.490 ± 0.082	0.697 ± 0.011	0.624 ± 0.016	0.678 ± 0.010
13	0.930 ± 0.013	0.920 ± 0.014	0.908 ± 0.019	0.934 ± 0.018	0.931 ± 0.013
14	0.924 ± 0.004	0.914 ± 0.006	0.926 ± 0.006	0.938 ± 0.002	0.920 ± 0.005
15	0.850 ± 0.008	0.806 ± 0.061	0.895 ± 0.014	0.880 ± 0.006	0.926 ± 0.009

and

Table 8. Results (mean ± std (rank)) of the intersection similarity (the larger the better).

No.	PT-Bayes	PT-SVM	AA-BP	AA-kNN	FFRS-w
1	0.660 ± 0.016	0.949 ± 0.006	0.877 ± 0.008	0.958 ± 0.002	0.962 ± 0.003
2	0.681 ± 0.015	0.948 ± 0.006	0.891 ± 0.007	0.953 ± 0.003	0.957 ± 0.003
3	0.695 ± 0.019	0.951 ± 0.003	0.899 ± 0.006	0.955 ± 0.001	0.959 ± 0.001
4	0.764 ± 0.008	0.926 ± 0.004	0.922 ± 0.004	0.937 ± 0.002	0.939 ± 0.003
5	0.773 ± 0.018	0.935 ± 0.003	0.925 ± 0.005	0.936 ± 0.002	0.939 ± 0.001
6	0.765 ± 0.010	0.906 ± 0.008	0.902 ± 0.007	0.908 ± 0.004	0.915 ± 0.005
7	0.802 ± 0.012	0.934 ± 0.004	0.934 ± 0.004	0.936 ± 0.003	0.940 ± 0.002
8	0.801 ± 0.011	0.954 ± 0.002	0.950 ± 0.003	0.955 ± 0.002	0.958 ± 0.001
9	0.789 ± 0.011	0.907 ± 0.006	0.906 ± 0.006	0.904 ± 0.005	0.909 ± 0.006
10	0.810 ± 0.017	0.909 ± 0.005	0.911 ± 0.005	0.907 ± 0.004	0.912 ± 0.005
11	0.470 ± 0.062	0.781 ± 0.010	0.671 ± 0.025	0.742 ± 0.014	0.784 ± 0.009
12	0.350 ± 0.014	0.364 ± 0.055	0.499 ± 0.012	0.544 ± 0.018	0.556 ± 0.014
13	0.846 ± 0.016	0.839 ± 0.015	0.824 ± 0.022	0.855 ± 0.021	0.848 ± 0.016
14	0.834 ± 0.003	0.827 ± 0.009	0.847 ± 0.008	0.872 ± 0.002	0.839 ± 0.007
15	0.725 ± 0.011	0.711 ± 0.052	0.788 ± 0.015	0.780 ± 0.007	0.864 ± 0.015

. The best results are highlighted in boldface. In order to evaluate the performance of each algorithm on all datasets, the average ranks of each algorithm are listed in the bottom row.

The results demonstrate that the proposed FFRS-w algorithm exhibited the best performance. AA-kNN exhibited better performance than all other algorithms, while FFRS exhibited better performance than AA-kNN in most datasets. The average ranks of these algorithms are shown in Figure 7, where FRFS-w always ranks at the top among all four measurements.

AA-kNN maintained the structure information of the labels. However, PT-SVM destroyed the distribution structure during the process of transforming the data, and therefore, PT-SVM exhibited results that were worse than AA-kNN. PT-Bayes exhibited good performances when the data matched the Gaussian assumption required by the algorithm. However, its performance fell when operated on other real datasets. Too many parameters in AA-BP led to poor performance because it is difficult to find the optimized settings.

The FFRS-w algorithm learns more information from the data. The lower and upper approximations are the measures of necessity and possibility. They are combined to form the interval, of which the weights of each label are trained and used to evaluate the test data. This process improves the robustness of the algorithm. Therefore, FFRS-w exhibits better performances than AA-kNN.

5. Conclusions

In this paper, we propose a fuzzy learning-based expression recognition method for human emotion prediction. We construct a fuzzy label distribution system using fuzzy sets and present two fuzzy label distribution prediction methods based on fuzzy rough sets. To compare our algorithm with the label distribution approach, we conduct experiments on the fuzzy label distribution task and the label distribution task, respectively. The experimental results show that the accuracy of the proposed method is superior to the other methods on both tasks. The other methods cannot directly deal with the human emotion fuzzy label distribution task; therefore, the proposed method is a more general label distribution prediction method that is more suitable for deployment on AI IoT.

This work only proposes the framework of fuzzy learning for label fuzzy distribution, and there are still some drawbacks: firstly, the accuracy of the proposed method, although better than the label distribution learning, still has some room for improvement; secondly, the efficiency of the proposed method and how to quickly output the fuzzy prediction of multiple composite labels; finally, the scalability of the proposed method, and how to extend the proposed method to multi-modality data.

Therefore, in future work, we will try to improve the following aspects: firstly, the fuzzy rough set can be replaced by a more advanced fuzzy learning method such as intuitionistic fuzzy rough set and neutrosophic rough sets, thus further improving the accuracy of the method; secondly, the parallel computing framework of the proposed method is increased to improve the efficiency of the framework; finally, we study how to extend the fuzzy relationship calculation method between samples so that it can be applied to multi-modality data.