A Unified Framework of Deep Learning-Based Facial Expression Recognition System for Diversified Applications

Abstract

This work proposes a facial expression recognition system for a diversified field of applications. The purpose of the proposed system is to predict the type of expressions in a human face region. The implementation of the proposed method is fragmented into three components. In the first component, from the given input image, a tree-structured part model has been applied that predicts some landmark points on the input image to detect facial regions. The detected face region was normalized to its fixed size and then down-sampled to its varying sizes such that the advantages, due to the effect of multi-resolution images, can be introduced. Then, some convolutional neural network (CNN) architectures were proposed in the second component to analyze the texture patterns in the facial regions. To enhance the proposed CNN model’s performance, some advanced techniques, such data augmentation, progressive image resizing, transfer-learning, and fine-tuning of the parameters, were employed in the third component to extract more distinctive and discriminant features for the proposed facial expression recognition system. The performance of the proposed system, due to different CNN models, is fused to achieve better performance than the existing state-of-the-art methods and for this reason, extensive experimentation has been carried out using the Karolinska-directed emotional faces (KDEF), GENKI-4k, Cohn-Kanade (CK+), and Static Facial Expressions in the Wild (SFEW) benchmark databases. The performance has been compared with some existing methods concerning these databases, which shows that the proposed facial expression recognition system outperforms other competing methods.

1. Introduction

Facial expressions [1] are a crucial non-verbal method of indicating meaning and represent a unique, universal way for people to communicate. The facial expression recognition system (FERS) is a contactless recognition system in which the image of a human face of a person can be captured from a distance without any intervention or interruption, even when he/she is moving around, walking, sitting, or performing activities [2]. Facial expressions play an essential role in our daily communication with people and in social interactions [3]. The FERS is mainly used to identify types of human facial expression [4]. According to Ekman et al. [5], there are six basic expressions, including a neutral face as a baseline reference. Figure 1 shows some examples of human facial expressions, e.g., fear (FA), anger (AN), disgust (DI), happy (HA), neutral (NE), sad (SA), and surprise (SU).

The FERS [6] is an emergent research topic in computer vision research areas. It has comprehensive potential in a diversified field of applications [7] with various challenges in healthcare, education, marketing research, business organization, customer and retail fields, government, entertainment, and within the Internet of Things (IoT). There exists several vendors such as Microsoft, IBM, Amazon, and Google that provide some application programming interfaces using facial expressions but with limited solutions. Here, these diversified fields of application for FERS are discussed in terms of the four major concerns, which are as follows:

■

e-Healthcare:- The real-time FERS incorporates the healthcare system via the system that is used to analyze and detect the image’s visualization of the patient’s feelings remotely by identifying facial expressions [8] for patients with different ages, puberty levels, and genders collected from a giant cloud and from social networks. The m-Health provides mobile device-based practices to patients to support their medicine administration and daily healthcare facilities. e-Health is an electronic health service that uses information and communication technology for delivering facilities digitally and for processing patients and doctors through computers for drug administration. Both e-Health and m-Health provide immense support to healthcare industries in building e-Healthcare systems to ensure that patients, doctors, medical professionals, and businesses benefit, as well as to ensure the establishment of a healthy civilization with technological advancements in smart cities. Electronic healthcare systems provide services to patients to physically localize and monitor through recognizing their voice, speech, gesture movement, and facial expressions. Our proposed facial expression recognition system (FERS) will improve the services of healthcare systems. It is a significant challenge to obtain good results in the context of more efficient and less costly health services. Hence, while integrating the FERS into the healthcare framework, all healthcare requirements, such as automated intelligent sensors, sophisticated tools, security, authenticity, access, and privacy, should also be considered.

■

Social IoT:- Social IoT systems represent an evolution of IoT-based systems. It establishes a platform for interconnecting subjects or objects worldwide through social relationships. It provides better services to users by relaxing the common interests between the users. Now, the services of social IoT are exploited in emotion-recognition as these emotions relate to the social activities of humans in their daily life. Hence, the integration of social IoT services will make life easier with several social care facilities for people [9]. The proposed FERS is useful for developing IoT-based smart devices and appliances. It can be used for several entities such as education, marketing research, retail, government, media and content, gaming, and finance. During online teaching, the facial expressions of students can be compared to their interest and understanding of topics that have been taught to them. The sentiments from online trading and investment strategies will be beneficial for the financial development of the organization. The emotion analysis using customer reviews and shopkeepers’ experience will bring good marketing research for the organization.

■

Emotion AI:- Emotion AI [10] has wide applications in human resource management, such as in any business organization. It helps the human resource management system (HRMS) during the recruitment of a candidate for selection. This emotion AI considers several traits such as voice and text to analyze the sentiments in candidates.

■

Cognitive AI:- Cognitive AI [11] provides methods and technologies to build a decision-making system based on the behavior and reasoning ability of a person. It helps a person to make decisions through a system. Job searching, salary prediction, carrier path selection for job-seeker problems, cyber-security with enabled AI, and natural language processing for sentiment analysis problems are under cognitive AI categories. Thus, social interaction, planning, interpretation, decision-making, competence of emotion, and self-learning capabilities are the processes of cognitive AI.

These applications use facial images for recognizing expressions in humans. The psychology of facial expression [12] states that the face is the key to understanding emotions. Linking the face to emotions may be an important idea in the psychology of emotions. The facial expression recognition system works on the facial movements [13], which are described by the facial action coding system (FACS). The FACS breaks down facial expressions into action units that introduce a distinct change in the facial appearance. There are various uses of FACS for discovering disorders in neuropsychiatric and social–emotional development that are performed through psychological research. The FACS is an immediate, powerful, and effective non-verbal communication tool to transit messages and convey emotional information. In most of the implementation cases of FERS, the facial region is analyzed as a texture where numerous techniques such as statistical and structural-based methods have been employed to extract discriminant features [14]. Apart from these techniques, recently, deep learning-based approaches with convolution neural networks [15] have been employed to extract more discriminant and distinctive features to ensure that a better performance can be obtained. However, most of these methods are database-dependent and these databases have been captured spontaneously under controlled environments [16] with tightly controlled illumination, age, and pose variation conditions.

Despite the current state-of-the-art methods for the FERS and their significant progress in effective computing, they still suffer from some limitations: (i) The employed datasets are either laboratory-controlled or wild. These images are captured under unconstrained environments and the images suffer from several challenging issues such as illumination, poor resolution, occlusion, pose, age, and expression variations. Thus, the extraction of the face region from the input images in optimal time is also a challenging issue. (ii) Due to limited domain knowledge, the local to global feature representation schemes generate less discriminative and distinctive patterns. (iii) The assumption of the feature selection might not be valid, i.e., the extraction of local geometric information or action units’ geometric features is not valid. Hence, we have proposed a novel deep learning-based framework for the FERS to address these problems and improve its usefulness in diversified fields of applications such as in e-Healthcare, social IoT, and emotion AI. The contributions of the proposed work are as follows:

• We have designed a fast and efficient end-to-end deep learning-based framework using the convolutional neural network approach for learning face representation by adding some extra levels of feature representation schemes to improve the robustness and generalization of the model.
• The obtained predictive model detects and learns powerful high-level features from the input image and extracts more distinctive and discriminant features that provide effective results for the proposed FERS under various illumination changes as well as pose and age variation artifacts.
• To enhance the performance of the FERS, several experiments have been carried out with a trade-off between the batch vs. epoch, data augmentation, progressive image resizing, hyper-parameter tuning, and transfer learning techniques for the better prediction of expression types on the human face and for improvement of the performance as well as robustness of the proposed system.
• The proposed method finds the solution for the challenging issues of FERS. At the same time, a series of experiments have been conducted to reduce the training loss and over-fitting problems that arise due to inadequate training data and bias in the expressions’ variation.

The organization of this paper is as follows: Section 2 describes the related work for the proposed system. The proposed facial expression recognition system (FERS) is discussed in Section 3, which describes the face pre-processing techniques and the proposed CNN architectures for the feature computation of both frontal and profile facial images. The database description, experimental results, and discussions are described in Section 4. Finally, Section 5 concludes this paper.

2. Related Work

An automatic facial expression recognition and classification for multi-pose and multi-level face images have revealed to be an attractive and challenging problem since the last thirty years [17]. A literature review stated that early stages of research has focused on several statistical and structural-based methods [14]. In contrast, some [17] template-based and feature-based approaches have also been investigated. The classical methods, such as the Histogram-of-Orientation Gradient (HOG) [18], the Scale Invariant Feature Transform (SIFT) [19], LBP (Local Binary Pattern) [20] features, and some spatio-temporal features (STM-ExpLet [21]), have been adopted by many researchers to obtain texture features in statistical ways; however, these methods require great effort to achieve high performance. Recently, researchers have used convolutional neural networks (CNN, ConvNets) [22] and have achieved great success for large-scale static images and sequences of video recognition [23]. The CNN has been widely applied for the FER system and has significantly improved state-of-the-art practices as well as analyzed the performance of ImageNet classification challenges [22]. Earlier CNN models were used to solve character recognition tasks [24], but nowadays, CNN is widely used in various object recognition problems. Here, the most important ingredient for the success of CNN is the availability of large quantities of training data, i.e., the use of image augmentation techniques [15]. Additionally, the CNN achieves high performance by learning powerful high-level features by combining global appearances to local geometric features rather than conventional handcrafted features. However, the training image samples suffer from the lack of intensity noises, illumination, pose and expression variation, motion blur, low resolution, and occlusion by hair artifacts. The CNN aims towards the application of people-sentiment analysis; application to multi-modal human–machine or computer interactions; and application to intelligent systems with their challenges that arise when capturing images under an unconstrained imaging environment.

Depending on the existing state-of-art methods for face representation and facial expression, recognition could be broadly classified and analyzed into two categories: appearance-based methods and facial action units-based methods. In the appearance-based methods, the entire face region is divided into several blocks or patches and the features are extracted from these patches using the Local Binary Pattern (LBP) [25], Histogram of Oriented Gradients (HOG) [26], and Scale Invariant Feature Transform (SIFT) [19], as explored by Zhao et al. [27] and others. Facial action unit-based methods usually exploit the face geometrical information or face action units-driven representation for facial expression classification. Tian et al. [28] used the positions of facial landmarks for facial action unit recognition and then performed expression classification. The appearance-based method [29] is the most successful and well studied for face recognition. There are several works in which the whole face image captured in controlled-lab conditions was taken as the input image ${\mathsf{ℐ}}_{m\times n}$ to create a subspace based on the reduction of inconsistent and redundant face space dimensionality reduction techniques [30]; for instance, Fisher LDA, PCA, and LPP [31] had been adopted. A comparative literature review of these methods for facial expression recognition have been done in [32]. The LDA and PCA practically are based on the kernel methods. The Euclidean structure and miscellaneous learning methods [33] have been employed for face recognition [34]. The computational cost of these techniques is expensive and some of these systems may fail due to the system explicitly exhibiting the exact structure of the manifold. However, these are powerful tools based on statistical signal-modeling, which is known as sparse coding. The sparse coding provides beautiful results for the facial expression recognition [35] system. Instead of these handcrafted features, deep learning methods have been assumed to be a breakthrough in computer vision and have broken the world record in the field of recognition task problems.

Many state-of-the-art methods and deep learning frameworks use hand-labeled points and CNN architecture for both feature extraction and built facial expression recognition systems. Gutta et al. [36] proposed a model with an ensemble radial basis function, a grayscale image, and inductive decision trees for the four classes (i.e., Asian, Caucasian, African, and Oriental) ethnicity recognition problem. Zhang and Wang [37] proposed a method for two-class racial classification using multi-scale LBP (Local Binary Pattern) texture features while combining 2D and 3D texture features. Zhang et al. [38] described two types of features, namely the geometry-based features and Gabor-wavelets-based features for the FER System. Bartlett et al. [39] applied the Gabor filters coupled with feature selection and machine learning techniques for recognizing facial expressions on a human face. In [40], Rose applied Gabor and log-Gabor filters on low-resolution images for facial expression recognition. Wu et al. [41] explored the Gabor motion energy filters [42] to recognize the dynamic facial expressions of individuals. Gabor filters together with genetic algorithms (GA) and SVM for the analysis of six basic facial expressions from video sequences were employed in [25]. In [43], Gu et al. proposed a method for facial expression recognition based on the radial encoding of local Gabor features with classifier synthesis. Almaev et al. [44] proposed a new dynamic feature descriptor called the Local Gabor Binary Patterns from Three Orthogonal Planes (LGBP-TOP) by combining LBP-TOP [45] and Gabor filters.

The major problems that occur during the development of the FER System concern shallow features and bias caused by various cultures and collection conditions. Current datasets have a strong build-in bias and the corresponding proposed methods show that the conditional probability distribution between training and testing datasets are different. We will assess this bias and present novel deep CNN models to address these issues. In our proposed methodology, we considered face recognition as an image classification problem. This face recognition definition has been extended to our work for the classification of facial expressions on human faces. The proposed FER system is based on two backbones: (1) face preprocessing and (2) the design and analysis of features from the proposed CNN architectures. The proposed CNN architecture is built using several convolutional layers, max-pooling, batch normalization, and dropout layers with an optimizer followed by the soft-max classifier for the final classification tasks. Our extensive random experimental results show that our proposed deep-CNN method achieves superior results for facial expression recognition problems for both lab-controlled and real-world databases. The principal issues involved in the facial expression recognition system design are face representation and classifier selection [31]. The face representation concerns extracting feature descriptors from the input face image that minimize the intra-class similarities and maximize the inter-class dissimilarities. In the case of classifier selection, it does not make sense that the high-performance classifiers always find a better separation between different classes even if there are significant similarities. Sometimes, the most sophisticated classifier may fail to execute the facial expression recognition and classification tasks due to inadequate face representations. We cannot achieve high-performance recognition accuracy if we employ good face representation but do not select a good classifier. Hence, the below sections describe the proposed FER system.

3. Proposed Methodology

In this work, we have proposed a facial expression recognition system (FERS) in the diversified fields of applications, such as e-Healthcare, social IoT, emotion AI, and cognitive AI. The block diagram of the proposed method is demonstrated in Figure 2. Since these fields belong to interdisciplinary research areas, the algorithms and techniques employed during the implementation of these frameworks are interconnected. Thus, the proposed FERS will be used as the common platform for analyzing expressions in the applications of these frameworks. Furthermore, the implementation of a basic FERS is discussed in the following paragraphs.

A facial expression recognition system generally consists of face representation, feature extraction, and classifier components. Regarding the importance of face recognition in computer vision research areas, we have proposed robust, efficient, and accurate deep convolution neural network (CNN) models for facial expression recognition systems. Here, an image ${\mathsf{ℐ}}_{m\times n}$ with a valid face region is used as an input to the system. Our objective is to predict the types of expressions, such fear, anger, disgust, surprise, sadness, happiness, and neutral, from the input face region ${\mathsf{ℐ}}_{m\times n}$. The proposed facial expression recognition system (FERS) has been implemented in four steps: (i) pre-processing, wherein the bounded box face region is detected from the input image using the tree structure part model [38]; (ii) feature extraction, wherein the global generic CNN features are extracted from the detected bounding box face region and are prepared for the next level through the deep learning model; (iii) the representations are further modified by using multi-stage progressive image resizing followed by transfer learning methods, wherein image augmentation and fine-tuning of parameters have also been adopted; and (iv) classification, which concerns predicting the type of expression classes of the facial region. Each of these steps is described in the block diagram of the proposed system, which is presented in Figure 2. Recently, deep convolution neural network (CNN) techniques have been successfully developed to learn discriminative features in various fields. It is widely being used in deep FER representation. Deep FER suffers from the over-fitting problem due to the lack of sufficient training samples, age variations, head poses, identity bias, and illumination variations. The proposed method focuses on these issues and overcomes the computational complexity of the proposed system.

3.1. Face Preprocessing

We have implemented a deep learning framework to recognize discrete human facial expression categories in this proposed work. The input face image has been resized to the same size and was normalized to a fixed size face image ${\mathsf{ℐ}}_{m\times n\times 3}$. Here, these input images are mapped to the same locations, i.e., eye locations, the tip of the nose, etc., are known as a feature map. At the lowest level of abstraction, it is assumed that preprocessing is a standard term that concerns computing over intensity images. These input and output intensity images are the same as the original data captured by the sensor. A matrix of image function values usually represents an intensity image. The goal of preprocessing is to enhance the expression of the region of interest and to suppress the unwanted, redundant, and inconsistent noises in the image. Image preprocessing methods are classified into four categories according to the size of the pixel neighborhood that is used for the calculation of new pixel brightness: pixel brightness transformations; geometric transformations; certain preprocessing methods that use a local neighborhood of the processed pixel; and image restoration that requires knowledge about the entire image. Here, the required face region is detected from the input image using a tree-structured part model [46]. The detected face has been resized to a fixed image ${\mathsf{ℱ}}_{n\times n}$. These face images are used as input to the proposed CNN models. During preprocessing, we extracted the face region from each input image ${\mathsf{ℐ}}_{m\times n}$. Since the facial expressions contained very minute details, it is important to be conscious about analyzing both expressive or non-expressive characteristics of the facial region. During face preprocessing, we applied the tree-structured part model, which works better for both frontal and profile face regions compared to Haar-like features [47]. This model has outstanding performance results compared to the other face detection algorithm in computer vision. The tree-structured part model works on the principle of a mixture of trees with a global mixture of topological viewpoints changing. For an unconstrained image with an unknown face region, this model locates all the facial landmarks in ${\mathsf{ℐ}}_{m\times n}$. For facial landmark localization, we consider ${\mathsf{ℒ}}_q^p=({\mathbf{x}}_q^p,{\mathbf{y}}_q^p)$ as the coordinate for the pixel location of part q. Hence, the tree-structured part model computes thirty-nine landmark points for profile faces, while computing sixty-eight landmark points for the frontal faces. These landmark points undergo the computation of four corner points of the face region ${\mathsf{ℱ}}_{n\times n}$. The face preprocessing steps are shown in Figure 3.

3.2. Feature Representation for Expression Classification

Feature extraction is a crucial task to extract discriminating features from the input image ${\mathsf{ℱ}}_{n\times n}$ to ensure that the extracted feature contains more distinctive patterns [48]. Here, the input image may be a grayscaled or RGB color image. In the field of computer vision and in image processing research areas, feature extraction starts from an initial set of measured data and builds the features which are supposed to be informative and non-redundant, facilitating the subsequent learning and generalization steps. In many cases, the texture feature extraction techniques [49] lead to better human interpretations. Moreover, it is related to dimensionality reductions, i.e., when the input image size is too large to be processed as the representation for that image, it is transformed into a reduced set of features, also called a feature vector. The modern state-of-the-art technique for generic CNN feature representations for facial expression recognition problems could compete with the statistical and structural-based methods of computer vision. These generic CNN features can cope with the articulation and occlusion face images captured in an unconstrained environment and can achieve better performance.

The proposed method describes a complex CNN baseline model with two components, i.e., feature extraction and classification parts. The proposed CNN model uses a convolutional neural network architecture with five to seven deep image perturbation layers. The model performs convolution operations using the ReLU activation function followed by max-pooling and batch normalization operations for feature extraction. Finally, two flatten layers, which are fully connected, are used for classification tasks on the extracted feature maps from the top of the layers. The performance for the proposed CNN has been increased through adding new levels by applying image augmentation and progressive image resizing methods. These also help the model to prevent the over-fitting and imbalanced data problem. Progressive image resizing methods support the model to avoid the use of excessive computational power. Considering only the pre-trained weights of the last few layers are being used, these weights have to be learned properly. We take advantage of image augmentation, batch normalization, the activation function, and regularization methods including the mix-up optimizer and label smoothing techniques. The convolution operation is the primary operator and main building block of a CNN architecture. The term convolution is a mathematical operation that combines two functions and generates a third function. Here, it is used to extract features from the images. In the case of a CNN model, the convolution operation is executed over the input image with the help of a $t\times t$ sized kernel or filter and then generates feature maps. The convolution operation is performed by sliding the filter followed by non-linearity over the input. At every location, matrix multiplication is performed and sums the result onto the feature map. Finally, we used a fine-tuning of the parameters and fusion methods to enhance the performance of the proposed recognition system. Thus, the descriptions of the employed layers for the proposed CNN architecture are as follows:

• Convolution:- The Convolution layer is the core building block of a CNN model that performs most of the computation operations. Convolution is a linear matrix operation consisting of some set of kernels or filters ${\mathrm{W}}_{t\times t}$. The kernel is a small-sized matrix of weights that slide over the input [50] and performs element-wise matrix multiplication. The convolution operation essentially performs dot products between some sets of learnable filters ${\mathrm{W}}_{t\times t}$ and local regions of the input image ${\mathsf{ℱ}}_{n\times n}$, and produces an output matrix of dimension $n^{\prime }\times n^{\prime }$. Here, $n^{\prime }$ is calculated by $n^{\prime }=\frac{n-t+2\times \mathsf{𝒫}}{\mathsf{𝒮}}+1$, where $\mathsf{𝒮}$ is the stride that governs how many numbers of cells will be moved by the filter to the right and down, from the top-left corner to the bottom-right corner, in the input image to calculate the next cell in the result. Additionally, $\mathsf{𝒫}$ is the padding that shrinks the height and width of the volumes. Mathematically, the formulation of the convolution operation is denoted as follows [51]: for input feature vector $\mathrm{F}=\mathbf{f}\left(v\right)$ and a filter vector $\mathrm{W}=\mathbf{w}\left(v\right)$, the convolution operation is obtained as $\mathbf{F}★\mathbf{W}={\sum }_{u\in \mathbf{U}}\mathbf{f}\left(u\right)\mathbf{w}(v-u)=\langle \mathbf{f}\left(u\right),\mathbf{w}(v-u)\rangle$, where the operator ★ denotes the convolution operation and $\langle .,.\rangle$ represents the sliding vector inner product between the input feature $\mathbf{f}\left(u\right)$ and the flipped kernel $\mathbf{w}(v-u)$. It measures the similarity between the two vectors. The primary benefits of the convolution operation are: (i) parameter or weight sharing, as a feature detector is used in one part and transfers into other parts of the image; (ii) the fact that it reduces the number of effective parameters and image translation; and (iii) the sparsity of connections, i.e., the hidden layers’ input and output dependencies.
• Max-pooling:- A pooling operation is a mathematical operation that performs pixel-wise average or median operations to reduce the input image size by half its size. The effective advantages of using pooling operations concern a means of removing noise, correcting images, and overcoming incidental occlusions [52]. The pooling layer is used to reduce the size of the representation to speed up the process as well as to make some of the features it detects more robust. There are different types of pooling operations, such as average pooling, fractional max-pooling, and max-pooling. Max-pooling is a commonly used pooling operation that is used in most CNN models. Max-pooling calculates the maximum value for patches of a feature map and uses it to create a down-sampled feature map. It is usually used after a convolutional layer. The primary benefits of max-pooling are as follows: (i) it is a translation invariance, i.e., it translates the image by a small amount that does not significantly affect the values of most pooled outputs; (ii) has reduced computational costs; (iii) has faster matching; and (iv) has improved accuracy.
• Fully Connected Layers:- It has been stated that fully connected layers and convolutional layers are distinct, but it has been observed that fully connected layers are a special case of convolutional layers [53]. In our proposed CNN model, we used two fully connected layers denoted as ${\mathsf{ℱ}\mathsf{𝒞}}_1$ and ${\mathsf{ℱ}\mathsf{𝒞}}_2$. Here, $n_2$ neurons in ${\mathsf{ℱ}\mathsf{𝒞}}_2$ have full connections to all activation $n_1$ in ${\mathsf{ℱ}\mathsf{𝒞}}_1$. The activation function can be computed with a matrix multiplication followed by a bias offset. Let $x\in {\mathbf{R}}^{n_1\times 1}$ represent the single output vector of layer ${\mathsf{ℱ}\mathsf{𝒞}}_1$ and let $\mathbf{W}\in {\mathbf{R}}^{n_1\times n_2}$ denote the weight matrix of the ${\mathsf{ℱ}\mathsf{𝒞}}_2$. Suppose $w_i$ is the weight vector of the corresponding $i\mathrm{th}$ neuron of the column vector of $\mathbf{W}$ in layer ${\mathsf{ℱ}\mathsf{𝒞}}_2$ [54]. Then, the output of ${\mathsf{ℱ}\mathsf{𝒞}}_2$ is obtained by ${\mathbf{W}}^{\mathcal{T}}\times x$. The output of fully connected layers is independent of the input image size. Fully connected layers of a CNN architecture will reduce the full image size, compute the single vector of class scores, and produce a resulting vector of size $[1\times 1\times {\mathsf{𝒞}}_i]$.
• Dense Layers:- The dense layer is a type of fully connected connection layer in deep neural networks [55]. In a dense layer, all input layers are connected to the output layers by a weight. It performs linear operations with ${\mathsf{𝒳}}_{inputs}$ parameters and generates ${\mathsf{𝒳}}_{output}$ parameters [56] that are also connected to the next layer as inputs. It utilizes dense connections between layers with matching feature map size ${\mathsf{𝒳}}_l^{\prime }=g^{\prime }\left({\mathbf{W}}^T{\mathsf{𝒳}}_{l-1}^{\prime }\right)$, where $g^{\prime }$ is the activation function, e.g., ReLU defined as $p\left(x\right)=max(0,x)$.
• Batch Normalization:- Batch is used to normalize the inputs of the previous layers at each batch, maintaining the values in a comparable range with the mean equal to 0 and the standard deviation equal to 1. This helps the CNN model to prevent skews at any one particular point and increases the computation speed. We applied the batch normalization after every convolution layer and then passed these values to the ReLU activation function. Batch normalization acts as a regularizer and allows the model to use higher learning rates [57]. It is used in various image classification problems and achieves higher accuracy with fewer training steps. Batch normalization also has a beneficial effect on the gradient flow through the network by reducing the dependence of gradients on the scale of their parameters or initial values. It also regularizes the model and reduces the need for dropout layers. We calculated the batch normalization mathematically as follows: For a a mini-batch $\chi$ of size m and with values of $x^{\left(l\right)}$, i.e., activation and omit $\mathit{l}$ for clarity, the mini-batch is expressed as $\chi =\left(x_{1\dots m}\right)$. $\theta$ and $\psi$ are the learning parameters, $\left({\widehat{x}}_{1....m}\right)$ are normalized values, and $y_{1....m}$ are their corresponding linear transformations denoted by batch normalizing transform, i.e., ${\mathsf{ℬ}\mathsf{𝒩}}_{\theta ,\psi }:x_{1...m}\to y_{1...m}$. Thus, consider the following: mini-batch mean, ${⪯}_{\succ }=\frac{1}{m}{\sum }_{i=1}^m{x}_i$; mini-batch variance, ${\supset }_{\succ }^{\in }=\frac{1}{m}{\sum }_{i=1}^m{(x_i-{\mu }_{\chi })}^2$; normalization, $\widehat{x_i}=\frac{x_i-{\mu }_{\chi }}{\sqrt{{\supset }_{\succ }^{\in }+\zeta }}$; and scale and shift, $y_i\leftarrow \theta \widehat{x_i}+\psi ={\mathsf{ℬ}\mathsf{𝒩}}_{\theta ,\psi }\left(x_i\right)$.
• Regularization:- Regularization strategies are designed to reduce the test error of a machine learning algorithm, possibly at the expense of the training error [58]. The popular regularization methods that exist in the field of deep learning [59] are dropout, R1-regularization, and discriminative regularization, among others. We employed the dropout regularization technique on the penultimate layer $\alpha =\left[{\overrightarrow{a}}_1,{\overrightarrow{a}}_2,...,{\overrightarrow{a}}_{\mathrm{F}}\right]$ ($\mathrm{F}$ are the numbers of filters) for our proposed deep CNN model with constrain: ${\ell }_2$ norms of the weight vector [60]. The dropout regularization technique drops a unit during the training time with a specified probability. Dropout prevents co-adaptation of the network’s hidden units by randomly dropping out a portion or setting the hidden units to zero during forward and backward propagation. The neural network becomes too reliant on particular connections. Instead of using $\gamma =\omega \times \alpha +\delta$ for output hidden unit $\gamma$ in forward propagation, here, dropout uses $\gamma =\omega \times (\alpha \otimes \beta )+\delta$, where the operator ’⊗’ performs element-wise matrix multiplication and $\beta \in {\mathbf{R}}^{\mathrm{F}}$ is the masking vector of the Bernoulli random variable. At test time, all units are present and the learned weight vectors are scaled by $\mathsf{𝒫}$ such that $\overrightarrow{\omega }=\mathsf{𝒫}\times \omega$, where $\overrightarrow{\omega }$ represents the class score computed without dropout. The advantage of using dropout is that it prevents artificial neural networks from over-fitting. Intuitively, dropout can be thought of as creating an implicit ensemble of neural networks. This means that a selected subset of units for each training sample, including their incoming and outgoing connections, are temporarily removed from the network. Suppose a dropout probability of 0.5 is used; in this case, roughly half of the activation in each layer is deleted for every training sample, thus preventing hidden units from relying on other hidden units present.
• Optimisation:- The proposed FERS problem has been solved by stochastic optimization methods to optimize our CNN models. In this study, we used the popular first-order gradient-based Adam optimizer of the stochastic objective function. The popular optimization methods used for solving FERS problems are Adagard, SGD, RMSProp, SGD with momentum, AggMo, Demon, Demon CM, DFA, and Adadelta optimization methods. They use their stochastic mini-bath method. This method estimates the learning rate based on lower-order momentum. Adam [61] uses only the first two moments of gradient ${\tilde{v}}_t$ and the learning rate or steps size $\eta$. The weight updates for the Adam optimizer are mathematically calculated as $w_t=w_{t-1}-\eta \frac{\widehat{h}}{\sqrt{{\tilde{v}}_t}-ϵ}$, where $ϵ$ is a smaller number. The primary advantages of using the Adam optimizer are that it works well and is suitable for problem-solving for large training data sets. Adam can handle non-stationary objective functions as in RMSProp while overcoming the sparse gradient issue drawbacks that appear in RMSProp. Adam is favorable compared to other stochastic optimizers. The implementation of Adam is straightforward and computationally efficient with less memory required.

The proposed CNN architectures are based on several blocks as discussed in the previous section. Here, an input image ${\mathsf{ℱ}}_{n_H\times n_W}$ is convolved with a set of kernels of size $t\times t$. These convolution layers are called feature maps. The feature maps are stacked to provide multiple filters on the input. We used $3\times 3$ sized filters with a stride of 1 for each convolution layer. The activation function for each convolution layer was ReLU. The computational complexity of the CNN models was reduced by using $d\times d$ pooling layers, which reduces the output size from one layer to the next in the hidden network layers. To select maximum elements, we used $2\times 2$ max-pooling operations to preserve the important features [62]. Hence, these layers reduce the size of the input image by half. To feed the pooled output from the stacked featured map to the final layer, the maps were flattened into one column. The final layers of the CNN had two fully connected layers with M number of nodes each. Fully connected layers also used the ReLU activation function. These two layers were regularized by using the dropout layers with the regularization technique. Finally, the Softmax layer was employed, followed by two fully connected layers, and the number of nodes of this layer was equal to the number of expression classes.

During the feature representation of images using deep learning approaches, it was observed that the CNN models obtained better representation when patterns were analyzed from the multi-resolution of images. Additionally, increasing some layers in the architecture while increasing the resolution of the images results in more deeply analyzing some hidden patterns in the feature maps. Inspired by these observations, we applied multi-resolution of the facial images with varying layers in different CNN architectures. During feature representations, we considered facial images with three different resolutions such that the original facial image ${\mathsf{ℱ}}_{n\times n}$ was down-sampled to ${\mathsf{ℱ}}_{n_1\times n_1}$, ${\mathsf{ℱ}}_{n_2\times n_2}$, and ${\mathsf{ℱ}}_{n_3\times n_3}$, $n_3=2\times n_2=4\times n_1$. Here, for facial images, namely ${\mathsf{ℱ}}_{n_1\times n_1}$, ${\mathsf{ℱ}}_{n_2\times n_2}$, and ${\mathsf{ℱ}}_{n_3\times n_3}$, three different CNN architectures, namely $CN{N}_1$, $CN{N}_2$, and $CN{N}_3$, were proposed. These architectures are shown in Figure 4, Figure 5 and Figure 6, whereas the detailed descriptions of these architectures, including the employed input–output hidden layers, the output shapes of the convoluted images, and the input image sizes and parameters generated at each layer, are shown in

Table 1. The proposed CNN architecture for the input size 48 × 48 with layers, output shape, image size, and parameters.

Layer	Output Shape	Image Size	Parameters	Layers	Output Shape	Image Size	Parameters
Block-1				Block-3
Conv2D (3 × 3)@32	($n,n,$32)	(48, 48, 32)	((3 × 3 × 3) + 1) × 32 = 896	Conv2D (3 × 3)@96	($n_2,n_2$,96)	(12,12,96)	((3 × 3 × 64) + 1) × 96 = 55,392
Batch Norm	($n,n,32$)	(48, 48, 32)	4 × 32 = 128	Batch Norm	($n_2,n_2$,96)	(12,12,96)	4 × 96 = 384
Activation ReLU	($n,n,32$)	(48, 48, 32)	0	Activation ReLU	($n_2,n_2$,96)	(12,12,96)	0
Maxpool2D (2 × 2)	($n_1,n_1$,32) $n_1$ = n/2	(24, 24, 32)	0	Maxpool2D (2 × 2)	($n_3,n_3$,96) $n_3$ = $n_2$/2	(6,6,96)	0
Dropout	($n_1,n_1$,32)	(24, 24, 32)	0	Dropout	($n_3,n_3$,96)	(6,6,96)	0
Block-2				Block-4
Conv2D (3 × 3)@64	($n_1,n_1$,64)	(24, 24, 64)	((3 × 3 × 32) + 1) × 64 = 18,496	Conv2D (3 × 3)@96	($n_3,n_3$,96)	(6,6,96)	((3 × 3 × 96) + 1) × 96 = 83,040
Batch Norm	($n_1,n_1$,64)	(24, 24, 64)	4 × 64 = 256	Batch Norm	($n_3,n_3$,96)	(6,6,96)	4 × 96 = 384
Activation ReLU	($n_1,n_1$,32)	(24, 24, 64)	0	Activation ReLU	($n_3,n_3$,96)	(6,6,96)	0
Maxpool2D (2 × 2)	($n_2,n_2$,64) $n_2$ = $n_1$/2	(12, 12, 64)	0	Maxpool2D (2 × 2)	($n_4,n_4$,96) $n_4$ = $n_3$/2	(3,3,96)	0
Dropout	($n_2,n_2$,64)	(12, 12, 64)	0	Dropout	($n_4,n_4$,64)	(3,3,96)	0
Block-5
Conv2D (3 × 3)@64		($n_4,n_4$,64)		(3,3,64)		55,360
BatchNorm		($n_4,n_4$,64)		(3,3,64)		256
ActivationReLU		($n_4,n_4$,64)		(3,3,64)		0
Maxpool2D (2 × 2)		($n_5,n_5$,64), $n_5$ = $n_4$/2		(1,1,64)		0
Dropout		($n_5,n_5$,64)		(1,1,64)		0
Layer		Output Shape		Image Size		Parameter
Flatten		(1, $n_5\times n_5\times 64$)		(1, 64)		0
Dense		(1, 256)		(1, 256)		(1 + 64) × 256 = 16,640
Batch Normalization		(1, 256)		(1, 256)		1024
Activation Relu		(1, 256)		(1, 256)		0
Dropout		(1, 256)		(1, 256)		0
Dense		(1, 256)		(1, 256)		(256 + 1) × 256 = 65,792
Batch Normalization		(1, 256)		(1, 256)		1024
Activation Relu		(1, 256)		(1, 256)		0
Dropout		(1, 256)		(1, 256)		0
Dense		(1, 7)		(1, 7)		(256 + 1) × 7 = 1799
Total parameters for the input image size:						300,871
Total number of trainable parameters:						299,143
Non-trainable parameters:						1728

,

Table 2. The proposed CNN architecture for the input size 96 × 96 with layers, output shape, image size, and parameters.

Layer		OutputShape		ImageSize		Parameters
Block-1
Conv2D (3 × 3)@32		($n,n,32$)		(96, 96, 32)		896
BatchNorm		($n,n,32$)		(96, 96, 32)		128
ActivationReLU		($n,n,32$)		(96, 96, 32)		0
Maxpool2D (2 × 2)		($n_1,n_1$, 32), $n_1=n/2$		(48, 48, 32)		0
Dropout		($n_1,n_1$,32), $n_1=n/2$		(48, 48, 32)		0
Block-2				Block-4
Conv2D (3 × 3)@32	($n_1,n_1$,32)	(48,48,32)	((3 × 3 × 3) + 1) × 32 = 9248	Conv2D (3 × 3)@96	($n_3,n_3$,96)	(12,12,96)	((3 × 3 × 64) + 1) × 96 = 55,392
Batch Norm	($n_1,n_1$,32)	(48,48,32)	4 × 32 = 128	Batch Norm	($n_3,n_3$,96)	(12,12,96)	4 × 96 = 384
Activation ReLU	($n_1,n_1$,32)	(48,48,32)	0	Activation ReLU	($n_3,n_3$,96)	(12,12,96)	0
Maxpool2D (2 × 2)	($n_2,n_2$,32) $n_2=n_1/2$	(24, 24, 32)	0	Maxpool2D (2 × 2)	($n_4,n_4$,96) $n_4=n_3/2$	(6,6,96)	0
Dropout	($n_2,n_2$,32)	(24, 24, 32)	0	Dropout	($n_4,n_4$,96)	(6,6,96)	0
Block-3				Block-5
Conv2D (3 × 3)@64	($n_2,n_2$,64)	(24, 24, 64)	((3 × 3 × 32) + 1) × 64 = 18,496	Conv2D (3 × 3)@96	($n_4,n_4$,96)	(6,6,96)	((3 × 3 × 96) + 1) × 96 = 83,040
Batch Norm	($n_2,n_2$,64)	(24, 24, 64)	4 × 64 = 256	Batch Norm	($n_4,n_4$,96)	(6,6,96)	4 × 96 = 384
Activation ReLU	($n_2,n_2$,32)	(24, 24, 64)	0	Activation ReLU	($n_4,n_4$,96)	(6,6,96)	0
Maxpool2D (2 × 2)	($n_3,n_3$,64) $n_3=n_2/2$	(12, 12, 64)	0	Maxpool2D (2 × 2)	($n_5,n_5$,96) $n_5=n_4/2$	(3,3,96)	0
Dropout	($n_3,n_3$,64)	(12, 12, 64)	0	Dropout	($n_5,n_5$,64)	(3,3,96)	0
Block-6
Conv2D (3 × 3)@64		($n_5,n_5$,64)		(3,3,64)		55,360
BatchNorm		($n_5,n_5$,64)		(3,3,64)		256
ActivationReLU~		($n_5,n_5$,64)		(3,3,64)		0
Maxpool2D (2 × 2)		($n_6,n_6$,64), $n_6=n_5/2$		(1,1,64)		0
Dropout		($n_6,n_6$,64)		(1,1,64)		0
Layer		Output Shape		Image Size		Parameter
Flatten		(1, $n_6\times n_6\times 64$)		(1, 64)		0
Dense		(1, 256)		(1, 256)		(1 + 64) × 256 = 16,640
Batch Normalization		(1, 256)		(1, 256)		1024
Activation Relu		(1, 256)		(1, 256)		0
Dropout		(1, 256)		(1, 256)		0
Dense		(1, 256)		(1, 256)		(256 + 1) × 256 = 65,792
Batch Normalization		(1, 256)		(1, 256)		1024
Activation Relu		(1, 256)		(1, 256)		0
Dropout		(1, 256)		(1, 256)		0
Dense		(1, 7)		(1, 7)		(256 + 1) × 7 = 1799
Total parameters for the input image size:						310,247
Total number of trainable parameters:						308,455
Non-trainable parameters:						1792

and

Table 3. The proposed CNN architecture for the input size 192 × 192 with layers, output shape, image size, and parameters.

Layers		Output Shape		Image Size		Parameters
Block-1
Conv2D (3 × 3)@16		($n,n,16$)		(192, 192, 16)		448
BatchNorm		($n,n,16$)		(192, 192, 16)		64
ActivationReLU		($n,n,16$)		(192, 192, 16)		0
Maxpool2D (2 × 2)		($n_1,n_1$,16), $n_1=n/2$		(96, 96, 16)		0
Dropout		($n_1,n_1$,16)		(96, 96, 16)		0
Block-2
Conv2D (3 × 3)@32		($n_1,n_1$,32)		(96, 96, 32)		4640
BatchNorm		($n_1,n_1$,32)		(96, 96, 32)		128
ActivationReLU		($n_1,n_1$,32)		(96, 96, 32)		0
Maxpool2D (2 × 2)		($n_2,n_2$, 32), $n_2=n_1/2$		(48, 48, 32)		0
Dropout		($n_2,n_2$,32)		(48, 48, 32)		0
Layers	Output Shape	Image Size	Parameters	Layers	Output Shape	Image Size	Parameters
Block-3				Block-5
Conv2D (3 × 3)@32	($n_2,n_2$,32)	(48,48,32)	((3 × 3 × 3) + 1) × 32 = 9248	Conv2D (3 × 3)@96	($n_4,n_4$,96)	(12,12,96)	((3 × 3 × 64) + 1) × 96 = 55,392
Batch Norm	($n_2,n_2$,32)	(48,48,32)	4 × 32 = 128	Batch Norm	($n_4,n_4$,96)	(12,12,96)	4 × 96=384
Activation ReLU	($n_2,n_2$,32)	(48,48,32)	0	Activation ReLU	($n_4,n_4$,96)	(12,12,96)	0
Maxpool2D (2 × 2)	($n_3,n_3$,32) $n_3=n_2/2$	(24, 24, 32)	0	Maxpool2D (2 × 2)	($n_5,n_5$,96) $n_5=n_4/2$	(6,6,96)	0
Dropout	($n_3,n_3$,32)	(24, 24, 32)	0	Dropout	($n_5,n_5$,96)	(6,6,96)	0
Block-4				Block-6
Conv2D (3 × 3)@64	($n_3,n_3$,64)	(24, 24, 64)	((3 × 3 × 32) + 1) × 64 = 18,496	Conv2D (3 × 3)@96	($n_5,n_5$,96)	(6,6,96)	((3 × 3 × 96) + 1) × 96 = 83,040
Batch Norm	($n_3,n_3$,64)	(24, 24, 64)	4 × 64 = 256	Batch Norm	($n_5,n_5$,96)	(6,6,96)	4 × 96 = 384
Activation ReLU	($n_3,n_3$,32)	(24, 24, 64)	0	Activation ReLU	($n_5,n_5$,96)	(6,6,96)	0
Maxpool2D (2 × 2)	($n_4,n_4$,64) $n_4=n_3/2$	(12, 12, 64)	0	Maxpool2D (2 × 2)	($n_6,n_6$,96) $n_6=n_5/2$	(3,3,96)	0
Dropout	($n_4,n_4$,64)	(12, 12, 64)	0	Dropout	($n_6,n_6$,64)	(3,3,96)	0
Block-7
Conv2D (3 × 3)@64		($n_6,n_6$,64)		(3,3,64)		55,360
BatchNorm		($n_6,n_6$,64)		(3,3,64)		256
ActivationReLU~		($n_6,n_6$,64)		(3,3,64)		0
Maxpool2D (2 × 2)		($n_7,n_7$,64), $n_7=n_6/2$		(1,1,64)		0
Dropout		($n_7,n_7$,64)		(1,1,64)		0
Layer		Output Shape		Image Size		Parameter
Flatten		(1, $n_7\times n_7\times 64$)		(1, 64)		0
Dense		(1, 256)		(1, 256)		(1 + 64) × 256 = 16,640
Batch Normalization		(1, 256)		(1, 256)		1024
Activation Relu		(1, 256)		(1, 256)		0
Dropout		(1, 256)		(1, 256)		0
Dense		(1, 256)		(1, 256)		(256 + 1) × 256 = 65,792
Batch Normalization		(1, 256)		(1, 256)		1024
Activation Relu		(1, 256)		(1, 256)		0
Dropout		(1, 256)		(1, 256)		0
Dense		(1, 7)		(1, 7)		(256 + 1) × 7 = 1799
Total parameters for the input image size:						314,503
Total number of trainable parameters:						312,679
Non-trainable parameters:						1824

, respectively, to allow for greater understanding and clarity about the models.

3.3. Factors Affecting the Performance of the Proposed FERS

• Data Augmentation:- The data augmentation technique is used to expand the training samples in order to improve the performance of recognition and the ability to generalize the models. In machine learning, image augmentation techniques artificially increase the amount of training data by applying transformation methods to the existing data [63]. The classical augmentation techniques that were employed are bilateral filtering, unsharp filtering, horizontal flip, vertical flip, Gaussian blur, additive Gaussian noise, image scale, image cropping, translation, image rotation, shear mapping, image zooming, image filling, and contrast normalization methods from [15] for the purpose of image augmentation. The whole training images were flipped horizontally by applying simple image data augmentation techniques. In this work, we applied these techniques for each resolution of the images.
• Fine Tuning:- Fine-tuning allows for higher-order feature representations in the base model to make them more relevant for the face recognition tasks. For example, VGG used many layers and generated a higher dimensional feature vector, and thus thw inference was quite costly at run-time due to huge parameters. In this case, fine-tuning techniques were applied when freezing some layers and the number of parameters, and the model was retrained to reduce computational overheads.
• Progressive Resizing:- Progressive image resizing is an eminent technique that sequentially resizes all images while training the CNN models on smaller, i.e., tinier images to larger image sizes. The progressive resizing technique is used to train a CNN with $n\times n$ image size, saving the weights, and then the CNN is retrained again for other iterations with the images of increased sizes greater than n. This technique was used for super-resolution [64], where low-resolution images gradually increased to the image with a higher resolution during training processes. The advantages of using progressive resizing are that it improves generalization and reduces overfitting problems.
• Transfer Learning:- The principle concept behind transfer learning for facial expression recognition and classification problems is that a model trained on large data sets for one problem is effectively used as a generic model in some way on other related problems. The model that has been trained earlier is known as the pre-trained model. Our proposed deep learning convolution neural network model uses a transfer learning technique in which the weights of the pre-trained model and/or a set of layers from the pre-trained model $CN{N}_1$ are used for the new model $CN{N}_2$ to solve similar problems. Similarly, the weights of $CN{N}_2$ have been adopted to solve the $CN{N}_3$ model. The benefits of using transfer learning are that it reduces the training time and can result in lower generalization errors.
• Scores Fusion:- In the proposed system, three CNN architectures have been proposed. These architectures take images of different sizes as inputs. Thus, during the recognition of facial expressions on the test sample F, there are three different classification score vectors, namely $s_1=(a_1^1,a_2^1,\cdots ,a_7^1)$, $s_2=(a_1^2,a_2^2,\cdots ,a_7^2)$, and $s_3=(a_1^3,a_2^3,\cdots ,a_7^3)$, where each $a_j^i$ is the classification score by the $CN{N}_i$ architecture and for $j\mathrm{th}$ expression class. These classification scores are fused together using score-level post-classification fusion approaches [14] to increase the performance of the recognition system. In this work, two score-level fusion techniques, namely Sum-rule and Product-rule, were employed. The Sum-rule and Product-rule techniques are defined as follows:

  $$\begin{array}{c}\hfill \underset{i\ne j, k=\{1,\cdots ,7\}}{max}\{a_k^i+a_k^j\}\end{array}$$

  (1)

  $$\begin{array}{c}\hfill \underset{i\ne j, k=\{1,\cdots ,7\}}{max}\{a_k^i\ast a_k^j\}\end{array}$$

  (2)

4. Experimentation

In this section, the experimentation of the proposed FER system is discussed and for this purpose, four challenging benchmark facial expression databases were experimented on. Each database was randomly divided into 50% of the dataset for training, while the remaining 50% was used for testing purposes. Finally, this partitioning of the datasets were done ten times and the average performance was reported, corresponding to each database. As there were no particular benchmark datasets specifically built for the healthcare scenario, social IoT, emotion AI, and cognitive AI, the employed datasets were assumed as backbones for e-Healthcare, social IoT, emotion AI, and cognitive-AI diversified applications as discussed in this paper. The proposed system has not been tested in a real-time scenario. Still, the employed datasets were very challenging. The proposed method can accept and handle all the unconstrained situations of facial expression recognition in the real-time strategy for e-Healthcare, social IoT, emotion AI, and cognitive AI applications.

4.1. Database Used

The first employed database was Karolinska-directed emotional faces (KDEF) [42] which contains seventy different subjects (thirty-five male and thirty-five female) with five different pose variations labeled with seven basic expression categories. Here, we used only 1210 samples as training sets, whereas 1213 samples were used as testing sets, as only these samples were available from the license agreement downloaded site. Figure 7 shows some examples from this database.

The second employed database is the GENKI database [65], which is composed of 4000 facial images that have been labeled as two classes: (i) happy and (ii) non-happy. Additionally, for this database, two thousand images were randomly selected as training sets, while the remaining two thousand images were considered for the testing set. Some examples of this database are shown in Figure 8.

The Extended Cohn-Kanade (CK+) [66] is our third database, which is composed of 593 video sequences from 123 subjects captured between the ages of 18 to 50 years. Here, only 309 image sequences were labeled with six basic expressions. During the experimentation, we randomly split this database into the training and testing sets. Figure 9 presents some image samples from the CK+ database.

Our fourth database was Static Facial Expressions in the Wild (SFEW) [26], which was created from the AFEW video database by selecting the keyframes based on facial point clustering. The challenging SFEW dataset contains 700 images which were divided into the training set (346 images) and testing set (354 images). This database has seven facial expression classes, namely afraid, anger, disgust, happiness, neutral, sadness, and surprise. Figure 10 presents image samples from the SFEW database.

Table 4. Presentation of the employed facial expression databases.

Database	Class	Training	Testing
KDEF	7	1210	1213
GENKI	2	2000	2000
CK+	6	663	146
SFEW	7	346	354

presents the detailed descriptions of the KDEF, GENKI, CK+, and SFEW facial expression databases. The objectives of the selection of these databases were (i) to obtain expressions that were common and generic in people; (ii) to ensure that the other expressions used in the affecting computing research areas were composed of a mixture of these basic facial expressions; and (iii) to ensure that the good recognition system for these expressions would be very beneficial for several real-world applications, such as in e-Healthcare frameworks, in the social Internet of Things (IoT), and in emotion AI in business organizations.

4.2. Results and Discussion

This section describes and explains the experimentation of the proposed facial expression recognition system (FERS). The proposed FERS was implemented using Python 3.7.9 version, Tensorflow 2.3.1 version, Keras 2.4.3 version, CUDA version 11.2, and NVIDIA-SMI 460.79 Driver Version in Windows 10 Pro 64-bit, Intel(R) Core(TM)-i7-9700 CPU, 3.30 GHz (8 CPU) Processor, and in a 8 GB NVIDIA GeForce RTX 2070 SUPER XLA GPU device with 16 GB RAM. During experimentation, we employed both gray-scaled and RGB-colored images as some databases have RGB images while others only have gray-scaled images. During image preprocessing, from each input image $\mathsf{ℐ}$, we detected the face region by applying the methods discussed in Section 3.1. Furthermore, the detected face region $\mathsf{ℱ}$ was normalized to a fixed size image $\mathsf{ℱ}\in {\mathbb{R}}^{200\times 200}$. For recognizing the expression classes on the human face, in this work, we employed deep learning-based approaches where three convolutional neural network (CNN) architectures (Figure 4, Figure 5 and Figure 6) were designed. These CNN architectures were trained in such a way that they would perform both feature computation and expression classification tasks. For better understanding the functionality of these architectures, at first, we started the experiment using $CN{N}_1$ architecture (Figure 4), where the input to this system is an image $\mathsf{ℱ}\in {\mathbb{R}}^{n_1\times n_1}$, $n_1=48$, i.e., the training ${\mathsf{ℱ}}_{48\times 48}$ samples were used to train $CN{N}_1$ architecture while the performance of the trained $CN{N}_1$ model was evaluated using the remaining testing samples. Learning the parameters in any CNN architecture is a very important task and depends on two factors, i.e., epochs and batches. Both these factors affect the learning capabilities of the architecture during the training of samples in the network. Thus, a trade-off between epochs and batches was established, which improved the performance of FERS using $CN{N}_1$ architecture (Figure 4). The demonstration of the performance with the trade-off between epochs and batches is shown in Figure 11 and from this figure, it is observed that the performance gradually improved with increasing epochs (best performance at nearly 700 epochs) while keeping 16 batches fixed.

Inspired by the experiment shown in Figure 11, another experiment was conducted using $CN{N}_1$ architecture while keeping the fixed batch = 16 with varying epochs with respect to the KDEF, CK+, and SFEW database; the performance is shown in Figure 12. From this figure, we can observe that during epochs between 700 and 800, the performance for each database was good.

4.2.1. Effect of Data Augmentation Techniques

The data augmentation techniques were applied to training samples to increase the number of samples. The increased training samples learned the parameters of $CN{N}_1$ architecture well and obtained a better performance. Moreover, in order to adapt the diversity of the training data and to avoid overfitting problems, data augmentation plays an important role. In this work, each sample of the training images was horizontally and then vertically flipped. Then, Affine transformations such as rotation, scaling, zooming, and shearing operations were performed. For the data augmentation technique, we employed the methods mentioned in [15], which derives seventeen images for each sample. Figure 13 shows the effect of data augmentation techniques on the performance of FERS using $CN{N}_1$ architecture and from this figure, we can observe that the performance of the proposed FERS increases due to the employed data augmentation techniques.

4.2.2. Effect of Progressive Image Resizing

The progressive image resizing technique has been discussed in Section 3.3. During experimentation, the preprocessed face region $\mathsf{ℱ}\in {\mathbb{R}}^{200\times 200}$ was further down-sampled into $n_1\times n_1\times 3$, $2n_1\times 2n_1\times 3$, and $4n_1\times 4n_1\times 3$ size images. Here, we already considered $n_1=48$ in the above experiments. Hence, in progressive image resizing, we down-sampled $\mathsf{ℱ}\in {\mathbb{R}}^{200\times 200}$ to $\mathsf{ℱ}\in {\mathbb{R}}^{48\times 48}$, $\mathsf{ℱ}\in {\mathbb{R}}^{96\times 96}$, and $\mathsf{ℱ}\in {\mathbb{R}}^{192\times 192}$. In other words, the $CN{N}_1$ architecture (Figure 4) was trained with ${\mathsf{ℱ}}_{48\times 48\times 3}$ images. Then, ${\mathsf{ℱ}}_{96\times 96\times 3}$ images were used to train $CN{N}_2$ architecture (Figure 5). Lastly, the $CN{N}_3$ architecture was trained with ${\mathsf{ℱ}}_{192\times 192\times 3}$ images. The purpose behind learning these architectures with the increasing image sizes concern the fact that (i) the high-resolution images are trained in the network; (ii) the effect of multi-resolution approaches can be introduced in the network such that the texture patterns at the higher level of abstraction will be reflected during the learning of parameters; and (iii) the system will provide deeper information that would be beneficial for the hierarchical representations of features. Hence, the use of progressive image resizing not only increases the performance of the recognition system but also reduces the overfitting problems. The effect of progressive image resizing on the performance of the proposed system is reported in

Table 5. Effect of the progressive image resizing on the performance of the proposed FERS where the first, second, and third row for each database shows the accuracy in percentage using $CN{N}_1$ ($\mathsf{ℱ}\in {\mathbb{R}}^{48\times 48}$), $CN{N}_2$ ($\mathsf{ℱ}\in {\mathbb{R}}^{96\times 96}$), and $CN{N}_3$ ($\mathsf{ℱ}\in {\mathbb{R}}^{192\times 192}$) models, respectively.

Database	Data Augmentation	No Data Augmentation
	75.95	71.67
KDEF	78.18	74.92
	80.92	78.21
	92.45	87.91
GENKI	94.13	89.77
	95.59	91.16
	95.89	88.15
CK+	96.22	92.67
	96.71	95.20
	34.05	32.56
SFEW	34.91	33.11
	35.72	33.34

and from this table, we can observe that for the KDEF, GENKI, CK+, and SFEW databases, the proposed FERS exhibits a better performance for ${\mathsf{ℱ}}_{192\times 192\times 3}$ images than for both ${\mathsf{ℱ}}_{96\times 96\times 3}$ and ${\mathsf{ℱ}}_{48\times 48\times 3}$ images. Moreover, it is evident that both progressive image resizing and data augmentation techniques together are very effective for the proposed CNN models for recognizing facial expression on facial regions.

4.2.3. Effect of Transfer Learning

For this technique, we used two different approaches: (i) in the first approach, we freshly trained $CN{N}_1$, $CN{N}_2$, and $CN{N}_3$ architectures with the corresponding image sizes, i.e., refreshed models were used; (ii) in the second approach, we used the trained $CN{N}_1$ model as a retrained model for $CN{N}_2$ such that only upper layers of the $CN{N}_2$ model were trained with ${\mathsf{ℱ}}_{96\times 96\times 3}$ images. Similarly, the upper layers of the $CN{N}_3$ architecture were trained by ${\mathsf{ℱ}}_{192\times 192\times 3}$ images, while for the remaining layers, the weights of the trained $CN{N}_2$ model were used. Figure 14 presents the effect of transfer learning approaches on the performance of the proposed FERS. Here, only the performance of the $CN{N}_3$ model trained with ${\mathsf{ℱ}}_{192\times 192\times 3}$ images is shown using both progressive image resizing and data augmentation techniques.

4.2.4. Effect of Score Fusion

Score fusion techniques defined in Equations (1) and (2) were applied on the classification scores obtained by the $CN{N}_1$, $CN{N}_2$, and $CN{N}_3$ architectures, and the results are reported in

Table 6. Effect of score-level fusion approaches on the performance of the proposed FERS.

Method	KDEF	GENKI
$CN{N}_1$	75.95	92.45
$CN{N}_2$	78.18	94.13
$CN{N}_3$	80.92	95.59
Sum-Rule	81.53	96.03
Product-Rule	82.63	96.75
Method	CK+	SFEW
$CN{N}_1$	95.89	34.05
$CN{N}_2$	96.22	34.91
$CN{N}_3$	96.71	35.72
Sum-Rule	97.07	36.15
Product-Rule	97.32	36.79

.

To better understand the performance of the proposed FERS, the confusion matrices are shown in Figure 15, corresponding to the KDEF, GENKI, CK+, and SFEW facial expression database. Here, each confusion matrix represents the product-rule-based fusion performance of the proposed FERS.

4.2.5. Comparison

To compare the performance of the proposed methodology, we computed features from the competing methods and obtained the performance under the same training–testing protocol. Here, the performance of methods of Vgg16 [67], ResNet50 [68], that of Zavare et al. [42], Inception-v3 [69], and that of Rao et al. [70] were compared with the performance of the proposed system for the KDEF database, as presented in

Table 7. Performance comparison of the proposed FERS for the KDEF database.

Method	Accuracy (%)	Remarks
Vgg16 [67]	65.08	Images used (980), expression class (7), train/test split
ResNet50 [68]	72.32	Images used (980), expression class (7), train/test split
Zavare et al. [42]	72.55	Images used (980), expression class (7)
		Images type (frontal), 10-fold cross validation
Inception-v3 [69]	75.04	Images used (980), expression class (7), train/test split
Rao et al. [70]	74.05	Images used (720), expression class (6)
		Images type (frontal), 10-fold cross validation
Proposed	82.63	Images used (980), proposed CNN for seven expression classes

. For the GENKI database the performance of the proposed system was compared with Vgg16, ResNet50, Inveption-v3, that of An et al. [29], that of Zhang et al. [71], and that of Gao et al. [72], and the competing methods are presented in

Table 8. Performance comparison of the proposed FERS for the GENKI database.

Method	Accuracy (%)	Remarks
Vgg16 [67]	72.08	VGG16 CNN for seven expression classes
ResNet50 [68]	82.30	ResNet 50 CNN for seven expression classes
Inception-v3 [69]	85.38	Inception-v3 CNN for seven expression classes
An et al. [29]	88.50	Feature (HOG), classifier (ELM)
Zhang et al. [71]	94.21	Feature (CNN), classifier (Softmax)
Gao et al. [72]	94.33	Feature (ensemble), classifier (ensemble)
Proposed	96.75	Proposed CNN for seven expression classes

. Similarly, we compared the performance of the proposed system with that of Sun et al. [73], ResNet50, and Inveption-v3 for the CK+ database in

Table 9. Performance comparison of the proposed FERS for the CK+ database.

Method	Accuracy (%)	Remarks
ResNet50 [68]	91.87	Images used (981), expression class (7), train/test split
Inception-v3 [69]	94.07	Images used (981), expression class (7), train/test split
Sun et al. [73]	94.67	Images used (510), expression class (7), k-fold cross-validation
Proposed	96.81	Images used (981), proposed CNN for seven expression classes

. For the SFEW database, the performance of Liu et al. [74], Vgg16, ResNet50, and the Inveption-v3 methods were compared in

Table 10. Performance comparison of the proposed FERS for the SFEW database.

Method	Accuracy (%)	Remarks
Vgg16 [67]	24.78	Images used (700), expression class (7), train (346)/test (354)
ResNet50 [68]	24.98	Images used (700), expression class (7), train (346)/test (354)
Inception-v3 [69]	29.52	Images used (700), expression class (7), train (346)/test (354)
Liu et al. [74]	26.14	Images used (700), expression class (7), train (346)/test (354)
Proposed	36.79	Images used (700), expression class (7), Train (346)/Test (354)

. The comparison of performance, as presented in Table 7, Table 8, Table 9 and Table 10, shows the superiority of the proposed system.

5. Conclusions

A novel method for facial expression recognition systems has been proposed in this work. The objective of the proposed system is to predict the seven basic types of expressions on the human face. The applications of this proposed system have been well described and demonstrated in the diversified fields of e-Healthcare, social IoT, emotion AI, and cognitive AI. The implementation of the proposed system has three components. In the first component, an image preprocessing task has been performed where a face region was extracted from a body silhouette image using the facial landmark points. Then, in the second component, from the extracted face region, the multi-resolution images were considered. The convolutional neural network architectures have been proposed for each resolution of the images. Here, the images undergo the CNN architectures and are classified into seven basic facial expression classes based on learning the parameters of CNN models. To enhance the performance of the recognition system and better handle the challenging issues of the facial expression recognition system, some advanced techniques such as image augmentation, progressive image resizing, transfer-learning, and fine-tuning of parameters were employed in the third component. Finally, fusion methods were applied to the best performance of the different CNN models to achieve a better performance than the existing state-of-the-art methods. Extensive experimentation has been performed using four benchmark databases, namely KDEF, GENKI-4k, CK+, and SFEW, and the performance of the proposed system has been compared with some existing methods concerning each database. The comparison of the performance of the proposed method with the competing methods shows the superiority of the proposed system.