Beauty in the Eyes of Machine: A Novel Intelligent Signal Processing-Based Approach to Explain the Brain Cognition and Perception of Beauty Using Uncertainty-Based Machine Voting

Abstract

The most mysterious question humans have ever attempted to answer for centuries is, “What is beauty, and how does the brain decide what beauty is?”. The main problem is that beauty is subjective, and the concept changes across cultures and generations; thus, subjective observation is necessary to derive a general conclusion. In this research, we propose a novel approach utilizing deep learning and image processing to investigate how humans perceive beauty and make decisions in a quantifiable manner. We propose a novel approach using uncertainty-based ensemble voting to determine the specific features that the brain most likely depends on to make beauty-related decisions. Furthermore, we propose a novel approach to prove the relation between golden ratio and facial beauty. The results show that beauty is more correlated with the right side of the face and specifically with the right eye. Our study and findings push boundaries between different scientific fields in addition to enabling numerous industrial applications in variant fields such as medicine and plastic surgery, cosmetics, social applications, personalized treatment, and entertainment.

1. Introduction

Plato once wrote in the Symposium that “if there is anything worth living for, it is to behold beauty”. For centuries, beauty has captivated humans, and philosophers have attempted to explain this mysterious human phenomenon without any appropriate means of measuring and comprehending beauty. Facial attractiveness (or facial beauty) profoundly affects numerous facets of a person’s social life. In the US alone, money spent on beauty products exceeds what is spent on both social services and education [1]. Researchers from many different disciplines, including engineering, human science, and medicine, have been interested in beauty [2]. The field of computing has only recently begun studying facial beauty. However, the psychological community has thoroughly researched this [3]. Gustav Fechner, a pioneer in experimental psychology, was interested in quantifying beauty [4]. However, in contemporary models of aesthetic experiences, beauty still remains a mysterious aesthetic response [5,6].

With advancements in psychology, scientists have detected internal and external factors that affect beauty judgment. Fertility and hormone levels are examples of internal influences, whereas social contexts, temporal contexts (e.g., long-term versus short-term relationships), environmental contexts, and visual experiences (e.g., parental familiarity) are examples of external elements [7,8]. Another study has found that facial symmetry is a major factor in determining facial beauty, as the human brain is sensitive to symmetry [9]. Furthermore [10], in addition to facial symmetry and averageness, the major attributes that affect facial attractiveness are full lips, thin eyebrows, tiny nose, high forehead, high cheekbones, and thick hair, which are probably applicable in the case of the attractiveness of men as well. However, the face is more feminine and attractive because of certain major characteristics, such as large lips, a small nose, and high cheekbones, which are indicators of youth, fertility, and high estrogen levels [10].

Existing studies explain the common features that contribute to beauty perception, but fail to explain and quantify the reasons why the human brain decides for a specific face to be beautiful, and why two persons could argue whether full lips are beautiful.

A pioneer neurobiologist Semir Zeki, who has extensively researched the question of beauty from a neuroscience perspective for many years, has defined beauty as, “Beauty is an experience that correlates quantitatively with neural activity in a specific part of the emotional brain, namely, in the field A1 of the medial orbito-frontal cortex (A1mOFC); the more intense the declared experience of beauty, the more intense the neural activity there” [11].

In one recent fMRI-based study [12], Zeki et al. [11] investigated how beautiful faces are manifested in the brain and whether beauty can be decoded on the basis of pattern activity. They showed 120 faces to seventeen volunteers, who rated the attractiveness of each face on a seven-point Likert scale while having their brain activity measured. They discovered that the emergence of various patterns of activity in the fusiform face area (FFA) and occipital face area (OFA), and the concurrent emergence of activity in the medial orbital frontal cortex are the results of the sense of beauty (mOFC). The right FFA (61.17%) and the mOFC (62.35%) exhibited the highest cross-participant classifications of the brain’s activity. However, researchers have concluded their research as, “The precise features that render a face beautiful, beyond the accepted general properties of symmetry, proportions, and precise relationships—which are not in themselves necessarily sufficient to render a face beautiful—may be unknown”.

The irrational integer of order 1.618 known as the golden ratio, symbolized by the symbol Φ (phi), was first referenced in writing in Euclid’s Elements in 300 BC [13].

Many have considered the golden ratio, commonly referred to as the divine proportion, to be the solution to the phenomenon of aesthetics; regularity; alignment; and human physiology, psychology, and beauty [14,15,16]. The golden ratio has been also used in surgical anatomy [17], dental and maxillofacial surgery [18,19], and in cosmetic surgery as a paradigm to measure aesthetics in both soft tissue or hard tissue (cephalometric) levels [20,21]. However, is the face that perfectly matches the golden ratio considered to be beautiful, and who decides whether the face is beautiful or not? Holland’s research [22] shows that there are no sufficient studies supporting the relationship between the golden ratio and facial beauty.

Owing to extensive advancements in the artificial intelligence field and strong links between deep learning techniques and the human brain [23,24], in recent years, the automatic human-like facial attractiveness prediction (FAP) or face beauty prediction (FBP) and its applications in machine learning and computer vision have attracted considerable research interest [25]. Despite significant advancements in determining and estimating the beauty of faces, the majority of studies have concentrated on predicting beauty without a thorough grasp of what constitutes beauty. A machine approach to understanding how the human brain processes such deep human concepts is necessary, as it is still not comprehensively understood in almost all fields [26]. Few studies [3,27,28] have attempted to investigate the constituents of beauty and understand the human brain’s judgments of beauty. However, such studies are all based on objective datasets obtained by either voting or average scores.

To appropriately understand beauty, we propose a novel methodology to train the machine regarding beauty and a new framework to explain the features it depends on in deciding beautiful faces.

Table 1. Review of certain existing research on facial beauty models.

Research	Data Size	Participants	Facial Data Property	Data Type	Explaining Beauty Decisions	Learned Features Visualization
(Gunes et al., 2006) [29]	215	46	Diverse	Score on average Rating	No	No
(Chen et al., 2010) [30]	23,412	Unknown	Asian	unknown	No	No
(Fan et al., 2012) [31]	432	30	Computer Generated	Score on average Rating	No	No
(Xu et al., 2015) [32]	500	75	Asian	Score on average Rating	No	CNNs Filters
(Zhang et al., 2016) [33]	799	25	Diverse	Score on average Rating	No	No
(Zhang et al., 2017) [28]	9415	Unknown	Asian/Computer Generated	Score on average Rating	No	No
(Liang et al., 2018) [34]	5500	60	Asian/Caucasian	Score on average Rating	No	No
(Liu et al., 2019) [3]	No data proposed	0	Diverse/Computer Generated	Voting/Score on average Rating	No	No
(Lebedeva et al., 2021) [35]	No data proposed	0	Asian/Caucasian	Score on average Rating	No	No
(Sano, 2022) [27]	No data proposed	0	Asian/Caucasian	Score on average Rating	No	Averaged Grad-CAM
(Bougorzi et al., 2022) [25]	No data proposed	0	Asian/Caucasian	Score on average Rating	No	No
Proposed	50,000	10	Diverse	Subjective	Yes	Proposed MVF Algorithm

summarizes the related studies. The research is organized as follows: Section 2 presents the details of the proposed approach, Section 3 presents the experimental results, Section 4 discusses the results, and Section 5 concludes the research and mentions future studies. In summary, the major contributions of this paper include the following:

• We extract objective general patterns of facial beauty attributes learned subjectively by deep convolutional neural networks (CNNs).
• This is the first study, wherein correlations between beauty and facial attributes are subjectively analyzed based on a quantitative approach. We conclude general patterns of statistically significant attributes of attractiveness.
• We propose a novel framework and algorithm to train CNNs, visualize and extract the learnings of the machine about beauty, and automatically explain every machine decision in the entire dataset.
• We validate existing psychological, biological, and neurobiological studies of beauty and discover new patterns.
• We propose a novel method to prove the relationship between the golden ratio and facial beauty.

2. Related Work

In a recent study [35], researchers used deep learning to build a facial attractiveness assessment model by training a CNN on the SCUT-FBP5500 dataset [34], and they used transfer learning techniques to improve the training accuracy. SCUT-FBP5500 [34] is a new dataset that contains 5500 Asian and Caucasian male and female faces for a multi-paradigm facial beauty prediction. In addition to a shallow prediction model with hand-crafted features, a deep learning model is used to predict facial beauty scores based on the input of 60 volunteers who labelled faces with a score ranging from 0 to 5, where 5 represents the most attractive face. However, does the perception of the 60 volunteers adequately represent beauty in general? Does the predicted score of 5 accurately indicate an extremely beautiful face and a lower score a less beautiful face? What does less beauty mean and according to whom? The main issue with this approach is that it attempts to establish a general beauty prediction model while the concept of beauty is not addressed. First, we need to understand why each volunteer judges a face to be beautiful or not and draw conclusions and establish a model upon this. In addition, the dataset used to train the machine would never manifest how the majority perceive beauty, and considering an average score does not satisfy anyone, including the 60 volunteers themselves, which causes the prediction output to be random and meaningless.

A geometry-based beauty assessment has been proposed to test the effect of geometric shape representation on facial beauty, regardless of other appearance features [28]. Apart from the fact that the proposed method is based on average score data, two faces can have similar geometric shapes; yet the beauty decision is different.

In addition to building beauty prediction models, researchers have attempted to visualize the research on beauty by CNNs trained on Asian faces through visualization of filters for understanding the reasons behind beauty decisions. However, the visualization of filters did not explain the reasons behind beauty decisions or show features of beauty because filters alone are uninterpretable [32]. The correlation between facial features was investigated by a different approach [3]. In this approach, researchers pre-assumed that the correlation between facial features renders a face beautiful; accordingly, their neural network was trained not to learn beauty but extract and estimate facial features based on majority voting and average score datasets, without investigating the reasons behind beauty decisions. Recent research [27] used an effective approach based on the gradient-weighted class activation mapping (Grad-CAM) technique [36] to visualize the learnings of neural networks about beautiful faces. This study used the previously mentioned dataset SCUT-FBP5500 [34] to train the CNN model and obtain activation maps, and the top 50 images with the highest attractiveness scores were averaged by overlaying images of the face activation maps. However, this approach has two limitations: first, training is based on average score data; second, an average of activation maps cannot convey how the machine determines beauty for each face and based on what features.

3. Methods

Deep human feelings, such as beauty, are common and highly subjective and are affected by many internal and external factors such as hormones, media, and environment, or even certain scenes, sounds, and smells [37,38]. The way these experiences evolve and accumulate causes an individual’s decisions to be unique and subjective. Therefore, to study such human phenomena and arrive at a general pattern and conclusion, we propose closely investigating each decision subjectively.

In our approach, we assume that beauty decisions are the result of brain cognition and perception and inherit all the accumulated information that eventually leads to such decisions. Since there is no sufficient information on the constituents of beauty in different fields of science, and since there is a strong relation between deep learning and the human brain, if we successfully train a deep learning network to mimic the human brain by identifying and classifying beauty subjective to a specific person, then we can claim that studying the learnings of the machine can explain what the individual’s brain learned.

To perform this study, data were subjectively collected and fed to the machine to teach it to mimic each participant’s brain decisions using deep learning. Once the machine successfully adopts the human brain’s decisions, we can study the human brain by studying the machine. Instead of assuming predefined features, the machine completely learns the features that contribute to beauty decisions. The proposed methodology is illustrated in Figure 1.

3.1. Data Preparation

In previous studies, the datasets used in training CNNs, determining facial ratios, and assessing facial geometric features were based on either group voting or average score rating, by assuming that beauty is universal, which is not necessarily true. Figure 2 illustrates the problem of such an approach, which can be summarized as follows:

• Predictions are not satisfactory and meaningless.
• However, when the majority rate or vote for a face is beautiful, they still disagree on the reason behind voting, which aggravates the problem that renders predictions meaningless.
• Because the model is trained on average scores and/or voting and produces meaningless predictions, the model learning is ambiguous, and it is impossible to interpret neurons to explain beauty, as there is no basis to justify beauty accordingly.

Since “beauty lies in the eyes of the beholder”, we believe every face is beautiful, at least to someone. Thus, in this study, we use the terms “Beauty” and “Different Beauty” that indicate the subjectivity of beauty. In this study, we collected 50,000 female faces’ images and included 10 male participants. The participants were from the Middle East and were aged between 25 and 35 years. The dataset contained different ethnicities, ages, accessories, and backgrounds. Images were crawled from sources under permissive licenses and automatically aligned and cropped using image processing techniques. To avoid any potential ethical or copyright issues, all faces illustrated in this paper have been synthetically generated by a computer and do not belong to any real person. We developed an interactive GUI application that all participants utilized to automate the classification process and data collection. This resulted in a clear and simple classification approach in which all participants employed the same classification strategy for the same dataset and were provided the same instructions. They were told to base their judgment solely on facial beauty. To avoid order effects, photos were shown on separate pages, each in random order. Participants could look at a face without time limits and were free to return to faces that they had already seen and changed their evaluation. The GUI is programmed to show each face with three labels: “Beauty”, “Different Beauty”, and “Another”, which a participant can choose if the face is perceived to neither belong to the “Beauty” or “Different Beauty” category. Every participant collected 3000 female face images for each class; the total number of faces per participant was 6000, and the average labeling time was 28 h. Since the 50,000 dataset was randomly displayed for each participant, it was not necessary that all participants rated the same faces, as our main objective was to construct a subjective dataset for each person and subsequently derive a general pattern at the end of the study.

3.2. Ensemble Learning

Ensemble learning promotes the “wisdom of crowds”, which holds that the judgment of a larger group of people is often superior to that of a single expert. Similarly, ensemble learning describes a collection (or ensemble) of basic learners or models who collaborate to produce a more accurate final prediction. A single model, also known as a base or weak learner, may not function effectively alone, owing to excessive variation or strong bias. However, weak learners can collaborate to build a strong learner, which outperforms any individual base model, since their combination minimizes bias or variation [39,40].

Let $S$ be a sample space containing samples $x_i\left(i=1,2,\dots ,s\right)$ and belong to classes $c_j\left(j=1,2,\dots ,N\right)$ which comprises class set $C=\left\{c_1,c_2,\dots ,c_N\right\}$. Consider $K$ independent voters (basic classifiers) as $f_k\left(k=1, 2, \dots ,K\right).$ Given sample $x_i$, the prediction (vote) by $f_k$ is $f_k\left(x_i\right)$. Accordingly, the simple ensemble voter (classifier) system is described as

$$F\left(x_i\right)=\{\begin{array}{c}{c}_{j^{*}} if \exists c_{j^{*}}\in C \wedge T_F\left(x_i\in c_{j^{*}}\right)=\underset{1\le j\le N}{\max }{T}_F\left(x_i\in c_{j^{*}}\right) \\ \wedge T_F\left(x_i\in c_{j^{*}}\right)>\alpha \times K\\ c_{reject} Else \end{array},$$

(1)

$$T_F\left(x_i\in c_j\right)={{\displaystyle \sum }}_{K=1}^K{y}_k\left(x_i\in c_j\right),j=1,2,\dots ,N,$$

(2)

$$y_k\left(x_i\in c_j\right)=\{\begin{array}{c}1 if f_k\left(x_i\right)=c_j\left(j=1,2,\dots ,N\right) \\ 0 Else \end{array},$$

(3)

where $\alpha \times K$ represents the threshold for voting, and $\wedge$ implies the logic operator AND. In this approach, each voter has the same weight, regardless of their performance. However, by examining each voter’s strengths in recognizing and assigning different weights, the ensemble system judges more accurately and produces considerably better performance. The ensemble-weighted majority voting system is described as follows:

$$F\left(x_i\right)=c_{j^{*}}if\exists c_{j^{*}}\in C\wedge T_F\left(x_i\in c_{j^{*}}\right)=\underset{1\le j\le N}{\max }{T}_F\left(x_i\in c_{j^{*}}\right),$$

(4)

$$T_F\left(x_i\in c_j\right)={{\displaystyle \sum }}_{K=1}^K{w}_k\left(x_i\in c_j\right),j=1,2,\dots ,N,$$

(5)

$$V_{winner}\left(x_i\in c_{j^{*}}\right)=\left\{f_{1,}{f}_2,\dots ,f_k\right\},$$

(6)

$$W\left(x_i\in c_{j^{*}}\right)=\left\{w_1,w_2,\dots ,w_k\right\},$$

(7)

where $w_k\left(x_i\in c_j\right)$ represents the weight of voter $f_k$ voting to $c_j$ for a given $x_i$, and $W\left(x_i\in c_{j^{*}}\right)$ represents all weight lists of the majority votes of winner voters $V_{winner}\left(x_i\in c_{j^{*}}\right)$. For optimum learning, we propose using varied training data with k-fold cross-validation for each voter, wherein for each fold model $m$, we create $s$ snapshots. The primary idea behind creating model snapshots is to train a single model while continuously lowering the learning rate to reach a local minimum and to save a snapshot of the weight of the current model. Later, to move away from the current local minimum, it will be essential to actively accelerate learning. The process continues until all cycles are completed. Using cyclic cosine annealing, one of the primary techniques for producing model snapshots for CNN is to gather multiple models during a single training session [41]. The cyclic cosine annealing approach begins with the initial learning rate, progressively drops to the bare minimum, and then rapidly shoots back up. Each epoch’s learning rate for cyclic cosine annealing is given by the following expression:

$$\alpha \left(t\right)=\frac{{\alpha }_0}{2}\left(\cos \left(\frac{\pi mod(t-1,\left[T/s]\right)}{[T/s]}\right)+1\right),$$

(8)

where $T$ is the entire count of the training iterations, $s$ the count of cycles, $\alpha \left(t\right)$ is the learning rate at epoch $t$, and ${\alpha }_0$ is the initial learning rate. The weight of the snapshot model is defined as the weight at the bottom of each cycle. The next learning rate cycle makes use of these weights but permits the learning algorithm to reach various conclusions, thereby producing a variety of snapshot models. We obtained $s$ model snapshots when s training cycles were completed, each of which was used for ensemble prediction. By using $s$ model snapshot predictions, we applied simple average ensemble predictions for each fold model m of each base classifier (voter). The output of the simple average ensemble predictions for each voter was used to obtain the final weighted voting ensemble, as illustrated in Algorithm 1. In the next section, we discuss the use of different policies as voting weights.

Algorithm 1: Machine-weighted voting algorithm

$\mathrm{Inputs}:\mathrm{Data}\mathrm{set}D=\left\{\left(x,y\right):x\in {ℝ}^{n\times p},y\in {ℝ}^n\right\}$; K base learning algorithms Output: Machines Vote Initialization; $\mathrm{for}k=1,\dots ,K$do 1: Split D into $D^{Train},D^{Test}$ $\mathrm{for}i=1,\dots ,msplits$do $\mathrm{Split}{D}^{Train}$$\mathrm{into}{D}_i^{train},D_i^{validation}$ for ith split. $\mathrm{Basic}\mathrm{classifier}{f}_k$$\mathrm{train}\mathrm{on}{D}_i^{train}$$\mathrm{and}\mathrm{validate}\mathrm{on}{D}_i^{validation}$ $P_{ik}=$$\mathrm{Predict}\mathrm{on}{D}^{Test}$ End Concatenate m $\mathrm{predictions}\mathrm{on}{D}^{Test}$ ${\widehat{Y}}_k=\left(P_{1k},\dots ,P_{mk}\right)$ $\mathrm{Compute}\mathrm{optimal}\mathrm{voting}\mathrm{weights}(w_1^{*},...,w_k^{*})$ Apply weighted class voting (4). End

3.3. Optimal Voting Weights

3.3.1. Best Combination

Grid searching for weight values between 0 and 1 for each ensemble member, such that the weights across all ensemble members add to one, is the simplest and possibly the most exhaustive approach. According to a study [39], however, using the optimization model proposed by Perrone et al. (1992) that aims to combine predictions from fundamental classifiers by choosing the most appropriate weight to combine them, such that the resulting ensemble reduces the overall expected prediction error (MSE) is a more effective approach than other methods [42].

$$\begin{gathered} Min MSE\left(w_1{\widehat{y}}_1+w_2{\widehat{y}}_2+\dots +w_k{\widehat{y}}_k,Y\right) \\ s.t.{{\displaystyle \sum }}_{k=1}^k{w}_k=1, \\ {w}_k\ge 0,\forall f_k\left(k=1,2,\dots ,K\right), \end{gathered}$$

(9)

where ${\widehat{Y}}_k$ denotes the vector of the basic classifier $f_k$ predictions on the cross-validation samples; $Y$ is the vector of true response values, and $w_k$ is the weight corresponding to the base model $f_k\left(k=1,2,\dots ,K\right)$. Assuming that $n$ is the total number of instances, the optimization model is as follows: $y_i$is the true value of observation $i$, and ${\widehat{y}}_{ik}$ is the prediction of observation $i$ of the base model $f_k$.

$$\begin{gathered} Min \frac{1}{n}{{\displaystyle \sum }}_{n=1}^n{\left(y_i-{{\displaystyle \sum }}_{j=1}^k{w}_k{\widehat{y}}_{ik}\right)}^2 \\ s.t.{{\displaystyle \sum }}_{k=1}^k{w}_k=1, \\ w_k\ge 0, \forall f_k\left(k=1, 2, \dots ,K\right) \end{gathered}$$

(10)

The aforementioned formulation is a nonlinear convex optimization problem. Computing the Hessian matrix shows that the objective function is convex since the constraints are linear. Therefore, the best solution to this problem is shown to be the global best solution because the local optimum of a convex function (the objective function) in a convex feasible region (the feasible region of the preceding formulation) is guaranteed to be a global optimum [43].

3.3.2. Priori Recognition Performance Statistics

A basic classifier is assigned more weight based on how well it recognizes patterns [40] Let the confusion matrix of voter $f_k$ be

$$C{M}_k=\left[\begin{array}{cccc}{n}_{11}^k& n_{12}^k& \cdots & n_{1N}^k\\ n_{21}^k& \ddots & \cdots & n_{2N}^k\\ ⋮& ⋮& n_{j_1{j}_2}^k& ⋮\\ n_{N1}^k& n_{N2}^k& \cdots & n_{NN}^k\end{array}\right]\left(k=1,2,\dots ,K\right).$$

(11)

When $j_1=j_2$, $n_{j_1{j}_2}^k$ denotes the count of samples that belong to class $c_{j_1}$ and are classified accurately as $c_{j_1}$ by voter $f_k$. When $j_1\ne j_2$, $n_{j_1{j}_2}^k$ represents the number of samples belonging to class $c_{j_1}$ that are misclassified as $c_{j_2}$ by voter $f_k$. The instances classified as $c_{j_2}$ become

$$n_{j_2}^k={{\displaystyle \sum }}_{j_1=1}^N{n}_{j_1{j}_2}^k\left(j_2=1,2,\dots ,N\right).$$

(12)

Consequently, the conditional likelihood that this sample belongs to class $c_{j_1}$ is reflected as

$$P{M}_k=\left[\begin{array}{cccc}{P}_{11}^k& P_{12}^k& \cdots & P_{1N}^k\\ P_{21}^k& \ddots & \cdots & P_{2N}^k\\ ⋮& ⋮& P_{j_1{j}_2}^k& ⋮\\ P_{N1}^k& P_{N2}^k& \cdots & P_{NN}^k\end{array}\right]\left(k=1,2,\dots ,K\right).$$

(13)

When classifier $f_k$ classifies instance $x_i$ as class $c_{j_2}$, its voting weight for class $c_{j_1}$ is $P_{j_1{j}_2}^k$.

3.3.3. Model Calibration

Neural networks are usually calibrated inadequately [44], implying that they are overconfident in their predictions. In the classification process, neural networks produce “confidence” scores along with the predictions. These confidence levels ideally coincide with the actual likelihood of correctness. For instance, if we provide 100 predictions with a confidence level of 80%, we would anticipate that 80% of the predictions will come true. If such a case, the network is calibrated. Model calibration is the process of obtaining a trained model and applying a post-processing procedure to enhance its probability estimation.

Let input images $X\in x$ and class labels $Y\in \mathcal{Y}=\left\{1,\dots ,k\right\}$ be random variables following the joint ground-truth distribution $\pi \left(X,Y\right)=\pi (Y|X)\pi \left(X\right)$, and let $h$ be a CNN with $h\left(X\right)=\left(\widehat{Y},\widehat{P}\right)$, where $\widehat{Y}$ is the predicted class, and $\widehat{P}\in \left[0,1\right]$ is the attributed confidence level. The objective is to calibrate $\widehat{P}$, such that it represents the true class probability. In practice, the accuracy of deep learning networks is typically lower than confidence. Perfect calibration is defined as

$$\mathbb{P}\left(\widehat{Y}=Y|\widehat{P}=p\right)=p,\forall p\in \left[0,1\right]$$

(14)

The calibration error which describes the deviation in expectation between confidence and accuracy is

$${\mathbb{E}}_{\widehat{p}}=\left[\left|\left(\widehat{Y}=Y|\widehat{P}=p\right)-p\right|\right].$$

(15)

Calibration techniques for classifiers seek to convert an uncalibrated classifier’s confidence score to a calibrated one $\widehat{Q}\in \left[0,1\right]$ that corresponds to the precision for a specific level of confidence [45]. This calibration technique is a post-processing technique that requires a separate learning phase to establish a mapping $g:\widehat{P}\to \widehat{Q}$ along with $\widehat{\theta }$ which denotes the calibration parameters and can be considered a probabilistic model $\widehat{\pi }(Y|\widehat{P},\widehat{\theta })$. The calibration parameters $\widehat{\theta }$ are typically estimated using the maximum likelihood (ML) for all scaling methods while minimizing the NLL loss. The calibration parameter $\widehat{\theta }$ can be calculated in this case by applying an uninformative Gaussian prior $\pi \left(\theta \right)$ with a wide variance over the parameters and inferring the posterior by

$$\pi \left(\theta |\widehat{P},Y\right)=\frac{\pi \left(Y|\widehat{P},\theta \right)\pi \left(\theta \right)}{{{\displaystyle \int }}_{\Theta }\pi \left(Y|\widehat{P},\theta \right)\pi \left(\theta \right)d\theta },$$

(16)

where $\pi \left(Y|\widehat{P},\theta \right)$ is the likelihood. We can map a new input ${\widehat{p}}^{*}$ with the posterior predictive distribution defined by

$$f\left(y^{*}|{\widehat{p}}^{*},\widehat{P},Y\right)=\underset{\theta }{{\displaystyle \int }}\pi \left(y^{*}|{\widehat{p}}^{*},\theta \right)\pi \left(\theta |\widehat{P},Y\right)d\theta .$$

(17)

We modeled the epistemic uncertainty of calibration mapping as opposed to Bayesian neural networks (BNNs). The distribution $f_i$ acquired by calibration for a sample with index $i$rather reflects the uncertainty of the classifier for a specific degree of confidence than the model uncertainty for a particular prediction. A distribution is obtained as a calibrated estimate. We utilized stochastic variational inference (SVI) as an approximation because the posterior cannot be calculated analytically. SVI uses a variational distribution (often a Gaussian distribution) whose structure is simple to evaluate [45]. We sample $T$ sets of weights and utilize them to construct a sample distribution consisting of $T$ estimates for a new single input ${\widehat{p}}^{*}$, with the parameters of the variational distribution optimized to match the true posterior using the evidence lower bound (ELBO) loss.

We utilized the vector scaling method which is an extension of Platt scaling [46]. Temperature scaling is a popular method for calibrating deep learning models [44]. Temperature scaling is a parametric calibration approach optimized with respect to negative-log-likelihood (NLL) on validation data [44]. It learns a single parameter temperature ($T)$ for all classes to produce the calibrated confidences:

$${\widehat{q}}_i=\underset{k}{\max }{\sigma }_{SM}{\left(\frac{z_i}{T}\right)}^{\left(k\right)},$$

(18)

where $k$ is the class label $\left(k=1,\dots ,K\right)$, $z_i$ is the logit vector, and ${\sigma }_{SM}\left(z_i\right)$ is the predicted confidence. As $T\to \infty$, ${\widehat{q}}_i$ approaches the minimum that indicates maximum uncertainty.

Calibration Evaluation

A typical metric for determining the calibration error of neural networks is expected calibration error (ECE) [44]. Let $B_m$ be the set of sample indices whose predicted confidence falls within the range $I_m=\left(\frac{m-1}{M},\frac{m}{M}\right]$, $m\in M$. The accuracy of $B_m$ is

$$acc\left(B_m\right)=\frac{1}{\left|B_m\right|}{{\displaystyle \sum }}_{i\in B_m}\mathbb{𝟙}\left({\widehat{y}}_i=y_i\right),$$

(19)

where ${\widehat{y}}_i$ and $y_i$ are the true label and predicted value of the instance $i$. The average predicted confidence of bin $B_m$ can be formulated as

$$conf\left(B_m\right)=\frac{1}{\left|B_m\right|}{{\displaystyle \sum }}_{i\in B_m}{\widehat{p}}_i,$$

(20)

where ${\widehat{p}}_i$ is the confidence of sample $i$.

The expected calibration error (ECE) takes the weighted average of the bins’ accuracy/confidence differences of $n$ number of samples:

$$ECE={{\displaystyle \sum }}_{m=1}^M\frac{\left|B_m\right|}{n}\left|acc\left(B_m\right)-conf\left(B_m\right)\right|.$$

(21)

The maximum calibration error (MCE) [44] focuses more on high-risk applications, where the maximum accuracy/confidence difference is more important than the average which represents the worst-case scenario.

$$MCE=\underset{m\in \left\{1,\dots ,m\right\}}{\max }\left|acc\left(B_m\right)-conf\left(B_m\right)\right|$$

(22)

Uncertainty Evaluation

Prediction interval coverage probability (PICP) is a metric used for Bayesian models to determine the quality of uncertainty estimates, that is, the likelihood that an instance’s true value falls within the predictive range. The mean prediction interval width (MPIW) is another metric used to measure the mean width of all prediction intervals to evaluate the sharpness of the uncertainty estimates.

A prediction interval around the mean estimate can be used to express epistemic uncertainty. We obtained the interval boundaries by selecting quantile-based constraints of the range for a specific confidence level $\tau$, while assuming a normal distribution.

$$C_{\tau ,i}=\left(l_i,u_i\right): \mathbb{P}\left(l_i\le prec\left(i\right)\le u_i\right)=1-\tau$$

(23)

where $prec\left(i\right)$ represents the observed precision of sample $i$ for specific ${\widehat{p}}_i$. If all samples of the measured accuracies lie in a $100\left(1-\tau \right)\%$ prediction interval (PI) around $100\left(1-\tau \right)\%$ of the time, the uncertainty is appropriately calibrated [45], with $g$ being a calibration model that produces a PDF $f_i$ for input with index $i$ out of $N$ samples. The PICP was calculated as follows:

$$PICP=\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N \mathrm{𝟙}(prec\left(i\right)\in C_{\tau ,i})$$

(24)

The definition of the PICP is usually applied when performing calibrated regression, and when the true target value is known. However, the true precision of the classification is not easily obtainable. As a result, we apply a binning method to all available quantities $\widehat{P}$ with $N$ samples to estimate the precision for each sample. It is necessary for PICP $\to \left(1-\tau \right)$ as $N$ $\to$ $\infty$ for flawless uncertainty calibration. Using this concept, we can calculate the uncertainty by measuring the difference between PICP and $\left(1-\tau \right)$. The PI width for a certain $C_{\tau ,i}$ is averaged over all N samples to obtain the MPIW, which is a complementary metric. With regard to the two metrics, we want the models to have larger PICP values while reducing the MPIW. By utilizing PICP and MPIW, we can assess both the quality of the calibration mapping and epistemic uncertainty quantification. In our study, we propose using PICP as an ensemble voting weight.

3.4. Proposed Framework to Explain CNNs

Even though CNN achieves extremely high accuracy in many detection and classification problems, the CNN is considered a “Black box”. Although we understand the CNN architecture and process and how features are extracted, it is still difficult for humans to know how the network decides its classification and based on what features the decision is made. This is extremely important in vital areas where the decision reason is important, such as in military operations and medicine fields. Explainable AI (XAI) research has grown significantly in recent years owing to the rapid advancement in deep learning and the need for reliable machine decisions. Deep Taylor decomposition is a powerful technique for explaining CNN decisions by identifying the features in an input vector that have the greatest impact on a neural network’s output based on redistributed relevance [47]. In a recent Alzheimer’s disease detection study, deep Taylor decomposition produced more reasonable and accurate results than Grad-CAM [48]. Layer-wise relevance propagation (LRP) is the foundation for deep Taylor decomposition [49] that seeks to create a relevance metric R over the input vector, such that we can represent the network output as the sum of the values of R, $f\left(x\right)={\displaystyle \sum }_{d=1}^V{R}_d$, where$f$ is the neural network forward pass function. We can use the deep Taylor decomposition function in terms of its partial derivatives to approximate the relevance propagation function illustrated in Figure 3.

Consider a neuron taking input vector $x_i$ that outputs

$$x_j=\max (0,{{\displaystyle \sum }}_i{x}_i{w}_{ij}+b_j),b_j\le 0,x_j\ge 0.$$

(25)

To decompose the neuron output in terms of the input variables, the output is rewritten as a first-order Taylor expansion:

$$x_j={{\displaystyle \sum }}_i{\frac{\partial x_j}{\partial x_i}|}_{{\left(x_i\right)}_{i=}{\left({\tilde{x}}_i\right)}_i}.(x_i-{\tilde{x}}_i),$$

(26)

where ${\left({\tilde{x}}_i\right)}_i$ is the root point of the forward-propagation function. The local decision boundary is where the root points of the forward propagation function are; therefore, the gradients along that boundary point provide the greatest details regarding how the function categorizes the input. The deep Taylor decomposition equation can be rewritten as follows:

$$R_i={\frac{\partial x_j}{\partial x_i}|}_{{\left(x_i\right)}_{i=}{\left({\tilde{x}}_i\right)}_i}.\left(x_i-{\tilde{x}}_i\right),{{\displaystyle \sum }}_i{R}_i=R_j.$$

(27)

By searching the root point in the direction $v_{ij}$ of the input space, we obtain the explicit decomposition equation

$$R_i={{\displaystyle \sum }}_j\frac{q_{ij}}{{{\displaystyle \sum }}_{i^{\prime }}{q}_{i^{\prime }j}}{R}_j,q_{ij}=v_{ij}{w}_{ij},q_{i^{\prime }j}=v_{i^{\prime }j}{w}_{i^{\prime }j}.$$

(28)

Another contemporary XAI method based on popular local interpretable model-agnostic explanations (LIME) [50] is the Bayesian local interpretable model-agnostic explanations (BayLIME) [51]. This method can be used with any machine-learning model because it is model-agnostic. By changing the input of the data samples and observing how the predictions vary, the technique aims to understand the model. The output of the LIME is a list of explanations indicating the contribution of each characteristic to the prediction of a data sample. This offers local interpretability and makes it possible to identify feature changes that have the greatest influence on the prediction. Let $X=\left\{x_1,\dots ,x_n\right\}$ be the input set with $n$ samples, where $x_i$ is instance $i$ with $m$ features (i.e., $X=\left(x_{ij}\right)\in {ℝ}^{n\times m}$), and let $Y={\left[y_1,\dots ,y_n\right]}^T$ be the corresponding $n$ target values. Accordingly, the maximum likelihood estimates of the weighted samples $\left(X^{\prime },Y^{\prime }\right)$ for the linear regression model is $\beta ={(X^{T}WX)}^{-1}{X}^{T}WY$, and $W=diag\left(w_{1,}\dots ,w_n\right)$ is the diagonal matrix, which is determined by a kernel function based on the proximity of the new samples to the original instance. BayLIME proposes embedding prior knowledge as follows:

$${\mu }_n={(\lambda I_m+\alpha X^{T}WX)}^{-1}\lambda I_m{\mu }_o+{(\lambda I_m+\alpha X^{T}WX)}^{-1}\alpha X^{T}WX\beta ,$$

(29)

where ${\mu }_o$ is the mean vector matrix to be estimated, $\lambda I_m$ is the “pseudo-count” of the prior sample size based on which we form our prior estimates of ${\mu }_o$ ($I_m$ is $m\times m$ identity matrix), and $\alpha X^{T}WX$ is the “accurate-actual-count” of observation data size, i.e., the true inference of the $n$ perturbed data $X^{T}WX$ scaled by precision $\alpha$. Let $m=1$ which symbolizes one feature instance with a simpler kernel function that yields a constant weight $w_c$ then ${\mu }_n$ becomes

$$\frac{\lambda }{\lambda +\alpha w_{c}n}{\mu }_o+\frac{\alpha w_{c}n}{\lambda +\alpha w_{c}n}\beta .$$

(30)

Using $\lambda$ point data prior to our new experiment, we created our prior estimate of ${\mu }_o$. We collected $n$ samples in the experiments and obtained the MLE estimate $\beta$ by considering the precision $\alpha$ and weights $w_c$ of the new samples. Depending on the distributions of the efficient data size employed, that is, $\lambda$ and $\alpha w_{c}n$, we combine ${\mu }_o$ and $\beta$. Finally, all of the effective samples employed capture the confidence in our new posterior estimate, that is, $\lambda +\alpha w_{c}n$ (the posterior precision).

In our research, we also use another effective method that uses perturbation masks to alter Sobol indices in the context of black-box models to determine whether parts of the sample have an impact on output predictions [52]. Let $(X_1,\dots ,X_d\}$ be independent variables, and assume that $\mathit{f}\in {\mathbb{L}}^2 \left(\mathcal{X}, P\right)$. Let $\mathcal{U}=\left\{1,\dots ,d\right\};$ $u$ is $a subset of \mathcal{U}$, its complementary $~\mathit{u}$ and $\mathbb{E}(·)$ is the expectation over the perturbation space.

Hoeffding decomposition enables the expression of the function $\mathit{f}$ as sums of increasing dimensions, with the symbol ${\mathit{f}}_u$ signifying the partial contribution of the variables ${\mathit{X}}_u={(X_i)}_{i\in u}$ to the score $\mathit{f}\left(\mathit{X}\right)$:

$$\begin{gathered} \mathit{f}\left(\mathit{X}\right)={{\displaystyle \sum }}_{u\subseteq \mathcal{U}}{\mathit{f}}_u\left({\mathit{X}}_u\right), \\ Var\left(\mathit{f}\left(\mathit{X}\right)\right)={{\displaystyle \sum }}_{u\subseteq \mathcal{U}}Var({\mathit{f}}_u\left({\mathit{X}}_u\right)). \end{gathered}$$

Following the constraint,

$$\forall \left(u,v\right)\subseteq {\mathcal{U}}^{2}s.t.u\ne v,\mathbb{E}\left({\mathit{f}}_u\left({\mathit{X}}_u\right){\mathit{f}}_v\left({\mathit{X}}_v\right)\right)=0.$$

(31)

The sensitivity index $S_u$ which quantifies the contribution of the variable set contributed. ${\mathit{X}}_u$ to the model response $\mathit{f}\left(\mathit{X}\right)$ in terms of fluctuation are given by Sobol indices:

$$S_u=\frac{Var({\mathit{f}}_u\left({\mathit{X}}_u\right))}{Var\left(\mathit{f}\left(\mathit{X}\right)\right)}=\frac{Var(\mathbb{E}\left(\mathit{f}\left(\mathit{X}\right)|\left({\mathit{X}}_u\right)\right)-{{\displaystyle \sum }}_{v\subset u}Var(\mathbb{E}\left(\mathit{f}\left(\mathit{X}\right)|\left({\mathit{X}}_u\right)\right)}{Var\left(\mathit{f}\left(\mathit{X}\right)\right)}$$

(32)

With regard to the model decision, Sobol indices provide a quantification of the significance of any subset of features where ${\displaystyle \sum }_{u\subseteq \mathcal{U}}{S}_u=1$.

The total Sobol index $S_{T_i}$ which quantifies the contribution of variable ${\mathit{X}}_i$ to the variance of the output of the model and its interactions of any order with any other input variables are as follows:

$$S_{T_i}={{\displaystyle \sum }}_{\begin{array}{c}u\subseteq U\\ i\in u \end{array}}{S}_u=1-\frac{Var{\mathit{X}}_{~i}({\mathbb{E}}_{X_i}(\mathit{f}\left(\mathit{X}\right)|\left({\mathit{X}}_{~i}\right))}{Var\left(\mathit{f}\left(\mathit{X}\right)\right)}=\frac{{\mathbb{E}}_{{\mathit{X}}_{~i}}(Va{r}_{X_i}(\mathit{f}\left(\mathit{X}\right)|\left({\mathit{X}}_{~i}\right))}{Var\left(\mathit{f}\left(\mathit{X}\right)\right)}$$

(33)

A more efficient estimator uses the Jansen estimator with the quasi-Monte Carlo (QMC) sampling strategy. Let ${\mathit{A}}_{ji}$ and ${\mathit{B}}_{ji}$ be the elements of the matrices, such that $i=1, \dots , d$ represents the number of variables investigated, and $j=1, \dots , N$ is the number of instances in matrices A and B obtained in the same size as the perturbed inputs. The new matrix $C^{\left(i\right)}$ formed is similar to A, except that the column of $\mathit{B}$ replaced the corresponding column of variable i. We express $f_{\varnothing }=\frac{1}{2N}{{\displaystyle \sum }}_{j=1}^N\mathit{f}({\mathit{A}}_j)$ and the empirical variance $\widehat{V}=\frac{1}{N-1}{{\displaystyle \sum }}_{j=0}^N{(\mathit{f}\left({\mathit{A}}_i\right)-f_{\varnothing })}^2$. The empirical estimators for the first order (${\widehat{S}}_i$) and total order (${\widehat{S}}_{T_i}$) can be formulated as

$${\widehat{S}}_{T_i}=\frac{\widehat{V}-\frac{1}{2N}{{\displaystyle \sum }}_{j=1}^N{(\mathit{f}({\mathit{B}}_j)-\mathit{f}(C_j^{(i)}))}^2}{\widehat{V}}.\hspace{1em}\hspace{1em}\hspace{1em}{\widehat{S}}_{T_i}=\frac{\frac{1}{2N}{{\displaystyle \sum }}_{j=1}^N{(\mathit{f}\left({\mathit{A}}_j\right)-\mathit{f}\left(C_j^{\left(i\right)}\right))}^2}{\widehat{V}}.$$

(34)

3.5. Proposed Approach to Explain Facial Beauty

XAI techniques are excellent tools for inferring and visualizing the learnings of neurons; however, examining every output manually is a tedious process, and it is an almost impossible task in the case of enormous datasets. Furthermore, examining only some samples provides less information about the learning of the machine, and in some cases, it can be extremely dangerous, as in medical diagnosis and army equipment. In general, to have a reliable machine system, we need to know what the neurons learned and how they made their decision for every input before a real application deployment. To overcome this challenge and visualize and understand what the machine learned about beauty, we propose the most voted feature (MVF) algorithm, Algorithm 2: a novel approach that enables explanation of every decision a machine makes and determination of a general learned pattern by considering the dataset as an input and the learned features that the machine decision depends on in the entire dataset as the output. The proposed MVF algorithm acting as an independent component can be applied as part of any CNN and XAI method, and it has the possibility of being deployed to different machine learning tasks.

The MVF algorithm starts the process by reading an image as input data in RGB channels $I:u\to {\left[0,1\right]}^3$ and obtaining the coordination points of the interesting area $f_n=\left[(x_1,y_1\right),\dots ,\left(x_n,y_n\right)]$ using object and landmark detection techniques. Subsequently, these coordination points are encoded and grouped into all possible and interesting features to examine $F_n=\left[f_1,\dots ,f_n\right]$. The possible beauty features are presented in

Table 2. Facial features.

Feature	Area
Feature 1	Right eye
Feature 2	Left eye
Feature 3	Right eyebrow
Feature 4	Left eyebrow
Feature 5	Nose
Feature 6	Center of lips
Feature 7	Right of lips
Feature 8	Left of lips
Feature 9	Chin
Feature 10	Around hairline
Feature 11	Right cheek
Feature 12	Left cheek

. The next process is based on the output of the XAI technique. For the activation of visualization-based methods such as Sobol and Deep Taylor, we predict the class and subsequently obtain the learned features using a pre-trained model. The output is normalized and converted into a new grayscale image$I_N:u\to {\left[0,1\right]}^1$, where Gaussian convolution is applied (Equation (35)). Gaussian convolution is used to average the image pixels and obtain the coordination of the maximum pixel values that represent the most learned feature coordinate $M_p$ (Equation (36)).

$$G{[I_N]}_p:{{\displaystyle \sum }}_{q\in u}{G}_{\sigma }\left(p-q\right){(I_N)}_q,p=\left(p_x,p_y\right),q=\left(q_x,q_y\right),\in u$$

(35)

$$M_p=\underset{p}{argmax}{(I_N)}_p$$

(36)

For segmentation visualization-based methods such as BayLIME, we first obtain the contours [53] as objects $N_z\left(z=1,2,\dots ,n\right)$, and subsequently, calculate the raw moments. For a 2D continuous function $f\left(x,y\right)$, the moment of order $\left(p+q\right)$ is defined as

$$M^N{}_{pq}={{\displaystyle \iint }}_{-\infty }^{\infty }{x}^p{y}^{q}f\left(x,y\right)dxdy$$

(37)

for; adapting this to a scalar (grayscale) image with pixel intensities $I\left(x,y\right)$, raw image moments $M_{ij}$ for a segment $N_z$ are calculated by

$$M^{N_z}{}_{ij}={{\displaystyle \sum }}_x{{\displaystyle \sum }}_y{x}^i{y}^{j}I\left(x,y\right)$$

(38)

Once we calculate the raw moments for each object, we can obtain the maximum contour area $M^{max}{}_{ij}=max${$M^{N_1}{}_{ij},\dots ,M^{N_z}{}_{ij}\}$ and derive different image properties, such as the centroid coordinate $M_p=\left\{\overline{x},\overline{y}\right\}=\left\{\frac{M^{max}{}_{10}}{M^{max}{}_{00}},\frac{M^{max}{}_{01}}{M^{max}{}_{00}}\right\}$.

After obtaining the most learned feature coordinate $M_p$, we calculate all distances $D=\left[d\left(M_p,F_1\right),\dots ,d\left(M_p,F_n\right)\right]$ between interesting features $F_n$ and the most learned feature $M_p$ using $l^{2}norm$ (Equation (39)). All calculations are based on the Cartesian pixel coordinate system, where the origin (0, 0) is in the upper-left corner. The final step is to obtain the most dependent feature (MVF) which is the minimum distance (Equation (40)).

$$d\left(M_p,F_n\right)=\sqrt{{{\displaystyle \sum }}_{i=1}^n{\left|M_p-F_i\right|}^2}$$

(39)

$$MVF=Min\left(D\right)$$

(40)

Beauty Feature Voting

After obtaining the learned features of each voter, we propose weighted feature voting. Let $MLF=\left\{l_{1k}, \dots ., l_{mK}\right\}$ be the output features set by each voter $f_{k }\in V_{winner}$ for a given sample $x_i$. Equation (4) becomes

$$E_k\left(x_i\right)=l_{m{k}^{*}}if\exists l_{k^{*}}\in MLF\wedge T_E\left(x_i\in l_{k^{*}}\right)=\underset{1\le j\le m}{\max }{T}_E\left(x_i\in l_{mk}\right),$$

(41)

$$T_E\left(x_i\in l_{mk}\right)={{\displaystyle \sum }}_{K=1}^K{w}_k\left(x_i\in l_{mk}\right),$$

(42)

where $w_k\left(x_i\in l_{mk}\right)$ represents the weight of the voter $f_{k }\in V_{winner}$ voting for feature $l$ explained by $m$ explainer. The framework of the proposed method is shown in Figure 4 and Algorithm 2.

Algorithm 2: Most voted feature (MVF) algorithm

$\mathrm{Input}:I:u\to {\left[0,1\right]}^3$$;V_{winner}$$; w_k^{*}$; Landmarks Detector $\mathrm{Output}:\mathrm{The}\mathrm{Most}\mathrm{Voted}\mathrm{Feature}\left(MVF\right)$ initialization; $\mathrm{for}k=1,\dots ,K$do    $\mathrm{for}j=1,\dots ,mexplainers$do       $1:\mathrm{Read}\mathrm{Image}I:u\to {\left[0,1\right]}^3$.       $2:\mathrm{Get}\mathrm{object}\mathrm{features}\mathrm{points}\mathrm{by}\mathrm{Landmarks}\mathrm{Detector}{f}_n=\left[(x_1,y_1\right),\dots ,\left(x_n,y_n\right)]$;       $3:\mathrm{Group}\mathrm{and}\mathrm{encode}\mathrm{features}\mathrm{points}\mathrm{into}\mathrm{all}\mathrm{possible}/\mathrm{interest}\mathrm{features}{F}_n=\left[f_1,\dots ,f_n\right]$;       4: Get class using Voters       If Activations based method do             $5:\mathrm{Get}\mathrm{learned}\mathrm{features}\left(\mathrm{Activations}\right)\mathrm{by}{V}_{winner}$              $6:\mathrm{Normalize}\mathrm{output}\mathrm{and}\mathrm{convert}\mathrm{it}\mathrm{to}\mathrm{a}\mathrm{new}\mathrm{grayscale}\mathrm{image}{I}_N:u\to {\left[-1,1\right]}^1$;             $7:\mathrm{Apply}\mathrm{Gaussian}\mathrm{convolution}G{[I_N]}_p:{\displaystyle \sum }_{q\in u}{G}_{\sigma }\left(p-q\right){(I_N)}_q,p=\left(p_x,p_y\right),q=\left(q_x,q_y\right),\in u$;             $8:\mathrm{Get}\mathrm{the}\mathrm{most}\mathrm{learned}\mathrm{feature}\mathrm{coordinate}{M}_p=\underset{p}{argmax}{(I_N)}_p$;             $9:\mathrm{Get}D=\left[d\left(M_p,F_1\right),\dots ,d\left(M_p,F_n\right)\right],d\left(M_p,F_n\right)=\sqrt{{\displaystyle \sum }_{i=1}^n{\left|M_p-F_n\right|}^2}$             $10:\mathrm{Get}\mathrm{the}\mathrm{most}\mathrm{learned}\mathrm{features}{l}_{mK}=Min\left(D\right)$          Else if Segmentation based method do             $5:\mathrm{Get}\mathrm{learned}\mathrm{features}\left(\mathrm{Segmentation}\mathrm{mask}\right)\mathrm{by}{V}_{winner}$.             $6:\mathrm{Get}\mathrm{contours}\mathrm{objects}{N}_z\left(z=1,2,\dots ,n\right)$             $7:\mathrm{Get}\mathrm{raw}\mathrm{image}\mathrm{moments}\mathrm{for}\mathrm{each}\mathrm{contour}\mathrm{object}{M}^{N_z}{}_{ij}={\displaystyle \sum }_x{\displaystyle \sum }_y{x}^i{y}^{j}I\left(x,y\right)$             8: Get $\mathrm{maximum}\mathrm{contour}\mathrm{area}{M}^{max}{}_{ij}=max${$M^{N_1}{}_{ij},\dots ,M^{N_z}{}_{ij}\}$$\mathrm{and}\mathrm{Centroid}\mathrm{coordinate}{M}_p=\left\{\overline{x},\overline{y}\right\}=\left\{\frac{M^{max}{}_{10}}{M^{max}{}_{00}},\frac{M^{max}{}_{01}}{M^{max}{}_{00}}\right\}$             $9:\mathrm{Get}D=\left[d\left(M_p,F_1\right),\dots ,d\left(M_p,F_n\right)\right],d\left(M_p,F_n\right)=\sqrt{{\displaystyle \sum }_{i=1}^n{\left|M_p-F_n\right|}^2}$             $10:\mathrm{Get}\mathrm{the}\mathrm{most}\mathrm{learned}\mathrm{features}{l}_{mK}=Min\left(D\right)$ $11:\mathrm{Get}\mathrm{the}\mathrm{most}\mathrm{learned}\mathrm{features}MLF=$ $\left\{l_{1k}, \dots ., l_{mK}\right\}$ 12: Apply weighted feature voting and get The Most Voted Feature (MVF) (4). end

3.6. Golden Ratio

The golden ratio of the two-line segments equals the ratio of their sum to the longer segment, which equals their ratio [54], as illustrated in Figure 5. To investigate the golden ratio effect and its relation to facial beauty, we propose to calculate 23 golden facial geometric features in addition to the main facial features, including the eyes, nose, and mouth, as illustrated in Figure 6 and

Table 3. Proposed 23 golden facial geometric features.

Feature	Distance	Description
Feature 1	D3 vs. D5	Midpoint between eyes to nose tip vs. nose tip to chin
Feature 2	D7 vs. D19	Left eye length vs. distance to eyes midpoint
Feature 3	D8 vs. D18	Right eye length vs. eyes midpoint
Feature 4	D4 vs. D6	Nose tip to lips center vs. lips center to chin
Feature 5	D3 vs. D4	Nose length vs. nose width
Feature 6	D1 vs. D2	Length of the face vs. width of the face
Feature 7	D11 vs. D17	Hairline to right pupil vs. pupil to lips center
Feature 8	D12 vs. D16	Hairline to left pupil vs. pupil to lips center
Feature 9	D5 vs. D6	Nose tip to chin vs. lips to chin
Feature 10	D5 vs. D14	Nose tip to chin vs. right pupil to nose tip
Feature 11	D5 vs. D15	Nose tip to chin vs. left pupil to nose tip
Feature 12	D4 vs. D20	Nose width vs. nose tip to lips
Feature 13	D21 vs. D11	Outside eyes vs. hairline to the right pupil
Feature 14	D21 vs. D12	Outside eyes vs. hairline to the left pupil
Feature 15	D4 vs. D21	Nose width vs. lips length
Feature 16	D13 vs. D1	Forehead vs. face length
Feature 17	D10 vs. D8	Right pupil to eyebrow vs. right eye length
Feature 18	D23 vs. D7	Left pupil to eyebrow vs. left eye length
Feature 19	D9 vs. D8	Right eyebrow length to right eyes length
Feature 20	D22 vs. D7	Left eyebrow length to left eyes length
Feature 21	D26 vs. D8	Right eye height vs. width
Feature 22	D25 vs. D7	Left eye height vs. width
Feature 23	D24 vs. D21	Height of lips vs. width

. Unlike previous studies that examined facial symmetry, the 23 geometric features contained almost all possible facial features that provided sufficient information to test feature symmetry based on the golden ratio and provided a suitable assessment of the overall facial symmetry.

3.7. Evaluation Metrics

During classification training, the evaluation metric is crucial for obtaining the best classifier. Therefore, choosing an appropriate assessment measure is crucial for differentiating and achieving the best classifier [55]. To statistically evaluate our proposed methods, we used multiple metrics to evaluate both performance and correlation. To measure the correlation between features and beauty, we applied the point-biserial correlation coefficient (r_pb) [56] which can be considered a case of the Pearson correlation coefficient (PCC) in the case of a dichotomous variable. The dichotomous variable Y is assumed by r_pb to have two values, 0 and 1, which in our study correspond to the two binary classes of Beauty and Different Beauty. The point-biserial correlation coefficient is calculated as follows: the data set is divided into two groups: the Beauty group, which received the value “1” on Y, and the Different Beauty group, which received the value “0” on Y.

$$r_{pb}=\frac{M_1-M_0}{S_n}\sqrt{\frac{n_1{n}_0}{n^2}},$$

(43)

$$s_n=\sqrt{\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{\left(X_i-\overline{X}\right)}^2},$$

(44)

where $M_0$ represents the mean value of the continuous variable X for the total sample points in Group 2, $M_1$ is the mean value of the continuous variable X for the total sample points in Group 1, and $s_n$ is the standard deviation. The number of data samples in Group 1 is $n_1$, those in Group 2 are $n_0$, and the overall sample size is $n$. The point-biserial correlation coefficient, which ranges from −1 to +1, assesses the degree of relationship between two variables. A value of −1 denotes a perfect negative association, a value of +1 denotes a perfect positive association, and a value of 0 indicates no association.

For performance evaluation, we applied the Matthews correlation coefficient (MCC) [57] to evaluate how well binary (two-class) classifications were performed. The MCC returns a number between 1 and +1 and is essentially a correlation coefficient between the observed and anticipated binary classifications. A coefficient of +1 denotes a correct prediction, 0 denotes a prediction that is no better than chance, and 1 denotes a complete discrepancy between prediction and observation. In addition, we considered the following evaluation measures to statistically assess the efficacy of our suggested method: area under the curve (AUC), a popular ranking metric; accuracy, which measures the ratio of correct predictions over the total number of instances evaluated; precision, which measures the proportion of positive patterns that are correctly classified; recall, which measures the proportion of positive patterns that are correctly predicted; and F1-score, which represents the harmonic mean between recall and precision values [55]. Let $TP,TN$ represent true positive and true negative, respectively, and $FP,FN$ represent false positive and false negative, respectively. Accordingly, the performance metrics are calculated as follows:

$$MCC=\frac{TP\times TN-FP\times FN}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TN+FN\right)}},$$

(45)

$$ACC=\frac{\left(TP+TN\right)}{\left(P+N\right)},$$

(46)

$$Precison=\frac{TP}{TP+FP},$$

(47)

$$Recall=\frac{TP}{TP+FN},$$

(48)

$$F1=\frac{2TP}{\left(2TP+FP+FN\right)}.$$

(49)

4. Empirical Results

4.1. Learning Beauty

Quantitative evaluations of the proposed approach were performed for both beauty classifications and explanations. In our study, we deployed ResNet50 [58], SENet [59], and VGG16 [60] as base models. A convolutional neural network (CNN) architecture VGG16 was employed to win the 2014 ILSVR (ImageNet) competition. This is regarded as one of the best vision model architectures created to date. The most distinctive feature of VGG16 is that it emphasizes having convolution layers of 3 × 3 filters with a stride of 1 and always utilizes the same padding and MaxPool layer of 2 × 2 filters with a stride of 2. Throughout the entire architecture, the convolution and max pool layers are arranged in the same manner. It ends with two fully connected layers (FC) and a softmax layer for the output. The 16 layers in VGG16 indicate that there are 16 layers with weights. This network incorporates 138.4 million parameters, rendering it sizable. ResNet50 comprises 48 convolutional layers, one MaxPool layer, and one average pool layer. The framework ResNets introduced enabled the training of extremely deep neural networks, implying that the network may include hundreds or thousands of layers and still function suitably. The squeeze and excitation network (SENet), which adds a channel-wise transform to the current deep neural network (DNN) building blocks, such as the residual unit, has achieved outstanding image classification results. SENet provides CNNs with a novel channel-wise attention technique to enhance channel interdependencies. The network includes a parameter that adjusts each channel weight, such that it is more responsive to important features and less sensitive to unimportant features.

To accomplish the model formulation and evaluation processes, we used three-fold cross-validation. For each fold cross-validation, three snapshot models were established, which resulted in nine models for the three folds, and the data were split into three equivalent subsets that were mutually exclusive for the three-fold cross-validation. Two of the three subsets were used as the training set in each iteration, and the third subset was used as the validation set. The final evaluation of each model depended on the average ensemble performance of nine snapshots created for each model. Before the final weighted voting ensemble stage, each base model was calibrated to obtain the base model performance, calibration errors, model confidence, and uncertainty. All experiments were repeated 10 times in order to reduce statistical variability. The calibration output and performance of our experiments are reported in

Table 4. Average calibration output of each voter for all participants.

Voters	ECE-B	ECE-A	MCE-B	MCE-A	PICP	MPIW
VGG16	35.99 (34.81–37.02)	12.66 (11.45–13.68)	41.05 (40.26–41.14)	26.93 (26.1–27.71)	81.29 (80.31–81.93)	13.57 (9.76–17.91)
SENet50	44.40 (43.56–45.08)	3.93 (2.92–4.93)	51.35 (49.79–52.73)	30.66 (29.67–32.5)	81.23 (79.37–83.76)	15.85 (12.87–17.66)
ResNet50	41.49 (39.74–42.85)	8.61 (6.79–10.38)	55.21 (54.96–56.48)	17.43 (16.46–18.49)	81.39 (80.69–82.86)	17.06 (14.55–19.99)

In each cell M (P25–P75), M is the median, P25 is the 25th percentile, and P75 is the 75th percentile of 10 tests. For each model, the best performance of each column is indicated in bold. A indicates “after calibration”, while B indicates “before calibration”.

,

Table 5. Average performance of each voter for all participants.

Voters	Precision	Recall	F1	MCC	ACC
VGG16	79.00 (77.41–80.01)	77.40 (76.58–78.48)	78.58 (78.02–79.1)	60.88 (59.31–62.43)	79.48 (77.42–81.73)
SENet50	83.01 (81.53–84.46)	87.14 (85.6–87.49)	85.47 (83.1–89.44)	71.67 (70.28–76.26)	84.51 (82.74–85.37)
ResNet50	85.94 (84.96–87.09)	87.20 (84.16–88.64)	86.31 (85.67–87.82)	74.11 (72.3–75.89)	87.34 (86.46–88.22)

In each cell M (P25–P75), M is the median, P25 is the 25th percentile, and P75 is the 75th percentile of 10 tests. For each model, the best performance of each column is indicated in bold. A indicates “after calibration”, while B indicates “before calibration”.

,

Table 6. Calibration output of the ensembled models on the test data.

Strategy	ECE-B	ECE-A	MCE-B	MCE-A	PICP	MPIW
Majority Voting	34.04 (32.82–35.36)	4.98 (3.52–6.25)	49.57 (48.01–51.58)	14.22 (11.78–17.51)	52.55 (51.79–53.25)	23.47 (22.02–25.39)
Best Combination	31.22 (30.53–32.02)	4.57 (4–5.15)	52.39 (50–54.17)	7.89 (6.2–9.21)	48.76 (45.92–50.69)	26.12 (25.53–26.64)
Priori Recognition Performance	31.98 (31.16–33.67)	4.84 (4.25–5.89)	41.02 (39.12–43.84)	9.80 (8.57–11.43)	48.58 (46.28–50.91)	23.10 (22.41–23.64)
ECE (ours)	32.87 (30.65–35.23)	2.83 (1.38–2.95)	47.25 (44.15–50.41)	8.42 (5.6–9.74)	45.46 (44.22–46.49)	24.52 (23.29–25.48)
MCE (ours)	26.18 (25.33–26.9)	2.37 (1.72–2.73)	51.60 (48.98–53.34)	5.80 (4.17–7.22)	48.56 (47.31–50.04)	25.19 (23.93–26.33)
PICP (ours)	34.37 (33.95–35.24)	4.52 (3.11–6.43)	49.51 (48.68–50.24)	6.79 (5.69–8.37)	45.90 (44.72–46.59)	24.39 (23.63–25.34)

In each cell M (P25–P75), M is the median, P25 is the 25th percentile, and P75 is the 75th percentile of the 10 tests. For each model, the best performance of each column is in bold. A indicates “after calibration”, while B indicates “before calibration”.

and

Table 7. Performance of the ensembled models on the test data.

Strategy	Precision	Recall	ACC	F1	MCC	AUC
Majority Voting	88.73 (88.21–90.02)	89.97 (89.15–91.21)	89.81 (87.01–91.52)	90.26 (89.29–90.82)	78.09 (77.86–78.52)	90.01 (88.43–92.69)
Best Combination	90.00 (89.96–90.03)	88.35 (87.78–88.63)	89.42 (89.09–90.09)	89.15 (88.63–89.44)	78.25 (77.02–79.32)	89.95 (89.71–90.35)
Priori Recognition Performance	90.48 (88.33–91.33)	88.75 (87.63–89.66)	89.03 (87.3–91.28)	89.43 (89.22–89.59)	78.69 (78.29–79.13)	89.93 (89.8–90.09)
ECE (ours)	91.40 (90.18–92.63)	90.74 (89.56–91.67)	91.02 (90.94–91.1)	91.00 (90.06–91.83)	82.38 (81.17–83.5)	91.93 (91.26–93.07)
MCE (ours)	90.21 (89.36–90.88)	90.94 (90.75–91.18)	90.96 (90.52–91.75)	90.57 (90.29–90.81)	81.01 (80.57–81.49)	91.57 (91.09–93.25)
PICP (ours)	93.89 (93.2–94.71)	92.16 (90.09–92.42)	92.25 (91.8–92.71)	92.98 (92.3–93.75)	85.56 (85.26–86.16)	93.66 (93.43–94.01)

In each cell M (P25–P75), M is the median, P25 is the 25th percentile, and P75 is the 75th percentile of the 10 tests. For each model, the best performance of each column is in bold. A indicates “After calibration”, while B indicates “before calibration”.

.

4.2. Most Dependent Features

In the previous section, we enabled the machine in understanding beauty, and here, we expect the machine to teach us what leads to beauty decisions. We must study the pattern and understand why and how the machine made its decisions based on what information. Using our proposed Algorithm 2, we are able to detect which features the model mostly depends on for each face in the entire dataset in both Beauty and Different Beauty classes.

Table 8. Top-five facial attributes correlation with beauty decision.

Feature	Variance	Standard Deviation	rpb	t-Test	p-Value	Correlation
Right eye area	13,232.09	115.03	0.87	7.37	<0.001	Positive
Left eye area	13,024.69	114.13	−0.93	10.60	<0.001	Negative
Right eyebrow area	623.15	24.96	0.58	3.06	0.007	Positive
Nose	245.13	15.66	−0.53	2.63	0.02	Negative
Right cheek area	627.25	25.04	0.06	0.27	0.79	Positive
Right corner of lips	181.16	13.46	0.11	0.48	0.64	Positive

shows the Top-5 facial attribute correlations with beauty decisions. Figure 7 shows the output of Algorithm 2: the most dependent feature (MVF) for both classes.

Figure 8 shows the face’s side correlation with the machine’s sentimental decisions based on the side of the face the machine paid more attention to while deciding and categorizing into Beauty and Different Beauty classes.

4.3. Golden Beauty

In this section, we investigate the relationship between the golden ratio and beauty by calculating the golden ratios of the proposed 23 golden facial geometric features for every face in the datasets in both classes. The results of the golden ratio for every feature are shown in Figure 9, and the total mean golden ratio of both classes for each participant is shown in Figure 10.

Using the MVF in the previous section, we found that beauty is correlated more with the right side of the face and specifically to the right eye. The golden ratio correlates more with beauty face decisions. However, it is not understood what causes the right eye to be more influential in beauty decisions than the left eye. To investigate this further, we calculated the average golden ratio of both the right and left eyes of all datasets of the participants. The results of the average eye representation, along with its golden ratio, are shown in

Table 9. Beauty class average eye golden ratio of 10 participants.

	Right Eye Average Representation	Golden Ratio	Left Eye Average Representation	Golden Ratio
Beauty		19.60%		19.60%
Different Beauty		10.95%		7.31%

.

5. Discussion

The experimental results presented in the previous section show a unique and interesting pattern of beauty. We noticed that most attention was paid to the eyes for both classes of Beauty and Different Beauty. This is extremely interesting as a recent social psychology study concluded that eyes are not only a “window to the soul, but also a benchmark of beauty” [61]. However, in our study, we were able not only to detect the most important feature in beauty decisions, but also to identify which sides of the eyes the beauty decision depends on. The decisions pertaining to the Beauty class were mostly based on the right eye area, while the decisions pertaining to the Different Beauty class were based on the left eye area beside the nose and lip areas. Another interesting finding is the side of the face on which beauty decisions mostly depend. The decisions pertaining to the Beauty class correlated with the right side of the face, while those regarding the Different Beauty class correlated more with the left side of the face. Interestingly, the fMRI-based study mentioned earlier in the Introduction [12] concluded that the cross-participant classification of the activity in the brain was higher on the right side of the fusiform face area (FFA). In another recent fMRI-based study, researchers found that the left dorsal lateral prefrontal cortex (dlPFC) strongly correlates with the Different Beauty class [62]. In the second part of the experiment, we investigated facial symmetry based on the golden ratio. The results show that a specific feature-based golden ratio is not necessarily an indicator of beauty. However, the overall golden ratio confirms a strong relationship between beauty and the golden ratio. In addition, the most interesting finding is that the right eye features a higher golden ratio than the left eye in both classes. Unlike previous approaches that classify beauty based on the golden ratio, we claim that our approach is a novel empirical approach that proves the correlation between beauty and the golden ratio without any previous assumption of beauty and is based on subjective pre-classified faces. These results could be affected by dataset quality, participant bias, and other factors such as education, environment, and personality characteristics of left-handed versus right-handed persons. A single source of face images of the same quality and poses should be considered in future studies. In addition, having different participants of different ethnicities and including additional metadata regarding participants in the study can produce more informative results. Reproducible results are available at https://github.com/waleed-aldhahi/LSLS/ (accessed on 20 December 2022).

6. Conclusions and Future Work

Existing psychological and biological studies explain the common features that contribute to beauty perception but fail to explain and quantify the reasons behind a person’s judgement of a specific face to be beautiful and why two persons could argue about whether full lips are beautiful. Despite considerable progress in estimating and predicting facial beauty, most studies have focused on predicting beauty without an adequate understanding of beauty. In this research, we propose a novel approach based on deep learning to address the question of beauty and how people process and perceive beauty. We evaluated state-of-the-art related research in psychology, neuroscience, biology, and engineering. To explain how the brain determines beauty, we proposed a subjective approach to teaching the machine to learn beauty using uncertainty-based ensemble machine voting. To obtain a quantifiable measure of beauty, we propose a novel algorithm and framework that addresses the limitations of the current explainable AI (XAI) and provides us the ability to understand every decision the machine makes and obtain the general learned patterns. In addition, we propose a novel approach to prove the relationship between beauty and facial symmetry based on the golden ratio. In future work, we will develop an additional dataset with male faces, conduct a larger experiment involving more participants of both genders and diverse ethnicities and ages, and apply our approach to obtain a deeper and general conclusion of beauty decision patterns. In deep learning, the network weights are updated and accumulated by each input; analogically, accumulated human experiences render beauty decisions highly subjective and distinct, even for the same person. Thus, there might be no ultimate truth or absolute correctness of the essence of beauty.