Integrated Multi-Model Face Shape and Eye Attributes Identification for Hair Style and Eyelashes Recommendation

Abstract

Identifying human face shape and eye attributes is the first and most vital process before applying for the right hairstyle and eyelashes extension. The aim of this research work includes the development of a decision support program to constitute an aid system that analyses eye and face features automatically based on the image taken from a user. The system suggests a suitable recommendation of eyelashes type and hairstyle based on the automatic reported users’ eye and face features. To achieve the aim, we develop a multi-model system comprising three separate models; each model targeted a different task, including; face shape classification, eye attribute identification and gender detection model. Face shape classification system has been designed based on the development of a hybrid framework of handcrafting and learned feature. Eye attributes have been identified by exploiting the geometrical eye measurements using the detected eye landmarks. Gender identification system has been realised and designed by implementing a deep learning-based approach. The outputs of three developed models are merged to design a decision support system for haircut and eyelash extension recommendation. The obtained detection results demonstrate that the proposed method effectively identifies the face shape and eye attributes. Developing such computer-aided systems is suitable and beneficial for the user and would be beneficial to the beauty industrial.

1. Introduction

The human face plays an important role in daily life. Pursuing beauty, especially facial beauty, is the nature of human beings. As the demand for aesthetic surgery has increased widely over the past few years, an understanding of beauty is becoming increasingly important for medical settings. The physical beauty of faces affects many social outcomes, such as mate choices and hiring decisions [1,2]. Thus, the cosmetic industry has produced various products that target to enhance different parts of the human body, including hair, skin, eye, eyebrow, and lips. Not surprisingly, research topics based on the face features have a long track record in psychology, and many other scientific fields [3]. During recent decades, computer vision systems have played a major role in obtaining an image from a camera to process and analyse it in a manner similar to natural human vision system [4]. Computer vision algorithms have recently attracted increasing attention and been considered one of hottest topics due to its significant role in healthcare, industrial and commercial applications [4,5,6,7,8,9]. Facial image processing and analysis are essential techniques that help extract information from images of human faces. The extracted information such as locations of facial features such as eyes, nose, eyebrows, mouth, and lips, can play a major role in several fields, such as medical purposes, security purposes, cosmetic industry, social media applications, and recognition [5]. Several techniques have been developed to localise these parts and extract them for analysis [10]. The landmarks used in computational face analysis often resemble the anatomical soft tissue landmarks that are used by physicians [11]. Recently, advanced technologies such as artificial intelligence and machine/deep learning algorithms have helped the beauty industry in several ways, from providing statistical bases for attractiveness and helping people alert their looks to developing products which would tackle specific needs of customers [12,13]. Furthermore, cloud computing facilities and data centre services have gained a lot of attention due to their significant role for customers’ access to such products by building web-based and mobile applications [14].

In the literature, there have been many facial attribute analysis methods presented to recognise whether a specific facial attribute is present in a given image. The main aim of developing facial attribute analysis methods was to build a bridge between feature representations required by real-world computer vision tasks and human-understandable visual descriptions [15,16]. Deep learning-based facial attribute analysis methods can generally be grouped into two categories: holistic methods which exploit the relationships among attributes to get more discriminative cues [17,18,19] and part-based methods that emphasise facial details for detecting localisation features [20,21]. Unlike the existing facial attribute analysis methods, which focus on recognising whether a specific facial attribute is present in a given face image or not, our proposed method suggests that all concerned attributes are present but in more than one label.

Furthermore, many automated face shape classification systems were presented in the literature. Many of these published face classification methods consider extracting the face features manually then passing them to three classifiers for classification, including linear discriminant analysis (LDA), artificial neural networks (ANN), and support vector machines (SVM) [22,23], k-nearest neighbours [24], and probabilistic neural networks [25]. Furthermore, Bansode et al. [26] proposed a face shape identification method based on three criteria which are region similarity, correlation coefficient and fractal dimensions. Recently, Pasupa et al. [27] presented a hybrid approach combining VGG convolutional neural network (CNN) with SVM for face shape classification. Moreover, Emmanuel [28] adopted pretrained Inception CNN for classifying face shape using features extracted automatically by CNN. The work presented by researchers showed progress in face shape interpretation; however, existing face classification systems require more effort to achieve better performance. The aforementioned methods perform well only on images taken from subjects looking straight towards the camera and their body in a controlled position and acquired under a clear light setting.

Recently, many approaches were developed for building fashion recommender systems [29,30]. Conducting a deep search in the literature seeking the existing recommendation systems leads to find out two hairstyle recommendation systems [27,31]. The system developed by Liang and Wang [31] considers many attributes such as age, gender, skin colour, occupation and customer rating for recommending hairstyle. However, the recommended haircut style might not fit the beauty experts’ recommendations based on face shape attributes. Furthermore, the hairstyle recommender system presented by [27] is not general and focuses only on women. To the best of our knowledge, our proposed eyelashes and hairstyle recommendation system is the first study conducted to automatically make a recommendation of a suitable eyelash type based on computer vision techniques.

Facial attributes such as eye features, including shape, size, position, setting, face shape, and contour determine what makeup technique, eyelashes extension and haircut styles should be applied. Therefore, it is important for beautician to know clients’ face and eye features before stepping into the makeup-wearing, haircut style and eyelashes extension applying. The face and eye features are essential for beauty expert because different types of face shape and eye features are critical information to decide what kind of eye shadows, eyeliners, eyelashes extension, haircut style and colour of cosmetics are best suited to a particular individual. Thus, automation of facial attribute analysis tasks based on developing computer-based models for cosmetic purposes would help to easing people’s life and reducing time and effort spent by beauty experts. Furthermore, virtual consultation and recommendation systems based on facial and eye attributes [27,31,32,33,34,35,36] have secured a foothold in the market and individuals are opening up to the likelihood of substituting a visit to a physical facility with an online option or use a specific-task software. Use of the innovation of virtual recommendation systems has numerous advantages; including accessibility, enhanced privacy and communication, cost saving, and comfort. Hence, our work’s motivation is to automate the identification of eye attributes and face shape and subsequently produce a recommendation to the user for the appropriate eyelashes and suitable hairstyle.

This research work includes the development of system with friendly graphical user interface that can analyse eye and face attributes from an input image. Based on the detected eye and face features, the system suggests a suitable recommendation of eyelashes and hairstyles for both men and women. The proposed framework integrates three main models: face shape classification model, gender prediction model for predicting the gender of user, and eye attribute identification model to make a decision on the input image. Machine and deep learning approaches with various facial image analysis strategies including facial landmark detection, facial alignment, and geometry measurements schemes are designed to establish and realise the developed system. The main contributions and advantages of this work are summarised as follows:

• We introduce a new framework merging two types of virtual recommendation, including hairstyle and eyelashes. The overall framework is novel and promising.
• The proposed method is able to extract many complex facial features, including eye attributes and face shape, accomplishing the extraction of features and detection simultaneously.
• The developed system could help the worker in beauty centres and reduce their workload by automating their manual work and can produce subjective results, particularly with a large dataset.
• Our user-friendly interface system has been evaluated on a dataset provided with a diversity of lighting, age, and ethnicity.
• The labelling of a publicly available dataset with three eye attributes, five face shape classes, and gender has been conducted by a beauty expert to run our experiments.
• In the face shape classification method, we developed a method based on merging the handcrafted features with deep learning features.
• We presented a new geometrical measurement method to determine the eye features, including eye shape, position, and setting, based on the coordinates of twelve detected eye landmarks.

The remainder of this paper is presented as follows: in Section 2, the materials and proposed framework are described and explained; results of the proposed system are revealed and discussed in Section 3, and finally, the work has been concluded in Section 4.

2. Development of Hairstyle and Eyelash Recommendation System

2.1. Data Preparation: Facial and Eye Attributes

A publicly available dataset, MUCT [37], is used to conduct our experiments and carry out the evaluation on the developed face shape and eye attribute identification system. The MUCT database consists of 3755 faces captured from 276 subjects with 76 manual landmarks. The database was created to provide more diversity of lighting, age and gender with an obstacle (some subjects wear glasses) than other available landmarked 2D face databases. The resolution of images is $480\times 640$. Five webcams were used to photograph each subject. The five webcams are located in five different positions but not on the left of the subject. Each subject was photographed with two or three lighting setting producing ten different lighting setups in all five webcams. To achieve diversity without too many images, not every subject was photographed with every lighting setup.

The MUCT dataset does not provide face shape and eye attribute labels. Therefore, we recruited a beauty expert to provide the ground truth by labelling the images in the dataset. To formulate our research problem, we initially set up the attributes required to be identified. Face shape and eye specifications/attributes required to be detected in this research work are defined as follow:

• Eye Shape: Two attributes (Almond or Round),
• Eye Setting: Three attributes (Wide, Close, or Proportional),
• Eye Position/Pitch: Three attributes (Upturned, Downturned, or Straight),
• Face Shape: Five attributes (Round, Oval, Square, Rectangle (Oblong), Heart Shape).

Identifying these attributes would help automatically report eyelashes and haircut style recommendations that meet the eye and face attributes. Illustrations of these attributes are shown in Figure 1 and Figure 2.

The process of determining the facial shape and eye attributes manually by beauty experts can be carried out in several stages, such as outlining the face, measuring the width and length of the face, jaw, forehead, cheekbones, the gap between eyes, and outer and inner corner of the eye. According to the beauty experts, the face shape can be recognised as five categories, namely: oval, square, round, oblong, and heart shapes [41]. The guidelines provided by beauty experts Derrick and Brooke [41,42] have been followed to label the face shape of 276 subjects into five classes. Furthermore, to provide the ground truth of eye attributes, the detailed guideline and described criteria presented in [43] has been followed by a beauty expert. The distribution of data labelled manually is depicted by exploratory data analysis in Figure 3 and Figure 4 describing the distribution of attributes over 276 subjects and per one camera, respectively.

2.2. Developed Multi-Model System

To develop a personalised and automatic approach to finding the appropriate haircut and/or eyelash style for men and women, we develop a multi-model system comprising three separate models, as shown in Figure 5. Each individual model is dedicated to achieving a specific task, including face shape identification, eye attribute identification, and gender identification. The diagram shows that the eye features extracted from an input image by the developed eye feature identification model are merged with identified face shape class. The gender of the individual in the input image is detected by the gender identification model. Thus, the face and eye attributes are combined with identified gender information. If the gender in the input image is a male, then a hairstyle recommendation for men is given; otherwise, both hairstyle and eyelashes recommendations are reported. The description of the three models are explained as follows:

2.2.1. Face Shape Identification Model

To identify the face shape, our developed model [44] that was designed by merging hand-crafted features with automatically learned features was trained and tested on data from MUCT. MUCT data has been randomly split into 3255 images for training, and 500 images was retained for testing. The developed model (shown in Figure 6) achieves the classification as follows: (1) detect the face region and crop it using a model trained on the histogram of oriented gradients (HOG) features with Support Vector Machine (SVM) model as classifier [45], (2) the detected face is aligned using the detected face landmarks (68 landmarks) by the ensemble of regression tree method (ERT) [46], and (3) Finally, the aligned images are used for training and evaluating Inception V3 convolutional neural network [47] along with hand-engineered HOG features and landmarks to classify the face into one of five classes. For further details about the face shape identification model, please refer to our developed system in [44].

2.2.2. Eye Attributes Identification

The geometry of face/eye measurements based on the coordinates of detected eye landmarks are determined to develop this model. Thus, the eye specifications (shape, setting, position) are obtained. First, the face region is detected, and the landmarks of the eye region are localised using the methods presented in our face shape classification system [44,45,46]. We further customised the landmark predictor to return just the locations of the eyes using dlib library. Only six landmarks per eye—12 in two eyes—are taken into consideration for predicting eye attributes, as shown in Figure 7. Given 12 eye landmark points $P_n=P_0,P_1,P_2,...,P_n$ where $n=0,1,2,...,11$, the eye specifications can be calculated using geometrical eye measurements as follows:

• Eye Setting: The setting of eye $Se$ can be determined by calculating the ratio of the distance between two eyes $L_1$ to the eye width $W_{eye}$. The width of the eye $W_{eye}$ can be found by measuring the distance from the start and the end of an individual eye. The distance between two facial landmark points d can be obtained using the Euclidean distance.

  $$d(X,Y)=\sqrt{{(x_2-x_1)}^2+{(y_2-y_1)}^2}$$

  (1)

  $$\begin{array}{c}\hfill W_{eye1}=d(P_0,P_3)\\ \hfill W_{eye2}=d(P_6,P_9)\end{array}$$

  (2)

  $$L_2=\frac{W_{eye1}+W_{eye2}}{2}$$

  (3)

  $$L_1=d(P_3,P_6)$$

  (4)

  $$Se=\frac{L_1}{L_2}$$

  (5)

  The threshold values for category classification are empirically chosen and estimated as follow:

  $$Classofeyesetting=\left\{\begin{array}{cc}Close,\hfill & \mathrm{if}Se<1.57\hfill \\ Proportional,\hfill & \mathrm{if}1.57\le Se\le 1.6\hfill \\ Wide,\hfill & \mathrm{Otherwise}\hfill \end{array}\right.$$

  (6)

  From Equation ( 6), the threshold value used to determine (Close) label is calculated by finding the median of subjects’ eye setting measurements. Thus, the eye setting value of a subject that is less than the median is labelled as (Close). Likewise, the middle number between the median and the highest number (the maximum) is harnessed to differentiate between (Proportional) and (Wide) labels. For example, the eye setting value of a subject that is greater than the median and less than the middle number is labelled as (Proportional); otherwise, it is labelled as (Wide).
• Eye Shape: The criterion for discriminating round eye from almond eyes is that, for round eyes, the white sclera area below the iris is visible [43] as illustrated in Figure 8. Our proposed criteria exploits eye-aspect-ratio rule to determine the eye shape as follows:

  $$Sh=\frac{d(P_1,P_5)+d(P_2,P_4)}{2d(P_0,P_3)}$$

  (7)

  The threshold values for category classification are empirically chosen and estimated as follow:

  $$Classofeyeshape=\left\{\begin{array}{cc}Almond,\hfill & \mathrm{if}Sh\le 0.28\hfill \\ Round,\hfill & \mathrm{Otherwise}\hfill \end{array}\right.$$

  (8)

  From Equation (8), the threshold value used to differentiate between (Almond) and (Round) labels is calculated by finding the median of subjects’ eye shape measurements. Thus, the eye shape value that is less than the median is labelled as (Almond); otherwise, it is labelled as (Round).
• Eye Position: Eye position categorised into up-turned, down-turned and straight can be determined using geometrical calculations by measuring the slope angle (degree). The slope angle is defined as the angle measured between a horizontal line and a slanted line (the line drawn between landmark points ($P_3$, $P_6$) and ($P_0$, $P_3$) for $ey{e}_1$, ($P_6$, $P_9$) for $ey{e}_2$). The equation of eye position identification can be defined as follows:

  $$Po=\frac{{\theta }_{eye1}+{\theta }_{eye2}}{2}$$

  (9)

  where $\theta$ represents the angle between the line drawn between landmark points ($P_3$, $P_6$) and ($P_0$, $P_3$) for $ey{e}_1$, ($P_6$, $P_9$) for $ey{e}_2$ and each point is represented as $(x,y)$.

  $${\theta }_{eye}=ta{n}^{-1}\frac{m_1-m_2}{1+m_1{m}_2}$$

  (10)

  where $m_1$ and $m_2$ are the slope of each line formulated to calculate the angle $\theta$ which is defined as follows:

  $$m=\frac{y_2-y_1}{x_2-x_1}$$

  (11)
  The threshold values for category classification are empirically chosen and estimated as follow:

  $$Classofeyeposition=\left\{\begin{array}{cc}Up-turned,\hfill & \mathrm{if}Po>6.9\hfill \\ Straight,\hfill & \mathrm{if}0\le Po\le 6.9\hfill \\ Down-turned,\hfill & \mathrm{Otherwise}\hfill \end{array}\right.$$

  (12)
  From Equation (12), the threshold value used to determine (Up-turned) label is calculated by finding the median of subjects’ eye position measurements. Thus, the eye position value that is greater than the median is labelled as (Up-turned). Likewise, the value between zero and the median is labelled as (Straight). Any value of less than zero (negative) is identified as (Down-turned).

2.2.3. Gender Identification

Gender identification is a pre-requisite step in the our developed framework. If the gender in the input image is male then the recommendation engine reports only the suitable hairstyle for men, otherwise; the system recommends for hairstyle and eyelash extension for women. The deep learning approach developed in [48] has been adopted to achieve this task. Convolutional neural network composed of three convolutional layers, each followed by a rectified linear operation and pooling layer is implemented. The first two layers also follow the normalisation using local response normalisation. The first convolutional layer contains 96 filters of $7\times 7$ pixels, the second convolutional layer contains 256 filters of $5\times 5$ pixels, the third and final convolutional layer contains 384 filters of $3\times 3$ pixels. Finally, two fully connected layers are added, each containing 512 neurons.

The dataset of the Adience Benchmark [48] is used to train the gender identification model. The Adience Benchmark is a collection of unfiltered face images collected from Flickr. It contains 26,580 images of 2284 unique subjects that are unconstrained, extreme blur (low-resolution), with occlusions, out-of-plane pose variations, and have different expressions. Images are first rescaled to $256\times 256$, and a crop of $227\times 227$ is fed to the network. All MUCT images composed of 52.5% female were used for testing.

2.2.4. Rule-Based Hairstyle and Eyelash Recommendation System

The right shape of artificial eyelash extensions can be used as a way to correct and balance facial symmetry and open the eyes wider. Eyelash Extensions can produce an illusion of various eye effects giving an appearance of makeup on the eye without applying it. It also can create an appearance of a lifted eye, or even a more feline look being produced [49]. For the best hairstyle, instead of choosing the latest fashion, it should be attempted instead to pick a style that fits face shape. The correct haircut that fits the face shape will frame and balance it expertly while showing the best features for a flattering and complementary appearance [50,51].

The guidelines from specialised websites [49,50,51] are followed to implement the rules of the recommendation engine for hairstyle and eyelashes in our proposed system. The rule-based inference engine system focuses on knowledge representation and applies rules to obtain new information by existing knowledge. When the data (eye attributes, face shape, gender) matches rule conditions, the inference engine can interpret the facts in the knowledge base by applying logical rules and deduce the suitable recommendation using if-then operations. For instance,

IF (Gender is Male) AND (Face Shape is Round) THEN Haircut Recommendation is Side part, French crop, pompadour.

Figure 9 and Figure 10 present the graphical user interface (GUI) of the developed system showing an input image (man and woman) with the automatically detected landmarks, automatically extracted facial/eye attributes and the recommended haircut and eyelash extension.

3. Results and Discussions

A new approach based on merging three models in one framework has been proposed for the simultaneous detection of the face shape, eye attributes, and gender from webcam images. The designed framework achieves the detection by extracting hand-crafted features, learning features automatically, and exploiting the face geometry measurements. To evaluate the performance of the proposed system, different measurements, including accuracy (Acc.), sensitivity (Sn.), specificity (Sp.), Matthews Correlation Coefficient (MCC), Precision (Pr), and $F_1$ score, have been calculated. These metrics are defined as follows:

$$Accuracy\left(Acc\right)=\frac{tp+tn}{tp+tn+fp+fn}$$

(13)

$$Specificity\left(Sp\right)=\frac{tn}{tn+fp}\times 100$$

(14)

$$Sensitivity\left(Sn\right)=\frac{tp}{tp+fn}\times 100$$

(15)

$$MatthewsCorrelationCoefficient\left(MCC\right)=\frac{tp\times tn-fp\times fn}{\sqrt{(tp+fp)(tp+fn)(tn+fp)(tn+fn)}}$$

(16)

$$F_{1}Score=\frac{2(Precision\times Recall)}{Precision+Recall}$$

(17)

where $Precision=\frac{tp}{tp+fp}$, $Recall$ = $Sensitivity$, $tp$: True Positive, $tn$: True Negative, $fp$: False Positive, $fn$: False Negative

Table 1. Confusion matrix representing the five classes of face shape.

Predicted
	Oval	Square	Rectangle	Round	Heart
Oval	59	11	16	9	5
Square	0	99	0	1	0
Rectangle	2	0	92	1	5
Round	1	1	4	91	3
Heart	6	0	7	0	87

depicts the confusion matrix of face shape classes, reporting an identification accuracy of 85.6% on 500 test images (100 image per each class). To compare our face shape classification model with the existing face shape classification methods,

Table 2. The performance of proposed face shape classification methodology compared to the existing methods in the literature.

Method	Accuracy
Inception V3 [28]	84.4%
Region Similarity, Correlation and Fractal Dimensions [26]	80%
Active Appearance Model (AAM), segmentation, and SVM [22]	72%
Hybrid approach VGG and SVM [27]	70.3%
3D face data and SVM [23]	73.68%
Geometric features [24]	80%
Probability Neural Network and Invariant Moments [25]	80%
Our model	85.6%

presents a comparison in terms of classification accuracy. It can be noticed that our developed face shape classification system [44] outperforms the other methods in the literature.

Initially, images belong to one subject were excluded from data because of failure detecting the eye landmarks of this subject in the eye attributes identification system due to extreme eye closing and obstacle. Thus, the number of subjects that will be considered for evaluation is 275. For the gender identification, the gender detection model has achieved prediction performance of $0.9018$, $0.8923$, $0.9103$, $0.8992$, and $0.8958$ in terms of accuracy, sensitivity, specificity, precision, and $f_1$ score measures; respectively, on 275 subjects. Likewise, the gender detection model achieved $0.9064$, $0.8989$, $0.9136$, $0.9088$, and $0.9038$; respectively, using the same metrics in terms of 748 image per camera.

Table 3. Confusion matrix representing the gender identification. The first part of matrix represents the identification in terms of number of subjects; whereas the second part explores the prediction in terms of images per one camera. F: Female, M: Male.

Predicted
	F	M	F	M
F	132	13	349	33
M	14	116	37	329

presents the confusion matrix obtained from gender identification system. The first part of Table 3 shows the gender prediction in terms of the number of subjects, whereas the second part explores the prediction in terms of images per camera. In terms of comparison with the existing methods, authors in [48] achieved an accuracy of $0.868$ on the Adience dataset [48]. On the same dataset, Duan et al. [52] reported accuracy of $0.778$ using convolutional neural network (CNN) and extreme learning machine (ELM) method. On Pascal VOC [53] dataset, De et al. [54] obtained accuracy of $0.8739$ based on a deep learning approach.

There are five webcams used to photograph each subject. In the following experiments, we name the webcams with five letters (A–E) including, Cam A, Cam B, Cam C, Cam D, and Cam E. Symbols X and Y (used in Cam X vs. Cam Y) refer to any individual webcam from the five cameras. Our developed system has provided flavour results on eye attributes identification.

Table 4. Performance evaluation of proposed system for eye setting attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, $F_1$: $F_1$ score, MCC: Matthews Correlation Coefficient, GT: Ground Truth, Cam: Camera.

Webcam Setting	Acc.	Sn.	Sp.	Pr.	${\mathit{F}}_1$	MCC
Cam A vs. GT	0.7086	0.6900	0.8503	0.7236	0.7002	0.5563
Cam B vs. GT	0.6885	0.6694	0.8425	0.7083	0.6767	0.5305
Cam C vs. GT	0.7273	0.7129	0.8625	0.7540	0.7166	0.5958
Cam D vs. GT	0.6845	0.6655	0.8389	0.7024	0.6753	0.5218
Cam E vs. GT	0.7019	0.6849	0.8453	0.7150	0.6939	0.5459
Average (all Cams)	0.7021	0.6845	0.8479	0.7207	0.6926	0.5501
Cam A vs. B	0.6939	0.7009	0.8446	0.6871	0.6927	0.5381
Cam A vs. C	0.6979	0.7142	0.8473	0.6932	0.7003	0.5500
Cam A vs. D	0.7019	0.7033	0.8477	0.6974	0.7001	0.5479
Cam A vs. E	0.7166	0.7192	0.8559	0.7195	0.7187	0.5740
Cam B vs. C	0.6885	0.6892	0.8438	0.6843	0.6862	0.5300
Cam B vs. D	0.7112	0.7037	0.8538	0.7104	0.7066	0.5611
Cam B vs. E	0.6791	0.6761	0.8376	0.6909	0.6819	0.5200
Cam C vs. D	0.6751	0.6691	0.8356	0.6811	0.6732	0.5112
Cam C vs. E	0.6872	0.6840	0.8418	0.7038	0.6910	0.5349
Cam D vs. E	0.6832	0.6845	0.8387	0.6914	0.6871	0.5255

,

Table A1. Confusion matrices depict eye setting attributes prediction in each webcam versus the ground truth.

Predicted		Desired
		Close	Average	Wide
Cam A	Close	111	72	1
	Average	23	162	61
	Wide	3	58	257
Cam B	Close	112	94	3
	Average	19	152	65
	Wide	6	46	251
Cam C	Close	122	95	7
	Average	13	168	58
	Wide	2	29	254
Cam D	Close	110	82	2
	Average	21	152	67
	Wide	6	58	250
Cam E	Close	108	75	3
	Average	25	171	70
	Wide	4	46	246
All Cameras	Close	563	418	16
	Average	101	805	321
	Wide	21	237	1258

and

Table A2. Confusion matrices depict eye setting attributes prediction in each individual camera versus another camera.

Cam X vs. Cam Y		Cam B
		Close	Average	Wide
Cam A	Close	154	29	1
	Average	53	128	65
	Wide	2	79	237
		Cam C
Cam A	Close	165	19	0
	Average	50	134	62
	Wide	9	86	223
		Cam D
Cam A	Close	147	37	0
	Average	44	133	69
	Wide	3	70	245
		Cam E
Cam A	Close	145	38	1
	Average	38	152	56
	Wide	3	76	239
		Cam C
Cam B	Close	169	40	0
	Average	47	125	64
	Wide	8	74	221
		Cam D
Cam B	Close	156	49	4
	Average	38	132	66
	Wide	0	59	244
		Cam E
Cam B	Close	153	54	2
	Average	31	133	72
	Wide	2	79	222
		Cam D
Cam C	Close	160	57	7
	Average	33	122	84
	Wide	1	61	223
		Cam E
Cam C	Close	161	54	9
	Average	25	140	74
	Wide	0	72	213
		Cam E
Cam D	Close	148	44	2
	Average	35	137	68
	Wide	3	85	226

in Appendix A report eye setting attributes detection performance using various evaluation metrics and state the obtained results from the proposed system by comparing the eye setting attribute prediction performance in one camera with ground truth using confusion matrices, and compare the prediction performance in each camera with another camera using confusion matrices, respectively. Likewise,

Table 5. Performance evaluation of proposed system for eye position attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, $F_1$: $F_1$ score, MCC: Matthews Correlation Coefficient, GT: Ground Truth.

Webcam Setting	Acc.	Sn.	Sp.	Pr.	${\mathit{F}}_1$	MCC
Cam A vs. GT	0.7005	0.7172	0.8453	0.7072	0.7093	0.5574
Cam B vs. GT	0.6778	0.7025	0.8327	0.6808	0.6867	0.5235
Cam C vs. GT	0.6631	0.6809	0.8261	0.6704	0.6724	0.5015
Cam D vs. GT	0.6898	0.7089	0.8395	0.6957	0.6986	0.5412
Cam E vs. GT	0.7032	0.7281	0.8456	0.7044	0.7121	0.5614
Average (all Cams)	0.6869	0.7075	0.8378	0.6917	0.6958	0.5370
Cam A vs. B	0.6123	0.6169	0.7961	0.6308	0.6230	0.4196
Cam A vs. C	0.6203	0.6312	0.8020	0.6323	0.6318	0.4338
Cam A vs. D	0.5802	0.5881	0.7804	0.5950	0.5913	0.3716
Cam A vs. E	0.6297	0.6363	0.8053	0.6474	0.6412	0.4469
Cam B vs. C	0.5668	0.5829	0.7716	0.5707	0.5760	0.3489
Cam B vs. D	0.5655	0.5780	0.7693	0.5715	0.5746	0.3443
Cam B vs. E	0.6003	0.6099	0.7869	0.6074	0.6086	0.3956
Cam C vs. D	0.5548	0.5638	0.7670	0.5698	0.5666	0.3335
Cam C vs. E	0.5922	0.5993	0.7859	0.6090	0.6035	0.3896
Cam D vs. E	0.5508	0.5612	0.7621	0.5651	0.5630	0.3250

,

Table A3. Confusion matrices depict eye position attributes prediction in each webcam versus the ground truth.

Predicted		Desired
		Downturned	Straight	Upturned
Cam A	Downturned	161	78	1
	Straight	52	192	65
	Upturned	0	28	171
Cam B	Downturned	156	70	0
	Straight	56	198	84
	Upturned	1	30	153
Cam C	Downturned	155	85	2
	Straight	56	180	74
	Upturned	2	33	161
Cam D	Downturned	165	68	0
	Straight	48	195	81
	Upturned	0	35	156
Cam E	Downturned	149	75	0
	Straight	64	205	65
	Upturned	0	18	172
All Cameras	Downturned	786	376	3
	Straight	276	970	369
	Upturned	3	144	813

and

Table A4. Confusion matrices depict eye position attributes prediction in each individual camera versus another camera.

Cam X vs. Cam Y		Cam B
		Downturned	Straight	Upturned
Cam A	Downturned	152	87	1
	Straight	74	179	56
	Upturned	0	72	127
		Cam C
Cam A	Downturned	158	80	2
	Straight	83	169	57
	Upturned	1	61	137
		Cam D
Cam A	Downturned	152	87	1
	Straight	81	160	68
	Upturned	0	77	122
		Cam E
Cam A	Downturned	152	88	0
	Straight	72	183	54
	Upturned	0	63	136
		Cam C
Cam B	Downturned	148	76	2
	Straight	92	164	82
	Upturned	2	70	112
		Cam D
Cam B	Downturned	145	81	0
	Straight	87	169	82
	Upturned	1	74	109
		Cam E
Cam B	Downturned	146	80	0
	Straight	77	187	74
	Upturned	1	67	116
		Cam D
Cam C	Downturned	145	96	1
	Straight	86	152	72
	Upturned	2	76	118
		Cam E
Cam C	Downturned	146	94	2
	Straight	77	171	62
	Upturned	1	69	126
		Cam E
Cam D	Downturned	135	98	0
	Straight	89	161	74
	Upturned	0	75	116

report eye position classes detection performance using various evaluation metrics and state the obtained results from the proposed system by comparing the eye position classes prediction performance in one camera with ground truth using confusion matrices, and comparing the prediction performance in each camera with another camera using confusion matrices, respectively. In the same way,

Table 6. Performance evaluation of proposed system for eye shape attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, $F_1$: $F_1$ score, MCC: Matthews Correlation Coefficient, GT: Ground Truth.

Webcam Setting	Acc.	Sn.	Sp.	Pr.	${\mathit{F}}_1$	MCC
Cam A vs. GT	0.7540	0.7111	0.7938	0.7619	0.7356	0.5072
Cam B vs. GT	0.7580	0.7171	0.7954	0.7619	0.7388	0.5146
Cam C vs. GT	0.7447	0.7066	0.7783	0.7381	0.7220	0.4865
Cam D vs. GT	0.7540	0.7159	0.7879	0.7500	0.7326	0.5055
Cam E vs. GT	0.7567	0.7333	0.7751	0.7202	0.7267	0.5075
Average (all Cams)	0.7535	0.7168	0.7861	0.7464	0.7311	0.5043
Cam A vs. B	0.9452	0.9389	0.9510	0.9468	0.9428	0.8902
Cam A vs. C	0.9452	0.9306	0.9588	0.9544	0.9423	0.8904
Cam A vs. D	0.9572	0.9444	0.9691	0.9659	0.9551	0.9145
Cam A vs. E	0.9332	0.8889	0.9742	0.9697	0.9275	0.8685
Cam B vs. C	0.9278	0.9160	0.9386	0.9316	0.9237	0.8553
Cam B vs. D	0.9452	0.9356	0.9540	0.9489	0.9422	0.8902
Cam B vs. E	0.9291	0.8880	0.9668	0.9606	0.9229	0.8598
Cam C vs. D	0.9372	0.9345	0.9395	0.9318	0.9331	0.8739
Cam C vs. E	0.9265	0.8917	0.9572	0.9485	0.9192	0.8532
Cam D vs. E	0.9439	0.9091	0.9747	0.9697	0.9384	0.8885

,

Table A5. Confusion matrices depict eye shape attributes prediction in each webcam versus the ground truth.

Predicted		Desired
		Round	Almond
Cam A	Round	256	104
	Almond	80	308
Cam B	Round	256	101
	Almond	80	311
Cam C	Round	248	103
	Almond	88	309
Cam D	Round	252	100
	Almond	84	312
Cam E	Round	242	88
	Almond	94	324
All Cameras	Round	1254	496
	Almond	426	1564

and

Table A6. Confusion matrices depict eye shape attributes prediction in each individual camera versus another camera.

Cam X vs. Cam Y		Cam B
		Round	Almond
Cam A	Round	338	22
	Almond	19	369
		Cam C
Cam A	Round	335	25
	Almond	16	372
		Cam D
Cam A	Round	340	20
	Almond	12	376
		Cam E
Cam A	Round	320	40
	Almond	10	378
		Cam C
Cam B	Round	327	30
	Almond	24	367
		Cam D
Cam B	Round	334	23
	Almond	18	373
		Cam E
Cam B	Round	317	40
	Almond	13	378
		Cam D
Cam C	Round	328	23
	Almond	24	373
		Cam E
Cam C	Round	313	38
	Almond	17	380
		Cam E
Cam D	Round	320	32
	Almond	10	386

present eye shape labels prediction performance.

From the reported results of eye attributes, it can be noticed that the model achieves the highest performance on eye shape detection, reporting accuracy of 0.7535, sensitivity of 0.7168, specificity of 0.7861, precision of 0.7464, F1 score of 0.7311 and MCC of 0.5043 while the lowest results were obtained from eye position class identification giving 0.6869, 0.7075, 0.8378, 0.6917, 0.6958, and 0.5370 using the same evaluation metrics, respectively. Likewise, the model reveals the results of 0.7021, 0.6845, 0.8479, 0.7207, 0.6926, and 0.5501 on eye setting detection in terms of the same aforementioned metrics. Although the system shows many misclassified prediction of eye attributes, it could be considered to be tolerated prediction cases (skew is small). By “tolerated prediction”, we mean that most misclassified cases are resulted from predicting the class into average labels, which is clear in all the reported confusion matrices. For example, in Table A3, the major predictions of images in Cam E were misclassified as straight rather than upturned label giving: downturned label: 149, straight: 64, upturned: 0. This proves that the predicted measurements of attributes are closer to the ground truth value. This also can be clearly noticed in the reported confusion matrix of the eye setting shown in Table A1.

To study the correlation of prediction among different webcams, all the evaluation metrics mentioned earlier, as well as confusion matrices, have been utilised to depict the variance and closeness of prediction. For instance, Table 4 presents the correlation between cameras (such as Cam A vs. Cam B and so on) in terms of evaluation metrics. Whilst, Table A2 reports the confusion matrices explaining the correlation of prediction among different webcams, which shows good prediction agreement among images captured from different cameras’ positions. Matthews Correlation Coefficient (MCC) is a correlation measurement that returns a value between +1 and −1, where −1 value refers to a total disagreement between two assessments and +1 represents the perfect detection (agreement). The range of obtained MCC value (between two webcams) was (0.3250–0.9145), which indicates of strong correlation of prediction among various webcam positions.

In comparison with the existing methods considering eye attributes detection, Borza et al. [55] and Aldibaja [56] presented eye shape estimation method. Yet, they have not reported the accuracy of the shape detection and instead reported eye region segmentation performance. To the best of our knowledge, our proposed method for eye attributes detection seeking the suitable eyelashes recommendation is the first work carried out automatically based on computer vision approaches. Thus, we could not find more related works in the literature to conduct further comparison with the results obtained from our framework. To further investigate our model’s performance, we validated the developed framework on external data represented by sample images of celebrities, as shown in

Table 7. Performance evaluation of proposed system on external image data. Red cells refer to incorrect detection while last column shows the true label.

Celebrity ID	Face Shape	Eye Shape	Eye Setting	Eye Position	Correction
Olivia Wilde	Square	Round	Wide	Upturned	Almond
Rihanna	Oval	Almond	Wide	Straight	Upturned
Cara Delevingne	Oval	Almond	Wide	Straight	Upturned
George Clooney	Oval	Almond	Wide	Downturned	Round
Adele	Round	Almond	Wide	Downturned	Straight
Scarlett Johansson	Round	Almond	Wide	Straight	Heart
Johnny Depp	Rectangle	Almond	Average	Downturned	Square
Chloe Grace Moretz	Heart	Almond	Wide	Upturned	Average
Camilla Belle	Round	Almond	Wide	Downturned	Average
Will Smith	Oval	Almond	Wide	Straight	-
Hilary Swank	Rectangle	Almond	Average	Straight	Wide
Sarah Jessica Parker	Rectangle	Almond	Wide	Straight	-
Selena Gomez	Round	Round	Wide	Straight	-
Naomi Campbell	Round	Round	Average	Upturned	Heart
Meryl Streep	Rectangle	Round	Average	Straight	Close

. Thus, we have demonstrated that the system generalises well on another data and performs well under variant conditions such as camera position, obstacle (wearing glasses), light condition, gender and age. Yet, on the other hand, it should be considered that the proposed framework brings up some limitations, including: (1) the developed system has not been evaluated on diverse ethnic and racial groups, (2) the presented research work has not considered eye situation characteristics analysis such as mono-lid, hooded, crease, deep-set, and prominent. However, these limitations are due to a lack of labelled data captured from variant races and ethnicities. These challenges can be overcome when larger annotated data becomes available. Moreover, the threshold parameter values used for converting scalar value into a nominal label of eye attribute are selected empirically. For future studies, studying the possibility of using automatic parameter tuning strategies (heuristic mechanisms) for determining and setting optimum threshold values could be investigated.

4. Conclusions

In this paper, a decision support system for appropriate eyelashes and hairstyle recommendation was developed. The developed framework was evaluated on dataset provided with diversity of lighting, age, and ethnicity. The developed system, based on integrating three models, has proven efficient performance in the identification of face shape and eye attributes. Face and eye attributes measurement tasks are usually carried out by an expert person manually before applying the treatment of individual eyelashes extension and hairstyle. Measuring face and eye characteristics costs time and effort. The proposed system could alleviate the need for additional time and effort made by expert before each treatment. Furthermore, face shape identification automatically is a challenging duty due to the complexity of face and the possible variations in rotation, size, illumination, hairstyle, age, and expressions. Moreover, the existence of face occlusion from hats and glasses also adds difficulties to the classification process. Our model trained on diverse images for automatic face shape classification would be viable in many other decision support systems such as eyeglasses style recommendation systems. In future work, we aim to extend our developed GUI desktop software and build a cloud-based decision support system using a mobile application. The extended system targets users’ facial and eye analysis via cloud server programs based on the images captured and uploaded from mobile phones to the server. Furthermore, the detected facial and eye attributes could be harnessed to develop a makeup recommendation system. Another important aspect that could be considered to be potential future work is studying the algorithm’s computational complexity to achieve the best and fastest convergence in the developed framework.