Facial Identity Verification Robust to Pose Variations and Low Image Resolution: Image Comparison Based on Anatomical Facial Landmarks

Abstract

Face comparison/face mapping is one of the promising methods in face biometrics which needs relatively little effort compared with face identification. Various factors may be used to verify whether two faces are of the same person, among which facial landmarks are one of the most objective indicators due to the same anatomical definition for every face. This study identified major landmarks from 2D and 3D facial images of the same Korean individuals and calculated the distance between the reciprocal landmarks of two images to examine their acceptable range for identifying an individual to obtain standard values from diverse facial angles and image resolutions. Given that reference images obtained in the real-world could be from various angles and resolutions, this study created a 3D face model from multiple 2D images of different angles, and oriented the 3D model to the angle of the reference image to calculate the distance between reciprocal landmarks. In addition, we used the super-resolution method of artificial intelligence to address the inaccurate assessments that low-quality videos can yield. A portion of the process was automated for speed and convenience of face analysis. We conclude that the results of this study could provide a standard for future studies regarding face-to-face analysis to determine if different images are of the same person.

1. Introduction

The face is one of the most important physical features of an individual that externally expresses identity via its own distinctive characteristics. Therefore, the face and face images are critical data in the identity verification process. Currently, with increasing utilization of security and surveillance systems, acquired images are commonly used in public safety and image forgery detection [1,2,3,4]. Therefore, numerous studies have attempted to develop methods to improve the accuracy of identity verification using facial images.

The methods for identity verification using facial images can be categorized into two major approaches. The first is called automated facial recognition (AFR), which uses computer algorithms to compare a database of facial images with the target facial image and identify the most similar database images. Another method is to compare and analyze facial images of the person side-by-side. The second approach is called facial comparison/mapping; this method can be relatively easy for people to use as compared with AFR, which requires a large image database and considerable development cost [5].

In a forensic context, landmark-based face comparison methods can be useful. A method that uses likelihood ratios (LRs) to assess the grouping of facial images based on the morphometric indices was presented in [6]. The landmark values, such as averages and SDs for the reference image, were estimated, and it calculated the probability of these values from the probability distributions for the suspect as well as the population to acquire the likelihood ratio. In addition, using the facial ratio from inter-landmark distances, it performed intra- and inter-sample comparisons using the mean absolute value, Euclidean distance and cosine distance between ratios [7]. The statistics of the two frontal-faces were examined to determine the calculation that best identifies a detectable correlation difference between faces belonging to the same person. In addition, a series of area triplets was constructed for geometric invariance including the area ratio and angle, then used as feature vectors in face identification by the detected landmarks [8].

Some more specific ways of verifying identity are photoanthropometry, morphological analysis, and facial superimposition [9,10,11,12]. When using these methods, many standards can be used in the analysis, such as facial landmarks, the distance between landmarks, the ratio and angle of the landmarks, and distinctive facial structures [13,14,15]. Facial landmarks have the same anatomical definition for every face. Therefore, such landmarks could be used as an objective standard if one could place reciprocal landmarks on test and comparison faces.

The human face has about 80 to 90 nodal points. Facial verification systems measure important aspects, such as the distance between the eyes, the width of the nose, and the length of the chin [16,17]. In addition, the reciprocal landmarks in the frontal or profile face are used directly or defined newly by extending the points [7,18,19,20]. As already demonstrated by many studies, deep facial features for face verification perform much better than the methods that use facial landmarks [18,21,22]. However, the facial landmarks are not only visually easier to explain than deep facial features, but also objective indicators that define the appearance of a face. Therefore, the face landmarks can be useful when forensic cases need to be discussed in court [23,24,25]. The face mapping/comparison is a task of finding out if two people are the same person, similar to inferring lookalike. Applications such as Google’s ‘Art & Culture’ app and Microsoft’s ‘CelebLike’ are famous for their ability to show the faces of celebrities who resemble the user. Finding a similar face is related to the face verification that selects one ID that is most likely to match the input in that it infers the face image that is most perceptually similar to the input [21,26,27]. Humans can identify two people as different identities, even if their faces are very similar [27]. To estimate the similarity between two face images, many studies have utilized the distance between low-level feature descriptors, such as SIFT (Scale-Invariant Feature Transform), facial shape, facial features, or facial texture [8,28,29,30,31]. Recent studies have explored improving facial verification performance by combining these traditional features and deep-learning technologies [32,33,34].

However, there can exist differences between identifiable features based on the image resolution or angle of the face. To handle the pose problem, a promising method is to use 3D facial information because the restored 3D face model is available to render target facial poses. It is an ill-posed problem to generate 3D information from a single image [35], therefore, prior-model based methods have been proposed, which require 3D scan data of a large number of people to reproduce various face shapes of the input [36,37,38]. In addition, a 3D geometry can be directly estimated from images using the SfM (Structure from Motion)—methods for estimating camera movements from image sequences with various viewpoints [39,40]. In addition, if there are several photos of the object illuminated in different directions, the depth can be estimated more accurately by combining this information. These photometric stereo techniques have been traditionally applied for 3D reconstruction [41,42]. Recently, deep-learning-based super-resolution studies have brought great progress in image quality improvement. There are many methods for super resolution that are based on deep-learning technology, such as super resolution using deep convolutional network (SRCNN), coupled deep convolutional auto-encoder (CDCA), and very deep convolutional network-based super resolution (VDSR) [43,44,45].

Theoretically, the distance between reciprocal landmarks would be close to zero if the facial pose of two images obtained from an individual were aligned perfectly. However, it is impossible for the calculated distance to be zero due to the bias that occurs when a professional analyst identifies landmarks or due to inaccuracies that occur when images of different resolutions are compared. Nevertheless, if one could minimize such potential errors and accurately locate landmarks on both images, then the distance between reciprocal landmarks would approach zero if the images being compared were of the same individual. Therefore, it might be possible to use the distance between reciprocal landmarks of images of the same individual as a standard for identity verification.

In general, large-scale datasets, such as MS-Celeb-1M [46] with 10 million face images, are required for face identification, and complex algorithms and deep-learning architectures need to be designed to process them. Our study has the novelty in that it provides effective landmarks for face identification by analyzing traditional facial landmarks that represent the structure of a face, as a way to determine whether it is the same person by comparing two input images. This can provide visual clues for identification in that it is simpler and contains more interpretable morphometric features than the deep-learning features. In addition, the images of suspects acquired from security cameras contain variations in facial pose and expression, low-image resolution, and occlusion of the face by hair or accessories. These factors can degrade the comparison performance of experts such as investigators [47,48].

This study aimed to resolve issues caused by different poses and image resolutions. First, to solve the problems caused by pose variation between reference and comparison images, we created a 3D face model of a subject via multiple images acquired from multiple cameras with various angles, then rotated the model pose to match that of the comparison image. Second, the comparison image obtained from CCTV (Closed-Circuit Television) generally has low resolution, which can degrade the performance of the face analyst who must manually locate facial landmarks. Thus, image enhancement could improve the accuracy of the location of landmarks. In this paper, a super-resolution method via deep learning is applied to solve this problem. In addition, to improve speed, convenience, and accuracy of face analysis, we utilized a machine learning-based facial landmark detector, which is widely used in computer vision. Further, we provide an index for identity verification based on facial landmarks and an associated threshold from which to determine whether different images have the same identity.

The contributions of this study can be summarized as follows:

• We present a landmark-based face mapping method which can represent the morphology of the face and easier provide visual and interpretable cues than deep features.
• We provide the landmark indices and associated thresholds by which to determine whether input images have the same identity.
• To cope with the images of low resolution and various poses, this study extracts more accurate facial landmarks from the input faces through restoring the 3D model to correct the poses and improving the image quality.

2. Materials and Methods

The overall flow of the identity verification method described in this paper, which uses the distance between reciprocal landmarks of a reference image and comparison image, is as follows.

First, to estimate a 3D geometry of the comparison face, we used photogrammetric range imaging technique from a sequence of facial images. The facial pose of the reconstructed 3D face model is rotated into the same pose as the comparison image obtained by security cameras such as CCTV. In addition, super-resolution technology is applied to the comparison image with low resolution to accurately extract facial landmarks. Next, the facial landmarks within each image are detected by an elastic model-based facial landmark detector; a professional facial analyst may adjust the detected landmarks for increased accuracy. Based on these facial landmarks, size normalization is performed to generate two images of same size and enable comparison of the locations of landmarks. The size of the facial image is adjusted by setting the interpupillary distance (IPD) to 100 pixels for a frontal presentation of the image. For other views, the distance between the nasion (midline depth of the nasal root) and gnathion (lowest median landmark on the lower border of the chin) are set to 100 pixels. Finally, we analyze the distribution of distances for the same individual and different individuals by calculating the distance between reciprocal landmarks of normalized facial images. The whole process is described in Figure 1.

To perform an identity verification experiment, we first constructed a Korean face dataset that consisted of over 2500 images (the K-Face DB [49]. https://github.com/k-face/k-face_2019, accessed on 21 December 2019). The dataset was collected from 50 Koreans (25 males and 25 females) who ranged in age from 20 to 50 years. By using a multiple-camera system with various angles and a light controller, over 50 images of each person were captured. Specifically, 40 images were collected as comparison images with 5 different facial angles, 2 different lighting conditions under 400 lux and 200 lux, and 4 different resolutions which include the ones higher than WQXGA (Wide Quad Extended Graphics Array) and WVGA (Wide Video Graphics Array) and lower than WVGA. The obtained image resolutions are 2592 × 1728, 864 × 576, 346 × 230, 173 × 115, respectively. Here, we assume the image with higher resolution than WQXGA and WVGA as high-resolution and mid-resolution image, respectively, and the rest as low-resolution image. Additionally, 12–14 images from each camera angle under uniform lighting were acquired to generate the 3D face models. The 3D face reconstruction process extracted 2D feature points from images, calculated their correspondences, extracted a mesh, and restored the facial textures based on the SfM technique [39]. We applied GPU (Graphics Processing Unit) acceleration such that the total execution time was less than one minute. Generally, there exists a problem of fixing the model’s scale and rotation coordinate system when restoring 3D data from photos. To address this issue, we post-processed all models to have physical dimensions and coordinate system based on measured distances and locations of some of the cameras of the multi-camera system. With these steps, we created a dataset that consisted of a 3D face model and 40 2D facial images per person, which we used to conduct the distance-calculation experiment for identity verification. Before collecting data, we informed the research participants about the general purpose of the research and allowed them to sign a consent form if they wished to participate. An example of a constructed dataset, which included obtained images and reconstructed 3D face models, is shown in Figure 2. The resolutions of images collected by real-world security cameras vary. If an image is low-definition, there are limitations in accurately determining facial landmarks. To address this issue, we utilized a VDSR network [45] which has simple and good performance by learning the mapping between low-resolution and high-resolution images based on residual learning. The overall structure is that it passes an input image through several convolutional layers and acquires an enhanced image as a result. To train the architecture, the CelebA and BERC databases were used [50,51]. Figure 3 shows an example of the result of super resolution based on deep learning.

Next, the facial landmarks are detected in the reference image and comparison image via the above steps. For the automatic landmark detection process, we utilized the wild-feature detector [52] for feature detection tasks. This assumes that the face shape is a tree structure, and uses a part-based model for face detection, pose estimation, and facial feature detection. In the current approach, the landmark detector extracts the facial region, facial pose angle, and facial landmarks concurrently, using 3D face structure and an elastic model. In particular, the facial landmarks considered in this paper are shown in

Table 1. Frontal pose facial landmark index numbers and definitions [2,7,20,53].

Landmark		Abbreviation	Definition
Index	Title
Midline
1	Gnathion	gn	The lowest median landmark on the lower border of the chin
2	Labiale inferius	li	The midpoint of the lower vermilion line
5	Labiale superius	ls	The midpoint of the upper vermilion line
6	Subnasale	sn	The point at which the columella merges with the upper lip in the midsagittal plane
9	Pronasale	prn	The most protruded point of the nasal tip
10	Nasion	n	The midline depth of the nasal root
Bilateral
3,4	Cheilion	ch	The point located at each labial commissure
7,8	Alare	al	The most lateral point of the alar contour
11,15	Endocanthion	en	The point at the inner commissure of the palpebral fissure
12,16	Exocanthion	ex	The point at the outer commissure of the palpebral fissure
13,17	Outer Eyebrow	oe	The point at the outer eyebrow
14,18	Inner Eyebrow	ie	The point at the inner eyebrow
19,21	Otobasion inferius	obi	The lowest point of attachment of the external ear to the head
20,22	Otobasion superius	obs	The highest point of attachment of the external ear to the head

and 

Table 2. Side pose facial landmark index numbers and definition [2,7,20,53].

Landmark		Abbreviation	Definition
Index	Title
Midline
1	Gnathion	gn	The lowest median landmark on the lower border of the chin
2	Labiale inferius	li	The midpoint of the lower vermilion line
3	Cheilion	ch	The point located at each labial commissure
4	Labiale superius	ls	The midpoint of the upper vermilion line
5	Subnasale	sn	The point at which the columella merges with the upper lip in the midsagittal plane
6	Pronasale	prn	The most protruded point of the nasal tip
7	Nasion	n	The midline depth of the nasal root
8	Exocanthion	ex	The point at the outer commissure of the palpebral fissure
9	Endocanthion	en	The point at the inner commissure of the palpebral fissure
10	Inner Eyebrow	ie	The point at the inner eyebrow
11	Outer Eyebrow	oe	The point at the outer eyebrow
12	Otobasion inferius	obi	The lowest point of attachment of the external ear to the head
13	Otobasion superius	obs	The highest point of attachment of the external ear to the head

. These landmarks were chosen with reference to previous studies of identity verification, and include 4 positions on eyebrows to utilize this part of the face in identity verification [2,7,20,53] as described in Figure 4.

To train the facial landmark detector, the Multi-PIE database was used [54]. The Multi-PIE database is a large-scale facial database that consists of different poses, illumination conditions, and facial expressions of 337 of a variety of races, with a male-to-female ratio of 7 to 3. Figure 5 is a sample from the Multi-PIE database.

After detecting the 22 facial landmarks and 13 facial landmarks for the frontal pose and side pose, respectively, professional facial analysts manually adjusted the detected facial landmarks. Before calculating the distance between reciprocal landmarks, image size normalization was performed. This procedure normalized the face sizes within a comparison pair of images with different face sizes. Different images of the same individual could differ in size because of the characteristics of the respective cameras, the distance between camera and subject, and so on. Here, we set the IPD to 100 pixels for frontal poses and the distance between the nasion and gnathion to 100 pixels for other poses. Figure 6 describes the examples of facial region size differences for two images of the same person and Figure 7 illustrates the example of size normalization, respectively.

3. Results

Using the materials and data described above, we experimentally assessed identity verification in 3 ways. First, we calculated the distance between each reciprocal landmarks. The nasion and gnathion were excluded when calculating the distances because they were used to normalize image size. Thus, 22 distances for frontal poses and 11 distances for side poses were calculated. Euclidean Distance (ED) was used as the distance measure, according to Equation (1).

$$Dist=\sqrt{\frac{{\sum }_{i=1}^n({(x_{in}-x_{jn})}^2+{(y_{in}-y_{jn})}^2)}{n}}$$

(1)

The dissimilarity of each landmark position was subject to pixel error. IPD is generally 63.36 mm [55]. Thus, a 1-pixel error was equivalent to 0.63 mm of distance because we set the IPD to 100 pixels. Accordingly, we analyzed the pixel error distribution for the same individual and for two different individuals to obtain standard values for identity verification. Before the experiment, we compared the average distance between reciprocal landmarks of each image resolution. Figure 8 shows the result of such comparison among high/mid-resolution, low-resolution, and enhanced low-resolution based on the super-resolution method. As mentioned in Section 1, low-image resolution can degrade the comparison performance of investigators. Thus, low-resolution images cause a problem of making the EDs larger, even if two images are identical. As shown in Figure 8, the EDs of low-resolution images (red bar) are more than double the distance of EDs of high/mid-resolution images (blue bar). By applying the super-resolution to low-resolution image (green bar), we could reduce the EDs for the same identity. Based on this result, we constructed a comparison image dataset which consists of high/mid-resolution images and enhanced low-resolution images obtained via super resolution. By using the constructed comparison dataset, we measured the EDs between each reciprocal landmarks according to 5 facial pose angles. The results are shown in Figure 8 and

Table 3. Mean and Standard deviation of ED between each reciprocal landmark.

Pose	Landmark	Same Identity		Different Identity		${\mathit{d}}^{\prime }$	Standard Value for Identity Verification
	Index	Mean	Std.	Mean	Std.
Left 90°	2	2.488	1.529	5.099	2.734	1.179	2.638
	3	2.413	1.485	5.269	2.701	1.310	2.559
	4	2.096	1.407	4.811	2.640	1.283	2.234
	5	2.395	2.134	4.808	2.615	1.011	2.605
	6	2.074	1.421	4.646	2.601	1.227	2.213
	8	11.030	8.582	32.078	18.404	1.466	11.871
	9	10.062	7.810	31.255	17.862	1.537	10.828
	10	9.241	7.044	31.508	17.684	1.654	9.931
	11	11.992	9.037	34.779	19.964	1.471	12.877
	12	13.438	10.393	35.322	20.461	1.349	14.456
	13	12.745	10.175	34.776	19.921	1.393	13.742
Left 45°	2	2.122	1.569	3.758	2.049	0.897	2.275
	3	2.565	1.833	4.978	2.558	1.084	2.745
	4	2.148	1.396	3.867	2.255	0.916	2.285
	5	2.679	1.645	4.003	2.169	0.688	2.840
	6	2.923	2.351	4.339	2.427	0.593	3.153
	8	10.920	8.989	32.345	20.319	1.364	11.801
	9	10.046	8.225	31.044	19.573	1.399	10.852
	10	9.828	7.765	31.769	19.726	1.464	10.589
	11	13.215	10.590	35.143	21.664	1.286	14.253
	12	13.501	10.516	33.153	20.810	1.192	14.531
	13	13.183	10.611	33.142	20.809	1.208	14.223
Front	1	7.158	4.325	13.977	8.871	0.977	7.582
	2	5.010	2.926	10.407	6.278	1.102	5.297
	3	5.739	3.375	10.816	6.519	0.978	6.070
	4	5.316	3.000	9.746	5.758	0.965	5.610
	5	4.612	3.040	9.285	5.985	0.984	4.910
	6	3.759	2.375	7.534	4.534	1.043	3.991
	7	4.288	2.360	7.731	4.439	0.968	4.520
	8	4.475	2.285	7.757	4.219	0.967	4.699
	9	5.056	2.884	7.641	4.565	0.677	5.338
	10	4.184	2.716	5.11	2.872	0.335	4.451
	11	2.772	1.878	3.048	1.725	0.153	2.956
	12	2.768	1.874	3.048	1.725	0.155	2.952
	13	10.955	7.253	11.996	6.908	0.147	11.666
	14	5.939	2.970	8.497	4.455	0.676	6.230
	15	2.969	1.840	3.049	1.741	0.045	3.149
	16	2.961	1.830	3.049	1.740	0.049	3.141
	17	10.586	6.525	11.593	6.542	0.154	11.225
	18	5.333	2.958	8.640	4.447	0.876	5.623
	19	5.600	3.453	16.809	9.272	1.602	5.938
	20	6.633	4.279	15.328	8.443	1.299	7.053
	21	5.090	3.152	17.843	9.812	1.750	5.399
	22	5.679	4.135	14.780	8.162	1.407	6.084
Right 45°	2	2.217	1.356	3.795	2.084	0.898	2.349
	3	2.655	1.916	5.414	2.828	1.142	2.843
	4	2.147	1.323	4.168	2.165	1.127	2.276
	5	2.500	1.471	4.109	2.226	0.853	2.644
	6	3.157	2.213	4.663	2.445	0.646	3.374
	8	8.245	7.020	29.181	17.338	1.583	8.933
	9	8.796	6.704	29.506	17.885	1.533	9.453
	10	8.768	5.929	29.004	17.363	1.560	9.349
	11	10.452	6.199	29.312	16.675	1.499	11.059
	12	8.189	7.216	29.471	17.168	1.616	8.897
	13	8.891	7.135	29.128	16.519	1.590	9.590
Right 90°	2	2.313	1.446	5.151	2.728	1.300	2.455
	3	2.464	1.612	5.621	2.950	1.328	2.622
	4	2.531	1.840	5.071	2.780	1.077	2.712
	5	2.658	2.205	4.924	2.625	0.935	2.874
	6	2.379	1.737	4.403	2.454	0.952	2.549
	8	9.587	7.266	34.803	21.469	1.573	10.299
	9	10.563	7.855	34.619	21.609	1.480	11.333
	10	10.010	7.380	34.026	21.604	1.488	10.734
	11	10.198	6.680	34.733	20.673	1.597	10.853
	12	9.720	7.383	35.084	21.022	1.610	10.444
	13	9.363	6.956	35.033	20.315	1.691	10.045

. In Figure 8, the X-axis, Y-axis, blue bars, and red bars refer to the landmark indexes defined in Table 1, calculated ED, distances for different images of the same individual, and distance for images of different individuals, respectively.

As shown in Figure 9 and Table 3, the distance between reciprocal landmarks tended to decrease as the pose angle approached that of a frontal pose. In particular, the landmarks near the eyes for the frontal pose had small values due to the size normalization based on the center of the two eyes; this decreased the distance between landmarks for different individuals. Thus, it was difficult to distinguish individuals using the landmarks near the eyes for frontal poses. For example, the right exocanthion (index number 16) had an average pixel ED of 2.96 for the same identity and 3.05 for different identities. If these distances are converted into mm, the values are 1.86 mm and 1.92 mm, respectively. Thus, if the facial analyst even slightly mis-positioned the landmark, two images obtained from different persons might be diagnosed as the same identity. In the case of landmarks, except the eye area, the distance between landmarks for the same identity was about half as small as that for different persons, and the average EDs of different identity near the ear area was almost three times larger than that of the same identity. Among the landmarks in the ear area, we could confirm that both otobasion inferius (index number 19, 21) are the most discriminable based on the average distance between same and different individual in case of frontal facial pose. For side poses, the ED between landmarks near the eye and ear were significantly different for the same identity versus different identities, which validates the importance of landmarks near the eye and ear for identity verification.

Based on the mean and standard deviation of ED of the landmark, we measured the $d^{\prime }$ value to select the reliability of the facial landmark [56]. The $d^{\prime }$ measurement is statistically used to calculate the distance between two distributions based on both mean and standard deviation. The equation of $d^{\prime }$ is as follows.

$$d^{\prime }=\frac{{\mu }_{imposter}-{\mu }_{genuine}}{\sqrt{\frac{{\sigma }_{genuine}^2+{\sigma }_{imposter}^2}{2}}}$$

(2)

In Equation (2), ${\mu }_{genuine}$ and ${\mu }_{imposter}$ denote the mean of ED for same identity and different identity, respectively. In addition, $\sigma$-genuine and $\sigma$-imposter are the standard deviations of them. As the distance between the two distributions increases, the value of $d^{\prime }$ also becomes greater. Based on this, it is possible to select the reliable landmarks that can distinguish the identity of the same and different individuals. As shown in Table 3, we can confirm that the landmarks in the eye and ear region have the reliability to verify the same identity compared to other landmarks.

In addition, we calculated the confidence interval of the population mean to be 95% confidence. In case of this experiment, the number of samples for the same identity was 400, so the confidence interval of the population of right otobasion inferius (index number 19) ranged from 5.261 to 5.938. From this result, we set the maximum value of the confidence interval as standard value for identity verification. Thus, we are confident that the probability of two images having the same identity is high if the ED from otobasion inferius is smaller than 5.938. The threshold for identity verification of all the landmarks are shown in the rightmost column in Table 3.

In the second experiment, we divided the face into 4 regions (eye, nose, lip, and ear) and calculated EDs using the landmarks in each region. This experiment enabled us to determine which facial regions were more effective for identity verification. The facial landmarks and index numbers in each region are shown in

Table 4. Facial landmarks and index numbers of each facial region.

Facial Region	Frontal Pose		Side Pose
	Landmarks	Landmark Index	Landmarks	Landmark Index
Eye	Endocanthion, Exocanthion, Outer Eyebrow, Inner Eyebrow	11,12,13,14, 15,16,17,18	Exocanthion, Endocanthion, Inner Eyebrow, Outer Eyebrow	4
Nose	Subnasale, Alare, Pronasale, Nasion	6,7,8,9,10	Subnasale, Pronasale, Nasion	5,6,7
Lip	Labiale inferius, Labiale superius, Cheilion	2,3,4,5	Labiale inferius, Labiale superius, Cheilion	2,3,4
Ear	Otobasion inferius, Otobasion superius	19,20,21,22	Otobasion inferius, Otobasion superius	12,13

.

In this experiment, the landmarks used for size normalization were excluded, as in the previous experiment. From the result, the $d^{\prime }$ in the eye region of the frontal face has a very low value as in the previous experiment, which means that it could not be utilized for same identity verification. Based on the $d^{\prime }$, the eye region, except the frontal pose and ear region, are the important facial parts to verify same identity which have more reliability than other regions. The results are shown in

Table 5. Mean and Std. of EDs between reciprocal landmarks in each region.

Facial Region	Pose Angle	Same Identity		Different Identity		${\mathit{d}}^{\prime }$	Standard Value
		Mean	Std.	Mean	Std.		for Identity Verification
Eye	Left 90°	5.431	3.917	16.316	9.075	1.557	5.815
	Left 45°	5.732	4.228	16.445	9.942	1.402	6.146
	Front	2.445	1.058	2.907	0.996	0.450	2.549
	Right 45°	4.694	3.023	14.757	8.435	1.588	4.990
	Right 90°	5.143	3.512	17.389	10.480	1.567	5.487
Nose	Left 90°	1.099	0.797	2.288	1.115	1.227	1.177
	Left 45°	1.388	0.855	2.050	0.921	0.745	1.472
	Front	2.117	0.786	3.43	1.463	1.122	2.194
	Right 45°	1.410	0.772	2.150	0.940	0.860	1.486
	Right 90°	1.234	0.874	2.261	1.081	1.046	1.319
Lip	Left 90°	1.421	0.825	3.033	1.326	1.509	1.492
	Left 45°	1.390	0.820	2.592	1.013	1.304	1.470
	Front	2.747	1.240	5.272	2.654	1.219	2.869
	Right 45°	1.432	0.771	2.741	1.080	1.395	1.508
	Right 90°	1.496	0.798	3.174	1.377	1.491	1.575
Ear	Left 90°	9.338	7.158	24.982	13.927	1.413	10.040
	Left 45°	9.502	7.369	23.584	14.480	1.226	10.224
	Front	3.122	1.467	8.528	3.629	1.953	3.266
	Right 45°	6.172	4.907	20.885	11.617	1.650	6.653
	Right 90°	6.887	4.870	24.956	14.330	1.688	7.364

.

In the last experiment, we calculated the ED based on entire facial landmarks, as shown in

Table 6. Mean and Std. of EDs between reciprocal landmarks in each region.

Pose Angle	Same Identity		Different Identity		${\mathit{d}}^{\prime }$	Standard Value for
	Mean	Std.	Mean	Std.		Identity Verification
Left 90°	2.694	1.891	7.639	4.010	1.577	2.879
Left 45°	2.806	2.002	7.488	4.363	1.379	3.002
Front	1.364	0.422	2.448	0.732	1.814	1.406
Right 45°	2.174	1.355	6.738	3.570	1.690	2.307
Right 90°	2.359	1.517	7.966	4.448	1.687	2.507

. The $d^{\prime }$ between two distributions of the overall pose angle were similar or higher than that of the previous experiment. In other words, as the use of overall landmarks, the probability of distinction between the same person and different person would increase, which means that it can be used more reliably in identity verification. All of the standard values for identity verification in each facial pose were 2.879, 3.002, 1.406, 2.307, 2.507, respectively. All proposed methods are described in Algorithm 1.

Algorithm 1 Proposed Method for Selecting Reliable Landmark Indices.

Input: Multi-view references images, comparison image Output: ED and landmark indices  1: Repeat:  2: Apply SfM [41] on multi-view images, generate a 3D face mesh  3: Rotate the 3D mesh to a comparison facial pose and acquire rendered image  4: Conduct VDSR [47] on the comparison image, acquire high-definition image  5: Conduct wild-feature detector [54] on the rendered and comparison image, obtain landmark points  6: Select the landmark indices for frontal and profile face images following [2,20,21,55]  7: Normalize the images by IPD and distance between the nasion and gnathion, both 100 pixels  8: Calculate distances between the selected landmarks following Equation (1).  9: Calculate $d^{\prime }$ measure following Equation (2)  10: Continue Until: Paired (genuine and imposter) images ended  11: Analyze the distribution for the same and different individuals  12: Select reliable landmark indices

4. Discussion and Conclusions

In this paper, by creating a 3D face model from multiple images taken from multiple angles and reorienting it to match the angle of a comparison image, we resolved the issues associated with comparisons of different face angles and poses. Inaccuracies caused by low-resolution images typical of security cameras frequently occurred. To resolve this problem, we applied the super-resolution method based on deep learning, which converted the input into a high definition image, and proved the effect of super resolution by comparing the average distance with that from a low-resolution image. In addition, we applied an automatic facial landmark detector to improve speed, convenience, and accuracy for face analysts who typically must manually indicate facial landmarks on each image.

Finally, we provided a standard ED value and more reliable facial landmarks for identity verification. By using the $d^{\prime }$ measurement between two distributions, which are from same identity and different identity, we could confirm the reliable landmarks according to each facial pose. As mentioned in the experimental results, the landmarks near the eyes and ears had a higher distinguish ability than others. The most reliable landmarks were otobasion inferius and otobasin superius in overall facial pose, and exocanthion, endocanthion, inner eyebrow, outer eyebrow were also reliability ones in the case of side pose. In addition, we proved that eyes and ears are significant facial components for identity verification by analyzing the distributions from landmark grouping comparison experiment. In addition, the standard threshold for identity verification was determined with 95% confidence by statistical analysis based on mean and standard deviation.

In future work, we plan to study standard measures by applying the proposed methods from various face datasets and, further, identity verification based on image comparisons that are robust to different facial expressions and occlusion of facial areas by hair or accessories. We also aim to improve on the reliability of our identity verification method by analyzing the relationships between lines and angles that define landmarks.