# Multi-Sensor Face Registration Based on Global and Local Structures

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. The Proposed Registration Method for Visible and Infrared Face Images

#### 2.1. Inner-Distance Shape Context for 2D Feature Descriptors

_{θ}= 12 denotes the numbers of orientation bins and the vertical axis n

_{d}= 5 denotes the numbers of logarithm distance bins.

**X**and a point ${q}_{j}$ in the point set

**Y**. Let

**C**

_{i}

_{,j}denote the cost of matching ${p}_{i}$ to ${q}_{j}$, ${\mathit{h}}_{\mathit{X},{p}_{i}}\left(k\right)$ and ${\mathit{h}}_{\mathit{Y},{q}_{j}}\left(k\right)$ denote the normalized histogram of the relative coordinates of the remaining points, respectively, and K be the number of histogram bins. As the IDSC is represented based on the histogram, it can be measured by the χ

^{2}test statistic:

**C**

_{i}

_{,j}is, the more similar the local appearance at points ${p}_{i}$ and ${q}_{j}$ are. Once the cost matrix

**C**for all correspondence points in the point set

**X**and

**Y**is obtained, the correspondences

**Ω**between the two point sets are an instance of an assignment problem, which can be solved by the Hungarian method.

#### 2.2. Student’s t Mixtures for Feature Point Set Registration

**X**

_{N}

_{×D}= (x

_{1}, …, x

_{N})

^{T}, which can be considered as an observation datum, the other point set

**Y**

_{M}

_{×D}= (y

_{1}, …, y

_{M})

^{T}is treated as the SMM centroids, where N and M represent the number of points in

**X**and

**Y**, respectively, and D represents the dimension of each point. Then the goal of registration is to align the centroids point set

**Y**to the observation data point set

**X**. Typically, the point sets contain noise and outliers, which can be supposed to be a uniform distribution 1/N, and the weight of the uniform distribution is denoted as η ∈ [0, 1]. Let w

_{ij}∈ [0, 1] denote a mixing proportion corresponding to the i

^{th}component of the SMM for point x

_{j}($\sum _{i=1}^{M}{w}_{ij}}=\text{}1$), then the mixture probability density function can take the form:

**ψ**= (

**ψ**

_{1},

**ψ**

_{2}, …,

**ψ**

_{M}) is mixture parameter set with

**ψ**

_{i}= (w

_{ij}, y

_{i}, ∑

_{i}, υ

_{i}), and

**f**(x

_{j}|y

_{i}, ∑

_{i}, υ

_{i}) is the Student’s t distribution probability density function for the i

^{th}component of SMM; we express that:

_{j}, y

_{i}; ∑

_{i}) is the Mahalanobis squared distance between observation point x

_{j}and centroid point y

_{i}, while ∑

_{i}, Γ and υ represent the covariance matrix, Gamma function and degrees of freedom, respectively.

#### 2.3. Multi-Spectral Face Registration Using Global and Local Structures

**ψ**in the mixture probability density function (2), according to [22,23], we first rewrite the function (2) as a complete data logarithm likelihood function $\mathrm{ln}{L}_{C}\left(\mathit{\psi}\right)={\displaystyle \sum _{j=1}^{N}\mathrm{ln}{\displaystyle \sum _{i=1}^{M}{w}_{ij}}}\mathit{f}\left({x}_{j}|{y}_{i},{\Sigma}_{i},{\upsilon}_{i}\right)$, and then use the Expectation Maximization (EM) to train the logarithm likelihood function. In general, the EM algorithm can be divided into the following two steps: Expectation step (E-step) and Maximization step (M-step), and the iterative process proceeds by alternating between the E- and M-steps until convergence.

**ψ**

^{(k)}to calculate the posterior probability ${\mathrm{\tau}}_{ij}{}^{(k)}$, and according to the Bayesian Theorem, it can be expressed in terms of the observation data x

_{j}belonging to the i

^{th}component of the SMM

^{th}iteration conditional expectation of the logarithm likelihood function $\mathrm{ln}{L}_{C}\left(\mathit{\psi}\right)$, which can be calculated as follows:

**Q**

_{1},

**Q**

_{2}and

**Q**

_{3}are respectively written as follows:

**Q**

_{2}that denotes the Digamma function, and the u

_{ij}

^{(k)}in

**Q**

_{3}that represents the conditional expectation about additional missing data in the complete-data; it can be written as:

**ρ**to denote the non-rigid transformation in the point set:

**X**=

**T**(

**Y**,

**ρ**) =

**Y**+

**ρ**(

**Y**), and add the regularization term

**φ**(

**ρ**) to the

**Q**

_{3}, which can enforce the smoothness of the displacement function in the alignment of two point sets. So this

**Q**

_{3}term can be rewritten as:

**ρ**can be denoted as:

**G**is a Gaussian kernel matrix, β determines the width of the smoothing Gaussian filter, and the Equation (8) can be conveniently denoted in matrix form as:

**ρ**(

**Y**) =

**GH**, where

**H**

_{M}

_{×D}= (h

_{1}, …, h

_{M}) is a matrix of coefficients. Thus, the Equation (7) can be expanded in the following matrix form:

^{th}iteration of the EM algorithm, considering that

**Q**

_{1},

**Q**

_{2}and

**Q**

_{3}in Equation (5) can be computed independently of each other, thus the maximization of the objective function

**Q**

_{1},

**Q**

_{2}and

**Q**

_{3}with respect to the parameters w, υ, ∑, ρ can be operated separately. The mixing proportion for SMM is updated by our consideration of the first term

**Q**

_{1}. Given the feature descriptors in Section 2.1, it can obtain the coarse correspondences

**Ω**between the point sets

**X**and

**Y**. For an observation data x

_{j}, we define ι (0 ≤ ι ≤1) as a confidence by IDSC feature matching, and then the mixing proportion w

_{ij}by incorporating the local structures among neighboring points can be updated by the following rule:

_{j}does not have a corresponding point y

_{i}in the label

**Ω**, the mixing proportion w

_{ij}

^{(k+1)}is given by the average of the posterior probabilities ${\mathrm{\tau}}_{ij}{}^{(k)}$ of the SMM component membership. If the observation datum x

_{j}corresponds to the centroid point y

_{i}in the label

**Ω**, the w

_{ij}

^{(k+1)}is given by a constant confidence ι.

_{ij}does not only depend on prior assignment by local structures, but also depends upon the posterior probability of observation data x

_{j}belonging to the i

^{th}component of the SMM by global structures. In addition, we obtain degree of freedom

**υ**by taking the corresponding derivative of

**Q**

_{2}to zero, and the updated value

**υ**

^{(k+1)}is a solution of the following equation:

**ρ**in Equation (7), we also need to take the derivative of ${\tilde{\mathit{Q}}}_{3}$, and the update values can be denoted as:

**1**is a column vector of all ones, and

**I**is an identity matrix, diag(•) denotes a diagonal matrix and

**G**(i,•) denotes the column vector in the kernel matrix

**G**. Furthermore, for our centroid point y

_{i}in the point set

**Y**, the non-rigid transformation

**T**(

**Y**,

**ρ**) =

**Y**+

**ρ**(

**Y**) can be expressed by Equation (8) as follows:

^{−5}in our experiment. Furthermore, to register the visible and infrared images accordingly, the image transformation and interpolation is performed on the visible image. Since our multi-spectral face registration method is implemented by using global and local structures, then the registration method is to be named as Face Registration using the Global and Local Structure (FR-GLS) in the rest of this paper.

## 3. The Proposed Fusion Strategies for Visible and Infrared Face Images

**F**is given by the guided filtering algorithm as follows:

_{G}**B**

_{n}and detail

**B**

_{n}layers of different source images:

**B**

_{n}and detail layers

**D**

_{n}via guided image filtering respectively. In order to optimize the initial fusion image

**F**, here the visible, infrared and fused images are defined by

_{G}**V**,

**I**and

**F**, respectively. Given that the thermal radiation information in the infrared image is typically characterized by the pixel intensity values, the pixel intensity values are quite different between the target and background. Thus, we constrain the fused image

**F**to have the similar pixel intensity distribution with the infrared source image

**I**, and the optimization problem can be formulated as:

_{1}norm. Besides, given that the pixel intensity distribution in the same physical location might be discrepant for infrared and visible face images, they are manifestations of two different phenomena. Note that the detailed information of object edge and texture is mainly characterized by the pixel gradient values in the visible image. Hence, we constrain the fused image

**F**to have the similar pixel gradient values with the visible source image

**V**, the optimization problem can also be formulated as:

**F**can be formulated as minimizing the following objective function:

**F**could be found by using gradient descent strategy. Furthermore, it is important to note that if the α is small, the fusion image preserves the more thermal radiation information of the infrared image, otherwise, the fusion image preserves more edge and texture information of the visible image. We set the weighting coefficient α = 5 in our experiment, because it can achieve good subjective visual quality in most cases.

## 4. Experimental Results

#### 4.1. Registration on Real Face Images with the UTK-IRIS Database

_{ij}of the SMM.

#### 4.1.1. Qualitative Evaluation

#### 4.1.2. Quantitative Evaluation

#### 4.2. Registration on Real Face Images of Self-Built Multi-Spectral Database

#### 4.2.1. Qualitative Comparison

#### 4.2.2. Quantitative Comparison

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Ma, J.; Ma, Y.; Li, C. Infrared and visible image fusion methods and applications: A survey. Inf. Fusion
**2019**, 45, 153–178. [Google Scholar] [CrossRef] - Ma, J.; Yu, W.; Liang, P. FusionGAN: A generative adversarial network for infrared and visible image fusion. Inf. Fusion
**2019**, 48, 11–26. [Google Scholar] [CrossRef] - Huang, Y.; Bi, D.; Wu, D. Infrared and visible image fusion based on different constraints in the non-subsampled shearlet transform domain. Sensors
**2018**, 18, 1169. [Google Scholar] [CrossRef] - Ma, J.; Chen, C.; Li, C. Infrared and visible image fusion via gradient transfer and total variation minimization. Inf. Fusion
**2016**, 31, 100–109. [Google Scholar] [CrossRef] - Paramanandham, N.; Rajendiran, K. Infrared and visible image fusion using discrete cosine transform and swarm intelligence for surveillance applications. Infrared Phys. Technol.
**2018**, 88, 13–22. [Google Scholar] [CrossRef] - Zhu, Z.; Qi, G.; Chai, Y. A geometric dictionary learning based approach for fluorescence spectroscopy image fusion. Appl. Sci.
**2017**, 7, 161. [Google Scholar] [CrossRef] - Singh, R.; Vatsa, M.; Noore, A. Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition. Pattern Recognit.
**2008**, 41, 880–893. [Google Scholar] [CrossRef] [Green Version] - Vizgaitis, J.N.; Hastings, A.R. Dual band infrared picture-in-picture systems. Opt. Eng.
**2013**, 52, 061306. [Google Scholar] [CrossRef] - Kong, S.G.; Heo, J.; Boughorbel, F. Multiscale fusion of visible and thermal IR images for illumination-invariant face recognition. Int. J. Comput. Vis.
**2007**, 71, 215–233. [Google Scholar] [CrossRef] - Oliveira, F.P.M.; Tavares, J.M.R.S. Medical image registration: A review. Comput. Method Biomech.
**2014**, 17, 73–93. [Google Scholar] [CrossRef] - Avants, B.B.; Epstein, C.L.; Grossman, M. Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain. Med. Image Anal.
**2008**, 12, 26–41. [Google Scholar] [CrossRef] [Green Version] - Pan, W.; Qin, K.; Chen, Y. An adaptable-multilayer fractional fourier transform approach for image registration. IEEE Trans. Pattern Anal.
**2008**, 31, 400–414. [Google Scholar] [CrossRef] - Zhuang, Y.; Gao, K.; Miu, X. Infrared and visual image registration based on mutual information with a combined particle swarm optimization—Powell search algorithm. Optik Int. J. Light Electron Opt.
**2016**, 127, 188–191. [Google Scholar] [CrossRef] - Sotiras, A.; Davatzikos, C.; Paragios, N. Deformable medical image registration: A survey. IEEE Trans. Med. Imaging
**2013**, 32, 1153–1190. [Google Scholar] [CrossRef] - Viergever, M.A.; Maintz, J.B.A.; Klein, S. A survey of medical image registration–under review. Med. Image Anal.
**2016**, 33, 140–144. [Google Scholar] [CrossRef] - Moghbel, M.; Mashohor, S.; Mahmud, R. Review of liver segmentation and computer assisted detection/diagnosis methods in computed tomography. Artif. Intell. Rev.
**2018**, 50, 497–537. [Google Scholar] [CrossRef] - Zhang, Y.; Zhao, D.; Sun, J. Adaptive convolutional neural network and its application in face recognition. Neural Process. Lett.
**2016**, 43, 389–399. [Google Scholar] [CrossRef] - Zhang, K.; Zhang, Z.; Li, Z. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal. Proc. Lett.
**2016**, 23, 1499–1503. [Google Scholar] [CrossRef] - Myronenko, A.; Song, X. Point set registration: Coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell.
**2010**, 32, 2262–2275. [Google Scholar] [CrossRef] - Ma, J.; Zhao, J.; Ma, Y. Non-rigid visible and infrared face registration via regularized Gaussian fields criterion. Pattern Recognit.
**2015**, 48, 772–784. [Google Scholar] [CrossRef] - Tian, T.; Mei, X.; Yu, Y. Automatic visible and infrared face registration based on silhouette matching and robust transformation estimation. Infrared Phys. Technol.
**2015**, 69, 145–154. [Google Scholar] [CrossRef] - Peel, D.; McLachlan, G.J. Robust mixture modelling using the t distribution. Stat. Comput.
**2000**, 10, 339–348. [Google Scholar] [CrossRef] - Gerogiannis, D.; Nikou, C.; Likas, A. Robust Image Registration using Mixtures of t-distributions. In Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, 14–21 October 2007; pp. 1–8. [Google Scholar]
- Gerogiannis, D.; Nikou, C.; Likas, A. The mixtures of Student’s t-distributions as a robust framework for rigid registration. Image Vis. Comput.
**2009**, 27, 1285–1294. [Google Scholar] [CrossRef] - Zhou, Z.; Zheng, J.; Dai, Y. Robust non-rigid point set registration using student’s-t mixture mode. PloS ONE
**2014**, 9, e91381. [Google Scholar] - Zhou, Z.; Tu, J.; Geng, C. Accurate and robust non-rigid point set registration using student’s t mixture model with prior probability modeling. Sci. Rep. UK
**2018**, 8, 8742. [Google Scholar] [CrossRef] - Maiseli, B.; Gu, Y.; Gao, H. Recent developments and trends in point set registration methods. J. Vis. Commun. Image Represent.
**2017**, 46, 95–106. [Google Scholar] [CrossRef] - Belongie, S.; Malik, J.; Puzicha, J. Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell.
**2002**, 24, 509–522. [Google Scholar] [CrossRef] [Green Version] - Ling, H.; Jacobs, D.W. Shape classification using the inner-distance. IEEE Trans. Pattern Anal. Mach. Intell.
**2007**, 29, 286–299. [Google Scholar] [CrossRef] - Li, H.; Wu, X.J. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Trans. Image Process.
**2018**, 28, 2614–2623. [Google Scholar] [CrossRef] - Chui, H.; Rangarajan, A. A new point matching algorithm for non-rigid registration. Comput. Vis. Image Underst.
**2003**, 89, 114–141. [Google Scholar] [CrossRef] - Riesen, K.; Bunke, H. Approximate graph edit distance computation by means of bipartite graph matching. Image Vis. Comput.
**2009**, 27, 950–959. [Google Scholar] [CrossRef] - Li, S.; Kang, X.; Hu, J. Image fusion with guided filtering. IEEE Trans. Image Process.
**2013**, 22, 2864–2875. [Google Scholar] [PubMed] - UTK-IRIS Database. Available online: http://www.cse.ohio-state.edu/otcbvs-bench/ (accessed on 10 September 2015).
- Dwith, C.; Ghassemi, P.; Pfefer, T. Free-form deformation approach for registration of visible and infrared facial images in fever screening. Sensors
**2018**, 18, 125. [Google Scholar] [CrossRef] [PubMed]

**Figure 2.**Schematic illustration of face silhouette feature descriptors based on Inner-Distance Shape Context (IDSC). (

**a**) Bellman-Ford shortest path graph built using face silhouette landmark points; (

**b**,

**c**) Four marked points and their corresponding IDSC feature histograms.

**Figure 4.**Registration and fusion results of our method on four typical unregistered visible/infrared image pairs in the database of UTK-IRIS. (

**a**) Charles; (

**b**) Heo; (

**c**) Meng; (

**d**) Sharon.

**Figure 5.**Quantitative comparisons of multispectral face image pairs with four individuals in Figure 4. (

**a**) Charles; (

**b**) Heo; (

**c**) Meng; (

**d**) Sharon.

**Figure 6.**4-DOF pan and tilt head vision platform for multi-spectral database acquisition: (

**a**) Design mechanical model of vision platform; (

**b**) Image acquisition scenario.

**Figure 7.**Registration and fusion results of our method considering six typical unregistered visible and infrared image pairs in the self-built multi-spectral face datasets. (

**a**) Visible image; (

**b**) Thermal infrared image; (

**c**) Canny edge maps; (

**d**) Superimposed checkerboard pattern of visible and infrared images; (

**e**) Fusion results.

**Figure 8.**Quantitative comparisons of multi-spectral face image pairs with 6 individuals in Figure 7.

**Table 1.**The average registration errors comparison of CPD, SMM, RGF and our FR-GLS method on four typical individuals in Figure 4.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Li, W.; Dong, M.; Lu, N.; Lou, X.; Zhou, W.
Multi-Sensor Face Registration Based on Global and Local Structures. *Appl. Sci.* **2019**, *9*, 4623.
https://doi.org/10.3390/app9214623

**AMA Style**

Li W, Dong M, Lu N, Lou X, Zhou W.
Multi-Sensor Face Registration Based on Global and Local Structures. *Applied Sciences*. 2019; 9(21):4623.
https://doi.org/10.3390/app9214623

**Chicago/Turabian Style**

Li, Wei, Mingli Dong, Naiguang Lu, Xiaoping Lou, and Wanyong Zhou.
2019. "Multi-Sensor Face Registration Based on Global and Local Structures" *Applied Sciences* 9, no. 21: 4623.
https://doi.org/10.3390/app9214623