# Exemplar-Based Face Colorization Using Image Morphing

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Image Morphing

#### 2.1. Morphing Model Based on [31]

- Fixing $\mathbf{I}$ and minimizing over $\mathbf{v}$ leads to the following K single registration problems:$$\underset{{v}_{k}}{\mathrm{arg}\hspace{0.17em}\mathrm{min}}{\mathcal{J}}_{\mathbf{I}}({v}_{k}):={\int}_{\mathsf{\Omega}}{|{I}_{k}-{I}_{k-1}\circ {\phi}_{k}|}^{2}+\mathcal{S}({v}_{k})\phantom{\rule{0.166667em}{0ex}}\mathrm{d}\mathbf{x},\phantom{\rule{1.em}{0ex}}k=1,\dots ,K,$$
- Fixing $\mathbf{v}$, resp., $\mathbf{\phi}$ leads to solving the following image sequence problem$$\underset{\mathbf{I}}{\mathrm{arg}\hspace{0.17em}\mathrm{min}}{\mathcal{J}}_{\mathbf{\phi}}(\mathbf{I}):=\sum _{k=1}^{K}{\int}_{\mathsf{\Omega}}{|{I}_{k}-{I}_{k-1}\circ {\phi}_{k}|}^{2}\phantom{\rule{0.166667em}{0ex}}\mathrm{d}\mathbf{x}.$$

#### 2.2. Space Discrete Morphing Model

#### Solution of the Registration Problems

#### Multilevel Strategy

Algorithm 1 Morphing Algorithm (informal). | |

1: | ${T}_{0}:=T,{R}_{0}:=R,{\mathsf{\Omega}}_{0}:=\mathsf{\Omega}$ |

2: | create image stack $({T}_{0},\dots ,{T}_{l}),({R}_{0},\dots ,{R}_{l})$ on $({\mathsf{\Omega}}_{0},\dots ,{\mathsf{\Omega}}_{l})$ by smoothing and downsampling |

3: | solve (3) for ${T}_{l},{R}_{l}$ with $K=1$, for $\tilde{v}$ |

4: | $l\to l-1$ |

5: | use bilinear interpolation to get v on ${\mathsf{\Omega}}_{l}$ from $\tilde{v}$ |

6: | obtain ${\tilde{K}}_{l}$ images ${\mathbf{I}}_{l}^{(0)}$ from ${T}_{l},{R}_{l},v$ by (9) |

7: | while $l\ge 0$ do |

8: | find image path ${\tilde{\mathbf{I}}}_{l}$ and deformation path ${\tilde{\mathbf{v}}}_{l}$ minimizing (3) with initialization ${\mathbf{I}}_{l}^{(0)}$ |

9: | $l\to l-1$ |

10: | if l > 0 then |

11: | use bilinear interpolation to get ${\mathbf{I}}_{l}$ and ${\mathbf{v}}_{l}$ on ${\mathsf{\Omega}}_{l}$ |

12: | for $k=1,\dots ,{\tilde{K}}_{l}$ do |

13: | calculate ${\tilde{K}}_{l}$ intermediate images between ${I}_{l,k-1},{I}_{l,k}$ with ${v}_{l,k}$ using (9) |

14: | $\mathbf{I}:={\mathbf{I}}_{0}$ |

## 3. Face Colorization

#### 3.1. Luminance Normalization

#### 3.2. Chrominance Transfer by the Morphing Maps

## 4. Variational Methods for Chrominance Postprocessing

Algorithm 2 Minimization of (13). | |

1: | ${u}^{0}\leftarrow b$, ${\overline{u}}^{0}\leftarrow {u}^{0}$ |

2: | ${p}^{0}\leftarrow \nabla {u}^{0}$ |

3: | $\sigma \leftarrow 0.001$, $\tau \leftarrow 20$ |

4: | for $n>0$ do |

5: | ${p}^{n+1}\leftarrow {P}_{\mathcal{B}}\left({p}^{n}+\sigma \nabla {\overline{u}}^{n}\right)$ |

6: | ${u}^{n+1}\leftarrow {\displaystyle \frac{{u}^{n}+\tau \left(\mathrm{div}({p}^{n+1})+\alpha b\right)}{1+\tau \alpha}}$ |

7: | $\theta =1/\sqrt{1+\tau \alpha}$ |

8: | $\tau =\theta \tau $ $\sigma =\sigma /\theta $ |

9: | ${\overline{u}}^{n+1}\leftarrow {u}^{n+1}+\theta ({u}^{n+1}-{u}^{n})$ |

10: | $\widehat{u}\leftarrow {u}^{+\infty}.$ |

Algorithm 3 Debiasing of Algorithm 2. | |

1: | ${u}^{0}=b$, ${\overline{u}}^{0}=b$ |

2: | $\delta \leftarrow b-\widehat{u}$ |

3: | ${\tilde{u}}^{0}=\delta $, ${\overline{\tilde{u}}}^{0}=\delta $ |

4: | ${p}^{0}\leftarrow \nabla u$, ${\tilde{p}}^{0}\leftarrow \nabla \tilde{u}$ |

5: | $\sigma \leftarrow 0.001$, $\tau \leftarrow 20$ |

6: | for $n\ge 0$ do |

7: | ${p}^{n+1}\leftarrow {P}_{\mathcal{B}}\left({p}^{n}+\sigma \nabla {\overline{u}}^{n}\right)$ |

8: | ${\tilde{p}}^{n+1}\leftarrow {\mathsf{\Pi}}_{{p}^{n}+\sigma \nabla {\overline{u}}^{n}}\left({\tilde{p}}^{n}+\sigma \nabla {\overline{\tilde{u}}}^{n}\right)$ |

9: | ${u}^{n+1}\leftarrow {\displaystyle \frac{{u}^{n}+\tau \left(\mathrm{div}({p}^{n+1})+\alpha b\right)}{1+\tau \alpha}}$ |

10: | ${\tilde{u}}^{n+1}\leftarrow {\displaystyle \frac{{\tilde{u}}^{n}+\tau \left(\mathrm{div}({\tilde{p}}^{n+1})+\alpha \delta \right)}{1+\tau \alpha}}$ |

11: | $\theta =1/\sqrt{1+\tau \alpha}$ |

12: | $\tau =\theta \tau $ $\sigma =\sigma /\theta $ |

13: | ${\overline{u}}^{n+1}\leftarrow {u}^{n+1}+\theta ({u}^{n+1}-{u}^{n})$ |

14: | ${\overline{\tilde{u}}}^{n+1}\leftarrow {\tilde{u}}^{n+1}+\theta ({\tilde{u}}^{n+1}-{\tilde{u}}^{n})$ |

15: | $\rho \leftarrow \left\{\begin{array}{cc}{\displaystyle \frac{\u2329{\tilde{u}}^{+\infty}|\delta \u232a}{\parallel {\tilde{u}}^{+\infty}{\parallel}_{2}^{2}}}& \mathrm{if}\phantom{\rule{4.pt}{0ex}}{\tilde{u}}^{+\infty}\ne 0\\ 1& \mathrm{otherwise}.\end{array}\right.,$ |

16: | ${u}_{\mathrm{debiased}}\leftarrow \widehat{u}+\rho {\tilde{u}}^{+\infty}.$ |

## 5. Numerical Examples

**Remark**(Limitations of our approach)

**.**

## 6. Conclusions

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Appendix A. Solution of the Image Sequence Problem

## Appendix B. Gauss-Newton Method for the Registation Problem

## References

- Levin, A.; Lischinski, D.; Weiss, Y. Colorization using optimization. ACM Trans. Graph.
**2004**, 23, 689–694. [Google Scholar] [CrossRef] - Irony, R.; Cohen-Or, D.; Lischinski, D. Colorization by example. In Proceedings of the 16th Eurographics Conference on Rendering Techniques, Konstanz, Germany, 29 June 29–1 July 2005; pp. 201–210. [Google Scholar]
- Pierre, F.; Aujol, J.F.; Bugeau, A.; Papadakis, N.; Ta, V.T. Luminance-chrominance model for image colorization. SIAM J. Imaging Sci.
**2015**, 8, 536–563. [Google Scholar] [CrossRef] - Yatziv, L.; Sapiro, G. Fast image and video colorization using chrominance blending. IEEE Trans. Image Process.
**2006**, 15, 1120–1129. [Google Scholar] [CrossRef] [PubMed] - Jack, K. Video Demystified: A Handbook for the Digital Engineer; Elsevier: Amsterdam, The Netherlands, 2011. [Google Scholar]
- Gupta, R.K.; Chia, A.Y.S.; Rajan, D.; Ng, E.S.; Zhiyong, H. Image colorization using similar images. In Proceedings of the 20th ACM International Conference on Multimedia, Nara, Japan, 29 October–2 November 2012; pp. 369–378. [Google Scholar]
- Horiuchi, T. Colorization algorithm using probabilistic relaxation. Image Vis. Comput.
**2004**, 22, 197–202. [Google Scholar] [CrossRef] - Welsh, T.; Ashikhmin, M.; Mueller, K. Transferring color to greyscale images. ACM Trans. Graph.
**2002**, 21, 277–280. [Google Scholar] [CrossRef] - Efros, A.A.; Leung, T.K. Texture synthesis by non-parametric sampling. In Proceedings of the 7th IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; Volume 2, pp. 1033–1038. [Google Scholar]
- Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; Süsstrunk, S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell.
**2012**, 34, 2274–2282. [Google Scholar] [CrossRef] [PubMed] - Chen, T.; Wang, Y.; Schillings, V.; Meinel, C. Grayscale image matting and colorization. In Proceedings of the Asian Conference on Computer Vision, Jeju Island, Korea, 27–30 January 2004; pp. 1164–1169. [Google Scholar]
- Charpiat, G.; Hofmann, M.; Schölkopf, B. Automatic image colorization via multimodal predictions. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2008; pp. 126–139. [Google Scholar]
- Chia, A.Y.S.; Zhuo, S.; Kumar, R.G.; Tai, Y.W.; Cho, S.Y.; Tan, P.; Lin, S. Semantic colorization with internet images. ACM Trans. Graph.
**2011**, 30, 156. [Google Scholar] [CrossRef] - Zhang, R.; Isola, P.; Efros, A.A. Colorful image colorization. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2016; pp. 1–16. [Google Scholar]
- Smythe, D.B. A Two-Pass Mesh Warping Algorithm for Object Transformation and Image Interpolation; Technical report; ILM Technical Memo Department, Lucasfilm Ltd.: San Francisco, CA, USA, 1990. [Google Scholar]
- Wolberg, G. Digital Image Warping; IEEE Computer Society Press: Los Alamitos, CA, USA, 1990; Volume 10662. [Google Scholar]
- Wolberg, G. Image morphing: A survey. Vis. Comput.
**1998**, 14, 360–372. [Google Scholar] [CrossRef] - Miller, M.I.; Younes, L. Group actions, homeomorphisms, and matching: A general framework. Int. J. Comput. Vis.
**2001**, 41, 61–84. [Google Scholar] [CrossRef] - Trouvé, A.; Younes, L. Local geometry of deformable templates. SIAM J. Math. Anal.
**2005**, 37, 17–59. [Google Scholar] [CrossRef] - Trouvé, A.; Younes, L. Metamorphoses through Lie group action. Found. Comput. Math.
**2005**, 5, 173–198. [Google Scholar] [CrossRef] - Christensen, G.E.; Rabbitt, R.D.; Miller, M.I. Deformable templates using large deformation kinematics. IEEE Trans. Image Process.
**1996**, 5, 1435–1447. [Google Scholar] [CrossRef] [PubMed] - Dupuis, P.; Grenander, U.; Miller, M.I. Variational problems on flows of diffeomorphisms for image matching. Q. Appl. Math.
**1998**, 56, 587–600. [Google Scholar] [CrossRef] - Trouvé, A. An infinite dimensional group approach for physics based models in pattern recognition. Int. J. Comput. Vis.
**1995**, 28, 213–221. [Google Scholar] [CrossRef] - Trouvé, A. Diffeomorphisms groups and pattern matching in image analysis. Int. J. Comput. Vis.
**1998**, 28, 213–221. [Google Scholar] [CrossRef] - Beg, M.F.; Miller, M.I.; Trouvé, A.; Younes, L. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis.
**2005**, 61, 139–157. [Google Scholar] [CrossRef] - Richardson, C.L.; Younes, L. Metamorphosis of images in reproducing kernel Hilbert spaces. Adv. Comput. Math.
**2016**, 42, 573–603. [Google Scholar] [CrossRef] - Hong, Y.; Joshi, S.; Sanchez, M.; Styner, M.; Niethammer, M. Metamorphic geodesic regression. Med. Image Comput. Comput. Assist. Interv.
**2012**, 15, 197–205. [Google Scholar] [PubMed] - Younes, L. Shapes and Diffeomorphisms; Springer: Berlin, Germany, 2010. [Google Scholar]
- Miller, M.I.; Trouvé, A.; Younes, L. Hamiltonian systems and optimal control in computational anatomy: 100 years since D’Arcy Thompson. Annu. Rev. Biomed. Eng.
**2015**, 17, 447–509. [Google Scholar] [CrossRef] [PubMed] - Miller, M.I.; Trouvé, A.; Younes, L. On the metrics and Euler-Lagrange equations of computational anatomy. Annu. Rev. Biomed. Eng.
**2002**, 4, 375–405. [Google Scholar] [CrossRef] [PubMed] - Berkels, B.; Effland, A.; Rumpf, M. Time discrete geodesic paths in the space of images. SIAM J. Imaging Sci.
**2015**, 8, 1457–1488. [Google Scholar] [CrossRef] - Christensen, G.E.; Johnson, H.J. Consistent image registration. IEEE Trans. Med. Imaging
**2001**, 20, 568–582. [Google Scholar] [CrossRef] [PubMed] - Fischer, B.; Modersitzki, J. Curvature based image registration. J. Math. Imaging Vis.
**2003**, 18, 81–85. [Google Scholar] [CrossRef] - Haber, E.; Modersitzki, J. A multilevel method for image registration. SIAM J. Sci. Comput.
**2006**, 27, 1594–1607. [Google Scholar] [CrossRef] - Han, J.; Berkels, B.; Droske, M.; Hornegger, J.; Rumpf, M.; Schaller, C.; Scorzin, J.; Urbach, H. Mumford–Shah Model for one-to-one edge matching. IEEE Trans. Image Process.
**2007**, 16, 2720–2732. [Google Scholar] [CrossRef] [PubMed] - Modersitzki, J. Numerical Methods for Image Registration; Oxford University Press: Oxford, UK, 2004. [Google Scholar]
- Modersitzki, J. FAIR: Flexible Algorithms for Image Registration; SIAM: Philadelphia, PA, USA, 2009. [Google Scholar]
- Deledalle, C.A.; Papadakis, N.; Salmon, J.; Vaiter, S. CLEAR: Covariant LEAst-square Refitting with applications to image restoration. SIAM J. Imaging Sci.
**2017**, 10, 243–284. [Google Scholar] [CrossRef] - Miller, M.I.; Christensen, G.E.; Amit, Y.; Grenander, U. Mathematical textbook of deformable neuroanatomies. Proc. Natl. Acad. Sci. USA
**1993**, 90, 11944–11948. [Google Scholar] [CrossRef] [PubMed] - Yuan, J.; Schnörr, C.; Steidl, G. Simultaneous higher order optical flow estimation and decomposition. SIAM J. Sci. Comput.
**2007**, 29, 2283–2304. [Google Scholar] [CrossRef] - Bertalmio, M. Image Processing for Cinema; CRC Press: Boca Raton, FL, USA, 2014. [Google Scholar]
- Nikolova, M.; Steidl, G. Fast Hue and Range Preserving Histogram Specification: Theory and New Algorithms for Color Image Enhancement. IEEE Trans. Image Process.
**2014**, 23, 4087–4100. [Google Scholar] [CrossRef] [PubMed] - Hertzmann, A.; Jacobs, C.E.; Oliver, N.; Curless, B.; Salesin, D.H. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA, 12–17 August 2001; pp. 327–340. [Google Scholar]
- Peter, P.; Kaufhold, L.; Weickert, J. Turning diffusion-based image colorization into efficient color compression. IEEE Trans. Image Process.
**2016**, 26, 860–869. [Google Scholar] [CrossRef] [PubMed] - Chambolle, A.; Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis.
**2011**, 40, 120–145. [Google Scholar] [CrossRef] - Joshi, S.C.; Miller, M.I. Landmark matching via large deformation diffeomorphisms. IEEE Trans. Image Process.
**2000**, 9, 1357–1370. [Google Scholar] [CrossRef] [PubMed] - Chen, D.; Hua, G.; Wen, F.; Sun, J. Supervised transformer network for efficient face detection. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 122–138. [Google Scholar]

**Figure 1.**Illustration of the image path and the diffeomorphism path, where ${I}_{0}:={I}_{\mathrm{temp}}$ and ${I}_{K}:={I}_{\mathrm{tar}}$ are the given template and target images.

**Figure 3.**Overview of the color transfer. The mapping $\mathsf{\Phi}$ is computed from Model (3) between the luminance channel of the source image and the target one. From this map, the chrominances of the source image are mapped. Finally, from these chrominances and the target image the colorization result is computed.

**Figure 4.**Illustration of the colorization steps of our algorithm. (

**a**) The color source image; (

**b**) The gray value target image; (

**c**) The transport of the $R,G,B$ channel via the morphing map is not suited for colorization; (

**d**) The result with our morphing method is already very good; (

**e**) It can be further improved by our variational post-processing.

**Figure 5.**Comparison of our approach with state-of-the-art methods on photographies. In contrast to these methods our model is not based on texture comparisons, but on the morphing of the shapes. Therefore it is able to handle faces images, where the background has frequently a similar texture as the skin.

**Figure 6.**Results including painted faces. Only our morphing method is able to colorize the target images in an authentic way.

**Figure 8.**Results on a color image turned into a gray-scale one for a quantitative comparison. The qualitative comparisons with the state-of-the-art methods are confirmed by the PSNR measures.

**Figure 9.**Multiple colorizations of known RGB-images with difference plots measured in Euclidean distance in ${\mathbb{R}}^{3}$.

Image | $\mathit{\mu}$ | K | $\mathit{\gamma}$ | $\mathit{\alpha}$ |
---|---|---|---|---|

Figure 5-1. row | 0.025 | 24 | 50 | 0.005 |

Figure 5-2. row | 0.05 | 24 | 25 | 0.005 |

Figure 5-3. row | 0.05 | 12 | - | - |

Figure 5-4. row | 0.05 | 24 | - | - |

Figure 6-1. row | 0.005 | 32 | - | - |

Figure 6-2. row | 0.0075 | 18 | 50 | 0.05 |

Figure 6-3. row | 0.04 | 24 | - | - |

Figure 7 | 0.0075 | 18 | - | - |

Figure 8 | 0.01 | 25 | - | - |

Figure 9-1. row | 0.005 | 25 | - | - |

Figure 9-2. row | 0.01 | 25 | - | - |

Figure 9-3. row | 0.01 | 25 | - | - |

**Table 2.**Comparison of the different PSNR values for the image pairs in Figure 9.

Gray | Welsh et al. [8] | Gupta et al. [6] | Pierre et al. [3] | Our | |
---|---|---|---|---|---|

Figure 9-1. row 1. pair | 24.8023 | 20.0467 | 26.3527 | 33.7694 | 44.7808 |

Figure 9-1. row 2. pair | 24.5218 | 23.9513 | 25.9457 | 34.4231 | 45.4682 |

Figure 9-2. row 1. pair | 24.3784 | 22.6729 | 27.5586 | 32.0119 | 41.1413 |

Figure 9-2. row 2. pair | 23.7721 | 23.2375 | 25.9375 | 30.1398 | 39.4254 |

Figure 9-3. row 1. pair | 24.5950 | 30.3985 | 24.3112 | 31.5263 | 42.3861 |

Figure 9-3. row 2. pair | 24.3907 | 27.7816 | 25.6207 | 31.8982 | 42.4092 |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Persch, J.; Pierre, F.; Steidl, G. Exemplar-Based Face Colorization Using Image Morphing. *J. Imaging* **2017**, *3*, 48.
https://doi.org/10.3390/jimaging3040048

**AMA Style**

Persch J, Pierre F, Steidl G. Exemplar-Based Face Colorization Using Image Morphing. *Journal of Imaging*. 2017; 3(4):48.
https://doi.org/10.3390/jimaging3040048

**Chicago/Turabian Style**

Persch, Johannes, Fabien Pierre, and Gabriele Steidl. 2017. "Exemplar-Based Face Colorization Using Image Morphing" *Journal of Imaging* 3, no. 4: 48.
https://doi.org/10.3390/jimaging3040048