Guided Facial Skin Color Correction

Abstract

This paper proposes an automatic image correction method for portrait photographs, which promotes consistency of facial skin color by suppressing skin color changes due to background colors. In portrait photographs, skin color is often distorted due to the lighting environment (e.g., light reflected from a colored background wall and over-exposure by a camera strobe). This color distortion is emphasized when artificially synthesized with another background color, and the appearance becomes unnatural. In our framework, we, first, roughly extract the face region and rectify the skin color distribution in a color space. Then, we perform color and brightness correction around the face in the original image to achieve a proper color balance of the facial image, which is not affected by luminance and background colors. Our color correction process attains natural results by using a guide image, unlike conventional algorithms. In particular, our guided image filtering for the color correction does not require a perfectly-aligned guide image required in the original guide image filtering method proposed by He et al. Experimental results show that our method generates more natural results than conventional methods on not only headshot photographs but also natural scene photographs. We also show automatic yearbook style photo generation as another application.

1. Introduction

Portrait photographs sometimes get undesirable color due to a background color reflection. In particular, changes in skin color sometimes give a bad impression, so that professional cameramen arrange lighting conditions to make the skin color and brightness uniform for each photograph. Meanwhile, for an amateur, more effort is required to achieve the same quality. Therefore, there is a need for a simple and automatic method not requiring any special technique to unify the color and brightness between multiple images. The undesirable color on portrait photographs is caused by the following:

• Each camera has its own camera response sensitivity, producing a different distribution of skin color in a color space.
• Background color is usually reflected to faces causing distorted colors.
• When each subject wears different colored clothes, color correction for the whole image region distorts the skin color, while skin color correction discolors the clothes.

To address the problems, we need a color correction scheme that affects only the face region.

In this paper, we attempt to correct an image with various color and brightness conditions with a guide image. Our method has various applications, e.g., the creation of a photo name list such as a yearbook without much effort or cost. If ones try to gather photographs taken with different cameras in different lighting conditions, correcting the images becomes a very difficult task even using off-the-shelf image processing software. In the situation considered, we need to unify several attributes of all images: the size of cropped facial image, facial skin color, brightness, and background color. Automatic correction of the attributes requires

(1)

Face detection and Facial skin color extraction;

(2)

Facial skin color correction.

A simple combination of these techniques gives boundary artifacts between the corrected region and the uncorrected region because the color correction only corrects the extracted region. This is described in Section 2.

In addition to the algorithm described in this paper, we propose a hybrid guided image filtering (GIF) method (the red box in Figure 1), which performs color transfer on the local face region while keeping colors in other regions, making GIF suitable for the correction of facial skin color and brightness. Because region extraction and color transfer are individual procedures, color transfer on part of an image generates a color gap between the original color part and the color transformed part. The hybrid GIF transforms a part of the image as an object based color transfer. In contrast to the fact that the original GIF [1] needs a perfectly-aligned guide image, our GIF only needs roughly aligned pixels. As for non-aligned regions, the filtering is achieved by aligned region propagation such as colorization [2] and matting [3]. Our method carries out nonlinear correction only on the face region and achieves better results than conventional methods. A preliminary version of this study without several improvements or new applications appeared in conference proceedings [4].

The rest of this paper is organized as follows: Section 2 introduces related work on color correction and facial image correction. Section 3 describes the proposed guide facial skin color correction method. The proposed algorithm is evaluated in experiments by comparison with various conventional methods in Section 4. Finally, this paper is concluded in Section 5.

2. Related Work

Reference based color correction is one of the essential research issues for image editing. Color grading methods [5,6,7] are usually used for color correction among multiple images. The method matches the color attributes of the input image, e.g., color distribution and tone curve for each color, with those of the target (reference) image. These existing methods, however, are for global image correction, and not suitable for local face area correction, and also assume that subjects wear similar color clothes. Colorization methods [2,8] are available for this purpose. By specifying coloring information for pixels, the method changes colors of a subject. However, the method requires a lot of coloring information to obtain natural results. Another possible approach is to apply color transfer based on GIF with guide images [1,9,10,11], where coloring information is collectively specified as a guide image. GIF, however, requires a perfectly aligned pair of input and guide images without any position gap, which is often not the case for the situation considered.

As for facial image correction methods, a wrinkle removal method is proposed in [12], but the purpose differs from color grading of facial skin color. The method proposed in [13] can also transfer the style of the original image to the target one. However, it also edits clothes and performs well only for headshot photos.

3. Proposed Method

Our algorithm transforms colors of a part of an input image using a target image color unlike general color transfer methods and corrects surrounding colors of the transformed part. Figure 1 illustrates the flow chart of our method. Our method mainly consists of two parts:

(1)

Face detection and Facial skin color extraction (the yellow box in Figure 1): It detects the face area and extracts its facial skin color.

(2)

Skin color correction (green box): The facial skin color extracted is corrected using the target image (a) in the color space. Then, the color of the face region is corrected using the image (b) as the guide image in the image space.

Note that this paper uses the two phrases: skin color and facial skin color. Skin color refers to the skin color of the whole body. Facial skin color is the color of the skin in the face region only. Additionally, region and area are distinguished. Area refers to the detected face area in Section 3.1. Region refers to the extracted facial skin color region in Section 3.2.

The novelty of the proposed method lies in (2), where the color correction with grading affects only a part of the image. The other steps consist of conventional methods with some modifications. Hereafter, we describe each procedure in more detail.

Incidentally, the accuracy of the conventional methods in (1) is sufficient for most of the images shown in this paper but sometimes affects the quality of the resulting images, and use of recent methods may improve the quality of the resulting images. We show that in Section 4.4 and Figure 16 as Limitations.

3.1. Face Detection

This step detects an area from head to shoulder (see Figure 2) by using the Viola–Jones algorithm [14], and the implementation in OpenCV as Haar-cascade detection for the face detection. We also examined the face detector in Dlib [15] and the face parsing methods based on deep learning, e.g., BiSeNet [16]. However, we chose the Viola–Jones method [14] because the detected data and parameters can be easily used, and the accuracy is sufficient if the face is facing closer to the front. The face area is described by n rectangle windows as shown in Figure 2b, where n is the number of candidates. As the parameters of each candidate, the barycentric coordinates $(x,y)$ and the width and height of a detected rectangle $(w,h)$ are obtained. We find the median of $x,y,w,h$ as

$$\begin{array}{cccc}\hfill \widehat{x}& :=\mathrm{median}\left(\{x_1,\dots ,x_n\}\right),\hfill & \hfill \widehat{w}& :=\mathrm{median}\left(\{w_1,\dots ,w_n\}\right),\hfill \\ \hfill \widehat{y}& :=\mathrm{median}\left(\{y_1,\dots ,y_n\}\right),\hfill & \hfill \widehat{h}& :=\mathrm{median}\left(\{h_1,\dots ,h_n\}\right),\hfill \end{array}$$

(1)

where the subscript is the index of the candidate. As the final face area, the rectangle with the median values of the barycentric coordinate $(\widehat{x},\widehat{y})$ is adopted. Finally, the size of the face area is adjusted as follows using the size of the face area $(\widehat{w},\widehat{h})$ and the prescribed scale factor l:

$${\displaystyle \left(2l\widehat{w}+1\right)}\times {\displaystyle \left(2l\widehat{w}+1\right)}.$$

(2)

3.2. Skin Color Extraction

This step roughly extracts the skin color region near the face. We use a common approach using clustering in the color space and morphological labeling in the image space:

(1)

The color of each pixel is classified by the color distribution of the entire image in the HSV color space. Each pixel is assigned the label of the cluster to which it belongs.

(2)

Some regions are generated by concatenating neighboring pixels with the same labels in the image space. Regions mainly in the detected face area ${\mathsf{\Omega }}_{\mathrm{rect}}$ (Equation (2)) are extracted, and regarded as the facial skin region.

As for the method for clustering color distributions, since the skin color in the detected face area (Section 3.1) is known, simpler approaches, e.g., k-means method [17] may give satisfactory results. We use k-means++ [18] to select stable seed clusters.

We use the above k-means clustering [18] in this paper. Additionally, for robustness to value (luminance) and saturation variation and also simplicity, we only use the hue values for clustering the 1D distribution. The aforementioned procedures (i) and (ii) are performed to detect the face region ${\mathsf{\Omega }}_S^{\mathit{hue}}$. Then, at each pixel contained in the region ${\mathsf{\Omega }}_S^{\mathit{hue}}$, the saturation and value components are thresholded. The condition for the skin color at pixel p is experimentally defined as follows, and the skin color region ${\mathsf{\Omega }}_S$ is obtained:

$${\mathsf{\Omega }}_S:=\left\{p\mid p\in {\mathsf{\Omega }}_S^{hue},s_p\in [\widehat{s}-0.2,\widehat{s}+0.2],v_p\in [0.15,0.95]\right\},$$

(3)

where $\widehat{s}$ is the median of the saturation in the face area. The saturation and the value are normalized to be in the range $[0,1]$ and the threshold values are set to excellently extract the facial skin color region on our dataset, which is available on the project page. The number of clusters depends on the photographic environment, especially if the skin color region needs to be perfectly extracted to the edge of the face because the edge color tends to change due to light reflection. However, the post-processing can handle this even in the case of rough extraction. We experimentally set $k=4$ in this paper.

For our guide image filtering, we define ${\mathsf{\Omega }}_B$ and ${\mathsf{\Omega }}_{\partial S}$ using the dilation function, which is a type of morphological operation. They are given as

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}_S^{\prime }& :=\mathrm{dilation}\left({\mathsf{\Omega }}_S\right),\hfill \\ \hfill {\mathsf{\Omega }}_B& :={\overline{{\mathsf{\Omega }}^{\prime }}}_S,\hfill \\ \hfill {\mathsf{\Omega }}_{\partial S}& :={\mathsf{\Omega }}_S^{\prime }\backslash {\mathsf{\Omega }}_S,\hfill \end{array}$$

(4)

where $\mathrm{dilation}(·)$ is the dilation function with a structuring element, which consists of a circle with a 20 pixel radius. The overline $\overline{·}$ is the complement and the operator ∖ is the set difference. An example of each region can be seen in Figure 3.

3.3. Color Grading

Figure 4 shows an example of color grading. Color grading brings the skin color in (b), the extracted region ${\mathsf{\Omega }}_S$, close to the target facial skin color in (a). More specifically, this method transforms the shapes of the color distributions, i.e., $(r,g,b)$ 3D coordinates, in the RGB color space. Note that we use the RGB color space instead of the HSV color space used in the preceding section. The reason is that the color grading method [5] has been developed for the RGB color space and the color-line image feature used in the next section is a feature related to the RGB color space. As a simpler technique, one may use a 1D tone curve to transfer each RGB component between the two images (see Section 3.1 in [5]). However, to improve the accuracy, most of the methods require pixel correspondences between the images [6], which is unsuitable in our case that handles the faces of different persons. As a method that does not require the correspondences, we employ Pitié et al.’s method [5]. The method realizes nonlinear transformation of the color distribution by iteratively changing the width of the color distribution along a particular axis chosen in each iteration so that the shape of the marginal distribution projected onto the axis matches that of the target distribution. The method, however, yields some artifacts, e.g., blurring artifacts around the edges, and requires a correction method such as joint/cross filtering as used in [11]. As the correction method, we describe our GIF in the next section.

3.4. Guide Image Filtering via Optimization

GIF is a sort of joint filtering (also called cross filtering) that uses a guide image to correct the input image. The energy function of GIF is based on the image matting method [3]. The data fidelity term is also known as the local linear model, and we focus on the data fidelity term, but the design objective is different. As a similar method, ref. [19] has been proposed in high dynamic range imaging.

Our guide image filtering reconstructs an image $\mathbf{x}\in {\mathbb{R}}^{3N}$ where the facial skin color region has a corrected color by using the input image $\mathbf{y}\in {\mathbb{R}}^{3N}$ and the color grading result, where N is a number of pixels. In the RGB color space (${\mathbb{R}}^3$), let the pixel value at pixel j of the color corrected image to be solved be ${\mathbf{x}}_j$, the input image be ${\mathbf{y}}_j$, and the guide image be ${\mathbf{g}}_j$ which are given by the color grading in Section 3.3. Figure 5 shows each image.

Using them, we formulate our guide image filtering as the following convex optimization problem:

$${\displaystyle \underset{\mathbf{x},\mathbf{A},\mathbf{b}}{min}\underset{\mathrm{data}\mathrm{fidelity}}{\underbrace{\sum _i\left(\sum _{j\in w_i}\parallel {\mathbf{x}}_j-{\mathbf{A}}_i{\mathbf{y}}_j-{\mathbf{b}}_i{\parallel }_2^2+\epsilon {\parallel {\mathbf{A}}_i\parallel }_F^2\right)}}+\underset{\mathrm{constraints}}{\underbrace{{\iota }_{{\mathcal{C}}_S}\left(\mathbf{x}\right)+{\iota }_{{\mathcal{C}}_B}\left(\mathbf{x}\right)}}.}$$

(5)

First, the data fidelity terms, which correspond to [3], reflect textures and local contrasts of $\mathbf{y}$ onto $\mathbf{x}$. As for the symbols, ${\mathbf{A}}_i\in {\mathbb{R}}^{3\times 3}$ and ${\mathbf{b}}_i\in {\mathbb{R}}^3$ are a scaling matrix and an offsetting vector to approximate ${\mathbf{y}}_j$ to ${\mathbf{x}}_j$ for each pixel i. $w_i$ is a square window around pixel i. ${\parallel ·\parallel }_2$ and ${\parallel ·\parallel }_F$ denote the ${\ell }_2$ norm and Frobenius norm, respectively. Next, the constraint terms are expressed by the following indicator function and conditions:

$${\iota }_{\mathcal{C}}\left(\mathbf{x}\right):=\left\{\begin{array}{cc}0\hfill & \mathrm{if}\mathbf{x}\in \mathcal{C},\hfill \\ +\infty \hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(6)

${\mathcal{C}}_S$ and ${\mathcal{C}}_B$ are conditions and given by

$$\begin{array}{cc}\hfill {\mathcal{C}}_S& :=\left\{\mathbf{x}\mid \sqrt{{\sum }_{p\in {\mathsf{\Omega }}_S}{\parallel {\mathbf{x}}_p-{\mathbf{g}}_p\parallel }_2^2}\le {\eta }_S\right\},\hfill \\ \hfill {\mathcal{C}}_B& :=\left\{\mathbf{x}\mid \sqrt{{\sum }_{q\in {\mathsf{\Omega }}_B}{\parallel {\mathbf{x}}_q-{\mathbf{y}}_q\parallel }_2^2}\le {\eta }_B\right\}.\hfill \end{array}$$

(7)

The term ${\iota }_{{\mathcal{C}}_S}\left(\mathbf{x}\right)$ works on the facial skin color region ${\mathsf{\Omega }}_S$, and brings the facial skin color close to the guide image $\mathbf{g}$. The term ${\iota }_{{\mathcal{C}}_B}\left(\mathbf{x}\right)$ works on the background region ${\mathsf{\Omega }}_B$, and keeps the color of the background the same as the original image $\mathbf{y}$. For reducing undesirable artifacts, we design the constraint that the summation of the color differences obtained at each pixel does not exceed $\eta$.

Note that we purposefully represent the second and third terms as constraint formulation over unconstrained one, e.g., the second term can be replaced with a regularizer $\lambda \sqrt{{\sum }_{p\in {\mathsf{\Omega }}_S}{\parallel {\mathbf{x}}_p-{\mathbf{g}}_p\parallel }_2^2}$, where $\lambda$ is a balancing parameter. This is because $\eta$ can be controlled more intuitively than $\lambda$, and it allows us to adaptively change $\eta$ depending on the area of ${\mathsf{\Omega }}_s$ (see the next Section 4). Such advantages of the formulation have been addressed in the literature of image restoration based on convex optimization [20,21,22,23,24,25,26].

Among convex optimization algorithms, we adopt a monotone version of the fast iterative shrinkage-thresholding algorithm (MFISTA) [27] to solve (5) because it is a first-order method that achieves a sublinear global rate of convergence, namely computationally efficient and very fast. The algorithm solves an optimization problem of the form:

$$\underset{\mathbf{x}}{min}f\left(\mathbf{x}\right)+h\left(\mathbf{x}\right),$$

(8)

where $f\left(\mathbf{x}\right)$ is a differentiable convex function with a Lipschitz continuous gradient and $h\left(\mathbf{x}\right)$ is a proper lower semicontinuous convex function. The problem (8) is calculated by the proximity operator that is defined by

$${\mathrm{prox}}_{\kappa h}\left(y\right):=arg\underset{x}{min}\kappa h\left(x\right)+\frac{1}{2}{\parallel y-x\parallel }_2^2,$$

(9)

where $h(·)$ is a proper lower semicontinuous convex function, and $\kappa$ is the index. For given ${\mathbf{x}}^0=:{\mathbf{v}}^1\in {\mathbb{R}}^N$ and $t^1:=1$, the iteration of the MFISTA consists of the following five steps:

$$⌊\begin{array}{cc}\hfill {\widehat{\mathbf{z}}}^k& {\displaystyle ={\mathbf{v}}^k-\frac{1}{L}\nabla f\left({\mathbf{v}}^k\right),}\hfill \\ \hfill {\mathbf{z}}^k& {\displaystyle ={\mathrm{prox}}_{h/L}\left({\widehat{\mathbf{z}}}^k\right),}\hfill \\ \hfill {\mathbf{x}}^k& {\displaystyle =argmin\{f\left(\mathbf{z}\right)+h\left(\mathbf{z}\right)|\mathbf{z}\in \{{\mathbf{z}}^k,{\mathbf{x}}^{k-1}\}\},}\hfill \\ \hfill t^{k+1}& {\displaystyle =\frac{1+\sqrt{1+4{\left(t^k\right)}^2}}{2},}\hfill \\ \hfill {\mathbf{v}}^{k+1}& {\displaystyle ={\mathbf{x}}^k+\frac{t^k}{t^{k+1}}({\mathbf{z}}^k-{\mathbf{x}}^k)+\frac{t^k-1}{t^{k+1}}({\mathbf{x}}^k-{\mathbf{x}}^{k-1}),}\hfill \end{array}$$

(10)

where “prox” is the proximity operator and $1/L$ is the step size.

In order to apply the MFISTA to our problem (5), we set

$$\begin{array}{cc}\hfill f\left(\mathbf{x}\right)& {\displaystyle :=\sum _i\left(\sum _{j\in w_i}\parallel {\mathbf{x}}_j-{\mathbf{A}}_i{\mathbf{y}}_j-{\mathbf{b}}_i{\parallel }_2^2+\epsilon {\parallel {\mathbf{A}}_i\parallel }_F^2\right),}\hfill \\ \hfill h\left(\mathbf{x}\right)& :={\iota }_{{\mathcal{C}}_S}\left(\mathbf{x}\right)+{\iota }_{{\mathcal{C}}_B}\left(\mathbf{x}\right).\hfill \end{array}$$

(11)

To compute the gradient $\nabla f\left(\mathbf{x}\right)$, we use a method similar to [28], which is an accelerated version of [3]. We compute each value of ${[\nabla f\left(\mathbf{x}\right)]}_i\in {\mathbb{R}}^3$ as follows:

$$\begin{array}{cc}\hfill {\mathbf{A}}_i^{*}& ={\Delta }_i^{-1}\left(\frac{1}{|w_i|}\left({\displaystyle \sum _{j\in w_i}{\mathbf{y}}_j{\mathbf{x}}_j^{\top }}\right)-{\overline{\mathbf{y}}}_i{\overline{\mathbf{x}}}_i^{\top }\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathbf{b}}_i^{*}& ={\overline{\mathbf{x}}}_i-{\mathbf{A}}_i^{*}{\overline{\mathbf{y}}}_i,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {[\nabla f\left(\mathbf{x}\right)]}_i& =|w_i|{\mathbf{y}}_i-\left(\left(\sum _{j\in w_i}{\mathbf{A}}_j^{*}\right){\mathbf{y}}_i+\left(\sum _{j\in w_i}{\mathbf{b}}_j^{*}\right)\right),\hfill \end{array}$$

(14)

where ${\overline{\mathbf{x}}}_i={\left[{\overline{x}}_i^R{\overline{x}}_i^G{\overline{x}}_i^B\right]}^{\top }$ and ${\overline{\mathbf{y}}}_i={\left[{\overline{y}}_i^R{\overline{y}}_i^G{\overline{y}}_i^B\right]}^{\top }$ are mean value vectors of each color in a square window $w_i$, $|w_i|$ is the number of pixels in $w_i$, $\Delta ={\mathsf{\Sigma }}_i+\frac{\epsilon }{|w_i|}\mathbf{U}\in {\mathbb{R}}^{3\times 3}$, ${\mathsf{\Sigma }}_i=\frac{1}{|w_i|}\left({\sum }_{j\in w_i}{\mathbf{y}}_j{\mathbf{y}}_j^{\top }\right)-{\overline{\mathbf{y}}}_i{\overline{\mathbf{y}}}_i^{\top }$ is a covariance matrix, and $\mathbf{U}$ is an identity matrix.

The proximity operator ${\mathrm{prox}}_{h/L}$ in (10) consists of two functions which are the second and third terms of (5) which handle different regions, ${\mathsf{\Omega }}_S$ and ${\mathsf{\Omega }}_B$, respectively, that satisfies, i.e., ${\mathsf{\Omega }}_S\cap {\mathsf{\Omega }}_B=\left\{0\right\}$. Therefore, ${\mathrm{prox}}_{h/L}$ can be calculated using

$$\begin{array}{cc}\hfill & {\mathbf{z}}_i:={\left[{\mathrm{prox}}_{h/L}\left(\widehat{\mathbf{z}}\right)\right]}_i=\hfill \\ \hfill & \left\{\begin{array}{cc}{\displaystyle {\widehat{\mathbf{z}}}_i+{\eta }_S\frac{{\widehat{\mathbf{z}}}_i-{\mathbf{g}}_i}{\sqrt{{\sum }_{p\in {\mathsf{\Omega }}_S}{\parallel {\widehat{\mathbf{z}}}_p-{\mathbf{g}}_p\parallel }_2^2}}}\hfill & \mathrm{if}\widehat{\mathbf{z}}\notin {\mathcal{C}}_S\wedge i\in {\mathsf{\Omega }}_S,\hfill \\ {\displaystyle {\widehat{\mathbf{z}}}_i+{\eta }_B\frac{{\widehat{\mathbf{z}}}_i-{\mathbf{y}}_i}{\sqrt{{\sum }_{q\in {\mathsf{\Omega }}_B}{\parallel {\widehat{\mathbf{z}}}_q-{\mathbf{y}}_q\parallel }_2^2}}}\hfill & \mathrm{if}\widehat{\mathbf{z}}\notin {\mathcal{C}}_B\wedge i\in {\mathsf{\Omega }}_B,\hfill \\ {\widehat{\mathbf{z}}}_i\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(15)

This process corresponds to a ${\ell }_2$-ball projection with a region constraint.

Finally, we update $\mathbf{x},\mathbf{z},\mathbf{v},t$ using the procedure in (10), then the solution $\mathbf{x}$ becomes our GIF result.

4. Results and Discussion

In this section, we show the results obtained through the proposed process. The experiment tests our algorithm on various sets, which are available from our website http://wwell.cs.shinshu-u.ac.jp/~keiichi/projects/auto_album (accessed on 14 August 2021). The range of RGB values is normalized to $[0,1]$. The prescribed scale factor l in Section 3.1 is set to $2.0$. The filter window sizes used in Section 3.4 and Appendix A are $19\times 19$ and $31\times 31$, respectively. ${\eta }_S=5\left|{\mathsf{\Omega }}_S\right|\times {10}^{-4}$ and ${\eta }_B=5\left|{\mathsf{\Omega }}_B\right|\times {10}^{-10}$ are used, where $|{\mathsf{\Omega }}_{(·)}|$ indicates the number of pixels contained in the region ${\mathsf{\Omega }}_{(·)}$. L in MFISTA is set to 500.

Our filtering often flattens the gradation of the input images caused by shadows. It yields unnatural images as shown in the top of Figure 6, where the input image is input to our algorithm, and the filtered image is a result of our guide image filtering. For the luminance correction, each pixel color value of the image is decomposed into a color component ${\mathbf{x}}_i^C\in {\mathbb{R}}^3$ and an intensity component $x_i^I\in {\mathbb{R}}^1$ as follows:

$$\begin{array}{cc}\hfill x_i^I& {\displaystyle =\frac{x_i^R}{\parallel {\mathbf{x}}_i{\parallel }_1}{x}_i^R+\frac{x_i^G}{\parallel {\mathbf{x}}_i{\parallel }_1}{x}_i^G+\frac{x_i^B}{\parallel {\mathbf{x}}_i{\parallel }_1}{x}_i^B,}\hfill \\ \hfill {\mathbf{x}}_i^C& {\displaystyle =\frac{{\mathbf{x}}_i}{x_i^I},}\hfill \end{array}$$

(16)

where the superscripts R, G and B indicate each color component. This decomposition procedure is the same as [10].

The input image and the filtered image are decomposed into the two components by (16), and then the intensity component of the input image $y_i^I$ and the color component of the filtered image ${\mathbf{x}}_i^C$ are combined as follows:

$${{\mathbf{x}}_i}^{\prime }=y_i^I{\mathbf{x}}_i^C.$$

(17)

Figure 6 shows this procedure and its effectiveness.

Figure 7 shows the result of our skin color correction. The area around the face in the input image has color distortion due to the lighting conditions and background color. Our result has a white balanced facial skin color similar to the target facial skin color. Sometimes, when we take a photograph in dark surroundings, the results sometimes have unnatural face colors (Figure 8b) due to the camera flash. Hence, we apply our proposed method to flash images in dark surroundings to reduce undesirable effects of artificial lights. Figure 8c shows the flash image editing results using our method. One can see that unnatural colors of the original image are corrected to natural colors by our method.

4.1. Automatic Yearbook Style Photo Generation

This section presents automatic yearbook style photo generation using our guided facial skin correction method and pre/post-processing procedure. It takes a long time to manually process a large number of images. Our algorithm generates a yearbook style photo in a short amount of time.

We first crop a photo with our face detection procedure as a pre-processing procedure (the red box in Figure 9), and then correct the facial skin color. Finally, the noisy background is replaced to clear the background as a post-processing procedure (the blue boxes in Figure 9).

4.1.1. Face Area Cropping

After the face area detection in Section 3.1, we resize the cropped images so that the sizes are unified among images. The crop size is roughly adjusted according to the image size. The final image size is $320\times 320$ ($h\times w$) in this experiment.

4.1.2. Background Replacement by Alpha Blending

We separate the image information of the background and foreground regions and assign a value (soft label) ${\alpha }_i:=[0,1]$ to them, where ${\alpha }_i=0$ and ${\alpha }_i=1$, respectively, denote the background and the foreground. Additionally, we define foreground, background, and original image, respectively, as ${\mathbf{f}}_i$, ${\mathbf{b}}_i$, and ${\mathbf{y}}_i$ (different from that in (5)). The relationship among them is given as follows:

$${\mathbf{y}}_i:={\mathbf{f}}_i+{\mathbf{b}}_i,{\mathbf{f}}_i:=(1-{\alpha }_i){\mathbf{y}}_i,{\mathbf{b}}_i:={\alpha }_i{\mathbf{y}}_i.$$

(18)

Here, $\alpha$ also works as the blending rate, and called alpha-mat. The background replacement with another background $\mathbf{z}$ is given by

$${\mathbf{y}}_i^{\prime }:={\mathbf{f}}_i+{\alpha }_i{\mathbf{z}}_i.$$

(19)

An estimation of the alpha-mats is described in Appendix A.

Figure 10 shows the results of our automatic yearbook style photo generation. Our algorithm generates the yearbook style photos from the original images using the target image.

We implement the whole algorithm in MATLAB and OpenCV (C++), respectively, and the total execution time is within 11 s. on a 3.20 GHz Core i5 CPU, i.e., face detection (Section 3.1) is 5 s, facial color extraction (Section 3.2) is 1 s, color grading (Section 3.3) is 2 s, GIF (Section 3.4) is 2 s, and matting (Section 4.1.2) is 1 s, respectively.

4.2. Comparison with Various Conventional Methods

Figure 11 compares our method with supplement methods [2,30]. For the methods, we process each RGB color layer because they are proposed for color components in YUV color space. Since colorization [2] just spreads colors to boundaries, the result does not have the details of the original image. The joint bilateral upsampling (JBU) [30] computes pixel values in hole regions by joint-filtering. We can see a contrast reduction in the top and the middle rows in Figure 11. The blue arrows and the red arrows indicate artifacts of each method. Meanwhile, our guide image filtering outputs sharp details of the original image while keeping the guide image color.

Figure 12 shows a comparison with existing color transfer methods. For the results (c), Pitié et al.’s method [5] transforms the color distribution of the input image to the target image one, then gain noise artifacts of the color transformed input image are removed by Rabin et al.’s method [11]. The gain noise artifacts can be seen in [5]. In the results of the color grading [11] with [5], the face and the background of the resultant image have a similar color since this method is a uniform color transfer method. In addition, this method takes a long time because it needs iterative bilateral filtering. In the non-rigid dense correspondence (NRDC) method [6], the facial color is improved, but the regions around the clothes are discolored. In Jaesik et al.’s method [31], the improvement in facial color is small in some images. On the other hand, in our method (e), color correction is successfully done over the entire face while keeping the color except for the face, and the result looks more natural than those of the other methods.

Figure 13 shows an example in which the target image and the original image are the same person. We prepared the target image as the ground truth image in which the background color is white and the skin color is not affected by the background color. For the comparison methods, we selected the aforementioned NRDC [6] and JBU [30] (respectively, (b) and (d) in this figure) since they also gave good results in Figure 11 and Figure 12. Additionally, (c) shows the result of color grading [5] performed in the first step by simply pasting on the original image. (e) shows the result using He et al.’s basic GIF [1] instead of our GIF, and the image (c) is used as the guide image in combination with the input image (a). In the results of NRDC (b), the skin color tends to include unnatural colors, and the background color changes. The results of JBU (d) and basic GIF (e) are natural if the skin region can be extracted well. Otherwise, we need to manually set the processing window size for smoothing the edge where there is a color difference. Finally, one can see that our results (f) are more similar to the ground truth target images (g).

Figure 14 shows a comparison with a background replacement result of the original image and [13]. The background replacement result has a distorted facial skin color due to the background color. Shin et al.’s method transfers a style from an input image to a target image. Although the facial skin color of the result is almost the same as the target image one, this method cannot correct the skin color of a person wearing glasses to the skin color of the target image. Our algorithm generates an image which has the target facial skin color even if a person wears glasses.

4.3. Semi-Automatic Color Correction

Our method can correct other images, such as animal photographs. Our color correction method requires some regions, which are ${\mathsf{\Omega }}_S$ as a foreground and ${\mathsf{\Omega }}_B$ as a background. Many methods have been proposed to detect face regions, but accurate object detection in natural scenes is still a challenging problem.

For color correction in natural scenes such as Figure 15c, we make each region manually as (b) and (d), where the colors indicate the region corresponding to Figure 3, respectively. Finally, our method adjusts the object color of the source image to the target one automatically. Figure 15e shows natural scene color correction results, and the red box shows that our method corrects the colors of main objects without boundary artifacts. By implementing an automatic object detection method, this application will be an auto-application.

4.4. Limitations

Figure 16 shows example images where our method does not work well. The face and hair are a similar color; hence, the facial skin color extraction process gets regions other than the face region (Figure 16b). As a result, our method outputs an image with a similar color between the face and hair (Figure 16c). Actually, the facial skin color extraction process seldom gets other regions even if the subject does not have white skin, so the guide image filtering results in the same color between the face and the other extracted regions.

On the other hand, we also show the good results in Figure 17 that are obtained by replacing the methods in facial skin color extraction with a face parsing method based on BiSeNet [16], and we use an implementation published on GitHub [32,33]. In the resulting images (d), the skin color and the hair color are correctly distinguished compared with the results in Figure 16c. Therefore, the facial skin color extraction process affects the quality of the final resulting images.

Incidentally, the aforementioned face parsing method was trained for the CelebA dataset [34], which mainly contains the area around the head, and the method does not handle the images shown in Figure 7 and Figure 8, in which the head area is smaller than the image width. Therefore, another method to detect the face region is needed. For the experiment of the good results, we used the Viola–Jones method [14].

5. Conclusions

We proposed a facial skin color correction method in combination with color grading and guided image filtering. In our approach, we consider only the face region and extract this region and correct the color because, when looking at portrait photographs, we are firstly interested in the facial area, so correction focused on this area results in a good impression. The color grading step corrects the skin color using a target (guide) image in a color space, while the guided image filtering step performs the color correction in an image space. In the current method, these two steps are used in simple heuristic combination, but there is room for improving how to combine the two steps on the basis of mathematical optimization.

With regard to this, there are color segmentation methods called soft color segmentation [35,36], and the algorithm consists of color clustering and image filtering as well as our method. The algorithm framework is derived based on mathematical optimization and gives better resulting images than those of the heuristic combination of independent methods. On the basis of the methods, we try to revise the algorithm framework of the proposed method.

On the other hand, as for the part of facial skin detection and extraction, good results tend to be obtained by using face parsing methods based on deep learning. Therefore, we try to consider the combination with the methods and how to use them appropriately.