Face Image Completion Based on GAN Prior

Abstract

Face images are often used in social and entertainment activities to interact with information. However, during the transmission of digital images, there are factors that may destroy or obscure the key elements of the image, which may hinder the understanding of the image’s content. Therefore, the study of image completion of human faces has become an important research branch in the field of computer image processing. Compared with traditional image inpainting methods, deep-learning-based inpainting methods have significantly improved the results on face images, but in the case of complex semantic information and large missing areas, the completion results are still blurred, and the color of the boundary is inconsistent and does not match human visual perception. To solve this problem, this paper proposes a face completion method based on GAN priori to guide the network to complete face images by directly using the rich and diverse a priori information in the pre-trained GAN. The network model is a coarse-to-fine structure, where the damaged face images and the corresponding masks are first input to the coarse network to obtain the coarse results, and then the coarse results are input to the fine network with multi-resolution skip connections. The fine network uses the a priori information from the pre-trained GAN to guide the network to generate the face images, and finally uses the SN-PatchGAN discriminator to evaluate the completion results. The experiment is performed on the CelebA-HQ dataset. Compared with the latest three completion methods, the qualitative and quantitative experimental analysis shows that our method has obvious improvement in texture and fidelity.

1. Introduction

Image completion, also often referred to as image inpainting, is the process of using information from undamaged areas of the original image or intact images of the same type to fill in the contents of missing areas of the image. The core problem of image completion is to synthesize semantically reasonable and realistic pixels in the region of missing information with the original image.

Image inpainting methods can be roughly divided into traditional methods and deep-learning-based methods. There are two main types of traditional image inpainting methods, including diffusion-based and patch-based methods. Although diffusion-based and patch-based traditional inpainting methods are very effective in texture completion, they are not capable of face completion with semantics. With the rapid development of deep learning, the emergence of feature-learning-based image completion methods exactly compensates the shortcomings of traditional image inpainting methods. Although the learning-based approach is capable of generating reasonable content, many existing image completion techniques generate over-smoothed and/or blurry regions, failing to reproduce fine details.

In particular, Goodfellow et al. proposed Generative Adversarial Networks (GAN) [1] in 2014, which again provided new ideas for image completion. This is because GAN models trained on large-scale natural images can obtain rich texture and shape priors. For example, the progressive growth generative adversarial networks PGGAN [2] and StyleGAN [3] proposed by karras et al. have achieved outstanding results in the work of generating human faces, and they can generate realistic images from a random noise (latent vector). GAN has been widely used in image inpainting [4], image recognition [5], image denoising [6], super resolution [7] and other image processing fields.

Inspired by GLEAN [7], in an encoder–decoder architecture, a pre-trained GAN can provide a priori information to guide the image to recover texture and details. In this paper, we propose a face image completion model based on GAN priors. The model uses a two-stage generative network from coarse to fine, and uses gated convolution instead of vanilla convolution in the generative network to distinguish valid pixels from invalid pixels of the image and improve the quality of the completion results. The coarse network is a typical encoder–decoder structure, in which the dilated convolution is added to enhance the perceptual field of the features and output to get the coarse result, and then the coarse result is input to the fine network. The fine network uses the a priori information from the pre-trained GAN to guide the network to generate the face image, and finally, the SN-PatchGAN discriminator is used to evaluate the completion result. Figure 1 shows the completion results of our method from coarse to fine. The coarse result in Figure 1c generates reasonable content but lacks details, while the fine network fills in the details using a priori information, as shown in Figure 1d.

The contributions of this paper are follows:

(1)

A coarse-to-fine face completion network based on pre-trained GAN prior to improve the texture and fidelity of the completion results.

(2)

Integration of a priori information from a pre-trained GAN into an encoder-decoder architecture with multi-resolution skip connections.

The rest of the paper is structured as follows: In Section 2, related works including multiple image inpainting methods are discussed. This is followed in Section 3 by details about our methods. Section 4 describes our experiments in detail. Section 5, ablation studies. Finally, conclusions and future works are given.

2. Related Work

2.1. Traditional Image Inpainting Methods

There are two main traditional methods for image compeltion, diffusion-based methods and patch-based methods. Diffusion-based [8,9] methods fill the missing region by smoothly propagating the image content from the boundary to the interior of the missing region. This type of method only fills the texture of the image, and it is difficult to capture the semantic information and global structure of the image, so this type of method cannot inpainting face images with complex information semantics. In addition, this approach does not work in the case of large missing areas.

On the other hand, the patch-based approach [10] can fill in the missing regions in an image by searching for similar patches in the image. Patch-based methods can synthesize plausible textures, but they usually fail badly in cases such as complex scenes and human faces. Additionally, these methods do not work in complex scenes with large missing areas, especially when the texture to be filled is not present in the image. Moreover, patch-based inpainting methods are slow, computationally expensive, and not based on semantic understanding of the image.

Therefore, traditional image completion methods are not competent for face image completion.

2.2. Deep-Learning-Based Image Inpainting Methods

In recent years, image inpainting methods based on deep learning have been widely studied. Pathak et al. [4] introduced adversarial loss into the image inpainting task, and they used adversarial loss to make the generated images as realistic as possible. Based on this work, Yeh et al. [11] used DCGAN [12] for image inpainting, which was able to successfully generate missing parts of the image and fill them. Although these methods achieved satisfactory results, there were obvious stitching traces in the border areas of the completed images. He et al. [13] successfully achieved the completion of high-resolution face images using a pre-trained stylegan.

Iizuka et al. [14] added a dual discriminator (local discriminator and global discriminator) to the image completion model, and this model ensured the local consistency and global consistency of the generated images. Li et al. [15] proposed a face image completion model on this structure and further ensured that the generated content was more realistic and enhanced the consistency between old and new pixels by using a pre-trained Parsing network. Yu et al. [16] proposed a coarse-to-fine network structure based on this structure and added a contextual attention layer to copy or borrow feature information from known backgrounds to generate real pixels. This dual discriminator structure is valid for the central rectangular mask, but in practice, there may be multiple arbitrarily shaped masks at any location.

Liu et al. [17] argued that vanilla convolution indiscriminately convolves valid and masked pixels, which usually leads to situations such as color differences and blurring, and thus proposed partial convolution to improve the quality of image completion and proposed a free-form mask generation method. Yu et al. [18] proposed a gated convolution, generalized partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers.

Nazeri et al. [19] proposed a two-stage adversarial image completion model that used an edge generator to predict the edge map of an image in the first stage, and then generated the image in the second stage based on structural information using the edge map. Yang et al. [20] also used a similar structure to achieve face image completion by predicting the landmarks of faces. Such methods rely heavily on the accuracy of the prediction, and the completion results are often unsatisfactory when the mask area is large.

Deep-learning-based methods have achieved remarkable success in image completion tasks. These methods use the learned data distribution to fill in the missing pixels. They are able to generate coherent structures in the missing regions, a feat that is almost impossible to achieve with traditional techniques. Although these methods are able to generate meaningful structures in the missing regions, the generated regions sometimes suffer from blurring, lack of context, and discontinuous color of the completion boundaries.

3. Our Method

To solve these problems, in this paper we design a coarse-to-fine architecture that allows the user to output completion results using a priori information in just one pass forward. The coarse network is a typical encoder–decoder structure. Adding a gated dilation convolution operation to this coarse network increases the range of the feature perception field, which allows the model to see a larger input image area and more global information, and outputs a coarse completion result. The coarse results are then fed into the fine network, and the prior information in the pre-trained GAN is used to guide the network to generate face images, and finally the completion results are evaluated using the SN-PatchGAN discriminator. An overview of the architecture is depicted in Figure 2.

3.1. Coarse Network

The coarse network is a encoder–decoder structure with a gated dilation convolution block, because the global information is crucial for image completion, and through the dilation convolution, the model can see a larger area of the input image than when using the standard convolution layer, so the coarse network can effectively learn better feature representation. With the input damaged image and corresponding mask, a coarse result can be obtained by the coarse network.

3.2. Fine Network

The fine network consists of three components, including a multi-resolution encoder, a pre-trained GAN, and a decoder.

3.2.1. Multi-Resolution Encoder

In order to extract features and latent vectors of the images and provide a priori information for the pre-trained GAN to generate features, we use gated convolution blocks to gradually reduce the resolution of the features.

$$f_i=\left\{\begin{array}{c}\hfill E_0\left(I_{coarse}\right),ifi=0\hfill \\ E_i\left(f_{i-1}\right),elsei\in 1,\dots N\end{array}\right.$$

(1)

where $E_i$ denotes a stack of a stride-2 gated convolution and a stride-1 gated convolution. N is the number of downsampling in the encoder stage of the fine network, and in our work, N is 6. To generate the latent vector, we first downsample the input image using multiple and multi-resolution features, and finally obtain the latent vector through the fully connected layer.

3.2.2. Pre-Trained Generator

Given convolutional features $f_i$ and latent vectors Z, we utilize a pre-trained PGGAN as a generator to provide a priori information for texture and detail generation. Since PGGAN uses a progressive generation approach and was originally designed for image generation tasks, it cannot be directly integrated into the proposed fine-grained network framework. In this work, we adapt PGGAN to the image completion network of our work with some modifications:

• The dimensionality of the latent vector output from the fully connected layer is the same as the input dimensionality of the pre-trained GAN
• In order to adjust the feature $S_i$ of the encoder, we use an additional gated convolution in each module for feature fusion:

  $$g_i=\left\{\begin{array}{c}\hfill S_0(Z,f_N)\hfill \\ S_i(g_{i-1},f_{N-i})\end{array}\right.$$

  (2)
  $S_0$ initializes Z into a 4 × 4 feature map using a pre-trained generator, and then fuses it with the features $F_N$ to $g_i$. The $S_i$ module first upsamples the $S_{i-1}$ features, then convolves the features with a priori information using the pre-trained generator, and finally fuses the features $F_{N-1}$ from the encoder to generate $g_i$.
• Instead of generating the output directly from the generator, we output the features $g_i$ and pass them to the decoder to better fuse the features from the generator and the encoder.

3.2.3. Decoder

We use an additional decoder with progressive fusion to integrate the features of the encoder and the generator to generate the output image. It takes the encoder’s features $E_N$ input and progressively fuses the multi-resolution features $g_i$ generated by the pre-trained generator.

$$d_i=\left\{\begin{array}{cc}\hfill D_0\left(f_N\right),ifi=0\hfill & \\ D_i(d_{i-1},g_i),otherwise\end{array}\right.$$

(3)

where $D_i$ and $d_i$ denote the 3 × 3 convolution and its output, respectively. In addition to the final output layer, each gated convolution is followed by a PixelShuffle [21] layer. With a skip connection between the encoder and decoder, the information captured by the encoder can be enhanced so that the generator can focus more on the generation of textures and details.

3.3. SN-PatchGAN Discriminator

While a normal discriminator maps the input to a scalar, where the magnitude of the value indicates the probability that the input sample is real or fake, a Patch-GAN [22] discriminator maps the input to a feature map, where the real or fake of each pixel block on the feature map is evaluated. The SN-PatchGAN [18] discriminator uses spectral normalization (SN) [23], where the weight matrix is divided by the corresponding Lipschitz constant. Theoretical analysis shows that spectral normalization can control the generalization error and stabilize the training.

3.4. Loss Function

In this paper, we use L1 loss, perceptual loss [24], style loss [25] and adversarial loss to optimize the network. For a given damaged image and the corresponding binary mask map M (value 0 for the missing region and value 1 for the valid pixel region), define the output of the first stage of the network as $I_{coarse}$, the output of the second stage as $I_{fine}$ and the real image as $I_{gt}$. Define L1 loss:

$$L_1=\parallel (1-M)(I_{coarse}-I_{gt}){\parallel }_1+{\parallel (1-M)(I_{fine}-I_{gt})\parallel }_1$$

(4)

The perceptual loss and style loss compare the differences in the deep feature maps between the generated and real images. Such a loss function can effectively transfer the structural and textural information of the image to the model. The perceptual loss function is defined as follows:

$$L_{perc}=\mathbb{E}\left[\sum _i\frac{1}{N_i}{\parallel {\varphi }_i\left(I_{fine}\right)-{\varphi }_i\left(I_{gt}\right)\parallel }_1\right]$$

(5)

where ${\varphi }_i$ is the activation map of the ith layer of a pre-trained network. In this work, ${\varphi }_i$ corresponds to the activation mappings of the $Relu1\_1$, $Relu2\_1$, $Relu3\_1$, $Relu4\_1$ and $Relu5\_1$ layers of the VGG-19 [26] network pre-trained on the ImageNet dataset [27]. These activation feature maps are also used to calculate the style loss. Given the feature map of $C_j\times H_j\times W_j$, define the style loss as:

$$L_{style}={\mathbb{E}}_j\left[\parallel G_j^{\varphi }\left(I_{fine}\right)-G_j^{\varphi }\left(I_{gt}\right){\parallel }_1\right]$$

(6)

where $G_j^{\varphi }$ is a $C_j\ast C_j$ Gram matrix constructed from activation maps ${\varphi }_j$. Moreover, we use LSGAN [28] to train the face image completion network. The adversarial loss is defined as:

$$L_{adv}=\frac{1}{2}{\mathbb{E}}_{x\sim p_r}{[D\left(x\right)-a]}^2+\frac{1}{2}{\mathbb{E}}_{z\sim p_z}[D{(G\left(z\right)-b]}^2+\frac{1}{2}{\mathbb{E}}_{z\sim p_z}[D{(G\left(z\right)-c]}^2$$

(7)

where D is the discriminator and G is the generator. The constants a, b denote the labels of the real image and the generated image, respectively. c is the value of the generator to make the discriminator think that the generated image is the real data. In our experiment, a = c = 1, b = 0. So, our overall loss function is:

$$L_{total}={\lambda }_1{L}_1+{\lambda }_2{L}_{perc}+{\lambda }_3{L}_{style}+{\lambda }_4{L}_{adv}$$

(8)

4. Experimental Results

In this paper, we use pre-trained PGGAN as generator and use publicly available codes of existing methods for comparison. The mask is generated by the method of Yu et al. [18], and a small number of small area rectangles are randomly added to increase the diversity of the mask. Classify masks in 10% steps according to their size relative to the whole image (e.g., 0–10%, 10–20%). To maintain fairness, both the method in this paper and the compared method are trained on the CelebA-HQ dataset to ensure that the difference in completion quality is mainly caused by the algorithm rather than the training distribution, while the test set is strictly excluded from the training set. The CelebA-HQ dataset contains 30k face images with a resolution of 256 × 256. We use 29k for training and the remaining 1k for testing. We use Adam [29] in our training to optimize the network and ${\beta }_1=0.5$, ${\beta }_2=0.99$. The learning rates of the generator and discriminator are $l{r}_g=1\ast e^{-4}$ and $l{r}_d=4\ast e^{-4}$, respectively. The weight values of the losses are ${\lambda }_1=1$, ${\lambda }_2=0.1$, ${\lambda }_3=250$ and ${\lambda }_4=1$, respectively. We compare the following completion methods: Gated Convolution(GC) [18], EdgeConnect (EC) [19], and Recurrent Feature Reasoning(RFR) [30].

4.1. Qualitative Analysis

The qualitative comparison results under free form and rectangular center masks are shown in Figure 3 and Figure 4, respectively. The completion results of GC suffer from blurring and lack of details, as in Figure 3b. The completion result of EC depends heavily on the accuracy of the predicted edge map, and EC does not work well when the key information is obscured in a large area, and even two pairs of eyes appear in the completion result in row 5 of Figure 4c. RFR uses recurrent feature reasoning, which recursively infers the hole boundaries of the convolutional feature mapping, and then uses them as cues for further reasoning. This recurrent feature inference does not work well on human faces, and its completion results are often blurred and do not understand the semantic information of the input, as in Figure 3d. The experimental results under free-form and rectangular center masks show that the completion results of our method have better texture and detail information than those of other methods.

4.2. Quantitative Analysis

We provide a quantitative comparison of the different methods in

Table 1. Quantitative comparison results when the proportion of the mask to the image is 10–50%.

	Mask	GC	EC	RFR	Ours
SSIM	10–20%	0.973	0.975	0.981	0.966
	20–30%	0.938	0.932	0.952	0.943
	30–40%	0.914	0.915	0.934	0.923
	40–50%	0.859	0.731	0.886	0.900
PSNR	10-20%	32.56	32.48	33.56	35.14
	20–30%	29.25	28.92	30.89	32.38
	30–40%	26.72	26.62	27.76	30.86
	40–50%	24.31	24.18	25.46	29.42

. For each category, we selected 10k images and calculated their average Structural Similarity(SSIM) and Peak Signal to Noise Ratio (PSNR). The experiments show that the PSNR of this method is better than those of the compared methods in the case of 10–50% mask, and the average improvement over the RFR model is 8.6%. For a mask size of 10–40%, the structural similarity between our method and the other methods compared is not significant; for a mask size of 40–50% of the image, our method achieves the highest structural similarity. This demonstrates the effectiveness of our approach.

5. Ablation Studies

In this section, we performed an ablation study in the CelebA-HQ dataset to show the effect of different sections on the complementation results.

5.1. Dilation Convolution

The global information of an image is crucial for image completion work. Dilated convolution can increase the perceptual field of the feature map. Adding dilated convolution to the network enables the network model to obtain a larger range of information. Figure 5 illustrates the effect of dilation convolution on the completion results. In the experiments, the four gated dilated convolutions in Figure 2 are replaced with gated convolutions. From Figure 5, it can be seen that the completion results of the model without the dilated convolutional products look reasonable but the quality of details is poor, while the model with the dilated convolutional products has a better quality because it obtains more global information.

5.2. Multi-Resolution Encoder

Figure 6 shows how the encoder features can be used to generate features with a priori information to help recover details and local structure. In the experiments, only the latent vectors are started and the effect of the features on the generated results is observed when they are gradually introduced into the generator. In order to avoid the interference of the decoder on the completion results, the experiments are tested using a variant of the decoder, and the output image is generated directly by the generator. When all convolutional features are dropped, the information of the image is poorly preserved and finer details are not retained. When coarse convolutional features (4 × 4 to 16 × 16 features) are passed to the pre-trained generator, more details are recovered and the output better approximates the original image. Further improvements in quality and fidelity are observed when finer features are passed to the generator. The above observations confirm that multi-resolution convolutional features are essential in guiding the recovery of fine details and local structures that cannot be reconstructed using only latent vectors.

5.3. Pre-Trained GAN

In order to analyze the contribution of pre-trained generators to the completion network, we investigated the impact of generating features from pre-trained generators. In the experiments, all pre-trained models are discarded at first, and then these features are gradually passed to the decoder. As Figure 7 gives two examples of the experimental results, the network is responsible for both generating realistic details and textures and maintaining the fidelity of the output results due to the lack of proper prior information. Such a demanding goal eventually leads to completion results that are deficient in both structure recovery and texture generation (as shown in Figure 7b). Since the pretrained generator already captures rich image priors, it can reduce the burden of generating textures and details. Therefore, when passing finer features (4 × 4, 8 × 8, …, 256 × 256 features) to the pre-trained generator, improvements in structure and texture can be observed (e.g., Figure 7c–e by gradually adding finer features, the completion results have more reasonable structure and finer texture).

5.4. Decoder

In the case of no decoder, although the overall perception is convincing, the enlarged output image clearly loses a lot of details. As shown in Figure 8, the completion result in row 1 (with decoder) has much more detail in the eyes and teeth than in row 2 (without decoder). This is because the decoder fused more feature information to produce more natural details. In addition, the multi-resolution skip connection between the encoder and decoder enhances the spatial information captured in the encoder features and makes the generator pay more attention to detail generation, which further improves the output quality.

5.5. SN-PatchGAN

In this paper, we consider the problem of free-form masks. Such masks can appear anywhere in the image, whereas previous work, for a single mask, often used global and local discriminators to discriminate the generated region and global information, respectively, which is not applicable to this work. In Figure 9, we show the difference between SN-PatchGAN discriminators and global discriminators. Although the global discriminator can also produce reasonable results, there is still a significant difference in details between the SN-PatchGAN. For example, the eyes generated in Figure 9b lose details compared to the completion results in Figure 9c, which illustrates the effectiveness of SN-PatchGAN in this work.

6. Conclusions

We propose a method to perform face image completion tasks by adding a pre-trained GAN to the encoder–decoder network structure. It is experimentally demonstrated that a pre-trained GAN can improve the quality of completion results by using its rich prior information. Compared with existing methods, the completion results in this paper are significantly improved in terms of texture and plausibility.

However, a priori-based methods have a prerequisite that a GAN with superior performance needs to be trained in advance. Such methods will fail when the experimental data cannot be trained to generate a GAN with excellent performance. In addition, recent research has shown that multiple reasonable results can be achieved [31,32], and we will study multiple-result tasks in the follow-up work.