A Two-Stage Image Inpainting Technique for Old Photographs Based on Transfer Learning

Abstract

To address the challenge of sparse old photo datasets, we apply transfer learning to image inpainting tasks. Specifically, we improve a two-stage image inpainting network that focuses on collaborative subtasks. We also design a transform module based on the cross-aggregation of windows to improve long-distance contextual information acquisition in image inpainting and enhance the integrity of images in terms of structure and texture. Our improved two-stage network has a significantly better repair performance compared to that of the current common inpainting methods. We further apply transfer learning techniques by utilizing the improved two-stage image inpainting network as the base network and decoupling the generator into a feature extractor and classifier, which consist of an encoder and a decoder, respectively. We obtain a domain-invariant feature extractor through minimax game training using source and target domain data. This feature extractor can be combined with the original encoder to restore old photo images. To verify the effectiveness of our approach, we conducted comparative experiments. Our results show that the PSNR, SSIM, and FID indexes of the model using transfer learning are 11.8%, 2.96%, and 44.4% higher than those without transfer learning, respectively. These findings suggest that applying transfer learning techniques can be an effective solution to address the challenge of sparse old photo datasets in image inpainting tasks.

1. Introduction

Photographs serve as important records of specific times and are a crucial tool for documenting historical development and family changes in modern times. However, early paper photographs were prone to damage due to the use of poor technology and preservation practices. To address the challenges of preserving paper photographs, digital image inpainting technology can be used to restore old and damaged photographs to their original appearance as much as possible [1,2]. Traditional image inpainting algorithms mainly include partial differential algorithms [3,4,5] and texture synthesis algorithms [6,7], which have obvious limitations due to the use of only the peripheral information surviving in the original image. When repairing complex and non-repetitive structures, these algorithms may produce unsatisfactory results.

In recent years, there has been significant progress in the use of deep learning techniques in computer vision and image processing, and the use of large numbers of images to train networks has enabled trained models to have a significant amount of prior knowledge, providing a new approach to image inpainting [8]. In particular, Generative Adversarial Networks [9] have been proposed and achieved promising results in image inpainting tasks [10]. Pathak et al. [11] designed and applied GANs based on traditional Convolutional Neural Networks and proposed Context Encoders to send the output of the network to a discriminator to detect the truth, which greatly improved the plausibility of the results. However, the performance of neural networks typically depends on a large amount of training data, which makes them unsuitable for areas with minimal training data, changing scenarios, and changing tasks. To address the challenge posed by sparse samples of old photos, this paper introduces transfer learning to transfer the trained model on open-source datasets to the small dataset collected by us, so that the final model can have a generalization performance using standard datasets. Specifically, the portability of deep features [12] is used to map images with intermediate- or high-level features using pre-trained models and train target-specific classifiers from them [13], which is a process commonly referred to as feature selection. Fine-tuning the source model for the target data is also feasible, and it usually is more effective as it optimizes the entire network for the target task. Therefore, fine tuning has become the rule of thumb for deep transfer learning with limited domain data [14,15]. During the fine-tuning process, the source model needs to be moderately tuned to avoid overfitting, as deep networks are over-parameterized for small-scale target tasks. Unlike traditional machine learning and deep learning, which require labeled data for model training, transfer learning can use previously accumulated knowledge to discover commonalities in problems and apply generic knowledge learned in certain tasks to similar tasks, enabling the model to learn in a more generalized manner.

As there have been only a few studies on the application of transfer learning in the field of image inpainting, this paper aims to address this gap by introducing transfer learning into the two-stage inpainting network for training. Due to the limited number of self-collected old photo data samples, overfitting and other issues are likely to occur during the training of deep neural networks. Therefore, this paper proposes a transfer learning-based approach for inpainting damaged old photos, which allows the effective training of a better model using only a small number of old photo samples. The main contributions of this paper are as follows:

• In this paper, a two-stage image inpainting network is constructed, which embeds images into two collaborative subtasks, which are structure generation and texture synthesis under structure constraints, and embeds the window cross aggregation-based transform module into the generator, which can effectively acquire the image long-range dependencies, thereby solving the problem that convolutional operations are limited to local feature extraction and enhancing the long-range contextual information acquisition capability of the model in image inpainting.
• Transfer learning is applied to image inpainting technology. The improved two-stage image inpainting network in this paper is used as the basic network, and the generator is decoupled into a feature extractor and classifier, which are an encoder and a decoder, respectively. A domain-invariant feature extractor is obtained through the training of the minimax game using source domain data and target domain data. The feature extractor can be combined with the original encoder to repair the old photo image, and the restoration of small sample old photo image dataset is realized.
• The experiments demonstrate that the two-stage network constructed in this paper has a better inpainting performance, and the inpainting of old photos using the transfer learning technique is better than that without the use of transfer learning, which proves the effectiveness of the method.

2. Related Work

2.1. Image Inpainting Technique

Deep learning’s ability to map deep features perfectly fits the requirements of image inpainting and shows the direction for new image inpainting methods. Recently, Nazeri et al. [16] proposed a two-stage GAN model named “EC”. The model combines the two stages of edge information prediction and image inpainting. It first generates the edge map of the missing region, and then sends it into the inpainting network as the guiding information for image inpainting, obtaining a relatively good restoration effect. Xiong et al. [17] demonstrated a similar model that, unlike EC, uses foreground object contours as a structural prior instead of edges for information. Ren et al. [18] pointed out that edge-preserving, smoothed images provide a better global structure due to them capturing more semantics, but these methods require a higher accuracy for structures, for instance, edges and contours. Some researchers have utilized the correlation between texture and structure to solve this problem. Li et al. [19] designed a progressive visual structure reconstruction network to progressively reconstruct structures and their associated visual features. They entangle the reconstruction of visual structures and visual features to benefit each other by sharing the parameters. Yang et al. [20] introduced a multitasking framework to generate sharp edges by adding structural constraints. Liu et al. [21] proposed a mutual coding decoding to simultaneously learn about convolutional features that correspond to different layers of structure and texture. However, a single shared framework is difficult for modeling textures and structures. Therefore, in order to effectively realize the restoration of image structure and texture information, Guo [22] et al. divided image inpainting into two subtasks, texture synthesis and structure reconstruction. They proposed a new dual-stream network for image inpainting to further improve the performance of image inpainting. Since existing image inpainting techniques are outputting only one restoration result for a broken image, but the nature of image inpainting is an uncertain task, and its output should not be limited. Based on this idea, Liu [23] et al. proposed a probabilistic diverse GAN algorithm. The closer to the center of the image hole that the area studied is, the higher its diversity is, and the more diverse it is, and thus, good results are obtained.

2.2. Transfer Learning Method

Deep learning-based transfer learning methods, that is, deep transfer learning aiming to reduce the time and cost of the training process, have become a popular research direction at present. In some domains (e.g., medical images, etc.) [24], there is the problem of difficulty in obtaining large datasets. Transfer learning provides a better solution for these domains. In addition, pre-trained models for specific jobs can be applied on simple edge devices with a limited processing power and training time, and the development of deep transfer learning opens the door to more intuitive and sophisticated artificial intelligence. Ge et al. [25] developed a method for fine-tuning by using additional data obtained from a large-scale dataset. Cui et al. [26] successfully applied the knowledge learned from large-scale datasets to domain-specific small-scale data through fine-tuning and won first place in the 2017 iNaturalist large-scale species classification challenge. As a result, their research provides new ideas for subsequent deep transfer learning. For example, Long et al. [27] proposed a new Deep Adaptation Network based on this, which generalizes deep Convolutional Neural Networks to new domain-adapted scenarios where Deep Adaptation Networks can learn transferable features with statistical guarantees and scale linearly using the unbiased estimation of kernel embeddings. Tzeng et al. [28] developed a method based on using GANs to train encoders for target samples. They use an adaptation layer to compute the Maximum Mean Difference between the source and target domain features [29]. Haeusser et al. [30] proposed the utilization of the association between source and target features in the training process. They aim to maximize the invariance of the learned feature domain, while minimizing the error in the source samples. In the field of image inpainting, the combination of deep neural networks and transfer learning technology has been successfully carried out in a preliminary exploration and has been applied to the image inpainting problem under the condition of large mask areas. Chen et al. [31] designed a transfer learning network that, for the first time, accomplished large mask image inpainting guided by a high-level understanding of abstract neuronal representation images. Zhao et al. [32] proposed a small-sample unsupervised joint transfer learning approach that combines fine-tuning with direct transfer training to enable the network to learn knowledge about the target domain.

3. Two-Stage Image Inpainting of Old Photos Based on Transfer Learning

The two-stage image inpainting network constructed in this paper adopts the BIFPN feature fusion network proposed by Li et al. [33], which effectively strengthens the fusion and interaction of information. The original network generator is a two-stream generator. In this paper, a two-stage network is adopted. By embedding the transform module into the generator network in the form of residuals, the distant image dependence relationship can be effectively obtained. This solves the problem that convolution operation is limited to local feature extraction and enhances the model’s capability to obtain long-distance contextual information in image restoration. The repair performance of the network has been improved. By applying transfer learning to train the image inpainting network, the large public dataset, CelebA, is used as the source domain, and old photo images are used as the target domain. A domain-invariant feature extractor is obtained, which can cope well with the restoration task of old photo images.

3.1. Two-Stage Image Inpainting Network

The image inpainting network is implemented as a Generative Adversarial Network, where two generators generate texture and structure information, respectively, modeled by a U-Net variant. As shown in Figure 1, both texture and structure generators are encoded–decoded structures, and a structure discriminator and texture discriminator are designed to distinguish the real image from the generated image by estimating the texture and structure features, respectively. The image generated by the generator is sent to the discriminator, together with the real image. The discriminator outputs “True” if the image is real and “False” if the image is generated, prompting the generator to generate images that are similar to the real ones. The structure of the two-stage inpainting network is shown in Figure 1.

A transform module based on cross-aggregation of windows is embedded to improve the information aggregation between the windows, without increasing the computational complexity. This improves the aggregation of information between the windows, allowing the effective acquisition of distant image dependencies. This solves the problem that the convolution operation is limited by local feature extraction. The two parallel coupled streams are modeled separately and combined to complement each other, further improving the structural and textural integrity of the generated images.

The two-stage network is described in detail below:

3.1.1. Generator Network

The two-stage image inpainting network uses dual generators, which are a structure generator and a texture generator. In this paper, the inpainting algorithm has different tasks depending on the mask area. In general, it can be seen to complement high-frequency information (structure) and low-frequency information (texture) modeled by a U-Net variant with an encoded–decoded form. This paper uses a dilated convolutional kernel instead of a partially normal convolutional kernel, with the kernel proposed in [34]. This choice simultaneously alleviates the problems of gradient dispersion and gradient explosion, while generating more complex predictions by combining low-level and high-level features at multiple scales through jump connections. The details of the generator structure are shown in Figure 2.

3.1.2. Dual Discriminators

Both the structure discriminator $D_1$ and the texture discriminator $D_2$ are selected as spectrally normalized Markov discriminators. The structural parameters are shown in

Table 1. Details of the structure of the discriminator network.

Layers Names	Convolution Kernel Size	Step Size	Activation Function	Output Feature Maps
Convolutional layer 1	4 × 4	2	LeakyReLU	64 × 128 × 128
Convolutional layer 2	4 × 4	2	LeakyReLU	128 × 64 × 64
Convolutional layer 3	4 × 4	2	LeakyReLU	256 × 32 × 32
Convolutional layer 4	4 × 4	1	LeakyReLU	512 × 16 × 16
Convolutional layer 5	4 × 4	1	LeakyReLU	512 × 4 × 4
Fully connected layer	-	-	Sigmoid	512 × 1 × 1

.

However, the structure discriminator and the texture discriminator also differ significantly. The former takes into account the enhancement of its own adversarial loss, which is achieved by inputting pairs of data. In terms of the input data, there are two main points: firstly, the edge map of the fused image derived using the edge detection algorithm, and secondly, the grayed-out image. After the above optimization steps, the structure discriminator is able to steadily improve the similarity between the restored image and the original image, and at the same time, it is able to discriminate whether the structured texture of the restored image is real or not.

The image inpainting algorithm of the two-stage network is divided into three steps to complete the inpainting:

(1) Pre-processing (edge detection). The edge information of the broken image is the prerequisite for the inpainting algorithm. The discovered excellent edge detection algorithm [35] is fused into our proposed two-stage image inpainting algorithm to make it perform the high-precision edge detection of the input broken image before inpainting.

(2) Structure and texture repair. The detected broken edge image, grayscale image, and mask are inputted into the structure encoding generator together to generate a preliminary complete structure image. The complete edge image is inputted into the texture repair generator network together with the broken image to generate a preliminary complete texture image.

(3) The initial structural and textural image information are fused to obtain the complete image, which is then discriminated by the structural discriminator $D_1$ and textural discriminator $D_2$. If the generated image is determined as being false, the information is fed back to the generator, and the generator is prompted to adjust the network parameters to generate an image that is more similar to the real image. This iterative process ends when both $D_1$ and $D_2$ are unable to distinguish between true and false images.

3.1.3. Aggregation Transform

The transformer [36] omits recursion and convolution and follows an encoder–decoder structure, where the encoder consists of six identical layers, the structure of which is shown in Figure 3, with two sub-layers each. The first layer is a multi-headed self-attentive mechanism, and the second layer is a simple multilayer perceptron. Residual connectivity [37] was used around each of the two sub-layers, followed by layer normalization [38]. The decoder is similar in structure to the encoder in that it also contains six identical layers, with each decoder layer having three sub-layers. The difference is that the attention mechanism layer is embedded in the decoder, ensuring that the prediction of position i depends only on the known output with positions less than i.

The structure of the multi-head self-attentive mechanism and the self-attentive mechanism are shown in Figure 3b,c. For the self-attentive mechanism, the three matrices Q (Queries), K (Keys) and V (Values) all come from the same input, and the calculation formula is shown in (1).

$$\mathrm{Attention}(Q,K,V)=\mathrm{SoftMax}(\frac{Q{K}^T}{\sqrt{d_k}})V$$

(1)

where $\frac{Q{K}^T}{\sqrt{d_k}}$ counts the raw score of attention (which is actually the similarity score derived from the dot product of $Q$ and $K$). $\sqrt{d_k}$ is a scaling factor that keeps the result from being too large or too small and avoids it being either 0 or 1 after softmax.

In the transformer, the self-attentive layer is further refined by adding a multi-headed attention mechanism by first mapping the Queries, Keys, and Values by h different linear transformations. Then, the different attentions are stitched together. Finally, a linear transformation is performed. Each set of attention is used to map the input them into a different sub-representation space, which allows the network model to jointly pay attention to the subspace feature information under different representatives of Queries, Keys, and Values at the same location, while acquiring more detailed features; the computational process can be expressed as:

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(hea{d}_1,\dots ,hea{d}_i)W^o$$

(2)

$$hea{d}_n=\mathrm{Attention}(Q{W}_n^Q,K{W}_n^K,V{W}_n^V)$$

(3)

where $W_n^Q\in R^{d_{\mathrm{mod}el}\times d_q}$, $W_n^K\in R^{d_{\mathrm{mod}el}\times d_K}$, and $W_n^V\in R^{d_{\mathrm{mod}el}\times d_V}$ are the Queries, Keys, and Values of the projection matrix with head number $k$. All the heads $\left(hea{d}_1,\dots ,hea{d}_i\right)$ are connected for linear projection to obtain the final result.

3.1.4. The Joint Loss Function

The two-stage inpainting network employs semantic-based joint loss training, including feature content loss, reconstruction loss, perceptual loss, style loss, and adversarial loss. By combining the above various loss functions, a complementary effect is achieved, which ultimately enables the restored images generated by the inpainting network to meet not only the visual match with the original image, but also the high accuracy requirements when the hard evaluation of metrics is performed.

(1)

Feature content loss

The feature content loss $L_f$ stabilizes the training process by comparing the activation mappings in the middle layer of the discriminator $D$ to constrain the generator $G$ to produce results that are more similar to the true structure map.

$$L_f=\mathrm{E}\left[{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}\frac{1}{N_{\mathrm{i}}}}{‖D_i\left(E_{in}\right)-D_i\left(E_{out}\right)‖}_1\right]$$

(4)

where n denotes the total number of convolutional layers of discriminator $D$, $N_i$ is the number of elements of the i-th layer, and $D_i$ denotes the activation function output of the i-th layer of discriminator $D$.

(2)

Reconstruction loss

The normalized $L_1$ distance is generally used as the reconstruction loss; however, in this paper, we use the distance $L_1$ between $I_{out}$ and $I_{gt}$ as the reconstruction loss, with the following equation:

$$L_{rec}={‖I_{out}-I_{gt}‖}_1$$

(5)

(3)

Perceptual loss [39]

In order to preserve the structural information of the global image and ensure the similarity of the high-level structure, image restoration requires a feature representation similar to that of the real image, rather than just pixel matching between images. The perceptual loss is calculated using a VGG-16 feature extractor [40] pre-trained on the ImageNet dataset [41] to extract the feature maps of the generated and real images separately and based on the $L_1$ distance between them; it is defined as follows:

$$L_{perc}=\mathrm{E}\left[{\displaystyle \sum _i{‖{\varphi }_i\left(I_{out}\right)-{\varphi }_i(I_{gt})‖}_1}\right]$$

(6)

where ${\varphi }_i(·)$ denotes the activation mapping obtained for a given input image $I^{*}$ through the i-th pooling layer of VGG-16.

(4)

Style loss

The perceptual loss helps to obtain a higher-level structure and avoid the generated image from deviating from the real image. In order to maintain the style consistency, this paper adds style loss to the joint loss function. The style loss calculates the $L_1$ distance of the Gram matrix of the feature map generated by the image after VGG-16. The perceptual loss defined as follows:

$$L_{style}=\mathrm{E}\left\{{\displaystyle \sum _i{‖\mathit{G}\mathit{r}\mathit{a}\mathit{m}\left[{\phi }_i\left(I_{out}\right)\right]-\mathit{G}\mathit{r}\mathit{a}\mathit{m}\left[{\phi }_i(I_{gt})\right]‖}_1}\right\}$$

(7)

where $\mathit{G}\mathit{r}\mathit{a}\mathit{m}$ denotes the Clem matrix, $\mathit{G}\mathit{r}\mathit{a}\mathit{m}\left[{\phi }_i(·)\right]={\varphi }_i{(·)}^T{\varphi }_i(·)$.

(5)

Adversarial loss [42]

Adversarial loss is used to enhance the gaming process of the generative and discriminative networks, aiming to make the data distribution of the generated image more similar to the real image, making the result more realistic. The objective function of the discriminative network is used here, which is defined in Equation (8) is shown as follows:

$$\begin{array}{c}{L}_{adv}=\underset{G}{\min } \underset{D}{\max } E_{I_{gt},E_{gt}}\left[\log D(I_{gt},E_{gt})\right]\\ +E_{I_{out},E_{out}}\log \left[1-G(I_{out},E_{out})\right]\end{array}$$

(8)

In summary, the joint loss function is as follows:

$$\begin{array}{c}{L}_{joint}={\lambda }_1{L}_f+{\lambda }_2{L}_{rec}+{\lambda }_3{L}_{perc}\\ +{\lambda }_4{L}_{style}+{\lambda }_5{L}_{adv}\end{array}$$

(9)

where ${\lambda }_1=10,{\lambda }_2=10,{\lambda }_3=0.1,{\lambda }_4=250,{\lambda }_5=0.1$.

3.2. Training of the Model

A large public dataset is used as the source domain, and the old photo dataset is used as the target domain, and the training process is made smoother by adding data to the source domain. The goal of this transfer learning is to train a domain-invariant feature extractor, and the training is divided into three main steps, as shown in Figure 4. The solid line indicates that the modules are being trained, and the dashed line indicates the trained models.

In the first step, this model is decoupled into a feature extractor $E$ and a classifier $C$, which are an encoder and a decoder, respectively. During training $E_S$, one performs feature extraction on the source domain samples, and C denotes the fully connected Softmax layer. Minimized cross-entropy loss is used.

$$L_{CE}=\underset{\theta E_s,\theta C}{\min }{E}_{(x_i)~(X_s)}H(C\circ E_s(x_i))$$

(10)

where $\theta E_s$ and $\theta C$ denote the parameters of $E_S$ and C, respectively, $X_s$ denotes the distribution of samples in the source domain, and H denotes the softmax cross-entropy function.

In the second step, the model S is trained to generate feature generators similar to the source domain features, and then training is performed using adversarial loss, as exemplified by the structural generation:

$$\begin{array}{c}{L}_{{}_{adv}}^1=\underset{\theta S}{\min } \underset{\theta f_d^1}{\max } E_{(z,e_i)~(p_z(z),E_i)}{‖f_d^1(S(z\parallel e_i)\parallel e_i)-1‖}^2\\ +E_{(e_i,x_i)~(E_i,X_i)}{‖f_d^1(E_s(x_i)\parallel e_i)‖}^2\end{array}$$

(11)

where $\theta S$ and $\theta f_d$ denote the parameters of S and $f_d$, respectively, $p_z(z)$ is the sample distribution extracted from $e$, and $e$ is the edge feature vector. In order to generate an arbitrary number of new feature samples, it is only necessary for S to take a cascade of noise vectors and an edge feature as an input and to output the desired generated feature vector:

$$F(z|e)=S(z\parallel e)$$

(12)

where $z~p_z(z)$ and F are feature vectors belonging to e.

In the third step, after initialization using the weights optimized in step one, the following maximum minimum training domain-invariant encoder $E_1$ is obtained to achieve optimal convergence.

$$\begin{array}{c}{L}_{{}_{adv}}^2=\underset{\theta E_1}{\min } \underset{\theta f_d^2}{\max } E_{x_i~X_s\cup X_t}{‖f_d^2(E_1(x_i))-1‖}^2\\ +E_{{}_{(z,e_i)~(p_z(z),E_i)}}{‖f_d^2\left(S\right(z\parallel e_i))‖}^2\end{array}$$

(13)

where $\theta E_1$ and $\theta f_d^2$ denote the parameters of $E_1$ and $f_d^2$, respectively. Since the model $E_1$ is trained using source and target domains, the results of the feature extractor are domain-invariant. It maps source and target samples in a common feature space, where features are indistinguishable from those generated using S. Since the latter is trained to produce features indistinguishable from the source samples, the feature extractor can be combined with the encoder of step one for image structure feature generation, and the texture generator is consistent with the above steps.

4. Experimental Analysis

4.1. Analysis of Experimental Results of Inpainting Model

We used the CelebA and Places datasets, which are widely used in the literature, to evaluate the proposed approach. We selected 10 categories from Places, each with 5000 training images, 900 test images, and 100 validation images. We used 30,000 images for training and 10,000 images for testing. The mask datasets for the experiments all used irregular masks obtained from [43], classified according to their hole size relative to the whole image in 10% increments; all images and corresponding masks were adjusted to 256 × 256 pixels, the batch size processed was 16 sheets, and 300,000, training iterations were used, optimized using the Adam optimizer [44] with the parameters set to ${\beta }_1=0.001$ and ${\beta }_2=0.9$. The model was first initially trained using a learning rate of 2 × 10⁻⁴, and then fine-tuned with a learning rate of 5 × 10⁻⁵, and the BN layer of the generator was frozen, and the discriminator was trained with a learning rate of 1/10 of the generator. The deep learning framework used for the experiments was Pytorch, the computer operating system was Windows 10, and the graphics card model was NVIDIA TITAN XP with 12G of video memory.

4.1.1. Qualitative Analysis

To verify the effectiveness of the two-stage image inpainting network proposed in this paper, test sets of the CelebA dataset and the Places dataset were used to compare the subjective as well as numerical results of this paper’s method with the MED [21], CTSDG, and BIFPN algorithms. Figure 5 shows the inpainting results of the method in this paper on the CelebA datasets.

Figure 5 shows the inpainting results of MED, CTSDG, BIFPN, and the methods proposed in this paper for different mask rates on the CelebA dataset. It can be observed that all the inpainting methods show a good performance when the mask area is small, specifically when the mask area is 10–20% of the original size, with only minor differences between the original and inpainted result maps. The MED and CTSDG methods (subfigure c2 and subfigure d2, respectively) exhibit varying degrees of blurring at 20–30% of the masked area, whereas the proposed methods maintain good inpainting, with sharper textural and structural properties. At a mask area of 30–40%, the MED, CTSDG, and BIFPN methods (subfigure c3, subfigure d3, and subfigure e3, respectively) all exhibit artifacts and even facial distortions, with some facial features being lost. Although the proposed algorithm also exhibits blurring and distortion, it still yields better inpainting results compared with those of the other three methods. At 40–50% of the masked area, it becomes clear that the restored image from the inpainting other algorithms is significantly distorted and no longer conveys the texture information that the original image should. In contrast, the proposed algorithm maintains a large amount of texture detail, with only a few differences with the ground truth.

Figure 6 shows the inpainting results of MED, CTSDG, BIFPN, and the proposed methods for different mask rates on the Places dataset. It can be observed that most of the inpainting methods show a good performance when the mask area is between 10–20% of the original size, with only MED exhibiting significant distortion. At a mask area of 20–30%, the image itself shows good inpainting due to its characteristics (e.g., the structure information is not obvious), but a detailed comparison reveals that the proposed method yields a clearer texture and structure inpainting performance. At a mask area of 30–40%, the MED, CTSDG, and BIFPN methods (subfigure c3, subfigure d3, and subfigure e3, respectively) exhibit artifacts and even facial distortions, with some image features being lost. At a mask area of 40–50%, it is evident that the remaining three algorithms have lost too many landscape details, with a significant portion of the area lacking texture details, which is significantly different from the results obtained by the proposed algorithm.

4.1.2. Quantitative Analysis

We refer to all the current studies in the field related to image inpainting and decide to adopt the three types of metrics, PSNR, SSIM, and FID, which were all used in quantitative evaluation to evaluate our restored images. At the same time, we designed the comparison experiments with other algorithms, and all algorithms used mask rates in inpainting based on 10–20% with 10% increments. Ultimately, the comparison of the quantitative analysis results clearly shows that the inpainting performance of our proposed algorithm model is much better than that of the other algorithms. The quantitative results of the three metrics for each specific algorithm are shown in

Table 2. Comparison of quantitative analysis results on CelebA.

	Mask Rate	MED	CTSDG	BIFPN	Ours
PSNR↑	10–20%	28.75	32.67	32.11	32.03
	20–30%	26.97	28.13	28.67	28.44
	30–40%	23.67	25.32	25.81	26.43
	40–50%	22.07	23.46	23.56	24.79
SSIM↑	10–20%	0.922	0.958	0.960	0.953
	20–30%	0.904	0.917	0.924	0.931
	30–40%	0.837	0.852	0.863	0.882
	40–50%	0.811	0.826	0.833	0.841
FID↓	10–20%	5.63	2.61	2.67	2.95
	20–30%	6.79	3.74	3.24	3.11
	30–40%	8.64	5.35	5.02	4.78
	40–50%	9.11	7.69	7.63	7.11

and

Table 3. Comparison of quantitative analysis results on Places.

	Mask Rate	MED	CTSDG	BIFPN	Ours
PSNR ↑	10–20%	28.05	30.54	31.09	31.86
	20–30%	25.44	26.55	26.61	27.14
	30–40%	22.89	23.73	24.17	25.71
	40–50%	21.76	22.54	22.78	23.64
SSIM ↑	10–20%	0.924	0.929	0.934	0.926
	20–30%	0.874	0.897	0.906	0.907
	30–40%	0.846	0.856	0.862	0.873
	40–50%	0.811	0.826	0.834	0.842
FID ↓	10–20%	5.71	4.11	3.88	3.16
	20–30%	6.59	5.21	4.16	4.07
	30–40%	9.14	7.68	7.11	6.89
	40–50%	11.54	9.13	8.75	8.18

.

From Table 2, it can be seen that the MED algorithm is clearly at a disadvantage when it needs to deal with the task of CelebA image inpainting at different mask rates. While our proposed algorithm model has its own advantages over the remaining two algorithms when facing different mask rates, it can be found that the model in this paper still significantly outperforms the rest of the algorithms, and this phenomenon can be clearly seen when targeting the image inpainting task of CelebA with high mask rates.

As for the Places dataset in natural scenes, Table 3 shows that the SSIM metrics of our proposed image inpainting model are slightly lower than those of the BIFPN model only at the mask rate interval of 10–20%. When restoration is performed at the rest of the mask rates, the restoration performance of our proposed inpainting model outperforms the rest of the algorithms, and the effect is especially prominent at high mask rates.

In summary, our algorithm outperforms current state-of-the-art image inpainting models.

4.2. Analysis of Old Photo Inpainting Results with Transfer Learning

In this paper, a modified two-stage image inpainting network is used as the base network by employing a model initialized using pre-trained source domain data in the initialization phase of the model. The pre-trained source domain model is trained on the CelebA dataset to obtain the weights. For training, the learning rate of all the fine-tuning layer parameters was 0.1 times the learning rate of the training layer parameters, using an Adam optimizer with momentum set to 0.95. The domain discriminators all use three fully connected layers and the ReLU activation layer. L2 regularization was also used for all the parameters with a factor of 0.1. In order to obtain a more stable representation of the data distribution, the data batch size is set to six for each iteration, and the number of model training iterations is based on the number of traversals of the small dataset, with the maximum number of traversals being 100. The mask dataset used in transfer learning training adopts the irregular mask obtained from [43]. In order to restore the old photo image more realistically, the mask used in the test experiment is extracted from the damaged old photo.

4.2.1. Experimental Content

In order to verify the practicality of the feasibility and effectiveness of transfer learning on old photo image inpainting tasks, we have prepared corresponding comparative experiments. Both sets of experiments used a two-stage inpainting network as the base network, and both sets of experiments were as follows:

Group 1 (without transfer learning): The hybrid dataset was used as the training data for the model, and the model was trained directly without transfer learning, when all the parameters in the model were initialized. (The hybrid dataset is the sum of the CelebA training set and the old photo training set).

Group 2 (using transfer learning): This group was training using transfer learning, and the steps are shown in Figure 4.

4.2.2. Dataset Acquisition and Pre-Processing

In the experiments and results analysis in this paper, it can be concluded that our proposed image inpainting model can obtain a good image inpainting performance; however, when the trained model is applied to the old photo inpainting task, it was found that the model does not restore the old photos well. After analysis, the main reason for the poor robustness of the model is that the open-source dataset is not exactly in the same style as the old photo dataset. Old photographs are subject to fading and blurring due to the limitations of past filming equipment and preservation conditions, and there are significant differences between them and the images obtained from the open-source dataset.

Due to the sparse sample of old photo image dataset, a transfer learning approach is used to implement the task of inpainting of old photos. A total of 316 images of complete old photo faces were collected from the internet, and 91 images of broken or blurred old photos were obtained after screening. The complete old photo images were divided into a training set, a validation set, and a test set in a ratio of approximately 8:1:1, and the results are shown in

Table 4. Old photo images segmentation dataset.

	Train Set	Validation Set	Test Set
Old photos	252	32	32

:

We use the random mask in this paper for training, and in order to make the broken old photos more realistic, the test mask for old photo inpainting uses the broken mask detected from the real broken old photo images as the test mask.

4.2.3. Analysis of Experimental Results

As the core work in this subsection is the task of old photo images inpainting, a subjective comparison was carried out with a focus on the overall inpainting of old photos.

• Subjective evaluation

An example plot of the comparison results for the first and second groups, respectively, on the old photo dataset is shown below, as shown in Figure 7.

As can be seen in Figure 7, the inpainting results for the first group have significant pixel inconsistencies, as seen in the third row where the shoulders appear to be colored, and in the second row where inconsistent facial colors are observed for the faces. On the other hand, the second group of inpainting results have better consistency, and it can be seen that the inpainted images are largely visually unobstructed.

In summary, the use of transfer learning methods in old photo image inpainting tasks gives more stability and accuracy in the results in terms of structural and texture features. It can produce more realistic results with more natural facial features for face images.

2.

Objective evaluation

In order to more objectively evaluate the strengths and weaknesses of the algorithms used in this paper, quantitative analysis experiments were set up. The inpainting results of the two groups of methods were quantitatively analyzed, and the experimental mask used the broken mask detected from the real broken old photo images as the test mask, and the test masks were consistent in both groups, and the quantitative results are shown in

Table 5. Objective quantitative comparison on old photos of human faces.

Methods	PSNR	SSIM	FID
Group 1	32.42	0.943	3.62
Group 2	36.25	0.971	2.01

.

From Table 5, it can be concluded that the use of transfer learning can make the inpainting of old photos more natural, and both PSNR and SSIM are far better than those without the use of transfer learning, which is basically consistent with the results of the qualitative analysis, thus proving the effectiveness of using the method of transfer learning in the inpainting of old photos images.

5. Discussion

At the early stage of the COVID-19 outbreak, it was not possible to conduct relevant research using neural networks because of the sparse dataset samples and the need for neural networks to rely on a large number of datasets for training. Due to this, in this paper, the authors applied transfer learning to the field of image inpainting for the first time. Specifically, the authors took the inpainting of old photo images as the research object, constructed a two-stage image inpainting model, and embedding Transform into the generator in the form of residual blocks, which solves the problem of long-distance feature acquisition that cannot be achieved by convolution. The generator was decoupled into a feature extractor and a classifier, and a domain-invariant feature extractor was trained with a large public dataset as the source domain and an old photo image as the target domain. Experimentally, the application of transfer learning to image inpainting in this paper has achieved initial success and proved the effectiveness of the method. And this work in this paper also provides new ideas for the subsequent research of image inpainting techniques for small sample datasets, so that it can be satisfied with more application scenarios.

6. Conclusions

In this paper, we introduce transfer learning techniques to address the problem of sparse samples of old photo datasets. In order to build an inpainting network more suitable for transfer learning, a two-stream network is decoupled into two parallel streams to form a two-stage network based on the original two-stream structure of the image inpainting model. A dual discriminator is designed to give the model a better inpainting performance by estimating texture and structure separately. Secondly, using a two-stage image inpainting network as the base model, the generator is decoupled into a feature extractor and a classifier, which are an encoder and a decoder, respectively, and a domain-invariant feature extractor is obtained by training the source and target domain data, which can be combined with the original encoder for the inpainting task of old photo images. The experiments show that the inpainting of old photos using transfer learning is better than that without transfer learning, maintaining pixel consistency and reasonableness. In the qualitative experimental analysis, the PSNR, SSIM, and FID indexes of the model using transfer learning are 11.8%, 2.96%, and 44.4% higher than those without transfer learning, respectively.