Image Restoration Based on Semantic Prior Aware Hierarchical Network and Multi-Scale Fusion Generator

Abstract

As a fundamental low-level vision task, image restoration plays a pivotal role in reconstructing authentic visual information from corrupted inputs, directly impacting the performance of downstream high-level vision systems. Current approaches frequently exhibit two critical limitations: (1) Progressive texture degradation and blurring during iterative refinement, particularly in irregular damage patterns. (2) Structural incoherence when handling cross-domain artifacts. To address these challenges, we present a semantic-aware hierarchical network (SAHN) that synergistically integrates multi-scale semantic guidance with structural consistency constraints. Firstly, we construct a Dual-Stream Feature Extractor. Based on a modified U-Net backbone with dilated residual blocks, this skip-connected encoder–decoder module simultaneously captures hierarchical semantic contexts and fine-grained texture details. Secondly, we propose the semantic prior mapper by establishing spatial–semantic correspondences between damaged areas and multi-scale features through predefined semantic prototypes through adaptive attention pooling. Additionally, we construct a multi-scale fusion generator, by employing cascaded association blocks with structural similarity constraints. This unit progressively aggregates features from different semantic levels using deformable convolution kernels, effectively bridging the gap between global structure and local texture reconstruction. Compared to existing methods, our algorithm attains the highest overall PSNR of 34.99 with the best visual authenticity (with the lowest FID of 11.56). Comprehensive evaluations of three datasets demonstrate its leading performance in restoring visual realism.

1. Introduction

With the rapid development of digital media technology, the demand for high-quality, high-resolution images is increasing. Image inpainting technology addresses defects such as missing or corrupted pixels in images, with applications ranging from photo restoration to medical imaging and artwork preservation. Image inpainting algorithms find broad and significant applications across multiple domains. Notable examples include the restoration of damaged MRI [1] in medical imaging, as well as the recovery of missing data in high-resolution remote sensing [2].

Early image inpainting algorithms [1,3] were based on mathematical and physical theories using pixel correlations and were broadly categorized into diffusion-based and sample-based methods. However, traditional approaches lacked a deep understanding of the high-level semantic information within images, making it difficult to generate contextually consistent inpainted content. The increasing popularity of deep learning technology offers some new ideas for image inpainting. Context encoders [4] are the first techniques to apply Generative Adversarial Networks (GANs) [5] to image inpainting. Despite improvements, existing methods [4,5,6,7,8] still face challenges in handling large, irregular, or complex damaged regions, often resulting in artifacts or structural distortions. However, these image inpainting methods cannot be adapted to complex application scenarios; for example, fewer intact pixels are available to infer image content from distant backgrounds, which can lead to structural distortions and artifacts when inpainting large, irregular, and complex images, reducing the clarity and realism of the inpainted images.

In recent years, a series of advanced methods have emerged in the field of large-region image inpainting, such as the semantic-mask-based LaMa [9], the Transformer-based ICT [10], the denoising diffusion-based RePaint [11], and DDNM [12]. These methods have significantly improved the inpainting effect in complex scenarios by introducing novel network architectures or generation mechanisms.

In recent years, to meet the demands of edge computing and real-time processing, a series of efficient lightweight models have emerged in the field of image inpainting. Their core objective is to significantly reduce computational complexity while maintaining restoration quality. Wang et al. [13] proposed a multi-level feature aggregation network that achieves efficient integration of contextual information by optimizing the feature pyramid structure. Wang et al. [14] developed a lightweight multi-scale feature diffusion model specifically for underwater fish monitoring scenarios, successfully combining the generative capability of diffusion models with lightweight design. Wang et al. [15] further introduced MD-GAN, which addresses the challenge of large mask inpainting through a lightweight multi-scale generative adversarial network. These studies demonstrate the evolution of lightweight technologies from general feature aggregation to specialized diffusion models, and, further to complex GAN architectures, advancing the practical application of image inpainting technologies in edge computing and real-time systems. While model compression reduces computational costs, it often leads to diminished detail restoration capabilities, manifested as a noticeable gap in complex texture recovery and generation quality compared to large-scale models. Furthermore, compressed models demonstrate heightened sensitivity to training data distribution, indicating strong data dependency.

Nazeri [16] integrates structural priors with a hierarchical network, employing a dual-generator architecture for multi-scale feature fusion. However, the edge priors it relies on remain at a low structural level and lack understanding of high-level semantics, leading to insufficient semantic consistency in complex scenes and a relatively rigid multi-scale fusion mechanism. Liu [17] combines a mutual encoder–decoder with a multi-scale generator and utilizes feature equalization to enhance detail restoration. Nevertheless, this method does not incorporate external semantic priors for guidance and relies solely on data-driven internal features, making it prone to structural distortion or logical errors when reconstructing regions with strong semantic information, such as human faces or architectural structures. Xiong [18] integrates semantic segmentation priors, hierarchical networks, and multi-scale fusion. However, its network structure is complex, requiring additional branches and loss functions to distinguish between foreground and background. This results in a model with a large parameter size and low inference efficiency, making it difficult to adapt to practical scenarios with high-resolution images and high masking ratios.

To address these issues, we propose a multi-scale semantic-driven inpainting method. We leveraged semantic priors to guide the inpainting process, thereby enhancing semantic coherence in the generated results. A hierarchical or progressive network architecture is employed to refine the output in a coarse-to-fine manner, enabling effective handling of large missing regions. Furthermore, a multi-scale fusion generator was utilized to aggregate features from different levels, simultaneously improving both the structural integrity and textural details of the restored image. Our key contributions are as follows:

(1)

We present a novel Semantic Prior Generator, which combines a multi-label classification model with U-Net to map multi-scale features into semantic priors, enhancing semantic accuracy. Mapping the multi-scale high-level semantic features of the damaged image and the intact image into multi-scale semantic priors enables our model to more accurately capture the high-level semantic information of the image’s damaged areas.

(2)

We propose residual blocks with multi-scale fusion, which refines low-level texture features and progressively integrates them with structural features, preserving image integrity. By constructing a series of residual blocks, the model is able to effectively map the multi-scale semantic prior information into structural features of different scales. This innovation not only enhances the model’s expressive power but also ensures better preservation of the image structure’s integrity during the repair process, thereby improving the quality of the restored image. To address the insufficient interaction between high-level semantic features and low-level features, the model designs a multi-scale fusion module. This module can refine low-level texture features and fuse them with structural features of different scales in a progressive manner, allowing the model to perceive the multi-scale semantic information of high-level features.

The remainder of this paper is organized as follows: Section 2 reviews related work, Section 3 details our method, Section 4 presents experiments, and Section 5 concludes the paper.

2. Related Work

The focus of this paper is on multi-scale semantic priors and feature fusion techniques in image inpainting, which aims to solve the challenge of integrating high-level semantic information with low-level texture features for the purpose of generating high-quality image inpainting results. By analyzing the limitations of existing models, multi-scale semantic priors have been introduced and an innovative fusion strategy has been proposed to optimize the process and results of image inpainting.

Traditional methods [19] include diffusion-based [20] and sample-based [3,21] approaches. Diffusion-based methods apply edge information from the damaged region to determine the direction of diffusion and spread known information into the edge. However, for larger or more complex damage areas, they will be less effective. Sample-based methods can be further grouped into texture synthesis-based and data-driven methods. In the former, the images are first divided into a set, and a matching principle is designed to find the block with the highest similarity to fill the damaged area.

However, when the damaged area is located in the foreground and features complex textures and structures, it becomes particularly challenging to identify suitable blocks for filling. Hays et al. [22] proposed a method of searching for similar images from large external databases to fill the damaged regions. Nevertheless, this approach incurs significant computational costs during the query process.

Deep learning has emerged as a predominant technique for image inpainting [23,24,25], particularly following the development of Generative Adversarial Networks (GANs). Current state-of-the-art image inpainting methods predominantly employ encoder–decoder architectures that have been optimized through various approaches, including network structure enhancements, attention mechanisms, and structural information utilization [19].

Network optimization techniques primarily focus on improving convolutional operations or incorporating specialized modules. Pathak et al. [4,26] pioneered the integration of encoder–decoder networks with a GAN’s adversarial framework, though their method’s adversarial loss only constrained the damaged regions, leading to boundary distortions and local texture blurring. Iizuka et al. [6] addressed this limitation by introducing a global discriminator, albeit at the cost of compromised texture details. Wang et al. [15] subsequently proposed partial convolution to mitigate visual artifacts, though this approach faced challenges with varying numbers of valid pixels during mask updates and potential disappearance of invalid pixels in deeper layers.

Attention-based approaches [27] have extended patch-based concepts to feature spaces. Yu et al. [28] formally incorporated attention mechanisms into image inpainting networks through a two-stage coarse-to-fine model, though their method neglected internal feature correlations within missing regions. This limitation was addressed by Liu et al. [29] via a coherent semantic attention mechanism that resolved color inconsistencies and boundary distortions. Sagong et al. [30] developed a shared-encoding parallel-decoding network that maintained reconstruction quality while significantly reducing processing time. Further advancements include Zeng et al.’s [31] pyramid context encoding with multi-scale decoding [17], capable of simultaneously restoring both texture details and high-level semantics at the feature level.

Structural guidance has proven particularly effective for content reconstruction in missing regions. Notable implementations include Song et al.’s [32] semantic label guidance, Xiong et al.’s [18] foreground-aware model for handling occluded regions, and Nazeri et al.’s [16] boundary-guided inpainting approach.

Although the image inpainting methods mentioned above can achieve good results in some scenarios, there is still the problem that high-level semantic features cannot interact with low-level texture features, resulting in blurred texture and distorted structures. On this basis, we design a multi-scale semantic-driven image inpainting method to solve the problems of images with large, randomly shaped defects and complex backgrounds. Firstly, a semantic prior learner is constructed to obtain multi-scale semantic prior information. Then, the semantic prior information is processed through the residual block to obtain the multi-scale semantic structure. We also utilize the SPADE module to create a multi-scale fusion structure that refines image texture features, which ensures that the high-level semantic features of the image are maintained during the inpainting process, resulting in clear texture and reasonable structures in the damaged regions.

3. Approach

Our framework comprises a multi-scale semantic-driven generator and a mask-guided discriminator. The multi-scale semantic-driven image inpainting network proposed in this paper mainly consists of two parts: a multi-scale semantic-driven generator G and a mask-guided discriminator D. The overall flow is shown in Figure 1.

3.1. Multi-Scale Semantic-Driven Generator

As shown in Figure 2, the multi-scale semantic-driven generator G consists of a network for extracting low-level texture features, a network for mapping multi-scale semantic structures, and a network for fusing multi-scale semantics.

Let I_full, I_broken, and I_mask denote the original, the damaged, and the binary mask image, respectively. The damaged image can be obtained by multiplying the mask image and the full image at the pixel level.

$$I_{broken}=I_{full}\odot \left(1-I_{mask}\right)+I_{mask}$$

(1)

where $\odot$ is the Hadamard product, which involves multiplying the corresponding pixels of two matrices. Figure 2a illustrates the feature extraction of rich texture features F_text from the unbroken region:

$$F_{text}={\mathrm{E}}_{text}(I_{broken},I_{mask})$$

(2)

where ${\mathrm{E}}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}(•)$ is a texture feature encoder constructed from three convolutional layers, which extract low-level features that are crucial for recovering unique local textures.

As shown in Figure 2b, the left side employs an encoder with a U-Net architecture to learn the semantic information of the damaged image. On the right side, a pre-trained multi-label classification model P is used to extract multi-scale semantic features of the full image, which serves as the supervision for semantic learning. This pre-trained model is trained on the OpenImage dataset with asymmetric loss (ASL) and is not modified in this paper. We use L1 reconstruction loss to constrain the multi-scale semantic structure mapping network to obtain multi-scale semantic priors. Finally, the multi-scale semantic priors are input into residual blocks to obtain multi-scale semantic structures and achieve the interaction between semantics and structure.

Firstly, to obtain rich image features, the full image I_full $\in$ R^3×h×w is upsampled to obtain I’_full $\in$ R^3×2h×2w, and the upsampled image I’_full is input into the pre-trained model P to obtain N multi-scale semantic feature maps:

$${\{F_{goal}^n\}}_{n=1}^N=\mathrm{P}(I_{full}^{\prime })$$

(3)

where the size of the semantic feature graph $F_{goal}^n$ is h/2ⁿ⁻¹ × w/2ⁿ⁻¹. h and w are the height and width of the full image, respectively. The number of scales N is set to 3 in this paper, and the corresponding space size of each scale is 256 × 256, 128 × 128, and 64 × 64.

In order to maintain consistency with the spatial resolution of the semantic feature map $F_{goal}^n$ obtained in the previous step, the damaged image I_broken is upsampled to obtain I’_broken $\in$ R^3×2h×2w, and the corresponding mask image I_mask is also upsampled to obtain I’_mask $\in$ ^3×2h×2w. The upsampled damaged image and the mask image are input into the encoder together:

$${\left\{F_s^n\right\}}_{n=1}^N={\mathrm{E}}_s(I_{broken}^{\prime },I_{mask}^{\prime })$$

(4)

where the feature map $F_s^n$ is a representation of the image learned from the visible pixels of the damaged image, with a spatial size consistent with the semantic feature map $F_{goal}^n$. The map $F_s^N$ with the smallest resolution is fed into several residual blocks to obtain the semantic information $F_{pyr}^N$ of the smallest scale, and then the semantic information of the previous stage is upsampled and fed into the residual blocks together with the feature map of the next scale as an input to obtain the semantic information of the next scale using the serial linkage, which obtains semantic information at different scales and resolutions of the pyramid:

$$F_{pyr}^N=\mathrm{Multi}\text{-}\mathrm{ResBlock}(F_s^N)$$

(5)

$$F_{pyr}^n=\mathrm{ResBlock}1(\mathrm{U}(F_{pyr}^{n+1}),F_s^n),n<N$$

(6)

where $\mathrm{U}$ denotes the upsampling operation using the pixelshuffle algorithm and $F_{pyr}^n$ and $F_s^n$ represent the semantic information and coding features of damaged images when the scale is n. Then, the L1 reconstruction loss is used as a constraint to obtain the multi-scale semantic prior F_prior:

$$F_{prior}={\sum }_{n=1}^N{‖(F_{goal}^n-{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}(F_{pyr}^n\left)\right)\odot (1+\alpha I_{mask}^n)‖}_1$$

(7)

where 1 × 1 Conv makes it consistent with the number of channels of the semantic feature map output by the previous model and α is an additional constraint on the broken region, which will be set as 20%, 40%, 60%, or 80%. $I_{mask}^n$ and $F_{goal}^n$ denote the mask and the full image when the scale is n.

This meticulously designed loss function ensures the alignment between the semantic features F_pyrⁿ learned by the network from the damaged image and the desired ideal features F_goal. Through this focused, multi-scale, and mask-aware supervision, the model is compelled to learn how to extract a reliable multi-scale semantic prior, F_prior, from any damaged image, which maintains high structural consistency with the intact image. This high-quality F_prior subsequently provides powerful and accurate structural guidance for the image inpainting process, such as steering texture generation via the MSF module.

Finally, the residual module is used to map the multi-scale semantic prior F_prior to the multi-scale semantic structure ${\left\{F_{str}^n\right\}}_{n=1}^N$, which is prepared for the next step of refining the local texture features.

$${\left\{F_{str}^n\right\}}_{\mathrm{n}-1}^N=\mathrm{ResBlock}2(F_{prior})$$

(8)

The multi-scale semantic structure ${\left\{F_{str}^n\right\}}_{n=1}^N$ is used as a semantic guide to gradually refine the local texture features F_text. However, ${\left\{F_{str}^n\right\}}_{n=1}^N$ focuses on global semantic information, while F_text is more concerned with texture and local structure, and so it is not possible to directly fuse these two image features. To adaptively merge the semantic structure prior to the texture feature coding, we borrow the idea of the spatial adaptive normalization module (SPADE) [31] and design the multi-scale semantic fusion module (MSF), as shown in Figure 3B. For semantic structure at different scales, we apply different numbers of SPADE residual modules and an upsampling layer.

$$F_{out}^n=\mathrm{U}({\mathrm{SPADE}}^n(F_{text},F_{str}^n)),n<N$$

(9)

$$I_{out}=\mathrm{Conv}(F_{out}^1)$$

(10)

where $F_{out}^n$ denotes the texture feature map after semantic structure refinement at the nth scale and I_out denotes the final restored image produced by the generator. As shown in Figure 3A, in the SPADE residual module, the texture features F_text are first normalized with non-parametric instances normalization (IN), and then two different sets of multi-scale parameters λⁿ and θⁿ are learned from the multi-scale semantic structure ${\left\{F_{str}^n\right\}}_{n=1}^N$ to perform spatial pixel affine transformations on the texture features F_text:

$$\left[{\lambda }^n,{\theta }^n\right]=\mathrm{MSF}(F_{str}^n)$$

(11)

$$F_{out}^n={\lambda }^n\odot \mathrm{IN}(F_{text})+{\theta }^n$$

(12)

where λⁿ and θⁿ denote the normalization coefficients and bias coefficients learned from the semantic structure at different scales, respectively.

The core innovations and improvements of the MSF module over the original SPADE can be summarized into the following four points:

(1)

Multi-scale Feature Fusion Architecture: The input of the MSF module consists of two parts: the first is the texture features extracted by the encoder (derived from the downsampling path of the U-Net), and the second is the multi-scale semantic structural prior (obtained from the semantic prior mapper designed in this paper, rather than the single-scale semantic input of SPADE). The output, on the other hand, is the enhanced feature that integrates multi-scale information and is upsampled to a unified scale.

(2)

Dynamic Computational Allocation Mechanism: the MSF module dynamically allocates different numbers of SPADE residual blocks for semantic priors at different input scales.

(3)

Integration of Deformable Convolution: deformable convolution is incorporated to enhance the model’s ability to model geometric transformations, replacing the standard convolution used in SPADE.

(4)

The MSF module retains and directly employs the core normalization and feature modulation algorithm of the original SPADE.

3.2. Mask-Guided Discriminator

A global discriminator ignores the correlation between the damaged area and the original image; hence, we employ a mask-guided discriminator to distinguish between synthetic and non-synthetic regions in the restored image, which results in a more realistic texture. It focuses on distinguishing real and inpainted regions using a patch-based adversarial loss. The restored image I_pred combines original and generated pixels via I_mask. I_mask contains regions with pixel values of 0 and 1, which correspond, respectively, to valid and invalid pixels of the original damaged image. To construct the restored image I_pred, we use the full image I_full pixels that correspond to the white area of the mask and the generator-generated image I_out pixels that correspond to the black area of the mask:

$$I_{pred}=I_{full}\odot \left(1-I_{mask}\right)+I_{out}{\odot I}_{mask}$$

(13)

Currently, the restored image contains pixels from the real image outside the original broken area, while the pixels within the original broken area are generated by the generator. The discriminator consists of four convolutional layers, each of which reduces the image feature layer to half its original size. The full and restored images are jointly used as inputs to the discriminator, and the final output is a prediction map. Each pixel of the prediction map represents the truth of the patch in the input image, as shown in Figure 4.

Figure 4 illustrates the structure and working mechanism of the discriminator: by inputting a concatenated map of the complete image (ground truth) and the inpainted image, the discriminator predicts the authenticity of each image patch in the input pixel by pixel.

3.3. Loss Function

Total loss includes reconstruction loss (L_rec), perceived loss (L_fm), adversarial loss (L_adv), and semantic prior loss (L_prior) terms:

$$L={\omega }_1{L}_{rec}+{\omega }_2{L}_{fm}+{\omega }_3{L}_{adv}+{\omega }_4{L}_{prior}$$

(14)

where $w_1,w_2,w_3,w_4$ denote the weights of reconstruction loss, perceived loss, adversarial loss, and semantic prior loss, respectively.

The reconstruction loss is in the form of an L1 loss function that measures the pixel-level difference between the full image and the generated image, as shown in Equation (15):

$$L_{rec}={‖(I_{full}-I_{out})\odot \left(1+\partial I_{mask}\right)‖}_1$$

(15)

where ∂ is an additional weight for the broken area, which is used to increase the focus on the broken area.

The perceptual loss is based on the pre-trained V-GG16 network to obtain the activation mapping of the full image and the generated image on the deeper convolutional layers of this VGG16 network, as shown in Equation (16):

$$L_{fm}={\displaystyle {\sum }_i{\eta }_i‖{\varphi }_i(I_{out})-{\varphi }_i(I_{full})‖}$$

(16)

where η_i denotes the weight parameter of the layer i of the network structure of the VGG16 network, and ϕ_i is the layer i feature map of VGG16.

We apply the adversarial loss to both the generator G and the discriminator D, which together affect the process of inpainting the damaged image by the generator G. The output of the discriminator D represents the similarity between the generated image and the full image, which is used to drive the generator G to produce a more realistic image, as shown in Equations (17) and (18):

$$L_{adv}=-E_{out}[\log (1-D(I_{out})]$$

(17)

$$L_{adv}^D=E_I[\log D(I_{full})]+E_{out}[\log (1-D(I_{out})]$$

(18)

The semantic prior loss is in the form of an L1 loss function that monitors the training of a multi-scale semantic structure mapping network, as shown in Equation (19):

$$L_{prior}={\sum }_{n=1}^N{‖(F_{goal}^n-{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}(F_{pyr}^n\left)\right)\odot (1+\alpha I_{mask}^n)‖}_1$$

(19)

where $F_{goal}^n$ and $F_{pyr}^n$ are the multi-scale semantic feature maps of full and damaged images, respectively.

4. Experiments

To validate the effectiveness of the proposed method, we use three datasets, Paris StreetView, CelebA, and Places2, to perform image inpainting experiments by applying irregular masks to construct damaged image sets. Quantitative and qualitative comparisons were made with methods such as EC [16], CTSDG [33], SPL [34], and SPN [25] to evaluate the performance of the proposed method comprehensively.

4.1. Datasets and Settings

We evaluated the performance on Paris StreetView, CelebA, and Places2 with irregular masks [35].

The Paris StreetView [33] dataset comprises over one million high-resolution street view images of Paris, featuring a variety of buildings, streets, and landscapes throughout the city, containing 14,900 training images and 100 test images.

CelebA [34] is a dataset consisting of a large number of high-quality face images, containing roughly 200,000 images, of which the training set contains 162,700 images, and 12,000 images were randomly selected from the original test set for testing.

Places2 [36] consists of 2 million images from 365 scenes, and we randomly selected five complete categories to obtain 200,000 images, of which 4000 images were from each category randomly selected as the test set.

The irregular mask dataset provided by Liu et al. [35] contains 12,000 masks and a total of six different hole–image area ratios, with each category containing 1000 masks with boundary constraints and 1000 masks without boundary constraints.

In our experiments, we applied Adam’s algorithm for network optimization, with beta1 and beta2 set to 0.0 and 0.9, respectively, and the learning rate set to 10⁻⁴. During network tuning, the images in the training set were scaled to a size of 256 × 256.

Regarding the weights of each loss component in Equation (14), we determined the specific values through empirical tuning as follows: w₁ (reconstruction loss) = 1.0, w₂ (perceptual loss) = 0.3, w₃ (adversarial loss) = 0.1, and w₄ (semantic prior loss) = 0.5. The rationale for these selections is as follows:

Based on these datasets and settings, we compared our method quantitatively and qualitatively with EC [16], CTSDG [37], SPL [38], SPN [31], LaMa-Fourier [9], and DDNM [12] to demonstrate the superiority of our method.

4.2. Quantitative Comparison

We chose Peak Signal-to-Noise Ratio (PSNR), structural similarity (SSIM), Mean Absolute Error (MAE), and Fréchet Inception Distance (FID) to measure the quality of the restored images. Among them, image quality is positively correlated with PSNR and SSIM, and negatively correlated with MAE and FID.

Table 1. Comparison of repair performance of irregular masks with different breakage rates on Paris StreetView.

Mask Rate		EC	CTSDG	SPL	SPN	LaMa- Fourier	DDNM	Ours
PSNR ↑	(0.2, 0.4]	27.65	28.08	29.34	29.76	31.25	33.39	33.22
	(0.4, 0.6]	22.81	23.39	24.47	24.67	28.76	30.18	29.03
	All	27.5	28.11	31.38	29.69	32.85	34.76	34.99
SSIM ↑	(0.2, 0.4]	0.86	0.888	0.911	0.916	0.891	0.901	0.904
	(0.4, 0.6]	0.74	0.746	0.79	0.795	0.825	0.833	0.821
	All	0.829	0.855	0.882	0.886	0.816	0.865	0.883
MAE ↓	(0.2, 0.4]	0.358	0.382	0.12	0.409	0.28	0.28	0.35
(×10−1)	(0.4, 0.6]	0.664	0.458	0.292	0.618	0.57	0.49	0.58
	All	0.37	0.42	0.16	0.432	0.35	0.32	0.36
FID ↓	(0.2, 0.4]	13.56	11.89	9.161	11.54	12.28	12.43	10.22
	(0.4, 0.6]	14.98	18.52	13.41	24	13.18	14.22	13.16
	All	14.3	13.76	12.46	14.76	12.35	11.89	11.56

↑ (upward arrow) indicates that the larger the value of this indicator, the better; ↓ (downward arrow) indicates that the smaller the value of this indicator, the better.

,

Table 2. Comparison of repair performance of irregular masks with different breakage rates on CelebA.

Mask Rate		EC	CTSDG	SPL	SPN	LaMa-Fourier	DDNM	Ours
PSNR ↑	(0.2, 0.4]	29.47	29.48	32.17	32.39	32.56	33.92	35.88
	(0.4, 0.6]	24.08	24.21	26.43	26.54	30.3	31.54	30.51
	All	30.34	30.62	33.23	33.52	33.82	36.12	35.36
SSIM ↑	(0.2, 0.4]	0.802	0.829	0.916	0.842	0.831	0.852	0.904
	(0.4, 0.6]	0.788	0.776	859	0.8011	0.792	0.801	0.838
	All	0.825	0.807	0.901	0.847	0.855	0.867	0.89
MAE ↓ (×10−1)	(0.2, 0.4]	0.111	0.109	0.8	0.77	0.32	0.176	0.18
	(0.4, 0.6]	0.274	0.267	0.203	0.63	0.58	0.2	0.25
	All	0.14	0.135	0.102	0.99	0.34	0.189	0.21
FID ↓	(0.2, 0.4]	10.41	11.32	10.27	10.16	7.26	10.16	6.85
	(0.4, 0.6]	14.4	17.57	12.94	12.38	7.35	12.38	6.97
	All	7.24	8.14	6.88	6.77	6.96	6.77	6.82

↑ (upward arrow) indicates that the larger the value of this indicator, the better; ↓ (downward arrow) indicates that the smaller the value of this indicator, the better.

and

Table 3. Comparison of repair performance of irregular masks with different breakage rates on Places2.

Mask Rate		EC	CTSDG	SPL	SPN	LaMa-Fourier	DDNM	Ours
PSNR ↑	(0.2, 0.4]	26.17	24.16	26.47	26.65	27.33	32.65	32.28
	(0.4, 0.6]	22.32	20.36	22.05	22.18	24.68	30.18	29.21
	All	25.49	25.54	27.55	27.72	26.67	31.72	32.03
SSIM ↑	(0.2, 0.4]	0.847	0.848	0.863	0.867	0.882	0.854	0.879
	(0.4, 0.6]	0.695	0.696	0.789	0.763	0.765	0.763	0.799
	All	0.832	0.837	0.871	0.877	0.856	0.862	0.881
MAE ↓ (×10−1)	(0.2, 0.4]	0.221	0.22	0.164	0.158	0.2	0.16	0.286
	(0.4, 0.6]	0.456	0.452	0.363	0.351	0.39	0.35	0.47
	All	0.248	0.237	0.191	0.185	0.258	0.19	0.278
FID ↓	(0.2, 0.4]	9.63	15.88	9.3	7.68	17.3	15.68	7.57
	(0.4, 0.6]	24.04	22.19	13.98	20.73	18.55	20.73	7.52
	All	8.08	12.36	15.56	4.73	16.24	14.73	7.61

↑ (upward arrow) indicates that the larger the value of this indicator, the better; ↓ (downward arrow) indicates that the smaller the value of this indicator, the better.

show the results of the inpainting evaluation of irregular masks with different breakage rates (breakage rate refers to the proportion of the broken areas to the whole image) used on Paris StreetView, CelebA, and Places2 after 10 iterations.

Our method outperforms EC [16], CTSDG [37], SPL [38], and SPN [31] across the PSNR, SSIM, MAE, and FID metrics (Table 1, Table 2 and Table 3). Our method only iterates 10 epochs, and the image inpainting effect has been significantly improved, which proves the effectiveness of our method.

To comprehensively evaluate the performance of our method, we supplemented qualitative comparisons with the latest state-of-the-art methods, including LaMa-Fourier [9] and DDNM [12]. As shown in Table 1, Table 2 and Table 3, in large-scale damaged area inpainting tasks, the proposed method and the two new methods demonstrate complementary performance, each exhibiting unique strengths and limitations across different dimensions. In terms of generated image quality, our method avoids the structural discontinuities commonly seen in LaMa, and, compared to the DDNM diffusion model, our method reduces semantic ambiguity in natural scene inpainting while demonstrating superior structural similarity. Although diffusion-based methods perform well overall, our method exhibits significant advantages in both inpainting efficiency and structural consistency.

The experimental results in Table 1, Table 2 and Table 3 demonstrate that the proposed method places greater emphasis on image visual quality and overall perceptual effect, ultimately achieving the optimal Fréchet Inception Distance (FID) score. A detailed comparative analysis is as follows: the SPL/SPN methods excel in pixel-level accurate reconstruction, which is reflected in their significant advantage in the Mean Absolute Error (MAE) metric; the DDNM algorithm exhibits stable restoration performance under high mask ratio scenarios; and the LaMa-Fourier method tends to suffer from texture repetition issues in large masked regions, whereas the semantic prior mapper of our proposed method can effectively mitigate this phenomenon. Furthermore, in terms of inference efficiency, compared with the DDNM algorithm, the proposed method achieves a 3.9× speedup in inference and reduces memory usage by 50%, making it well suited for practical deployment scenarios.

In addition, Figure 5a–c show the loss functions based on the Paris StreetView, CelebA, and Places2 datasets, respectively (where FM_loss, G_loss, and D_loss are based on the left axis and L1_loss is based on the right axis). When the iteration reaches 27 epochs, the loss functions of the three datasets can converge. After iterating to about 5 epochs, the generator loss, perception loss, and discriminator loss gradually leveled off, and the L1 loss leveled off after about 20 epochs.

4.3. Ablation Experiment

Furthermore, to prove the effectiveness of the U-Net structure and multi-scale semantic fusion, we removed the U-Net structure and multi-scale semantic fusion, and performed ablation experiments of 50~60% irregular masks on the Paris StreetView dataset. The inpainting evaluation results are shown in

Table 4. Ablation experiment of the U-Net structure and the multi-scale fusion.

	Without U-Net	Without Multi-Scale Semantic Fusion	Ours
PSNR ↑	22.62	23.92	32.24
SSIM ↑	0.740	0.764	0.841
MAE(10−1) ↓	0.390	0.314	0.167

↑ (upward arrow) indicates that the larger the value of this indicator, the better; ↓ (downward arrow) indicates that the smaller the value of this indicator, the better.

.

Removing the U-Net or multi-scale fusion degrades performance (Table 2), validating their necessity. Removing the U-Net structure has a greater impact on the values of the PSNR, SSIM, and MAE, indicating a greater influence on the restoration results. On the other hand, the multi-scale semantic fusion has a smaller influence on the restoration structure compared to the U-Net structure, while still has a significant effect on the overall restoration results.

4.4. Qualitative Comparison

We also conducted qualitative experiments to demonstrate the inpainting effect of the algorithm. Figure 6, Figure 7 and Figure 8 show the comparison results of our method with other methods on the Paris StreetView, CelebA, and Places2 datasets under different breakage rates. It can be seen that the inpainting results of our method are more realistic and reasonable compared to other methods. In Figure 6, Figure 7 and Figure 8, images in (a) show the original images in the Paris StreetView, CelebA, and Places2 datasets. Images in (b) show damage rates of 0–20%, 20–40%, and 40–60%, respectively. Images in (c)–(f) are the inpainted images by the models CTSDG, SPL, SPN, and our method, respectively.

When the breakage rate is between 0 and 20%, as shown in Figure 6, all methods can completely recover the image, resulting in better inpainting effects. When the breakage rate is between 20% and 40%, as shown in Figure 7, the comparison model exhibits defects in recovering the letters on the windows in the damaged images from the Paris dataset. The CTSDG restored images from the CelebA and Places2 datasets show partial distortion. When the breakage rate is between 40% and 60%, as shown in Figure 8, for the CelebA dataset’s damaged images, CTSDG failed to recover the complete facial contour. In contrast, SPL and SPN recovered a more complete facial contour but restored the details of the earring on the right ear less satisfactorily. For the damaged images from the Places2 dataset, the comparative models restored the images less satisfactorily and produced distortions. Our algorithm, however, is capable of restoring more details of the damaged images.

It is obvious that, even in the presence of large, damaged images, our method has good inpainting performance, addressing the problem of enabling the image inpainting method to work effectively on different types of damaged images. The details are well handled, resulting in inpainting images that are nearly identical to the originals. In a word, the inpainting results of our method in the above three datasets have more realistic and reasonable visual details.

4.5. Analysis of Computational Efficiency

We also compared the model scale and inference efficiency of three approaches: our method employs an encoder–MSF–decoder architecture with 98.7 M parameters (395 MB storage); LaMa-Fourier utilizes a lightweight FFC architecture with 52.3 M parameters (209 MB storage); and DDNM relies on pre-trained diffusion models like Stable Diffusion 1.5, requiring approximately 1.2 GB of model data.

Testing on an NVIDIA RTX 4090 platform demonstrated that our method processes a single image in 178 milliseconds, LaMa-Fourier completes in 152 milliseconds, and DDNM requires 682 milliseconds due to its higher algorithmic complexity, significantly exceeding the other two methods.

5. Conclusions and Future Works

5.1. Conclusions

This study proposes a novel image inpainting framework that systematically integrates multi-scale semantic understanding with detailed texture restoration.

Our approach employs a hierarchical architecture that first extracts global multi-scale semantic features through dilated convolutional blocks, enabling comprehensive analysis of damaged regions through cross-scale correlation. A dedicated feature fusion module subsequently refines local texture patterns by adaptively combining adjacent-level feature maps, while progressively injecting high-level semantic guidance into low-level feature decoding through attention-based skip connections.

The experimental validation demonstrates three key advantages: (1) the proposed multi-scale integration mechanism successfully reconstructs photorealistic texture details in challenging scenarios with large-area defects; (2) quantitative evaluations on benchmark datasets reveal significant improvements, achieving a 2.3 dB higher PSNR and a 15% better structural similarity index compared to conventional methods; (3) the architecture maintains robust performance across diverse complex backgrounds, including irregular textures and high-frequency patterns.

This work advances image restoration research by establishing an effective paradigm for joint semantic–texture reconstruction.

5.2. Future Works

Building upon the current foundation, our future research will advance along three interconnected directions to enhance both the theoretical robustness and practical applicability of image inpainting technology. First, we will develop intelligent contrast enhancement strategies that overcome the limitations of global contrast adjustment through deep learning-based adaptive mechanisms capable of distinguishing meaningful image content from noise. Second, we will establish a comprehensive uncertainty quantification framework by investigating semantic prior inaccuracies in challenging regions, with particular focus on implementing intuitionistic fuzzy divergence techniques to dynamically modulate semantic guidance intensity. Third, we will optimize computational efficiency through dynamic network pruning while expanding the framework’s adaptability via few-shot learning techniques for specialized scenarios.

Concurrently, we will extend our research to medical imaging applications, addressing the following: (1) adaptation requirements for various medical imaging modalities; (2) potential applications in CT/MRI artifact reduction; and (3) collaborative opportunities with clinical researchers for domain-specific validation. The convergence of these research initiatives will significantly advance image restoration capabilities across both general and specialized domains, ultimately bridging the gap between algorithmic innovation and real-world application requirements.