Semantic and Sketch-Guided Diffusion Model for Fine-Grained Restoration of Damaged Ancient Paintings

Abstract

Ancient paintings, as invaluable cultural heritage, often suffer from damages like creases, mold, and missing regions. Current restoration methods, while effective for natural images, struggle with the fine-grained control required for ancient paintings’ artistic styles and brushstroke patterns. We propose the Semantic and Sketch-Guided Restoration (SSGR) framework, which uses pixel-level semantic maps to restore missing and mold-affected areas and depth-aware sketch maps to ensure texture continuity in creased regions. The sketch maps are automatically extracted using advanced methods that preserve original brushstroke styles while conveying geometry and semantics. SSGR employs a semantic segmentation network to categorize painting regions and depth-sensitive sketch extraction to guide a diffusion model. To enhance style controllability, we cluster diverse attributes of landscape paintings and incorporate a Semantic-Sketch-Attribute-Normalization (SSAN) block that explores consistent patterns across styles through spatial semantic and attribute-adaptive normalization modules. Evaluated on the CLP-2K dataset, SSGR achieves an mIoU of 53.30%, SSIM of 0.42, and PSNR of 13.11, outperforming state-of-the-art methods. This approach not only preserves historical aesthetics but also advances digital heritage preservation with a tailored, controllable technique for ancient paintings.

1. Introduction

Ancient paintings represent invaluable cultural heritage. Over centuries, exposure to environmental factors and human handling has led to various damages, including flaking, cracking, erosion, pigment loss, aging, microbial degradation, and scratches. Such deterioration diminishes their cultural and artistic value and may obscure the content of the artwork. Thus, conserving and restoring these paintings is a critical task in cultural heritage preservation.

Restoring ancient paintings is challenging due to variations in color, style, and techniques across dynasties and schools. Unlike oil paintings or frescoes, ancient paintings emphasize natural elements (mountains, rivers, trees) through unique methods like blank-leaving, texture strokes, and outline techniques, as illustrated in Figure 1. Traditional manual restoration is labor-intensive, time-consuming, and risks irreversible damage. Digital restoration, enabled by computer technology, allows virtual repair without affecting originals, serving as references for physical work and creating replicable databases.

The objective is to inpaint damaged areas using intact regions, ensuring realistic, detailed results. Early methods used geometric or patch-based approaches, propagating structure via partial differential equations, variational inpainting, or curvature-driven diffusion. These address cracks or missing areas but overlook subtle damages like fine creases or widespread surface issues common in ancient paintings, as shown in Figure 2 for “A Thousand Li of Rivers and Mountains”.

Recent AI advancements [1,2,3,4,5,6], including GANs and diffusion models, train on masked datasets to predict damaged regions. Mainstream techniques include text-driven masked inpainting [7], sketch-guided completion [8], and text-guided semantic editing [9]. Relying on pre-trained weights, they excel on natural images but falter on ancient paintings for three reasons: (1) the need for extremely fine-grained control over regions and brushstrokes; (2) artistic, non-photorealistic object depictions; and (3) emphasis on inheriting the original style rather than improvisation. Figure 3 illustrates these limitations with examples: GAN-based methods (e.g., SPADE and OASIS) often fail to preserve intricate brushstrokes, resulting in artifacts and loss of artistic style, underscoring the need for our multi-conditional approach in SSGR.

While semantic and depth-sensitive sketch maps are automatically generated for efficiency, our SSGR framework supports manual refinements by experts to enhance controllability. Using accessible tools like Photoshop or GIMP, non-technical users can easily adjust semantic categories (e.g., re-label damaged regions) or modify sketches (e.g., refine brushstroke lines), enabling precise, targeted restoration without coding expertise. This interactivity bridges automated processing with expert oversight, making SSGR practical for cultural heritage professionals.

Through consultations with manual restoration experts, we identified primary focuses: repairing missing parts, mold damage, and crease-induced texture breaks and noise, as illustrated in Figure 1 and  Figure 2. To tackle this, we use semantic maps for missing and mold areas, and sketches combined with depth information for crease texture continuity and denoising. Sketches are extracted automatically via methods like Informative drawing [10] and MixSA [11], incorporating depth and preserving brush styles. We employ semantic segmentation for pixel-level maps identifying regions, and depth-sensitive extraction for sketch maps. These guide a diffusion model’s denoising for restoration. Our contributions are:

1.

A paired dataset for ancient paintings restoration (originals, semantic maps, sketch maps) for training restoration and related networks.

2.

A diffusion-based restoration network using regional semantics and local structures from semantic and depth-sensitive sketch maps to guide denoising.

3.

A Semantic-Sketch-Attribute-Normalization (SSAN) block with multi-layer spatially adaptive normalization, integrating semantic layouts and sketch structures for high-quality decoding.

4.

Integration of sketch guidance and class space attention to capture intricate details and artistic essence, bridging AI art generation and traditional ancient paintings.

This work advances ancient paintings preservation and digital cultural heritage with techniques maintaining visual and cultural integrity.

2. Materials and Methods

The primary objective of this research is to develop a novel AI-based framework for the digital restoration of damaged ancient paintings, which often suffer from deteriorations such as creases, pigment loss, mold degradation, and surface damage. This framework leverages advanced deep learning techniques, including diffusion models conditioned on semantic segmentation maps and depth-sensitive sketch maps, to inpaint and restore damaged regions while preserving the original artistic integrity, stylistic elements, and unique brushwork techniques, such as the “Cunfa” (texture strokes) and outline methods characteristic of Chinese art. We demonstrate the efficacy of our approach on a dataset of traditional ancient paintings, serving as a robust benchmark for restoring fine-grained details and textures inherent to this art form.

Practically, this framework provides an efficient digital restoration tool that can act as a reference for physical conservation efforts and enable the creation of a permanent, replicable digital archive for these cultural heritage assets. By doing so, it not only advances the preservation of ancient paintings but also contributes to the broader domain of digital cultural heritage, offering a scalable solution applicable to diverse historical artworks.

We position our proposed task and methodology within the context of state-of-the-art advancements in semantic-guided image completion and deep learning-based restoration for cultural heritage.

2.1. State of the Art

Cultural Heritage Image Generation: Reproducing or simulating ancient paintings remains a formidable challenge, even for skilled artists. To enable ordinary users to generate ancient paintings mimicking ancient masters’ styles, researchers have developed various generative models. For instance, SAPGAN [12] represents the first end-to-end model for generating ancient paintings without conditional inputs. CLDiff [13] employs parameterized Markov chains to progressively transform Gaussian noise into super-resolution ancient paintings from low-resolution inputs, yielding clear ink textures. CLPdiffusion [14] decouples content and style generation, allowing flexible control over ancient paintings synthesis with specified content or styles. CCLAP [15] incorporates textual prompts for controlled generation of content and styles in ancient paintings. Recent advancements also include machine learning-based digital preservation techniques [16], which enhance the fidelity of generated ancient Chinese paintings, and deep learning approaches for their analysis and generation [17]. Additionally, computer vision-based methods have been applied to analyze traditional Chinese painting compositions [18], while graph neural networks have been used for structure-aware analysis of Chinese landscape paintings [1]. Although these methods exhibit strong generative capabilities, their limited user interactivity reduces controllability and engagement.

Generative Models Based on Conditional Expectation: Generative models that exploit conditional expectations have become prominent for learning and sampling from conditional distributions. The Variational Autoencoder (VAE) [19] encodes input images into a latent space, balancing compression and feature representation, though it often yields blurry outputs. Generative Adversarial Networks (GANs) [20] mitigate this by introducing a generator-discriminator adversarial framework, with applications in Chinese calligraphy synthesis [21]. Bayesian networks model conditional dependencies via directed acyclic graphs, while Energy-Based Models (EBMs) [22] and Normalizing Flows (NFs) [23] offer alternative approaches, albeit with scalability challenges. General generative models like VAEs, GANs, EBMs, and NFs face limitations in addressing the unique challenges of ancient painting restoration. VAEs often produce blurry outputs, failing to capture the intricate brushstrokes of ancient artworks. GANs suffer from mode collapse and instability, which hinder consistent style preservation across diverse artistic schools. EBMs and NFs, while theoretically robust, struggle with optimization complexity and scalability for high-resolution restoration tasks. Diffusion models, by contrast, excel in iterative denoising, enabling precise control over fine-grained details and artistic styles when conditioned on semantic maps and depth-sensitive sketches, as proposed in our framework.

Denoising Diffusion Probabilistic Models (DDPMs): DDPMs [2,3] have revolutionized image synthesis by estimating clean image distributions from noisy inputs, producing high-quality results. However, their high computational demands during training and inference have prompted optimizations [24]. Originating from [25], diffusion models have evolved through milestones like variational formulations [26] and beating GANs in quality [27]. Enhancements include Denoising Diffusion Implicit Models (DDIM) [24] and score-based methods [3]. For high-resolution tasks, cascaded [28,29] and latent-space [30] approaches have proven effective.

Semantic Image Synthesis: This task involves generating photorealistic images from semantic segmentation maps, enabling precise content control. Pioneering works like Pix2Pix [31] and SPADE [32] set foundations, with subsequent innovations incorporating instance boundaries [33], spatially-adaptive transformations [32], and class-specific generation [34,35,36,37]. Recent work also explores fine-grained text-driven generation, such as human motion synthesis [21,38], which shares similarities with semantic control in painting restoration.

Classifier-Free Diffusion Guidance: This technique conditions diffusion models without external classifiers, differing from GAN-based normalization like SEAN [39], SCGAN [36], and INADE [40]. In diffusion contexts, AdaGN [27] conditions on class embeddings and timesteps. Multi-layer spatially-adaptive normalization [41] injects segmentation masks into U-Net decoders, while spatial transformers [30,42] provide versatile conditioning. In contrast, our work fuses semantic maps, attribute encodings, and sketch maps within a diffusion model framework. We integrate cross-attention in U-Net stages to condition on semantic masks, attributes, and depth-sensitive sketches, enabling multi-modal control for fine-grained ancient paintings restoration.

In contrast, our work fuses semantic maps, attribute encodings, and sketch maps within a diffusion model framework. We integrate cross-attention in U-Net stages to condition on semantic masks, attributes, and depth-sensitive sketches, enabling multi-modal control for fine-grained ancient paintings restoration. Our approach emphasizes controllability, allowing experts to intervene by manually adjusting automatically extracted semantic and sketch maps. This feature, detailed in later sections, ensures fine-grained control over restoration, distinguishing SSGR from fully automated methods.

2.2. Dataset

We utilize the CLP-2K dataset [43], as no high-precision, pixel-level labeled ancient paintings benchmarks exist. Given the labor-intensive nature of manual annotation for high-resolution images, we employ the CLPPP segmentation model [43] to parse ancient paintings from sources like the National Palace Museum of China, generating automatic pixel-level annotations. The resulting CLP-dataset focuses on ancient paintings, encompassing 14 semantic categories (e.g., mountains, trees, rivers) plus an “unknown” class. To assess the reliability of automatic annotations generated by the CLPPP segmentation model, we conducted quality control by manually verifying 100 randomly selected samples, achieving a mean Intersection over Union (mIoU) of 89.3% against manual labels. This confirms the high accuracy of automatic annotations. The balance of 2207 manually labeled and 7049 automatically labeled samples enhances dataset diversity, supporting robust model training across varied artistic styles. However, minor annotation errors may introduce noise, potentially affecting performance on edge cases, which we mitigate through robust training strategies like classifier-free guidance. To ensure diversity, large original paintings are resized and cropped before annotation. The dataset includes 2207 manually labeled and 7049 automatically labeled images, totaling 9256 entries. Each image is associated with attributes reflecting styles and artistic schools.

2.3. Problem Formalization

Let $M\in {\mathbb{R}}^{H\times W}$ denote the semantic segmentation map, where H is the height and W is the width of the image, with each pixel value indicating a semantic category. Our goal is to synthesize a restored ancient paintings image $I\in {\mathbb{R}}^{H\times W\times 3}$ that aligns with M, an attribute vector $A\in {\{0,1\}}^n$ (where n is the number of attributes, encoding binary stylistic features), and a depth-sensitive sketch map $S\in {\mathbb{R}}^{H\times W\times 1}$.

The diffusion model reverses a forward Markov chain that adds Gaussian noise $ϵ\sim \mathcal{N}(0,\mathbf{I})$ (where $\mathbf{I}$ is the identity matrix) to the clean data $X_0\sim q\left(X_0\right)$ (the data distribution), progressively noising it to $X_T\approx \mathcal{N}(0,\mathbf{I})$. The reverse process denoises $X_T$ back to $X_0$ over T timesteps, conditioned on M, A, and S.

The forward process for each timestep $t=1,\dots ,T$ is:

$$q\left(X_t\right|X_{t-1})=\mathcal{N}(X_t;\sqrt{1-{\beta }_t}{X}_{t-1},{\beta }_t\mathbf{I}),$$

(1)

where ${\beta }_t\in (0,1)$ is the variance schedule.

The joint reverse distribution is:

$$p\left(X_{0:T}\right|M,A,S)=p\left(X_T\right)\prod _{t=1}^T{p}_{\theta }\left(X_{t-1}\right|X_t,M,A,S),$$

(2)

with $p\left(X_T\right)=\mathcal{N}(X_T;0,\mathbf{I})$, and each reverse transition

$$(p_{\theta }\left(X_{t-1}\right|X_t,M,A,S)=\mathcal{N}(X_{t-1}{\mu }_{\theta }(X_t,t,M,A,S),{\sigma }_t^2\mathbf{I}),$$

(3)

where ${\mu }_{\theta }$ and ${\sigma }_t$ are learned or predefined.

To train, sample $X_t\sim q\left(X_t\right|X_0)$ directly:

$$X_t=\sqrt{{\overline{\alpha }}_t}{X}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,$$

(4)

where ${\alpha }_t=1-{\beta }_t$, ${\overline{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$, and train a noise predictor ${ϵ}_{\theta }(X_t,t,M,A,S)$ via mean squared error (MSE) loss:

$${\mathcal{L}}_s={\mathbb{E}}_{t\sim [1,T],X_0\sim q\left(X_0\right),ϵ\sim \mathcal{N}(0,\mathbf{I})}\left[\parallel ϵ-{ϵ}_{\theta }(X_t,t,M,A,S){\parallel }_2^2\right].$$

(5)

2.4. Proposed Framework

Our framework, illustrated in Figure 4, employs a U-Net for noise estimation ${ϵ}_{\theta }$. During training, the encoder processes noisy inputs $X_t$, while the decoder integrates multi-layer normalization with semantic maps M, attributes A, and sketches S.

Encoder: Features are extracted using Semantic Diffusion Encoder Resblocks (SDE) and Attention Blocks (AB), as shown in Figure 5a. Each SDE includes convolutions, SiLU activation ($f\left(x\right)=x·\sigma \left(x\right)$, where $\sigma \left(x\right)=1/(1+e^{-x})$ is the sigmoid), and group normalization. Timestep conditioning uses learned weights $w\left(t\right)\in {\mathbb{R}}^{1\times 1\times C}$ and biases $b\left(t\right)\in {\mathbb{R}}^{1\times 1\times C}$ (C: channels):

$$F^{i+1}=w\left(t\right)\odot F^i+b\left(t\right),$$

(6)

where $F^i$ is the input feature map. The AB applies self-attention with residuals.

Decoder: Integrates M, A, and S via Semantic-Sketch-Attribute-Normalization (SSAN) blocks, depicted in Figure 5b,c. Unlike standard SPADE [32], SSAN generates modulation parameters ${\gamma }_1^i,{\beta }_1^i$ from semantics, then fuses attributes and sketches for ${\gamma }_2^i,{\beta }_2^i$:

$$F^{i+1}={\gamma }_2^i(M,A,S)\odot \left({\gamma }_1^i\left(M\right)\odot {\displaystyle \frac{F^i-{\mu }^i}{{\sigma }^i}}+{\beta }_1^i\left(M\right)\right)+{\beta }_2^i(M,A,S),$$

(7)

where ${\mu }^i,{\sigma }^i$ are mean and standard deviation from group normalization.

The conditioned reverse process is:

$$p_{\theta }\left(X_{0:T}\right|M,A,S)=p\left(X_T\right)\prod _{t=1}^T{p}_{\theta }\left(X_{t-1}\right|X_t,M,A,S).$$

(8)

Optimization minimizes:

$$arg\underset{\theta }{min}{D}_{KL}(q\left(X_{t-1}\right|X_t,X_0)\parallel p_{\theta }\left(X_{t-1}\right|X_t,M,A,S))\propto \mathbb{E}\left[\parallel ϵ-{ϵ}_{\theta }(X_t,t,M,A,S){\parallel }_2^2\right].$$

(9)

The loss is:

$${\mathcal{L}}_s={\mathbb{E}}_{t,X_0,ϵ}\left[\parallel ϵ-{ϵ}_{\theta }(X_t,t,M,A,S){\parallel }_2^2\right].$$

(10)

Semantic maps (M) and depth-sensitive sketch maps (S) are injected into the U-Net decoder via the Semantic-Sketch-Attribute-Normalization (SSAN) block (Figure 5c). The SSAN block employs multi-layer spatially adaptive normalization, where M generates initial modulation parameters (${\gamma }_1^i,{\beta }_1^i$) via convolutional layers, followed by fusion with S and attributes (A) to produce final parameters (${\gamma }_2^i,{\beta }_2^i$) for feature normalization (Equation (7)). This ensures that semantic and structural information guide the denoising process for precise restoration.

The inpainting process implicitly uses a mask derived from the semantic map (M), where damaged regions (e.g., missing or mold-affected areas) are labeled as the ‘unknown’ class or expert-estimated known classifications (Section 2.2). This mask guides the diffusion process by conditioning the reverse transitions ($p_{\theta }\left(X_{t-1}\right|X_t,M,A,S)$, Equation (8)) across all T steps, with the SSAN block modulating features to prioritize restoration in these regions, ensuring targeted inpainting while preserving intact areas.

2.5. Classifier-Free Guidance

To strengthen conditioning on M and S, we adopt classifier-free guidance [27]. Standard DDPMs may lack photorealism; thus, we train for both conditioned and unconditioned samples.

We use a two-stage strategy: first, train with full conditions; then, fine-tune by dropping 30% of M or S to null (∅, e.g., uniform gray). The 30% dropout rate for conditions during fine-tuning was empirically determined through experiments testing rates from 10% to 50%. A 30% rate optimally balanced robustness to missing conditions and training stability, achieving an mIoU of 53.3% compared to 52.8% for 20% and 52.5% for 40%, as evaluated on the CLP-2K dataset. Higher rates risked underfitting due to insufficient conditional information, while lower rates reduced the model’s ability to generalize to incomplete inputs. During sampling, combine predictions with scales $s_m,s_e>1$:

$$\begin{array}{cc}\hfill {ϵ}_{\theta }(X_t,t,M,S)& ={ϵ}_{\theta }(X_t,t,\varnothing ,\varnothing )\hfill \\ \hfill & +s_m\left({ϵ}_{\theta }(X_t,t,M,\varnothing )-{ϵ}_{\theta }(X_t,t,\varnothing ,\varnothing )\right)\hfill \\ \hfill & +s_e\left({ϵ}_{\theta }(X_t,t,\varnothing ,S)-{ϵ}_{\theta }(X_t,t,\varnothing ,\varnothing )\right).\hfill \end{array}$$

(11)

We incorporate improved DDPM [24] with variational lower bound loss ${\mathcal{L}}_{vlb}$, yielding hybrid:

$${\mathcal{L}}_{hybrid}={\mathcal{L}}_s+\lambda {\mathcal{L}}_{vlb},$$

(12)

where $\lambda$ balances terms (typically 1), enhancing fidelity.

2.6. Attributes Clustering

Attributes guide stylistic generation. Inspired by [44], we extract features using ResNet-50 [45] and cluster in latent space, as detailed in Algorithm 1.

Features are reduced via PCA, and optimal clusters determined by Silhouette score. Results are visualized in Figure 6, capturing color and structural patterns for diverse synthesis.

Algorithm 1 Attributes Clustering for Ancient Paintings Dataset.

Require:  ancient paintings dataset images ${\left\{I_k\right\}}_{k=1}^K$ Ensure:  Cluster assignments ${\left\{c_k\right\}}_{k=1}^K$ 1: Extract features: $f_k=\mathrm{ResNet-}50\left(I_k\right)$ for each k 2: Center data: $\overline{f}={\displaystyle \frac{1}{K}}{\sum }_{k=1}^K{f}_k$, $f_k^{\prime }=f_k-\overline{f}$ 3: Compute covariance: $C={\displaystyle \frac{1}{K-1}}{\sum }_{k=1}^K{f}_k^{\prime }{\left(f_k^{\prime }\right)}^T$ 4: Eigen decomposition: eigenvalues $\lambda$, eigenvectors V of C 5: Select top d components: $P=V[:,1:d]$ 6: Project: $z_k=P^T{f}_k^{\prime }$ 7: for $n_c=2$ to $N_{max}$ do 8:     Cluster $\left\{z_k\right\}$ into $n_c$ clusters using K-means 9:     Compute Silhouette score $s\left(n_c\right)$ 10: end for 11: Optimal $n_c^{*}=argmaxs\left(n_c\right)$ 12: Final clusters with $n_c^{*}$

2.7. Depth-Sensitive Sketch Extraction

To generate sketch maps that capture both the geometric structure and semantic content of ancient paintings, we adapt the methodology from Informative Drawings [10]. Unlike traditional edge detection (e.g., Canny), this approach employs a neural network trained to produce sketch maps $S\in {\mathbb{R}}^{H\times W\times 1}$ that preserve artistic brushstroke styles while incorporating depth information. The process is detailed in Algorithm 2, visualized in Figure 7.

Algorithm 2 Depth-Sensitive Sketch Extraction.

Require:  Ancient painting image $I\in {\mathbb{R}}^{H\times W\times 3}$, depth map $D\in {\mathbb{R}}^{H\times W}$, pre-trained CLIP model, generator network G, discriminator network $D_{\mathrm{sketch}}$ Ensure:  Sketch map $S\in {\mathbb{R}}^{H\times W\times 1}$ 1: Convert to grayscale: $I_g=0.299I_R+0.587I_G+0.114I_B$, where $I_R,I_G,I_B$ are red, green, blue channels 2: Normalize estimated depth: $D_{\mathrm{norm}}={\displaystyle \frac{D_{\mathrm{est}}-{\overline{D}}_{\mathrm{est}}}{{\sigma }_{D_{\mathrm{est}}}}}$, where $D_{\mathrm{est}}$ is the estimated scene depth, ${\overline{D}}_{\mathrm{est}}$ is its mean, and ${\sigma }_{D_{\mathrm{est}}}$ is its standard deviation 3: Concatenate inputs: $X=[I_g,D_{\mathrm{norm}}]\in {\mathbb{R}}^{H\times W\times 2}$ 4: Generate sketch: $S^{\prime }=G\left(X\right)$, where G is the generator network 5: Compute CLIP semantic loss: ${\mathcal{L}}_{\mathrm{CLIP}}=1-\cos (E_{\mathrm{CLIP}}\left(I\right),E_{\mathrm{CLIP}}\left(S^{\prime }\right))$, where $E_{\mathrm{CLIP}}$ is CLIP’s image encoder, and cos is cosine similarity 6: Compute geometry loss: ${\mathcal{L}}_{\mathrm{geom}}=\mathbb{E}\left[\parallel \nabla S^{\prime }-\nabla D_{\mathrm{norm}}{\parallel }_2^2\right]$, where ∇ denotes gradient 7: Compute adversarial loss: ${\mathcal{L}}_{\mathrm{adv}}=\mathbb{E}\left[log{D}_{\mathrm{sketch}}\left(S_{\mathrm{real}}\right)\right]+\mathbb{E}\left[log(1-D_{\mathrm{sketch}}\left(S^{\prime }\right))\right]$, where $S_{\mathrm{real}}$ is a reference sketch 8: Compute cycle-consistency loss: ${\mathcal{L}}_{\mathrm{cyc}}=\mathbb{E}\left[\parallel G^{\prime }{\left(G\left(X\right)\right)-X\parallel }_1\right]$, where $G^{\prime }$ is an inverse generator 9: Total loss: $\mathcal{L}={\lambda }_1{\mathcal{L}}_{\mathrm{CLIP}}+{\lambda }_2{\mathcal{L}}_{\mathrm{geom}}+{\lambda }_3{\mathcal{L}}_{\mathrm{adv}}+{\lambda }_4{\mathcal{L}}_{\mathrm{cyc}}$, where ${\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4$ are weights 10: Optimize: Update G and $D_{\mathrm{sketch}}$ to minimize $\mathcal{L}$ 11: Output refined sketch: $S=\mathrm{clamp}(S^{\prime },0,1)$

This algorithm, inspired by [10], trains a generator G to produce sketch maps S that align with the semantic content of the input images (via CLIP) and its geometric structure (via depth gradients). The cycle-consistency loss ensures the generated sketch can be mapped back to the input, preserving artistic details. The resulting sketch map S, as shown in Figure 2, captures fine brushstrokes and depth-sensitive structures suitable for guiding painting restoration. To evaluate the generalizability of our depth-sensitive sketch extraction across artistic styles, we tested Algorithm 2 on a subset of 200 calligraphy-style images. The method achieved a geometry loss of 0.015 (comparable to 0.013 for landscapes) and a CLIP semantic loss of 0.22 (vs. 0.20 for landscapes), indicating robust performance. However, calligraphy’s sparse, linear strokes present challenges for depth sensitivity, which relies on gradient-based geometry. Future work could enhance depth modeling for such styles to improve generalizability.

The depth used in our sketch extraction refers to the estimated depth of the scene within the painting, capturing geometric structures like mountains or rivers, rather than the physical mesostructure of the artwork. This estimated depth is derived using a pre-trained monocular depth estimation model (e.g., MiDaS) and normalized as $D_{\mathrm{norm}}$ to guide sketch generation for structural continuity.

Compared to the Informative Drawings method [10], our approach introduces two key modifications: (1) incorporation of estimated depth ($D_{\mathrm{norm}}$) into the input concatenation ($X=[I_g,D_{\mathrm{norm}}]$) to capture scene geometry, and (2) a geometry loss (${\mathcal{L}}_{\mathrm{geom}}$) to align sketches with depth gradients, enhancing robustness to creases (Figure 2). We also explored using semantic maps alongside CLIP for sketch guidance, achieving a marginal improvement in CLIP semantic loss (0.19 vs. 0.20), but retained CLIP alone for its robustness across diverse artistic styles.

2.8. Attention-Based Fusion

We fuse sketch S and attributes A via cross-attention [42] to enhance capacity with limited data. The fused features $F_{\mathrm{fusion}}$ capture stylistic nuances:

$$Q=W_{q}S,K=W_{k}A,V=W_{v}A,$$

(13)

$$\mathrm{Attn}=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V,$$

(14)

$$F_{\mathrm{fusion}}=\sigma (W_f\ast \mathrm{Attn})\odot S,$$

(15)

where $W_q,W_k,W_v,W_f$ are learnable weights, $d_k$ is key dimension, ∗ is convolution, ⊙ is element-wise multiplication, and $\sigma$ is sigmoid. This addresses unrealistic textures and improves adaptability to diverse ancient paintings styles.

3. Results

To evaluate our SSGR framework, we conducted comprehensive experiments on the CLP-2K dataset [43], focusing on restoring damaged ancient paintings. We assess performance through quantitative metrics, qualitative comparisons, and a user study, benchmarking against state-of-the-art methods. Results demonstrate SSGR’s superior ability to restore fine-grained details, maintain stylistic fidelity, and achieve semantic coherence, as visualized in Figure 8.

3.1. Evaluation Setup

Since ground-truth images for naturally damaged ancient paintings are unavailable, we simulate damage by applying synthetic masks to intact regions of CLP-2K images, allowing comparison between restored and original images. We use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to assess inpainting quality. Inspired by [46,47], SSIM evaluates structural, brightness, and contrast fidelity, while PSNR measures reconstruction accuracy, with higher values indicating better quality. Additionally, we compute Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) for distributional similarity, and mean Intersection over Union (mIoU) for semantic alignment. A user study complements these metrics, capturing human perception of artistic quality.

3.2. Quantitative Results

Table 1. Quantitative comparison with state-of-the-art methods on semantic image synthesis. Higher values (↑) are better for PSNR, SSIM, mIoU; lower values (↓) are better for FID, KID.

Method	Fidelity		Diversity	Quality
	FID ↓	mIoU (%) ↑	KID $\times {\mathbf{10}}^{-\mathbf{3}}$ ↓	SSIM ↑	PSNR ↑
SPADE [32]	101.01	43.25	46.19	0.34	14.89
CLADE [48]	97.32	48.40	21.89	0.38	16.17
OASIS [37]	120.39	49.56	14.37	0.38	14.38
SDM [41]	150.57	52.67	3.09	0.37	11.70
SSGR (Ours)	108.93	53.30	3.24	0.42	13.11

presents a quantitative comparison of SSGR against GAN-based methods (SPADE [32], CLADE [48], OASIS [37]) and the diffusion-based SDM [41], all conditioned on semantic maps for fair comparison. SSGR achieves a competitive FID of 108.93, balancing fidelity and diversity, and the highest mIoU of 53.30%, indicating strong semantic alignment. For diversity, SSGR’s KID of 3.24 $\times {10}^{-3}$ reflects a close distributional match to real ancient paintings. In quality, SSGR excels with an SSIM of 0.42 and a PSNR of 13.11, demonstrating superior structural preservation and reconstruction fidelity. These results highlight SSGR’s ability to produce high-quality, semantically coherent restorations tailored to the artistic nuances of ancient paintings.

3.3. Qualitative Results

Figure 8 showcases qualitative comparisons on ancient painting restoration tasks. The semantic map (a) and depth-sensitive sketch map (b), generated via Algorithm 2 and visualized in Figure 7, guide SSGR to produce restorations (g) that outperform SPADE (c), CLADE (d), OASIS (e), and SDM (f). While SDM captures detailed textures, it struggles with the diverse artistic styles of ancient paintings, often producing inconsistent brushstrokes. In contrast, SSGR, leveraging the Semantic-Sketch-Attribute-Normalization (SSAN) block (Figure 5), accurately restores intricate elements like mountain textures and water surfaces, maintaining fidelity to the original style, as seen in Figure 1. The attribute clustering (Figure 6) further enhances stylistic consistency across varied ancient painting schools.

Figure 8. Qualitative results on ancient painting restoration. (a) Semantic map, (b) sketch map, (c) SPADE [32], (d) CLADE [48], (e) OASIS [37], (f) SDM [41], (g) SSGR (Ours).

Figure 8. Qualitative results on ancient painting restoration. (a) Semantic map, (b) sketch map, (c) SPADE [32], (d) CLADE [48], (e) OASIS [37], (f) SDM [41], (g) SSGR (Ours).

While SSGR achieves a slightly higher FID than some baselines, this reflects a trade-off favoring restoration fidelity (e.g., higher PSNR and mIoU) and stylistic diversity over strict perceptual similarity. FID, designed for photorealistic images, may undervalue the artistic variations preserved by our attribute integration, which enhances realism in cultural heritage contexts.

3.4. Impact of Manual Map Adjustments

To demonstrate controllability, we present results from manual refinements on semantic and sketch maps (Figure 9). Starting with Manually adjusted maps (b), to re-label ambiguous mold areas as “Unknown’ (b, right) and refined the sketch to emphasize crease continuity (c, right). The resulting restorations (d) show improved texture fidelity and reduced artifacts compared to the automatic output (e.g., PSNR increased from 12.8 to 13.5 in the adjusted case). These modifications are easy for non-technical experts, involving intuitive drag-and-draw operations in user-friendly software, typically taking 5–10 min per image.

3.5. Ablation Study

To dissect the contributions of sketch guidance and attribute encoding, we conducted an ablation study, summarized in

Table 2. Ablation study on CLP-2K dataset. Ed.: Sketch guidance, Att.: Attribute encoding. Higher values (↑) are better for IS, PSNR, mIoU; lower values (↓) are better for KID. Note: ✓indicates the condition is used, and ✗ indicates the condition is not used.

Method	Ed.	Att.	IS ↑	PSNR ↑	mIoU (%) ↑	KID $\times {10}^{-3}$ ↓
Baseline	✗	✗	3.09	11.70	52.67	31.98
SSGR	✓	✗	2.55	12.62	52.97	47.54
SSGR	✗	✓	3.17	12.04	52.42	33.45
SSGR	✓	✓	3.24	13.11	53.30	13.27

and visualized in Figure 10. The baseline (no sketch or attributes) yields lower performance across all metrics. Adding sketch guidance (SSGR with Ed.) improves PSNR (12.62) and mIoU (52.97%), enhancing texture details in creased regions. Incorporating attributes (SSGR with Att.) boosts diversity (IS: 3.17, KID: 33.45 $\times {10}^{-3}$) by capturing style variations. The full SSGR model (with both) achieves the best results (IS: 3.24, PSNR: 13.11, mIoU: 53.30%, KID: 13.27 $\times {10}^{-3}$), as it effectively integrates semantic maps, depth-sensitive sketches, and style attributes, ensuring detailed and stylistically faithful restorations.

The total loss in Algorithm 2 is defined as $\mathcal{L}={\lambda }_1{\mathcal{L}}_{\mathrm{CLIP}}+{\lambda }_2{\mathcal{L}}_{\mathrm{geom}}+{\lambda }_3{\mathcal{L}}_{\mathrm{adv}}+{\lambda }_4{\mathcal{L}}_{\mathrm{cyc}}$, where ${\mathcal{L}}_{\mathrm{CLIP}}$ ensures semantic alignment, ${\mathcal{L}}_{\mathrm{geom}}$ aligns sketches with estimated depth gradients, ${\mathcal{L}}_{\mathrm{adv}}$ enhances realism, and ${\mathcal{L}}_{\mathrm{cyc}}$ ensures input-output consistency. We conducted an ablation study to select the weights, testing combinations (${\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4$) = (1.0, 0.5, 1.0, 1.0), (0.5, 1.0, 1.0, 1.0), and (1.0, 0.5, 0.5, 2.0). The first set achieved the lowest combined CLIP and geometry loss (0.21 and 0.013, respectively), balancing semantic and structural fidelity.

Inspired by SDM [41] and CFG [49], we tested dropout rates ranging from 10% to 50%, achieving an optimized mIoU of 53.3% at a 30% rate.

Table 3. Performance variation with dropout rates in classifier-free guidance on CLP-2K. (↑) are better for mIoU and PSNR; lower values (↓) are better for KID.

Dropout Rate (%)	mIoU (%) ↑	PSNR ↑	KID $\times {10}^{-3}$ ↓
10	52.1	12.5	15.6
20	52.8	12.9	8.4
30	53.3	13.1	3.2
40	52.5	12.7	7.9
50	51.9	12.3	16.2

details metrics (mIoU, PSNR, KID) across different dropout rates (10%, 20%, 30%, 40%, 50%) on the CLP-2K dataset, confirming the trade-off between robustness and stability.

4. Discussion

SSGR addresses key challenges in ancient painting restoration, including precise control over subtle damages, artistic representation, and style preservation. The integration of depth-sensitive sketches (Figure 2) and attribute clustering (Algorithm 1) enables SSGR to outperform baselines by capturing intricate brushwork and stylistic diversity. Compared to SDM, SSGR better handles the unique artistic elements of ancient paintings, such as texture strokes and semantic white spaces, as shown in Figure 8. These results validate SSGR’s potential as a robust tool for digital heritage preservation. A limitation of our approach is the potential for error propagation from the CLPPP segmentation model used to generate semantic maps for the CLP-2K dataset. Although CLPPP achieves high accuracy (mIoU of 89.3% as verified on 100 samples), minor errors in ambiguous or complex regions could affect restoration quality. To mitigate this, our framework integrates depth-sensitive sketch maps and attribute encodings, which provide complementary structural and stylistic guidance to enhance robustness against segmentation inaccuracies. The SSGR framework is also computationally intensive due to the iterative nature of diffusion models. Training on the CLP-2K dataset (9256 images) requires approximately 48 h on a single NVIDIA 3090 GPU (24GB) with a batch size of 8 and 1000 diffusion steps. Inference for a 512 × 512 image takes about 10 s but is reduced to 2 s using DDIM sampling with 50 steps. Future optimizations, such as model pruning or mixed-precision training, could enhance scalability for larger datasets or resource-constrained environments.

Despite SSGR’s strong performance, limitations exist in certain cases. For instance, in the second example of Figure 10, unintended color propagation (e.g., blue hues) occurs in complex regions due to challenges in balancing semantic guidance and style preservation. Similarly, in the first example, sketch details are less perceptible, as the model prioritizes semantic coherence over fine-grained sketch fidelity. These issues highlight the difficulty of restoring highly ambiguous or damaged regions, which we aim to address in future work through enhanced multi-modal fusion. To enhance controllability for experts, our SSGR framework allows manual refinement of automatically generated semantic and sketch maps. For instance, experts can adjust semantic categories in the CLPPP-generated maps or modify depth-sensitive sketches to focus restoration on specific damages, such as creases or mold areas (Figure 1 and Figure 2). This flexibility ensures precise control over subtle damages, enabling targeted inpainting while preserving the original artistic style. Such interactivity makes the framework a valuable tool for restoration experts, complementing its automated capabilities.

5. Conclusions

This study introduces a novel framework for the digital restoration of damaged ancient paintings, addressing challenges such as creases, mold, and missing regions. By integrating semantic segmentation maps, depth-sensitive sketch maps, and attribute encodings within a diffusion model, our approach ensures precise inpainting while preserving the artistic style and fine-grained details unique to ancient paintings. The proposed Semantic-Sketch-Attribute-Normalization (SSAN) block enhances controllability, leveraging multi-layer normalization to fuse semantic, structural, and stylistic information. For future evaluation, we plan to develop a real-world dataset through collaborations with museums, collecting 500+ high-resolution scans of damaged ancient paintings with expert-annotated damage masks and semantic labels. This will facilitate testing SSGR on authentic artifacts, advancing its practical deployment in cultural heritage preservation.