Diff-GTISR: Guided Thermal Image Super-Resolution via Diffusion Model and Refinement

Abstract

This paper presents Diff-GTISR, a novel diffusion-based model for achieving super-resolution in thermal images guided by a high-resolution visible image. Thermal sensors are widely used in surveillance, safety, and industrial inspection; however, their limited spatial resolution constrains thermal image quality because of the low resolution. Thermal image super-resolution is thus critical to compensate for this limitation. The increasing prevalence of multisensor platforms has resulted in the availability of high-resolution visible images, providing effective guidance to enhance thermal image resolution. Recently, diffusion-based super-resolution has demonstrated strong capability in recovering perceptually plausible details; however, such models often underperform in distortion-oriented metrics compared with transformer-based approaches. To address this gap, the proposed Diff-GTISR method employs a modality-specific dual encoder to extract multiscale features and a cross-modal guidance attention module to transfer structural information from visible images into low-resolution thermal images. Also, a refinement network is employed to improve the method further. The experimental results indicate that Diff-GTISR consistently enhances perceptual quality in comparison to state-of-the-art diffusion-based methods. Furthermore, it is superior to transformer-based methods in terms of distortion performance.

1. Introduction

Thermal sensors detect long-wavelength infrared rays emanating from objects and convert them into electrical signals, which are subsequently reconstructed into images by thermal cameras. Thermal images reflect the temperature distribution of the captured scene and temperature differences between objects, making them advantageous for detecting heat sources. Consequently, they are widely utilized across various fields such as security, surveillance, industrial diagnostics, pedestrian detection, and disaster response [1,2]. On the other hand, visible light sensors detect rays within the visible spectrum perceivable by humans. Visible images are generally rich in spatial details such as boundaries, colors, and textures. However, image quality degrades under low illumination or sudden lighting changes, where identifying objects can become difficult since the color or texture contrast between the subject and background is low. Due to these complementary characteristics, multimodal applications combining thermal and visible light have been expanding [3,4].

Thermal image sensors have lower spatial resolution than visible light sensors generally, limiting thermal images to a lower quality. High-resolution (HR) thermal sensors are extremely expensive and impose a significant cost burden on an imaging system. Low-resolution (LR) thermal sensors are thus used in commercial and practical applications, which can potentially degrade recognition and analysis performance. To overcome this problem, thermal image super-resolution (TISR) that enhances the resolution of LR thermal images has emerged as a research area. A conventional approach to super-resolution (SR) research is single image super-resolution (SISR), which generates an HR image from its single LR image. Recently, deep learning-based SISR has achieved substantial progress primarily in the visible domain [5,6,7,8,9,10]. Subsequently, research applying SISR to the thermal imaging domain, as shown in Figure 1a, has been reported [11,12,13,14].

Figure 1. Thermal image super-resolution approaches. (a) SISR: single image super-resolution for an LR thermal image; (b) GTISR: guided thermal image super-resolution, where an aligned HR visible image is provided as guidance to reconstruct an HR thermal image from an LR thermal input.

Guided thermal image super-resolution (GTISR) has been introduced as a key research topic as deep learning techniques have been developed [15]. This method can enhance the image quality of thermal images by supplementing spatial details from HR visual images. As shown in Figure 1b, it utilizes HR visible images as guides in the multisensor environment that is increasingly being utilized. Consequently, various GTISR methods have been proposed to enhance thermal image resolution by learning spatial details from HR visible images [16,17,18,19,20,21,22]. Methods utilizing guided visible images are effective because they can provide spatial details while preserving the temperature information of the thermal image.

SR research can generally be discussed under two distinct objectives. One is minimizing distortion from the ground truth image, where transformer-based methods have demonstrated superior performance on distortion metrics [7,8]. The other seeks perceptually natural and realistic restoration, where diffusion-based methods have shown excellent performance on perceptual metrics [23,24,25,26]. Diffusion models can guide the generation process by utilizing visible images as conditions, making them highly applicable to GTISR settings. However, designing methods that reliably suppress distortion from the ground truth remains a significant challenge [27]. Additionally, the phenomenon of hallucination, where patterns unrelated to the actual signal appear in a result image, must also be considered [28].

In this paper, we propose Diff-GTISR, a novel GTISR method that consists of a diffusion model and refinement for improving perceptual quality and suppressing distortion. To achieve this, ResShift [26], which has demonstrated efficiency and excellent restoration performance in diffusion-based SISR, is adopted as the base model and redesigned for the proposed thermal–visible multimodal model. Here, the proposed model introduces a structure incorporating a modality-specific dual encoder and a cross-modal guidance attention (CMGA) module. The dual encoder extracts features from thermal and visible images through separate paths, providing multiscale representations. Subsequently, the CMGA module robustly injects structural cues from visible features into thermal features, enhancing detailed structure restoration while preserving the temperature-based information in thermal images. Also, a refinement network can suppress unrealistic patterns that can arise from diffusion-based restoration outputs. Magnification experiments with a factor of 8 and 16 indicate that the proposed Diff-GTISR method provides higher performance than existing methods in terms of subjective and objective quality, confirming its competitiveness and validity.

In summary, the main contributions of this paper are as follows:

(1)

Diff-GTISR, a diffusion model for guided thermal image super-resolution, which simultaneously improves the image quality subjectively and objectively.

(2)

Design of a modality-specific dual encoder and cross-modal guidance attention module, effectively injecting visible light structural details into thermal image restoration.

(3)

Utilization and analysis of an optional refiner to suppress unrealistic patterns that may appear in the resulting images.

2. Related Works

2.1. Single Image Super-Resolution

Single image super-resolution aims to transform an LR image acquired by an LR sensor into its HR image. Traditional signal processing-based interpolation methods focus on structural preservation by using local edges, directional information, and spatial correlations [29]. An input image can be decomposed into its signal components, which are interpolated by reflecting the characteristics of each component [30]. Recently, learning-based SISR methods have learned correspondences between LR and HR patches from training data. These methods then estimate high-frequency details via sparse coding or local regression to reconstruct an HR image [31,32].

The subsequent emergence of deep learning has brought significant advancements to SR methods. Deep learning-based methods jointly learn feature representations and restoration mapping functions from large-scale data. These methods have greatly improved restoration performance by compensating for the limitation of the previous SR methods. They are primarily developed using visible RGB images so that their collection is relatively easy. SRCNN was the first method to apply deep learning to SR, using a convolutional neural network (CNN) to surpass the performance of existing methods [5]. EDSR removed batch normalization in CNN caused by visual quality degradation and significantly improved performance by stacking deeper residual networks [6]. However, CNN-based methods require more complex and deeper networks to be constructed to capture long-range dependencies, leading to computational problems.

In contrast, transformer-based methods can efficiently model long-range dependencies by processing global information through their self-attention mechanism. SwinIR achieved state-of-the-art performance with fewer parameters by extracting low-level features using CNN and then learning the global context (long-range dependencies) through residual Swin transformer blocks [7]. HAT combines self-attention and channel attention to employ rich pixel information [8]. Consequently, transformer-based approaches have gained widespread adoption in SR to overcome the limitations of the CNN-based methods. However, most SR methods utilizing CNN or transformer backbones are trained to minimize distortion metrics. The restored HR images thus tend to be overly smoothed, limiting the fine details and textures.

Generative models introduced a new paradigm for SR. Considering that SR is inherently an ill-posed problem, generative models introduce the possibility of generating natural HR images by learning conditional distributions—that is, they shift the objective from conventional distortion minimization to perceptual quality enhancement. SRGAN applied a generative adversarial network (GAN) to SR and improved perceptual quality by proposing a perceptual loss [9]. ESRGAN refined the residual structure and loss function of SRGAN [10]. However, GAN-based SR methods are sensitive to the balance between the generator and the discriminator. Thus, these methods can be unstable and potentially suffer from artifacts such as mode collapse and hallucination textures, resulting in unrealistic HR images.

2.2. Guided Thermal Image Super-Resolution

Guided thermal image super-resolution aims to convert LR thermal images into HR ones by utilizing the corresponding HR visible images, acquired in a multisensor environment, as guides. This approach enhances the resolution of thermal images by injecting structural and boundary information learned from their visible images. CGTISR is a pioneering method that performs guided filtering based on a single RGB channel from the visible image showing the highest correlation with the thermal image [33]. TSRCNN converts the visible image into features of the same size as those of the thermal image using separate branches and then fuses them [16]. PAG-SR extracts a pyramid edge map from the visible image and injects edge information [17]. GSRIF-CS proposes a spectral mapping-based GTISR network for crack segmentation and fuses the restored HR thermal image with the visible image [18].

Recently, various methods have been proposed to utilize a visible image as a guide in the thermal image super-resolution challenge [15]. GuidedSR proposes a network that extracts shallow features from thermal and visible images, fuses their multimodal information, and restores the image via a reconstruction layer [15]. FW-SAT hierarchically learns global, local, and regional features using channel spatial attention, window-based self-attention, and flexible window-based self-attention [19]. SwinFuSR applies an image fusion model to SR, where it fuses the thermal–visible deep features extracted by two branches and then restores them into an HR thermal image [20]. Additionally, it enhances performance by stacking deeper transformer layers in the feature extraction and using a cross-domain fusion block in the feature fusion module.

2.3. Diffusion Model for Image Super-Resolution

Diffusion models are probabilistic generative models aimed at generating high-quality data similar to the distribution of training data. DDPM defines a Markov chain consisting of forward and reverse processes. The forward process adds noise to an image gradually, converting the image into a noisy image with a Gaussian distribution. The reverse process removes noise from the noisy image gradually, generating its image from the noisy image [34]. Diffusion models generate high-quality natural images compared to GANs because of stable learning. Thus, the models have been extended to super-resolution and are utilized as a powerful generative approach to improve the perceptual quality of restored HR images.

SR3 was an early study extending DDPM to conditional generation for image super-resolution methods, starting from the standard Gaussian noise with LR inputs as a conditional parameter. It gradually restores an HR image from the noise through the inverse process [23]. SRDiff restored the residual between an LR image and an HR image instead of directly restoring the HR image [24]. However, early diffusion-based SR methods suffered from high computational cost and time consumption during inference because of the numerous denoising steps in the inverse process. LDM achieved high-quality restoration while reducing computational cost by compressing images into a latent space using an autoencoder [35] in forward–inverse processes [25]. ResShift designed a novel Markov chain that directly shifts residuals between LR and HR images [26]. This residual shifting approach reduced the required denoising steps significantly, improving inference efficiency while achieving excellent restoration performance.

As the most successful of the diffusion models, diffusion-based SR methods have naturally been applied to thermal SISR. Studies have particularly focused on the characteristics of thermal images. DifIISR uses ResShift as a baseline and derives SR results by injecting gradient guidance into the reverse processing [36]. DroneSR proposes a novel diffusion-based Gaussian quantization learning method for thermal image datasets captured by drones. It can reduce overfitting and enhance robustness [37].

Furthermore, research has emerged extending the SISR diffusion model to GTISR, utilizing not only LR thermal images but also HR visible images as reference information. DuaDiff proposed a dual-conditioning diffusion model that extracts multiscale high-frequency components from visible images through a learnable Laplacian pyramid. It improved perceptual quality by promoting feature interaction between the two modalities in the latent space [21]. DiffGUA is a diffusion-based, generic, uncertainty-aware approach for thermal image super-resolution [22]. This method combines visible images as a channel dimension and injects them as conditions. It also reduces visual artifacts in the restored images by introducing a refiner network.

According to existing methods for GTISR, diffusion models provide a promising approach to thermal SR methods. Also, the image quality of restored HR images from a guided diffusion model depends on how thermal and visible images are conditionally injected into the network. Therefore, it is worth developing a diffusion model-based framework for GTISR by effectively integrating multimodal characteristics from thermal and visible images to reduce distortion while achieving natural results.

3. Proposed Method

As shown in Figure 2, we propose a diffusion-based guided thermal image super-resolution (Diff-GTISR) framework utilizing an HR visible image as a guide. Diff-GTISR employs ResShift [26], designed for SISR, as its baseline model. ResShift achieves stable restoration with fewer sampling steps by redefining the diffusion process of shifting the residual image between the HR and LR images. Specifically, the forward process starts from the original HR image at time step $t=0$ and generates a noisy LR image at time step $t$ through incremental residual shifts. Conversely, the reverse process iteratively executes the transition from $t$ to $t-1$ using a denoising U-Net, ultimately restoring the HR image at $t=0$ through incremental noise removal. Here, the denoising U-Net takes as input time step $t$, the noisy LR state at that step, and the original LR image, predicting the noisy LR image at $t-1$. For computational efficiency, ResShift first compresses the image into a latent space using a VQ-GAN-based autoencoder [35] and then executes the denoising U-Net in this reduced latent space.

Figure 2. Architecture of the proposed Diff-GTISR denoising network, consisting of a modality-specific dual encoder, a cross-modal guidance attention (CMGA) module, and a thermal decoder.

To extend ResShift to GTISR, we design a novel denoising network, $f_{\theta }$, to replace the existing denoising U-Net. The proposed $f_{\theta }$ architecture employs the HR visible image as auxiliary input to provide conditional information during restoration. It also remedies the issue of differences in modality between the guide and target images by use of a modality-specific dual encoder, cross-modal guidance attention module, and a thermal decoder. The proposed design effectively utilizes HR visible images as guidance to restore LR thermal images.

3.1. Modality-Specific Dual Encoder

The proposed $f_{\theta }$ denoising network performs reconstruction operations in the latent space. Autoencoders are a particularly good choice of dimensionality reduction method in the field of diffusion models, as they transform an image into its latent space. When an autoencoder pretrained on visible datasets is applied to thermal images, the specific distribution and signal characteristics of the thermal images are insufficiently reflected in the latent space. This can lead to a domain mismatch. To remedy this problem, we build a thermal image-specific autoencoder by further fine-tuning the autoencoder used in ResShift. This autoencoder is utilized for thermal images. In the proposed method, the encoder $E_{vis}^{AE}$ of the visible autoencoder and the encoder $E_{thr}^{AE}$ of the thermal image autoencoder are utilized to stably convert inputs $(x, y, v)$ into their latent representations $(z_x,z_y, z_v)$, which are represented by

$$z_x=E_{thr}^{AE}\left(x\right), z_y=E_{thr}^{AE}\left(y\right), z_v=E_{vis}^{AE}\left(v\right)$$

(1)

where $x$ and $v$ are the original HR thermal image and HR visible image aligned at the same spatial resolution, while $y$ denotes the input obtained by applying bicubic interpolation to the original LR thermal image and resizing it to the same resolution as $x$. Here, $z_x$ denotes the latent representation of the target HR thermal image, $z_y$ denotes the latent representation of the input LR thermal image, and $z_v$ denotes the latent representation of the guidance HR visible image. Furthermore, each latent representation has its spatial resolution reduced to 1/4 relative to the input by the autoencoder.

When combining different modalities in a single encoder path, the distribution of one modality may inhibit the learning of other modalities. To overcome this problem, we design the existing U-Net encoder as dual encoders with an identical structure but without parameter sharing. Thus, we introduce two separate paths—thermal features $F_{thr}^l$ and visible features $F_{vis}^l$—at all levels $l(l=\mathrm{1,2},\dots ,L)$, which is written as

$$F_{thr}^l=E_{thr}\left(z_t,z_y\right), F_{vis}^l=E_{vis}\left(z_v\right),$$

(2)

where $z_t$ represents the noise-containing latent representation at time step $t$ of the preprocessing process executed in the thermal image domain. The thermal encoder $E_{thr}$ receives $z_t$ and $z_y$ concatenated along the channel dimension as input, while the visible encoder $E_{vis}$ receives $z_v$ as input. As each encoder shares the same architecture, they compute corresponding thermal features $F_{thr}^l$ and visible features $F_{vis}^l$. In other words, the dual encoder learns domain-specific representations for thermal and visible features while ensuring dimensional alignment of multiscale features.

3.2. Cross-Modal Guidance Attention Module

In existing diffusion-based multimodal fusion methods, information from the reference modality is incorporated into the target modality representation either by directly concatenating the features of the two modalities along the channel dimension or by fusing them through cross-attention. A standard cross-attention module extracts the attention context from the target modality for the query and the reference modality for the key and the value. Then, it combines them with the input feature using a residual connection. This process transfers information from the reference modality to the target modality representation.

However, this standard cross-attention structure is not directly suitable for GTISR. In the GTISR task, the target is the thermal image and the reference is the guiding visible image. Here, the visible representation contains details from the HR input, whereas the thermal representation has limited structural information due to the low resolution of the LR input. This asymmetry between the two modalities makes it difficult to establish correspondences between thermal and visible features, leading to inaccurate feature fusion.

Reflecting these domain characteristics, we propose a cross-modal guidance attention (CMGA) module that derives guidance from HR visible structural cues and uses it to reconstruct thermal information. To do so, we split the attention operation across modalities by computing similarity in the visible branch and applying the resulting weights to thermal values. Specifically, we compute the affinity from the similarity between visible queries and keys. Also, we use the fused representation via a weighted sum of the thermal values. In this way, visible-based similarity works for spatial guidance and thermal-based value aggregation works for thermal information. In addition, instead of using a residual connection, we construct the fused feature by concatenating the attention context with the original thermal feature along the channel dimension. In this way, the fused information from the two modalities is reflected, while the thermal feature obtained from the forward process is explicitly preserved. Thus, the CMGA fusion feature map incorporates structural information from the visible modality, preserving the thermal characteristics.

The architecture of the proposed CMGA is illustrated in Figure 3. Here, the thermal and visible features at a level of $l$, $F_{thr}^l,F_{vis}^l\in {\mathbb{R}}^{C\times H\times W}$, which are generated by the dual encoders, are fused into the final feature $F_{fuse}^l\in {\mathbb{R}}^{2C\times H\times W}$, the output of the CMGA module, and the derivation of CMGA is based on Equation (3) through Equation (7). First, the two input features are converted into their token sequences by spatially flattening (flat) followed by layer normalization (LN), which is written as

$$S_{thr}=\mathrm{L}\mathrm{N}\left(\mathrm{f}\mathrm{l}\mathrm{a}\mathrm{t}(F_{thr}^l)\right), S_{vis}=\mathrm{L}\mathrm{N}\left(\mathrm{f}\mathrm{l}\mathrm{a}\mathrm{t}(F_{vis}^l)\right),$$

(3)

where $S_{thr}$ and $S_{vis}$ denote the normalized thermal and visible tokens, respectively. Also, the token length is $N=H\times W$.

Figure 3. Proposed cross-modal guidance attention module architecture.

In the multi-head attention, query $Q$ and key $K$ are constructed from the visible token sequence $S_{vis}$, whereas value $V$ is constructed from the thermal token sequence $S_{thr}$:

$$Q=S_{vis}{W}_Q, K=S_{vis}{W}_K, V=S_{thr}{W}_V,$$

(4)

where $W_Q,W_K,W_V\in {\mathbb{R}}^{C\times d}$ are learnable weights that linearly project each token into the attention embedding space and $Q,K,V\in {\mathbb{R}}^{N\times d}$. The attention embedding dimension $d$ is set equal to $C$. The attention weights $A$ and context $Y$ are computed as follows:

$$A=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(Q{K}^T/\sqrt{d}\right), Y=AV,$$

(5)

where $A\in {\mathbb{R}}^{N\times N}$ serves as a visible-guided spatial guidance and $Y\in {\mathbb{R}}^{N\times d}$ is the attention context obtained by the weighted aggregation of thermal values according to the guidance. For simplicity, the multi-head decomposition and concatenation processes are omitted in Equations (4) and (5). In practice, $Q,K$ and $V$ are split into $h$ heads, and for each head, the similarity between visible queries and keys is independently computed to obtain the attention weights, which are then applied to the corresponding thermal values. The outputs of all heads are concatenated to form the final attention context.

Normally, transformer blocks in a multi-head attention combine the attention context with the input using a residual connection. In the CMGA module, the thermal representation is explicitly constructed by a fused sequence, $S_{fuse}\in {\mathbb{R}}^{N\times 2C}$. This is obtained by channel-wise concatenation of the attention context and the original thermal token sequence $\mathrm{f}\mathrm{l}\mathrm{a}\mathrm{t}\left(F_{thr}^l\right)$, which is written as

$$S_{fuse}=\mathrm{L}\mathrm{N}\left(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(Y,\mathrm{f}\mathrm{l}\mathrm{a}\mathrm{t}(F_{thr}^l))\right).$$

(6)

Here, the Equation (6) process is based on consideration of the characteristics of the denoising U-Net in ResShift, where the reverse process takes the noisy LR and the original LR condition as inputs at each time step.

The final output of the CMGA module, the fused feature $F_{fuse}^l$, is obtained by applying a multi-layer perceptron (MLP) to $S_{fuse}$ and then updating and reshaping the processes using a residual connection, yielding the output form, which is written as

$$F_{fuse}^l=\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{p}\mathrm{e}\left(S_{fuse}+\mathrm{M}\mathrm{L}\mathrm{P}(S_{fuse})\right).$$

(7)

The output of our CMGA module is the fused feature with dual channel dimensionality, which is then fed into the pretrained thermal autoencoder.

3.3. Decoder and Loss Function

The decoder $D_{thr}$ in the proposed denoising network reconstructs the latent signal ${\tilde{z}}_{t-1}$ by progressively up-sampling from the fused feature $F_{fuse}^L$ at the deepest level $L$ of the CMGA module. In addition, at each level $l$, the decoder feature is combined with $F_{fuse}^l$ via a skip connection to reinforce multiscale information. Since CMGA produces spatially aligned fused features $F_{fuse}^l$ from the thermal and visible features at all levels, the decoder can directly use them as inputs without the additional spatial alignment. Consequently, the dual encoder and the CMGA module provide consistent representations across scales, enabling effective hierarchical fusion with the decoder.

Given the noisy latent $z_t$, the LR thermal latent $z_y$, the HR visible latent $z_v$, and the time step $t$, we define a loss function to enable the denoising network $f_{\theta }$ to predict the target high-resolution thermal latent $z_x$, which is written as

$${\mathcal{L}}_f\left(\theta \right)={\mathbb{E}}_{z_t,z_y,z_v,t}\left[{\Vert f_{\theta }\left(z_t,z_y,z_v,t\right)-z_x\Vert }_2^2\right].$$

(8)

During inference, the predictor ${\tilde{z}}_x$ for HR thermal latent $z_x$ is decoded into the image space, where the decoder $D_{thr}^{AE}$ of the thermal autoencoder generates the final HR thermal image $\tilde{x}$. This is written as

$$\tilde{x}=D_{thr}^{AE}\left({\tilde{z}}_x\right).$$

(9)

3.4. Refiner Network

It is evident that generative networks for diffusion-based image restoration and super-resolution are constrained by a trade-off between perception and distortion [27,38]. As images restored by generative models become closer to the natural image distribution, they tend to produce hallucinated details that are not present in the ground truth [28]. Although such outputs appear realistic perceptually, they can potentially degrade performance in terms of the numerical accuracy required in some applications. On the other hand, generative networks can produce blurry or unnatural images minimizing distortion alone.

To alleviate this issue, we introduce a refiner network, $r_{\varphi }$, which reduces distortion in the diffusion output as post-processing. Network $r_{\varphi }$ is a U-Net-based model proposed in DiffGUA [22]. It takes as input the up-scaled LR thermal image $y$, the guidance HR visible image $v$, and the super-resolved output $\tilde{x}$ from the denoising network $f_{\theta }$. Using these inputs, $r_{\varphi }$ suppresses fine-grained hallucinations in the restored results and encourages structures consistent with the real scene.

The refiner $r_{\varphi }$ updates the final HR thermal estimate $\widehat{x}$ as follows:

$$\widehat{x}=r_{\varphi }\left(y,v,\tilde{x}\right).$$

(10)

To train the refiner network, we use the MSE loss to minimize distortion with respect to the ground truth HR thermal image $x$. This loss function is defined by

$${\mathcal{L}}_r\left(\varphi \right)=MSE\left(x,\widehat{x}\right).$$

(11)

As shown in Figure 4, the proposed Diff-GTISR method is a two-stage framework consisting of the denoising network $f_{\theta }$ and the refiner network $r_{\varphi }$. The overall generated HR thermal image $\widehat{x}$ can be optionally updated by $r_{\varphi }$. This dual-stage configuration offers the advantage of selective utilization, allowing for customized deployment to meet subjective or objective requirements, depending on the application areas.

Figure 4. Overall architecture of the proposed Diff-GTISR framework, consisting of the diffusion-based GTISR network and the optional refiner network.

4. Experiment Results

4.1. Dataset

The dataset for the experiment is selected from the CIDIS dataset [39] from the PBVS 2024 TISR Challenge (track 2), an international contest for GTISR [15]. Related GTISR methods have also reported their performance on the CIDIS dataset, making it an appropriate benchmark dataset for comparative evaluation. The CIDIS dataset consists of HR visible, HR thermal, and LR thermal images. The HR images were acquired using visible and thermal cameras during daytime and were subsequently aligned to a 640 × 448 resolution. The visible camera was a Basler camera (Basler AG, Ahrensburg, Germany), and the thermal camera was a FLIR Tau 2 camera (FLIR Systems, Inc., Wilsonville, OR, USA). The LR thermal images are resized from their HR thermal images by a bicubic interpolation method with factors of 8 and 16, resulting in resolutions of 80 × 56 and 40 × 28, respectively. The dataset images consist of three parts: 700 images are used for training, 200 images for validation, and 40 images for testing. As contest data, the test images were unavailable. For a fair comparative evaluation, we split the 700 training images at a 9:1 ratio, resulting in 630 images for training and 70 images for validation, employing 200 images for official validation as the test images. This split configuration also enables a more stable comparative evaluation within the range of publicly available data. To increase the diversity of the training data, data augmentation techniques such as patch-based cropping and horizontal flipping were applied. Specifically, 12 HR patch images of size 256 × 256 were extracted from each HR image through overlapping cropping, and the corresponding 12 LR patches of size 32 × 32 or 16 × 16 were prepared accordingly. Consequently, a total of 7560 patch tuples were used to train the proposed model, and random horizontal flipping was additionally applied during training.

4.2. Implementation Details

The hyperparameters used to learn the proposed Diff-GTISR network, such as a diffusion process of 15 steps and a noise scale increasing from 0.04 to 0.99, are equal to those of the baseline ResShift [26]. The proposed denoising network, including the dual encoder, CMGA module, and single decoder, has 226 M trainable parameters. The denoising network was implemented using Python 3.10.16 and PyTorch 2.1.1 with CUDA 12.1. Training was conducted on an NVIDIA GeForce RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA) with a batch size of 16 for 38,000 iterations, taking approximately 23 h. Additionally, the inference time and GPU memory usage were measured on an RTX 4090 with an input size of 256 × 256 and a batch size of 1. The inference time was measured based on the model inference time, excluding data loading, file I/O, and post-processing, and the average value over 50 repeated runs after 10 warm-up runs. As a result, the proposed denoising network required an average inference time of 1.50 s per image, and the peak GPU memory usage during inference was 1.69 GB in allocated memory. Under the same conditions, the computational requirements of the baseline ResShift model were also measured. As the baseline model of the proposed method, ResShift has 118 M parameters, and it showed an average inference time of 0.86 s per image and a peak allocated memory of 1.02 GB. From this observation, we infer that the proposed technique includes a model capable of a thermal imaging modality while requiring an inference time below one second compared with the existing model. This finding indicates that the model size of the proposed technique poses no practical burden for real-world deployment. The refiner network was trained after the denoising network was trained. The refiner consists of a single encoder–decoder structure with two residual layers and skip connections, with 1.7 M trainable parameters. The refiner network was implemented using Python 3.10.18 and PyTorch 2.6.0 with CUDA 12.4. Training was also performed on an RTX 4090 with a batch size of 8 for 116,235 iterations, taking approximately 18 h. The refiner network used for the final restoration result showed an average inference time of 3 ms per image and a peak allocated memory of 0.15 GB under the same conditions. Therefore, most of the computational cost of the proposed full pipeline arises from the denoising stage, while the additional deployment burden caused by the refiner network is limited.

4.3. Evaluation Metrics

To quantitatively evaluate the proposed methods, three metrics are employed: the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM) [40], and the learned perceptual image patch similarity (LPIPS) [41]. PSNR is the most popular metric for measuring image interpolation quality. This metric is computed from the mean squared error between the reconstructed image and the ground truth, where higher values indicate lower distortion. SSIM measures structural similarity based on luminance, contrast, and structural components, where the values closer to 1 indicate higher agreement with the ground truth. LPIPS measures perceptual distance in the feature space of a pretrained deep network, where lower values indicate perceptually more similar reconstructions. LPIPS is computed with the AlexNet backbone.

4.4. Compared Methods

Table 1 provides a concise overview of the various benchmarking methods employed for comparative evaluation. The categorization of these models is based on two key factors: the specific task they are designed to perform and the baseline model from which they are derived. In terms of the task, single image super-resolution (SISR) methods were tested using only LR thermal images without HR visible images, while guided thermal image super-resolution (GTISR) methods were tested using HR visible images. With respect to the baseline model, the methods of benchmarking can be categorized as diffusion- or transformer-based methods.

Table 1. Summary of compared methods.

Method	Task	Model	Parameters
ResShift [26]	SISR	Diffusion	118.59 M
SwinIR [7]	SISR	Transformer	12.04 M
DiffGUA [22]	GTISR	Diffusion	123.7 M
SwinFuSR [20]	GTISR	Transformer	0.94 M

All comparison methods were retrained based on their official implementations with the CIDIS dataset of this experiment. Therefore, the comparative results in this study provide relative performance comparisons under the same protocol rather than representing final benchmark results on the undisclosed official test set. In particular, in the retraining of the ResShift model, our fine-tuned thermal autoencoder is employed to stably extract latent vectors from the thermal images since the model is insufficient for thermal images.

4.5. Objective Comparison

Table 2 and Table 3 summarize the quantitative evaluation results for the existing methods and the proposed method. Here, the best and second-best results are highlighted in bold and underlining, respectively. These results indicate that the proposed Diff-GTISR method overall is superior to the existing methods in terms of both distortion and perceptual metrics.

Table 2. Quantitative comparison results for ×8 SR on the CIDIS set.

Method	PSNR [dB]↑	SSIM↑	LPIPS↓
Bicubic	24.99	0.7650	0.4876
ResShift [26]	25.83	0.7999	0.2113
SwinIR [7]	26.79	0.8297	0.2648
DiffGUA [22]	26.97	0.8189	0.2405
SwinFuSR [20]	28.18	0.8633	0.2010
Diff-GTISR (Ours)	28.23	0.8613	0.1753

Table 3. Quantitative comparison results for ×16 SR on the CIDIS set.

Method	PSNR [dB]↑	SSIM↑	LPIPS↓
Bicubic	21.86	0.6894	0.6569
ResShift [26]	22.29	0.6962	0.3569
SwinIR [7]	22.92	0.7279	0.4953
DiffGUA [22]	24.16	0.7515	0.3273
SwinFuSR [20]	24.73	0.7855	0.3129
Diff-GTISR (Ours)	25.34	0.7992	0.2491

Table 2 shows the results of super-resolution with a factor of 8 (×8 SR). Here, the proposed method demonstrated the highest PSNR of 28.23 dB among all the comparison methods while also maintaining the second-best SSIM of 0.8613. Furthermore, it recorded the lowest LPIPS of 0.1753, indicating balanced results when considering both distortion-based performance and perceptual quality.

Table 3 presents the results of super-resolution with a factor of 16 (×16 SR). Here, our method achieved PSNR of 25.34 dB, SSIM of 0.7992, and LPIPS of 0.2491, providing the best performance for all three metrics in comparison with the state-of-the-art methods.

A comparison with diffusion-based baseline models is particularly significant, as the proposed method is based on a diffusion model. Residual shifting such as ResShift has proven highly effective for single image super-resolution [26]. Building upon this foundation, the proposed method, as a guided version of residual shifting, achieves substantial improvements: in the ×8 SR experiment, the PSNR increases from 25.83 dB to 28.23 dB, yielding a gain of 2.40 dB; in the ×16 SR experiment, the PSNR rises from 22.29 dB to 25.34 dB, corresponding to an improvement of 3.05 dB. These results confirm that the proposed method successfully extends the structural property of the baseline into a guided diffusion model via the proposed denoising network. Moreover, when compared with a state-of-the-art guided diffusion model such as DiffGUA [22], our method achieves additional gains of 1.26 dB at ×8 SR and 1.18 dB at ×16 SR. Consistent improvements are also observed in SSIM and LPIPS metrics, further substantiating the effectiveness of the proposed approach.

4.6. Subjective Comparison

Figure 5 and Figure 6 present visual comparisons of the enlarged images resulting from the ×8 and ×16 super-resolution (SR) experiments, respectively. In general, single-input SISR methods exhibit limited capability in recovering boundary and structural information compared with GTISR approaches that incorporate visible guidance, and they show deficiencies in reconstructing missing high-frequency components. By contrast, GTISR methods more effectively recover fine details and boundary structures by exploiting the structural cues available from the visible modality.

Figure 5. Qualitative comparison of prior methods for ×8 SR on the re-split CIDIS test set.

Figure 6. Qualitative comparison of prior methods for ×16 SR on the CIDIS test set.

An examination of the magnified images from existing methods highlights clear differences in their reconstruction behavior. Transformer-based methods generally deliver high PSNR values and maintain object contours with notable stability. However, from a visual standpoint, these methods tend to oversmooth regions containing fine textures and high-frequency patterns, which diminishes perceptual sharpness despite favorable distortion metrics. In contrast, diffusion-based methods subjectively appear more capable of recovering intricate textures and high-frequency structures, producing visually richer restorations. However, these gains come at the cost of hallucination artifacts in certain scenes, where spurious patterns unrelated to the true thermal signal are amplified. Such artifacts not only reduce the perceived fidelity of the restoration but also negatively impact objective measures such as PSNR. Thus, these observations suggest that while the transformer-based approaches emphasize stability and distortion-oriented performance, diffusion-based approaches prioritize perceptual detail but risk introducing misleading structures.

In comparison with the exiting methods, the proposed Diff-GTISR method demonstrates superior performance in thermal image super-resolution. By retaining the strengths of diffusion-based models while improving distortion-oriented metrics, the method achieves restorations that are particularly well suited to thermal imagery. Specifically, Diff-GTISR preserves boundary and structural information more consistently and reconstructs high-frequency components with greater accuracy. These improvements are evident in Figure 5 and Figure 6, where subjective comparisons across diverse regions of resulting images show closer agreement with those of the ground truth thermal images.

Furthermore, the proposed design more accurately preserves the thermal distribution characteristics than the existing diffusion-based methods, substantially reducing hallucination artifacts. This reduction not only enhances perceptual quality but also contributes to improved objective performance. As summarized in Table 2 and Table 3, Diff-GTISR surpasses state-of-the-art transformer-based approaches in PSNR, SSIM, and LPIPS metrics, confirming its ability to balance perceptual detail with distortion-oriented accuracy. Therefore, these findings highlight the effectiveness of the proposed Diff-GTISR method in achieving natural high-resolution thermal super-resolution while maintaining accurate thermal signals.

4.7. Ablation Study

To analyze the effect of the proposed CMGA module, an ablation study was conducted using several variants with different attention mechanisms and feature fusion structures, as summarized in Table 4. Here, the best and second-best results are highlighted in bold and underlining, respectively. Structure A denotes the baseline without attention, corresponding to the SISR-based ResShift architecture using only a single thermal image as input. Structure B introduces standard cross-attention (CA) into this baseline, and all three metrics are improved compared with the baseline, confirming the effectiveness of attention-based feature fusion. Structure C replaces the residual connection in B with channel-wise concatenation and yields further performance improvement. This suggests that explicitly preserving and utilizing the thermal features obtained from the forward process is more effective than simple residual addition. Structure D reverses the modality assignment of Q, K, and V in C, leading to additional improvement in performance. This implies that deriving queries from the high-resolution visible image, rather than from the low-resolution thermal image, may be more advantageous for reflecting structural information. Finally, structure E applies the proposed CMGA module, which computes similarity based on visible features while using thermal features as values, and achieves the best performance across all evaluation metrics. This indicates that the proposed design, which guides structural information through visible features while performing value aggregation on thermal features, is effective in terms of both structural guidance and preservation of thermal characteristics. In addition, although small alignment errors still exist in the CIDIS dataset, the proposed CMGA consistently achieves the best performance. This observation suggests that the proposed design can remain relatively stable even under minor visible–thermal misalignment. The visual comparison in Figure 7 shows that the proposed CMGA module reconstructs boundaries and structural details more clearly than the baseline and various standard cross-attention variants while also producing the most visually natural thermal image restoration results.

Table 4. Quantitative ablation study comparison of the CMGA module for ×8 SR on the CIDIS test set.

Structure	Attention	Q/K/V	Connection	PSNR [dB]↑	SSIM↑	LPIPS↓
A (baseline)	-	-	-	25.83	0.7999	0.2113
B	CA	thr/vis/vis	residual	26.20	0.8073	0.1950
C	CA	thr/vis/vis	concat	26.53	0.8158	0.1956
D	CA	vis/thr/thr	concat	27.64	0.8448	0.1565
E (Ours)	CMGA	vis/vis/thr	concat	27.64	0.8484	0.1520

Figure 7. Qualitative ablation study comparison of the CMGA module for ×8 SR on the CIDIS test set.

The proposed framework integrates a refiner network as a post-processing module, which can be optionally applied depending on application requirements. To assess its impact, an ablation study was conducted to evaluate the role of the refiner within Diff-GTISR. Quantitative comparisons in Table 5 and Table 6 show that removing the refiner (w/o Refiner) leads to decreases in PSNR and SSIM, confirming its effectiveness in improving distortion-oriented metrics. Here, the best result is highlighted in bold. Conversely, LPIPS increases without the refiner, reflecting its perceptual nature. This observation aligns with prior findings that distortion and perceptual metrics do not necessarily vary in the same direction [38]. Visual comparisons in Figure 8 further illustrate that applying the refiner reduces excessive contrast and mitigates hallucination artifacts in certain high-frequency regions, although some fine patterns may appear smoother as a result. Overall, the refiner network provides an additional mechanism for suppressing distortion in the diffusion output. Importantly, Diff-GTISR allows the refiner to be applied selectively, offering flexibility to balance numerical accuracy and perceptual quality according to the specific demands of different applications.

Table 5. Quantitative ablation study comparison of the refiner network for ×8 SR on the CIDIS test set.

Structure	PSNR [dB]↑	SSIM↑	LPIPS↓
w/o Refiner	27.64	0.8484	0.1520
with Refiner	28.23	0.8613	0.1753

Table 6. Quantitative ablation study comparison of the refiner network for ×16 SR on the CIDIS test set.

Structure	PSNR [dB]↑	SSIM↑	LPIPS↓
w/o Refiner	24.95	0.7869	0.2116
with Refiner	25.34	0.7992	0.2491

Figure 8. Qualitative ablation study comparison of the refiner network for ×8 SR on the CIDIS test set.

5. Conclusions

In this study, we have introduced the Diff-GTISR method, a diffusion-based framework for guided thermal image super-resolution that leverages a high-resolution visible image. While prior diffusion-based SR models have demonstrated strong perceptual quality, they often lag behind transformer-based methods in distortion-oriented performance. Diff-GTISR is designed to address this limitation by improving distortion-based quality while maintaining perceptual advantages. The core contribution lies in conditioning the diffusion denoising process with visible guidance, where multiscale features are extracted through the proposed modality-specific dual encoder, and the proposed cross-modal guidance attention module transfers structural cues from visible features to thermal features in a stable manner. This design enables effective use of visible guidance while preserving thermal domain characteristics, achieving balanced restoration quality. Experimental evaluations on the CIDIS dataset confirmed that Diff-GTISR surpasses prior diffusion-based methods and attains distortion-based performance nearly comparable to that of transformer-based approaches. These findings suggest a promising design direction for enhancing both perceptual and distortion-based quality within diffusion frameworks. For future work, further refinement of the lightweight refiner network, which can be optionally applied, will be pursued to enhance restoration quality. Improving the practical deployment feasibility of Diff-GTISR by reducing the computational cost and memory usage of the denoising network while preserving restoration quality will also be considered. In addition, extending Diff-GTISR to other modalities across diverse sensors and environments will be an important direction. With these advancements, Diff-GTISR will have the potential to evolve into a broadly applicable solution for multimodal guided super-resolution tasks.