Remote Sensing Image Super-Resolution via Multi-Scale Texture Transfer Network

Abstract

As the degradation factors of remote sensing images become increasingly complex, it becomes challenging to infer the high-frequency details of remote sensing images compared to ordinary digital photographs. For super-resolution (SR) tasks, existing deep learning-based single remote sensing image SR methods tend to rely on texture information, leading to various limitations. To fill this gap, we propose a remote sensing image SR algorithm based on a multi-scale texture transfer network (MTTN). The proposed MTTN enhances the texture feature information of reconstructed images by adaptively transferring texture information according to the texture similarity of the reference image. The proposed method adopts a multi-scale texture-matching strategy, which promotes the transmission of multi-scale texture information of remote sensing images and obtains finer-texture information from more relevant semantic modules. Experimental results show that the proposed method outperforms state-of-the-art SR techniques on the Kaggle open-source remote sensing dataset from both quantitative and qualitative perspectives.

1. Introduction

Remote sensing is an important ground observation technology that has developed rapidly in recent years. Given its unique advantages of extensive coverage, regular temporal regularity, and high accessibility, it has been widely used in disaster detection and early alerting [1], resource exploration [2], and land cover classification [3,4,5,6]. There are broad application prospects in environmental testing and other fields [7,8,9,10,11,12]. Due to space imaging equipment and communication bandwidth limitations, the resolution of images captured from satellite sensors is limited. In addition, the image quality is subject to various factors, including atmospheric turbulence, imaging process noise, and motion blur, leading to further reduced resolution and quality. Super-resolution (SR) algorithms can enhance acquired remote sensing images, thus, bypassing hardware limitations.

In the past few decades, SR technology has been widely applied to enhance the quality and resolution of remote sensing images. Existing remote sensing SR methods can be categorized into two main types: methods based on traditional algorithms [13] and methods based on deep learning [14,15]. The traditional learning-based super-resolution methods can learn prior knowledge from a data sample library by constructing learning models, thereby establishing a mapping relationship between low-resolution (LR) and high-resolution (HR) images. This process generates high-frequency details and texture information lost during the downscaling. Zheng et al. [16] first applied sparse representation theory to the field of remote sensing image SR, establishing overcomplete coupled high–low resolution dictionaries and solving sparse coefficients using an orthogonal matching pursuit strategy. Hou et al. [17] introduced a global joint dictionary model and non-local self-similarity for global and local constraints, establishing the intrinsic connections between image blocks and global information. Zhang et al. [18] proposed a sparse residual dictionary learning method, directly reconstructing residual images with a residual dictionary to alleviate the ringing effects of unstructured pixel-domain high-low resolution dictionaries. Wu et al. [19] proposed a SR framework based on a multi-feature dictionary, constructing multiple LR dictionaries for representation and overlay in the HR space.

With the widespread use of convolutional neural networks (CNN) in SR tasks, the deep learning-based remote sensing SR method has become the mainstream research direction of image SR methods. Deep learning-based methods usually learn complex mapping functions between LR and HR patches through CNN. Considering its outstanding learning ability, Shi et al. [20] built a sub-pixel CNN that builds a mapping mechanism from LR to HR. Kim et al. [21] proposed a novel network method with deep residual learning to ease the training process. Lai et al. [22] designed a pyramid network that fuses multi-scale residuals in the feature domain. Luo et al. [23] achieved excellent results by replacing zero paddings with self-similarity to prevent useless information. Wang et al. [24] proposed an SR method that preserves the temporal correlation between frames. Ledig et al. [25] introduced perceptual loss and proposed an SR method for generative adversarial networks (GAN) to obtain images with more realistic textures. The generation network guides image reconstruction with a perceptual loss so that the reconstruction result is close to the original HR image. The discriminant network compares the reconstructed image to the original HR image, and the reconstructed image is output if it can pass the discriminant network.

Current SR methods primarily target natural images, but they fall short when applied to the diverse shooting angles and multi-scale characteristics present in large-scale remote sensing data. Existing methods lack consideration for these factors, making it crucial to develop an SR method specifically tailored for accurately reconstructing remote sensing images with intricate textures. Drawing inspiration from the effectiveness of multi-angle priors in enriching prior information [26], this paper introduces reference images (Ref) to enhance the prior information of training samples. The algorithm is adapted for the reconstruction of remote sensing images by aligning feature texture information. The proposed network initially matches local texture information between the Ref image and the LR image, then conveys the identified texture features to the reconstruction network via the depth residual module. The final output is high-quality remote sensing images with detailed texture information. Extensive experimental results demonstrate the algorithm’s superiority over other state-of-the-art methods, both quantitatively and qualitatively.

We summarize the major contributions of this work as follows:

(1) We propose a novel remote sensing image RefSR method based on a multi-scale texture transfer network (MTTN), which can effectively address the issue of insufficient texture detail information in the reconstruction process of the single remote sensing image SR method.

(2) We propose a novel multi-scale texture transfer method. The proposed method uses residual blocks with different convolution kernel sizes to extract texture information of different scales from LR and Ref remote sensing images and take advantage of the texture details provided by Ref images to facilitate the reconstruction of LR images.

(3) We conduct extensive experiments on the challenging Kaggle open dataset, and the experimental results demonstrate the effectiveness and adaptability of our suggested MTTN method. In addition, the results also point to its great potential in subsequent remote sensing image classification tasks.

The remainder of this article is structured as follows. Section 2 presents related work, including existing SISR of remote sensing images and reference-based SR methods. Section 3 describes the principal method of this model in detail. Section 4 introduces the experimental results. Section 5 concludes this study and Section 6 discusses future work.

2. Related Work

Over the past decades, deep learning-based SR methods have demonstrated excellent reconstruction performance in evaluation indicators or visual quality compared to traditional methods. This section introduces the single image super-resolution (SISR) and reference-based super-resolution (RefSR) of remote sensing images.

2.1. SISR of Remote Sensing Images

The SISR method for remote sensing images has recently drawn increasing attention due to the unique value of high-quality remote sensing images in application scenarios. In addition to relying on advanced hardware, the SR method offers an efficient way to enhance the quality of remote sensing images. Recently, with the rapid popularity of convolutional neural networks in acquiring high-quality images, many researchers have proposed a large number of deep learning-based SISR methods for remote sensing images. Lei et al. [27] designed a CNN-based network framework to learn the local details and global prior information of remote sensing images. Xu et al. [28] proposed a deep memory connection network that reconstructs high-quality remote sensing images through local and global memory connections, so the reconstruction performance of remote sensing images is effectively improved. Lu et al. [29] employed a novel multi-scale residual neural network method. This method extracts image blocks of different sizes as multi-scale information and then fuses multi-scale high-frequency information to reconstruct images. Jiang et al. [30] designed an edge enhancement-based SR network to enhance the features of remote sensing images using edge detail information extracted by the network. Dong et al. [31] designed a multi-sensory attention method to obtain multi-level information of remote sensing features through the proposed feature extraction module. Haut et al. [32] designed a network to extract more related features through combining residual units and skipping connections. Qin et al. [33] designed a deep gradient-aware network with image-specific enhancement to preserve more gradient details in reconstructed images. Zhang et al. [34] proposed a novel attention module to capture finer remote sensing image texture information through this module. Lei et al. [35] created a new remote sensing image SR framework by introducing a transformer to extract multi-level features and better fuse the features of different dimensions. Liu et al. [36] designed a graph neural network (GNN) based SISR method, which can better consider the ill-conditioned phenomena in remote sensing images.

Despite the great progress in aforementioned SISR methods, most of these methods rely on paired images and local information. However, when confronted with certain specific and complex textures, they fail to capture sufficient prior information. Therefore, the SISR methods that handle remote sensing imagery have been criticized for their limitations in handling multi-source data.

2.2. Reference-Based Super-Resolution

In contrast to the SISR method, the RefSR method achieves the SR reconstruction process by introducing other related images. For the Ref image to effectively aid the SR reconstruction process, texture information similar to the LR image is typically required. Consequently, the Ref image is usually selected from related video frames [37,38] or photos taken from various angles [39]. Recently, SR methods based on Ref images have made great progress in general image methods. Zheng et al. [40] used cross-scale warping to create an end-to-end fully convolutional deep neural network. This network improves the accuracy of reconstructed images by performing end-to-end spatial alignment at the pixel level. Zhang et al. [41] employed a novel RefSR model that restores the target image through texture migration of the Ref image. Xie et al. [42] designed a matching and exchange module for the RefSR task to extract similar texture and high-frequency features from the Ref image by appropriately assigning the gradient to the previous feature encoding module. Yang et al. [43] designed a novel texture transformer model that transmits HR texture from the Ref image through a texture converter and attention module. Huang et al. [44] proposed a novel RefSR framework, which includes a texture extraction module and a texture transfer module to more fully transfer textures in an adaptive manner. Dong et al. [45] employed a GAN-based RefSR network, mainly using the relevant attention module that improves the network’s reconstruction ability through gradient-assisted alignment. Cai et al. [46] designed a texture transformer for remote sensing image SR algorithm, which reduces the SR model’s dependence on Ref images through U-Transformer.

The above-mentioned RefSR method introduces a new perspective for addressing remote sensing SR tasks. While the RefSR method offers more possibilities for remote sensing image super-resolution compared to SISR methods, existing RefSR methods often overlook the spatial diversity features of remote sensing images. These features contain valuable information that can significantly benefit remote sensing SR tasks. To address this gap, we have developed an innovative multi-scale texture transfer model to enhance the performance and robustness of RefSR. By extracting texture information from reference remote sensing images, this model further aids in extracting multi-scale information from the original low-resolution images, thereby reconstructing more refined remote sensing images.

3. Method

The main idea of the proposed method is to reconstruct SR images $I^{SR}$ from LR remote sensing images $I^{LR}$ and similar Ref images $I^{\mathrm{Re}f}$, and then synthesize a reasonable texture subject to $I^{\mathrm{Re}f}$, while maintaining consistency with the $I^{LR}$ content. The proposed model’s overall network structure is shown in Figure 1. The multi-scale texture transfer method takes into account the semantic and textural similarity of $I^{LR}$ and $I^{\mathrm{Re}f}$ images and can better suppress irrelevant textures while transferring related remote sensing image texture features.

Figure 1. Framework of the proposed multi-scale texture transfer network (MTTN). The specific structure of $RB{s}_A$, $RB{s}_B$, and $RB{s}_C$ are shown in Figure 2.

Figure 1. Framework of the proposed multi-scale texture transfer network (MTTN). The specific structure of $RB{s}_A$, $RB{s}_B$, and $RB{s}_C$ are shown in Figure 2.

Figure 2. Illustration of the residual blocks (RBs) in the proposed MTTN model. The $RB{s}_A$, $RB{s}_B$, and $RB{s}_C$ represent RBs with different convolution kernel sizes.

Figure 2. Illustration of the residual blocks (RBs) in the proposed MTTN model. The $RB{s}_A$, $RB{s}_B$, and $RB{s}_C$ represent RBs with different convolution kernel sizes.

3.1. Feature Swapping

The first step in our MTTN is to swap remote sensing image features. The design algorithm searches the entire remote sensing reference image for locally similar multi-scale textures, and these extracted multi-scale textures can be used to exchange the texture features of $I^{LR}$ with enhanced remote sensing SR image reconstruction. The multi-scale texture is then transferred directly to the final output $I^{SR}$ via feature search in HR space coordinates. Specifically, we first extract the corresponding feature map on the $I^{LR}$, and then sample the extracted feature map twice to obtain an enlarged LR image $I^{LR\uparrow }$ with the same resolution as the $I^{HR}$. We also apply the same upscaling factor for $I^{\mathrm{Re}f}$ before performing bicubic interpolation sampling to obtain a Ref image block $I^{\mathrm{Re}f\downarrow \uparrow }$ matching the up-sampled $I^{LR\uparrow }$. As the LR and Ref images’ data sources may differ, we compute texture similarity in the high-resolution feature space $\varphi \left(I\right)$ to emphasize structure and texture information, and we apply the inner product method to compute feature similarity between image pairs:

$$\begin{array}{c}\hfill h_{i,j}=〈K_i\left(\varphi \left(I^{LR\uparrow }\right)\right),\frac{K_i\left(\varphi \left(I^{\mathrm{Re}f\downarrow \uparrow }\right)\right)}{∥K_i\left(\varphi \left(I^{\mathrm{Re}f\downarrow \uparrow }\right)\right)∥}〉,\end{array}$$

(1)

where $K_i$ denotes the i-th patch obtained from the high-resolution space, $∥ ∥$ denotes feature normalization. The Ref feature block performs the best matching for the reference image features through a normalization function. Then we calculate the similarity between image pairs via convolution operations on the LR patch. The kernel operation expression of the Ref patch follows:

$$\begin{array}{c}\hfill N_j=\varphi \left(I^{LR\uparrow }\right)\ast \frac{K_j\left(\varphi \left(I^{\mathrm{Re}f\downarrow \uparrow }\right)\right)}{∥K_j\left(\varphi \left(I^{\mathrm{Re}f\downarrow \uparrow }\right)\right)∥},\end{array}$$

(2)

where $N_j$ represents the Ref patch’s similarity graph. $N_j\left(x,y\right)$ represents the texture similarity between LR and Ref. We represent LR texture patches based on similarity scores by creating swapped feature maps. The image patches in the feature map are represented as:

$$\begin{array}{c}\hfill K_{\omega \left(x,y\right)}=K_{j*}\left(\varphi \left(I^{\mathrm{Re}f}\right)\right),j^{*}=arg\underset{j}{max}{N}_j\left(x,y\right),\end{array}$$

(3)

where $\omega \left(\right)$ corresponds to the patch center and the patch index. Note that while $I^{\mathrm{Re}f\downarrow \uparrow }$ is used for matching (Equation (2)), the original Ref $I^{\mathrm{Re}f}$ is used for exchange (Equation (3)) to retain the high-frequency feature information. The final generated swap feature map is then fed to the subsequent multi-scale texture transfer by calculating the average between the swap feature maps.

3.2. Multi-Scale Texture Transfer

Our multi-scale texture transfer method merges the transferred texture data into the generation network of various feature layers corresponding to various scales after the layer-by-layer transfer of the feature maps extracted from the feature extraction network and high-resolution reference images. The above-described procedure builds the exchanging feature map $M_l$ for multi-scale layer l, where the texture encoder ${\psi }_l$ corresponds to the scale being used.

To build the fundamental generative network, we use residual blocks (RBs) with various convolution kernel sizes. The output ${\varphi }_l$ of layer l can be recursively expressed as:

$$\begin{array}{c}\hfill {\psi }_l=\left[{\mathrm{R}}_{es}\left({\psi }_{l-1}\nabla M_{l-1}\right)+{\psi }_{l-1}\right]{\uparrow }_{2\times },\end{array}$$

(4)

where ${\mathrm{R}}_{es}\left(\right)$ represents the RBs, ∇ represents the channel cascade, and ${\uparrow }_{2\times }$ represents the use of sub-pixel convolution layer for $2\times$ magnification. The remote sensing reconstruction image is generated once the L layer has reached the desired HR resolution:

$$\begin{array}{c}\hfill I^{SR}={\mathrm{R}}_{es}\left({\psi }_{l-1}\nabla M_{l-1}\right)+{\psi }_{l-1}.\end{array}$$

(5)

Unlike the traditional SISR method that only considers the texture difference between $I^{SR}$ and $I^{HR}$, the proposed MTTN network considers the texture variation between $I^{SR}$ and $I^{\mathrm{Re}f}$. In other words, the texture of $I^{SR}$ must be similar to the swap feature map $M_l$. Therefore, we define the texture loss $L_{tex}$ as:

$$\begin{array}{c}\hfill L_{tex}={\displaystyle \sum _l}{\lambda }_l{∥G_r\left({\varphi }_l\left(I^{SR}\right)\times N_l^{*}\right)-G_r\left(M_l\times N_l^{*}\right)∥}_F,\end{array}$$

(6)

where $G_r\left(\right)$ calculates the gamma matrices, and ${\lambda }_l$ is the normalized exponent of the l layer. $N_l^{*}$ represents the weight map of the input LR remote sensing image patch. The difference between this weight map and $I^{LR}$ is the fact that texture owns a lower weight, leading to lower texture loss. Thus, the texture transferred from $I^{\mathrm{Re}f}$ to $I^{SR}$ can be enforced adaptively based on the image quality of Ref image, resulting in more robust texture features.

3.3. Loss Function

To improve the visual quality of remote sensing reconstructed images, we design four loss functions. In addition to introducing texture loss $L_{tex}$, our algorithm also introduces reconstruction loss $L_{rec}$, perceptual loss $L_{per}$, and adversarial loss $L_{adv}$. The $L_{rec}$ is a commonly used loss function in SR, and the $L_{per}$ and the $L_{adv}$ are the losses introduced by the generation confrontation network in order to reconstruct a more realistic image. We describe these loss functions below.

3.3.1. Reconstruction Loss

The $L_{rec}$ is typically defined by the mean square error. The function aims to achieve a higher peak signal-to-noise ratio (PSNR), and the L1-norm is adopted:

$$\begin{array}{c}\hfill L_{rec}={∥I^{HR}-I^{SR}∥}_1,\end{array}$$

(7)

3.3.2. Perceptual Loss

Many SR networks have introduced perceptual loss to achieve more realistic visual quality. Therefore, our network also uses the perceptual loss function ($L_{per}$):

$$\begin{array}{c}\hfill L_{per}=\frac{1}{V}{\displaystyle \sum _{i=1}^C}{∥{\varphi }_i\left(I^{HR}\right)-{\varphi }_i\left(I^{SR}\right)∥}_F,\end{array}$$

(8)

where V and C denote the volume and channel of the feature map, respectively. ${\varphi }_i$ indicates the i-th channel of the output feature map, and ${∥∥}_F$ represents the Frobenius norm.

3.3.3. Adversarial Loss

The $L_{adv}$ function can improve the sharpness of SR reconstructed images. Our network uses the improved $L_{adv}$ function in WGAN [26]:

$$L_{adv}=-{\mathbb{E}}_{I^{HR}}\left[log\left(D\left(I^{HR},I^{SR}\right)\right)\right]-{\mathbb{E}}_{I^{SR}}\left[log\left(1-D\left(I^{SR},I^{HR}\right)\right)\right],$$

(9)

where D is a relativistic average discriminator [47].

4. Experiements

This section mainly introduces the experiments of the proposed MTTN network, including the dataset in Section 4.1, the evaluation indicators in Section 4.2, the implementation details of the experiments in Section 4.3, the quantitative and qualitative analyses with different methods in Section 4.4, and the ablation studies in Section 4.5.

4.1. Dataset

In this study, we use the Kaggle open-source remote sensing competition dataset (https://www.kaggle.com/c/draper-satellite-image-chronology/data). For the RefSR-based remote sensing image problem, the texture similarity between input remote sensing image pairs poses a significant impact on the SR results. In general, for training and evaluating RefSR algorithms, Ref images with various similarities to LR images are preferred. However, the Kaggle open-source dataset contains a wide variety of numbers and subject matter, including 324 different scenarios. Therefore, we constructed a dataset of reference images with various similarities based on the Kaggle dataset. We use the SIFT [48] feature-matching method to collect image pairs with different similarities. This feature-matching method collects Ref remote sensing images that are consistent with local texture-matching targets. We also segment these images with a size of $320\times 320$, leading to a total of 1200 remote sensing images, of which 600 are used as input images, and the other 600 are used as reference images for training. Note that the Ref image fed to the algorithm is the same scene as the input image but with different angles. Figure 3 presents some selected samples of our dataset.

4.2. Evaluation Indicators

In this experiment, we use five popular objective evaluation metrics, including peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [49], feature similarity (FSIM) [50], visual information fidelity (VIF) [51], and relative dimensionless global error in synthesis (ERGAS) [52].

Peak Signal-to-Noise Ratio (PSNR), with decibel (dB) as unit, evaluates the similarity between two images from the pixel similarity level. It is one of the most commonly used objective measures to evaluate image quality:

$$PSNR=10{log}_{10}\frac{{\left(2^N-1\right)}^2}{MSE}$$

(10)

$$MSE=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{\left[X\left(i,j\right)-Y\left(i,j\right)\right]}^2$$

(11)

where $MSE$ is the mean square error between the original image X and the reconstructed image Y, N is the number of bits of image pixels (most general optical images have 8 bits). The image’s width and height are denoted by W and H, respectively. The PSNR ranges from $[0,+\infty ]$, where a higher value indicates more similarity between the images.

Structural Similarity (SSIM) is an objective evaluation index designed based on the human visual system. By decomposing the image, it comprehensively evaluates the degree of similarity between two images based on image brightness, contrast, and structure. For images $I_x$ and $I_y$, SSIM can be calculated as:

$$SSIM\left(I_x,I_y\right)=\frac{\left(2{\mu }_{I_x}{\mu }_{I_y}+C_1\right)\left(2{\sigma }_{I_x{I}_y}+C_2\right)}{\left({\mu }_{I_x}^2+{\mu }_{I_y}^2+C_1\right)\left({\sigma }_{I_x}^2+{\sigma }_{I_y}^2+C_2\right)}$$

(12)

where ${\mu }_{I_x}$ and ${\mu }_{I_y}$ are the mean and luminance components between image pairs, respectively, and ${\sigma }_{I_x}^2$ and ${\sigma }_{I_y}^2$ are the variance and contrast components of the two images, respectively. ${\sigma }_{I_x{I}_y}$ is the covariance of images $I_x$ and $I_y$, used for structural contrast components. $C_1$, $C_2$, and $C_3$ are the balance parameters. The value range of SSIM is [0, 1]. The larger the value, the greater the degree of image similarity.

Feature Similarity Index Mersure (FSIM) uses low-level image features to replace image statistical features in SSIM and evaluates the feature similarity between two images from three feature levels of image phase consistency and gradient magnitude. For images $I_x$ and $I_y$, FSIM can be are calculated as follows:

$$FSIM\left(I_x,I_y\right)=\frac{\sum {}_{(I_x,I_y)\in \mathsf{\Omega }}{S}_L\left(I_x,I_y\right)·P{C}_m\left(I_x,I_y\right)}{\sum {}_{(I_x,I_y)\in \mathsf{\Omega }}P{C}_m\left(I_x,I_y\right)}$$

(13)

where $\mathsf{\Omega }$ represents the pixel domain of the whole image, $P{C}_m\left(·\right)$ and $S_L\left(·\right)$ represent the phase consistency and fusion similarity of the two images, respectively. The value range of FSIM is [0, 1]. The higher the value, the more similar the images are.

Visual Information Fidelity (VIF) is based on statistical models of natural scenes, image distortions, and the human visual system. For images $I_x$ and $I_y$, VIF can be calculated as:

$$VIF\left(I_x,I_y\right)=\frac{\sum I\left({\overline{C}}^N;{\overline{F}}^N\mid S^N\right)}{\sum I\left({\overline{C}}^N;{\overline{E}}^N\mid S^N\right)}$$

(14)

where $I\left({\overline{C}}^N;{\overline{F}}^N\mid S^N\right)$ and $I\left({\overline{C}}^N;{\overline{E}}^N\mid S^N\right)$ represent the information extracted from specific subbands of the input image. N is the number of sub-bands in the wavelet domain, C is the set of random vector fields, E is the random coefficient vector of the wavelet sub-band of the original image, F is the random coefficient vector of the wavelet sub-band of the distorted image, and S is an independent scalar factor.

The Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS) provides a global fusion evaluation. The smaller the value (the closer to zero), the better the performance. ERGAS is defined as:

$$ERGAS=100\frac{h}{l}\sqrt{\frac{1}{p}\sum _{i=1}^L\left(\frac{RMS{E}^2\left(i\right)}{MEAN\left(i\right)}\right)}$$

(15)

where h and l represent the spatial resolution of the panchromatic image and the multispectral image, respectively, P represents the number of bands of the fusion map, $MEAN\left(i\right)$ is the i-band of the mean Ref multispectral image, and $RMSE\left(i\right)$ is the Ref image and the fusion map MSE.

4.3. Experimental Details

In this study, we adopt VGG19 [53] as the pre-training model of the network and design an end-to-end remote sensing SR network. We first downsample the original remote sensing images to LR images of $40\times 40$ pixels, and then perform data augmentation by random horizontal flipping and 90-degree rotation during training, and adopt a learning rate decay strategy during training. The batch size of the network is set to 9, and the learning rate is initially set to $1\times 10^{-4}$ with the reduction rate of each epoch of 0.9. We use the ADAM optimizer [54] for network optimization, and the momentum parameter is set to ${\beta }_1=0.9$ and ${\beta }_2=0.999$.

For training and testing, we use a server with NVIDIA 1080Ti GPU and Intel I7-8700 CPU. Our MTTN network is implemented based on the TensorFlow framework under Ubuntu 16.04, CUDA 8.0 and CUDNN 5.1 systems. In our experiments, it takes about 20 h to train our model with a 1080Ti GPU under the above settings for 50 epochs.

4.4. Quantitative and Qualitative Comparison with Different Methods

To demonstrate the superiority of MTTN, we compare it to recent SISR and RefSR remote sensing imagery algorithms. The SR methods to be compared include Bicubic, SRCNN [55], VDSR [21], SRResnet [25], SRFBN [56], SRRFN [57], SEAN [58], HAN [59], MHAN [34], SRNTT [41], DLGNN [36], and HSENet [60]. Among them, DLGNN [36] and HSENet [60] are the latest remote sensing SR algorithms that demonstrate great reconstruction results. The competing algorithm also includes a RefSR method, i.e., SRNTT [41], a classic RefSR method.

All of the competing methods were trained and tested on the Kaggle public dataset that was recreated in this study to ensure a fair comparison.

Table 1. Comparison of average PSNR, SSIM, FSIM, VIF, and ERGAS results obtained from the Kaggle Open Source dataset. ↑ means that the larger the value, the better the reconstruction effect. ↓ means that the smaller the value, the better the reconstruction effect.

Algorithm/Metrics	PSNR/dB ↑	SSIM ↑	FSIM ↑	VIF ↑	ERGAS ↓
Bicubic	24.73	0.6972	0.7544	0.2558	2.7915
SRCNN	26.56	0.7675	0.8292	0.3368	2.2890
VDSR	27.20	0.8001	0.8461	0.3759	2.1461
SRResnet	26.42	0.7790	0.8480	0.3634	2.3560
SRFBN	28.02	0.8264	0.8617	0.4121	1.9713
SRRFN	28.61	0.8415	0.8794	0.4350	1.8559
SEAN	28.41	0.8374	0.8716	0.4261	1.8852
HAN	28.02	0.8259	0.8743	0.4157	1.9714
MHAN	27.86	0.8265	0.8687	0.4106	2.0071
SRNTT	30.28	0.8983	0.9243	0.6207	1.5731
DLGNN	28.04	0.8246	0.8609	0.4106	1.9688
HSEnet	28.42	0.8357	0.8746	0.4275	1.8907
MTTN (Ours)	30.48	0.9020	0.9333	0.6352	1.5355

displays the objective evaluation indicators for all the algorithms, and the best outcomes are highlighted in red. It can be seen from Table 1 that the proposed MTTN shows great performance on the five objective evaluation indicators, i.e., PSNR, SSIM, FSIM, VIF, and ERGAS.

The above results prove that certain SR models designed for natural images are not suitable for remote sensing images. The multi-scale texture transfer network we designed is more effective in learning the texture and structure of remote sensing images, as it uses texture information from Ref remote sensing images to assist in reconstructing texture detail information in LR images.

Figure 4 and Figure 5 presents a visual comparison of the experimental results and further the advantage of the proposed MTTN network. We selected typical scenes, including playgrounds, parking lots, ports, airports, buildings and roads, to evaluate the superiority of the MTTN network. From the zoomed-in details in Figure 4 and Figure 5, we notice the CNN-based approach can reconstruct some texture details. However, as some models are directly transferred to remote sensing SR tasks, the reconstructed remote sensing images contain notable blurred contours. The GAN-based method presents better visual effects, but the generated artifacts deteriorate the reconstructed remote sensing image. In comparison, our proposed MTTN algorithm produces sharper edges and finer details than other competing algorithms. We also observe that the proposed MTTN further enhances the visual performance of the reconstructed image by combining the multi-scale residual information of Ref images.

4.5. Ablation Studies

4.5.1. Effectiveness of Reference Image

To verify the effectiveness of Ref images on MTTN networks, we set up ablation experiments with different configurations.

Table 2. Comparative results of different modes of ablation experiments on the Kaggle Open Source dataset with a scale factor of 4.

Algorithm/Metrics	PSNR/dB ↑	SSIM ↑	FSIM ↑	VIF ↑	ERGAS ↓
MTTN without Ref images	30.31	0.8977	0.9319	0.6319	1.5866
MTTN without texture loss	30.10	0.8958	0.9314	0.6259	1.6130
MTTN without perceptual loss	29.73	0.8919	0.9268	0.6311	1.7056
MTTN (Ours)	30.48	0.9020	0.9333	0.6352	1.5355

lists the experimental results of various ablation experiments. The SISR model performs worse than the original network by 0.17 dB in terms of PSNR. However, our model still achieves comparable performance to some SISR algorithms without introducing Ref into remote sensing images. The MSE comparison chart in Figure 6 shows that after adding Ref remote sensing images, the reconstructed remote sensing images present richer texture details, indicating that SR performance can be enhanced when incorporating Ref images.

4.5.2. Effectiveness of Loss Function

Our proposed MTTN model uses a novel texture loss, making it distinct from most SR methods. Different from other texture transfer methods, the MTTN network transfers the most relevant texture of the reference image to the target image. Specifically, the local matching method maintains the texture consistency of remote sensing images; the adaptive neural migration improves texture transfer; and the spatial regularization achieves better global spatial consistency. The test results for texture loss are shown in Table 2. The PSNRs of MTTN with and without texture loss tested on the Kaggle dataset are 30.48 dB and 30.10 dB, respectively. It proves that the texture information from Ref cannot be effectively transferred to the target image without texture loss.

We also incorporate perceptual loss into this network to enhance the visual appeal of reconstructed remote sensing images. To verify the effectiveness of this loss, we conducted relevant ablation experiments. The experimental results are shown in Table 2. Regarding the test PSNR results on the Kaggle dataset, the PSNR values after introducing perceptual loss are significantly higher than the test results without perceptual loss. From Figure 6, we notice that the two loss functions in our proposed MTTN effectively reduce the disparity between SR images and the corresponding ground-truth images.

4.5.3. Effectiveness of Residual Blocks

In this study, we extract multi-scale texture information from remote sensing images using different numbers of residual blocks (RBs). Similar to other works [61,62], in order to make full use of the multi-scale texture information of remote sensing images, our different residual blocks adopt a combination of different convolution kernel sizes, and the specific structure is shown in Figure 2. According to the experience sharing of other work [41], we set the number of RBs at different scales to different numbers ($3\times 3$ convolution uses 16 $RB{s}_A$, $5\times 5$ convolution uses 8 $RB{s}_B$ and $7\times 7$ convolution uses 4 $RB{s}_C$), which proves that the MTTN network can ensure that the parameter quantity of the network does not increase additionally, and at the same time, it also guarantees a good reconstruction ability of the network. To verify the effects of different numbers of design methods of $RB{s}_A$ on the performance of MTTN, we designed a set of ablation experiments. As shown in Figure 7, the reconstruction performance of our network decreases as the number of residual blocks increases. Therefore, considering the network’s parameters and the reconstruction performance of remote sensing images, the MTTN network adopts the above settings to obtain better reconstruction performance when transmitting texture information.

4.5.4. Effectiveness of Remote Sensing Scene Classification

The SR algorithm for remote sensing images is considered a pre-processing step for some downstream tasks, e.g., change detection and semantic segmentation. To further support the efficacy of the suggested MTTN, we evaluate the reconstruction outcomes of various networks using an unsupervised semantic segmentation algorithm (ISODATA). We set the number of categories to five and the maximum number of iterations to five [63]. It is particularly challenging for ISODATA to obtain precise classification results because the results of the comparison experiments show significant spectral distortions and fuzzy patterns. The comparison of the experimental results of the classification algorithm is shown in Figure 8. In particular, we notice the misclassification of ships and the lack of texture information from the results of DLGNN and HSEnet, while SRNTT fails to recover detailed ship edge information. The results demonstrate that the reconstructed images by the MTTN network contain fine details, evidenced by the ISODATA classification results.

5. Conclusions

In this study, we propose MTTN, a novel RefSR method for remote sensing images with multi-scale residual transfer. Specifically, the proposed MTTN extends the prior information of training samples by introducing additional texture information from reference images. In addition, the use of multi-scale methods to transfer texture information also enriches the prior information required by the network at different stages. The MTTN network introduces multi-scale residual blocks to enhance the image edge and texture information. We perform a comprehensive evaluation of the experimental results in the Kaggle remote sensing competition dataset. Extensive experimental results demonstrate that MTTN can enhance the texture details and visual effects of reconstructed images. Quantitatively, the MTTN network outperforms other competing algorithms in all selected objective evaluation metrics. The self-reference ablation experiments demonstrate that MTTN achieves outstanding reconstructed results with and without reference images.

6. Discussion

This paper has conducted relevant research on remote sensing super-resolution algorithms based on reference images, laying a certain foundation for future related work. However, in the practical acquisition scenarios of remote sensing images, the process of obtaining low-resolution remote sensing images is complex, and there are factors such as noise, downsampling, and blur, making existing methods unsuitable for direct application in the super-resolution reconstruction of remote sensing images in real-world scenarios. Therefore, future research directions may consider investigating remote sensing image super-resolution reconstruction in complex real-world scenarios, aiming to enhance the model’s generalization capability and applicability.