Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks

Abstract

Image super-resolution (SR) techniques can improve the spatial resolution of remote sensing images to provide more feature details and information, which is important for a wide range of remote sensing applications, including land use/cover classification (LUCC). Convolutional neural networks (CNNs) have achieved impressive results in the field of image SR, but the inherent localization of convolution limits the performance of CNN-based SR models. Therefore, we propose a new method, namely, the dilated Transformer generative adversarial network (DTGAN) for the SR of multispectral remote sensing images. DTGAN combines the local focus of CNNs with the global perspective of Transformers to better capture both local and global features in remote sensing images. We introduce dilated convolutions into the self-attention computation of Transformers to control the network’s focus on different scales of image features. This enhancement improves the network’s ability to reconstruct details at various scales in the images. SR imagery provides richer surface information and reduces ambiguity for the LUCC task, thereby enhancing the accuracy of LUCC. Our work comprises two main stages: remote sensing image SR and LUCC. In the SR stage, we conducted comprehensive experiments on Landsat-8 (L8) and Sentinel-2 (S2) remote sensing datasets. The results indicate that DTGAN generates super-resolution (SR) images with minimal computation. Additionally, it outperforms other methods in terms of the spectral angle mapper (SAM) and learned perceptual image patch similarity (LPIPS) metrics, as well as visual quality. In the LUCC stage, DTGAN was used to generate SR images of areas outside the training samples, and then the SR imagery was used in the LUCC task. The results indicated a significant improvement in the accuracy of LUCC based on SR imagery compared to low-resolution (LR) LUCC maps. Specifically, there were enhancements of 0.130 in precision, 0.178 in recall, and 0.157 in the F1-score.

1. Introduction

Long time series and high-spatial-resolution remote sensing images play a crucial role in high-precision LUCC [1]. However, due to the limitation of hardware technology and cost, publicly available remote sensing data with high spatial resolution usually do not have long time series. For example, the Sentinel-2 satellite has a spatial resolution of up to 10 m, but its temporal coverage starts from 2015, and even expensive commercial satellite data are usually available from 2000 onwards. In contrast, remote sensing data with long time series usually do not have high spatial resolution. For example, Landsat series satellites have been providing valuable data since 1972. These data are frequently utilized for time series land use analysis. However, their spatial resolution is limited to 30 m, which restricts their application in long-term series and high-precision LUCC analysis. Therefore, it is crucial to improve the spatial resolution of long time series and low-spatial-resolution remote sensing images by means of algorithm development. Traditional SR methods for remote sensing images mainly include interpolation [2], Pansharp [3], sparse representation-based [4], and convex set projection [5]. The interpolation method has the advantage of simplicity and speed, but the interpolation results are usually blurred. The Pansharp method requires the sensor to have a high spatial resolution of the panchromatic bands, and can then improve the spatial resolution of other bands by data fusion. Methods based on sparse representation and convex set projection have high computational complexity and have difficulty recovering high-frequency details of the image. Convex set projection, in particular, demands a substantial amount of prior knowledge [6]. In recent years, deep learning techniques have been rapidly developed and have achieved impressive results in various computer vision (CV) tasks, including image super-resolution. Using deep learning techniques, LR data can be processed with super-resolution to improve the spatial resolution, which provides an opportunity to obtain higher-quality LUCC maps [7].

1.1. Deep Learning for Image Super-Resolution

The super-resolution convolutional neural network (SRCNN) [8] was the first convolutional neural network used for image super-resolution. SRCNN uses a stack of three convolutional layers to directly learn the mapping relationship between LR and HR end-to-end. Deep residual learning [9] was proposed to shift the structure of deep learning networks towards greater depth. The very deep super-resolution network (VDSR) [10] improves the super-resolution performance of the network by using residual concatenation and stacking very deep convolutional layers. SRCNN and VDSR up-sample the image when it is fed into the network, which results in slower training and high computational resource usage. The fast super-resolution convolutional neural network (FSRCNN) [11] and the enhanced deep super-resolution network (EDSR) [12] up-sample the feature maps in the last part of the network and achieved better super-resolution results in terms of the peak signal-to-noise ratio (PSNR) metric. Subsequently, the residual channel attention network (RCAN) [13] achieved better performance than deep convolutional networks such as VDSR and EDSR. This improvement was achieved by integrating the attention mechanism into the super-resolution network. This breakthrough showcased the mechanism’s ability to reconstruct intricate texture details within the image. Consequently, the attention mechanism has become a widespread inclusion in various super-resolution networks. The multispectral remote sensing images super-resolution convolutional neural network (msiSRCNN) [14] verified the feasibility of applying a convolutional neural network to the super-resolution of multispectral remote sensing images by fine-tuning SRCNN, and achieved better results than the traditional methods. The convolutional neural network became the mainstream method for the super-resolution of remote sensing images. Remote sensing images are distinct from ordinary optical images due to their diverse feature types and varying scales among these features. Aiming to address these problems, researchers have proposed many new structures [15,16,17,18,19], which are used to enhance the feature-learning capability of super-resolution networks for remote sensing images. Although CNN-based methods have achieved significant results in remote sensing image super-resolution tasks, the inherent localization of CNNs makes it difficult to model the global pixel dependencies of remote sensing images, which limits the further improvement of CNN performance in super-resolution tasks.

The Transformer [20] has quickly become the dominant approach in the field of natural language processing (NLP) with its powerful global modeling capabilities. The emergence of Vision Transformer (VIT) [21] introduced Transformer to the CV domain, achieving performance beyond CNNs on large datasets. Currently, there have been many works combining CNNs with Transformers for image super-resolution [22,23,24,25,26,27]. These methods use a CNN as a shallow feature extractor and Transformer for deep feature extraction, combining the local feature extraction capability of the CNN and the global modeling capability of Transformer to further improve the quality of SR images. Although Transformer can effectively compensate for the localization of CNN, in addition to the local–global learning capability, multi-scale feature learning is equally important for the super-resolution task of remote sensing images [17,18,27]. Unfortunately, Transformer does not have the ability of multi-scale feature learning. Numerous researchers have explored the integration of multi-scale information into the Transformer model [28,29,30,31]. However, these methods generally result in an increase in the number of network parameters, thereby further impeding the training process of the already large Transformer model. In addition to that, there are ways [29,32] to realize Transformer’s multiscale hierarchical representation of images by gradual down-sampling, but this is not applicable to the image super-resolution task. Although the CNN and Transformer-based methods are better than the traditional super-resolution methods and result in higher peak PSNR values, they are more ambiguous in visual perception.

Generative adversarial networks (GAN) [33] have powerful image-generation capabilities in image generation [34], style migration [35], and image super-resolution [36,37,38]. The GAN consists of two sub-networks, the generator and the discriminator, which are trained against each other in a ”zero-sum game”. The generator’s goal is to generate realistic images to deceive the discriminator, while the discriminator’s goal is to determine the authenticity of the input images. The generator updates the gradient based on the feedback from the discriminator. Adversarial training allows the GAN to generate images that are visually superior to the CNN. Super-resolution GAN (SRGAN) [36] uses the GAN network for the image super-resolution task and the pretrained VGG19 [39] network is used as a feature extractor to compute the perceptual loss to optimize the perceptual quality of the SR images. The enhanced super-resolution GAN (ESRGAN) [37] is an improvement of SRGAN; ESRGAN uses dense residual blocks to enhance the feature-learning capability of the network and removes the BatchNorm layer [40] from SRGAN. ESRGAN is still one of the most advanced image super-resolution methods. For the task of remote sensing image super-resolution, researchers have made many improvements to the GAN network, including the introduction of the attention mechanism [41,42], processing after super-resolution [43], and improvements to the discriminator [44], etc. GAN-based methods have more powerful image-generation capabilities than CNN-based methods, which generate SR images with more details. Therefore, we choose to train our model using the GAN framework.

1.2. Deep Learning for Land Use/Cover Classification

Land use/cover classification can extract information of natural land types as well as artificially utilized land types from remote sensing images, which is important in the fields of ecological protection, urban planning, and precision agriculture. The traditional LUCC methods [45,46,47] often rely on artificially designed features, such as spectral indices [48], and the spatial correlation of pixels is ignored. In contrast to traditional classification methods, deep-learning-based approaches eliminate the dependence on artificial features. They effectively capture both the spatial and spectral features inherent in remote sensing images [49], leading to superior classification accuracy and enhanced robustness. Fully convolutional neural networks (FCN) [50] represent an enlightening approach for the semantic segmentation task based on deep learning, which can realize the classification of images at pixel level. U-net [51] is a new approach for semantic segmentation, which was initially proposed for biomedical image segmentation, but has been widely used for image segmentation in various fields, including remote sensing, due to its superior performance. The Deeplab [52,53,54,55] family of models are another classic set of approaches for image segmentation tasks as well as U-net networks. In contrast to the stepwise down-sampling structure of U-net, Deeplab employs dilated convolutions [56] to facilitate multi-scale feature learning, thereby enhancing the segmentation accuracy. At present, in the remote sensing image LUCC task, the Transformer-based classification method is one of the hotspots in research. The self-attention mechanism of Transformer means that it can model the spectral features well. Many researchers have opted to integrate CNNs and Transformer by utilizing a CNN for extracting the spatial features and employing Transformer to capture the spectral features. These methods incorporating both spatial and spectral features have achieved better accuracy in the LUCC task [57,58,59,60]. The morphFormer, proposed by Roy et al. [61], integrates the learnable spectral morphological convolution operation and a self-attention mechanism. This combination enhances the interaction between the spectral features and improves the representation of the structure and shape information in tokens. When compared to traditional CNNs and other Transformer LUCC models, morphFormer achieves higher classification accuracy in experiments. Thus, it stands as one of the most advanced LUCC methods available at present. In this study, this method was directly employed for the SR data in the second stage, specifically for the LUCC task.

The objective of this study is to enhance the spatial resolution of remote sensing images using deep learning techniques. This enhancement aims to provide richer and more accurate surface information for LUCC tasks, thereby further improving the precision of LUCC. This study is mainly divided into two stages: remote sensing image SR and LUCC. In the SR stage, we propose a new model named the dilated Transformer GAN (DTGAN) for real remote sensing image super-resolution. The generator of this model combines CNN and Transformer, using a CNN as a shallow feature extractor and Transformer for deep feature extraction. At the same time, we seek to solve the problems of Transformer’s inability to learn multi-scale features and its slow computation and large resource consumption. This research is influenced by [32,62,63,64], with regard to the attention mechanism called dilated window multi-head self-attention (DW-MHSA), which can introduce multi-scale information into the Transformer and improve the computation efficiency of the self-attention without increasing the network parameters. The discriminator of DTGAN uses PatchGAN [38]. In the LUCC stage, we directly adopt morphFormer [61] in the LUCC of SR to verify the availability of the SR data.

2. Method for Remote Sensing Image Super-Resolution

2.1. The Overall Structure of the Generator

The generator structure of DTGAN is shown in Figure 1. Given a low-resolution input image $I_{\mathrm{LR}}$, it is first mapped from the real image to the feature space by a convolutional layer:

$$f_0=\mathrm{Conv}\left(I_{\mathrm{LR}}\right)$$

(1)

where the number of input channels of Conv is 4, the number of output channels is 64, the convolution kernel size is 3, and the padding width is 1. $f_0$ represents the initial shallow features.

As shown in Figure 1, the residual channel attention group (RCAG) in RCAN [13] is used as the base CNN backbone network to extract the shallow features. RCAG contains n residual channel attention blocks (RCAB). Each RCAB consists of two Conv&Relu layers, with the channel attention (CA) layer positioned between them. Moreover, the CA layer incorporates residual connections between the inputs and outputs. The structure of the CA layer is shown in Figure 2.

For the CA layer, if the input feature map is $f_{\mathrm{CA}}^{\mathrm{in}}$, the CA process can be represented as follows:

$$\begin{array}{cc}\hfill \mathrm{avg}& =\mathrm{AvgPool}\left(f_{\mathrm{CA}}^{\mathrm{in}}\right)\hfill \\ \hfill \max & =\mathrm{MaxPool}\left(f_{\mathrm{CA}}^{\mathrm{in}}\right)\hfill \\ \hfill \mathrm{merged}& =\mathrm{Relu}\left(\mathrm{Conv}\right(\mathrm{avg}+\max \left)\right)\hfill \\ \hfill f_{\mathrm{CA}}^{\mathrm{out}}& =\mathrm{SoftMax}\left(\mathrm{merged}\right)\otimes f_{\mathrm{CA}}^{\mathrm{in}}\hfill \end{array}$$

(2)

where AvgPool and MaxPool represent the global average pooling and maximum pooling operations, respectively, applied to $f_{\mathrm{CA}}^{\mathrm{in}}$ in the spatial dimensions. Subsequently, the global average and maximum information are fused through a Conv&Relu layer to obtain the merged result. The SoftMax function maps the fused information to the range [0,1], representing the importance of each channel. Element-wise multiplication of this attention with $f_{\mathrm{CA}}^{\mathrm{in}}$ results in $f_{\mathrm{CA}}^{\mathrm{out}}$. The process of $f_0$ through the RCAG backbone network can be expressed as:

$$\begin{array}{c}\hfill f_n={\mathrm{RCAG}}_n\left(f_{n-1}\right)={\mathrm{RCAG}}_n\left({\mathrm{RCAG}}_{n-1}(\dots {\mathrm{RCAG}}_1\left(f_0\right)\dots )\right)\end{array}$$

(3)

where ${\mathrm{RCAG}}_n$ represents the nth RCAG. In this study, we used four RCAGs as the CNN backbone network for feature extraction and each RCAG contains five RCABs.

Prior to feeding the features $f_1,f_2,\dots ,f_n$ into the enhanced encoder layer with multiscale hierarchical feature enhancement, a $1\times 1$ convolutional layer is employed to reduce the dimensionality of the feature maps. This step aims to minimize the redundant features and to improve the computational efficiency. Then the feature map is divided into $P\times P$ feature patches to obtain tokens by patch linear embedding. This embedding method is achieved using a convolutional layer with a convolutional kernel of the same size as the patch.

$$\begin{array}{cc}\hfill {\widehat{f}}_i& ={\mathrm{Conv}}_{k=1}\left(f_i\right)\hfill \\ \hfill {\mathrm{tokens}}_i& ={\mathrm{Conv}}_{k=P}\left({\widehat{f}}_i\right)\hfill \end{array}$$

(4)

where $f_i$ represents the feature map output from the RCAG backbone network, with $i=1,2,\cdots ,n$, ${\widehat{f}}_i$ represents the corresponding feature maps after dimensionality reduction, and ${\mathrm{Conv}}_{k=1}$ represents a convolutional layer with a convolutional kernel size of $1\times 1$. Then a convolutional layer with kernel size $P\times P$ and stride equal to P is used to divide the feature map into $P\times P$ patches and embed them into tokens.

Subsequently, ${\mathrm{tokens}}_i$ enter the Transformer-based encoder layer feature enhancement module. The encoder layer has different dilated rates to learn the multi-scale representation of the features when calculating the DW-MHSA. These multi-scale features are fused in the decoder layer. The query matrix of the last decoder layer comes from the output of the last encoder layer, while the query matrix of the remaining decoder layers comes from the output of the previous decoder layer. The key and value matrices of all the decoder layers come from the output of the multi-scale encoder layer. The decoder layer fuses the multi-scale features by mixed multi-head attention (mixed-MHSA). Figure 3 shows the detailed information flow of the last encoder layer and decoder layer. The hierarchical multiscale feature fusion enhancement module can be represented as:

$$\begin{array}{cc}\hfill f_{{\mathrm{De}.\mathrm{ly}.}_{i+1}}& ={\mathrm{De}.\mathrm{ly}.}_{i+1}[{\mathrm{En}.\mathrm{ly}.}_i\left({\mathrm{tokens}}_i\right)+{\mathrm{En}.\mathrm{ly}.}_{i+1}\left({\mathrm{tokens}}_{i+1}\right)]\hfill \\ \hfill f_{{\mathrm{De}.\mathrm{ly}.}_i}& =f_{{\mathrm{De}.\mathrm{ly}.}_{i+1}}+{\mathrm{En}.\mathrm{ly}.}_i\left({\mathrm{tokens}}_i\right)i=1,2,\cdots ,n-2\hfill \end{array}$$

(5)

where De.ly. and En.ly. denote the decoder layer and the encoder layer, and $f_{{\mathrm{De}.\mathrm{ly}.}_{i+1}}$ represents the output feature of the ith decoder layer. The output feature $f_{{\mathrm{De}.\mathrm{ly}.}_{\mathrm{out}}}$ from the decoder layer undergoes a depth-to-space transformation, reconstructing patches into feature maps. Subsequently, bicubic interpolation is applied to up-sample the feature maps by a specified factor. Finally, the up-sampled feature maps pass through two stacked Conv-Relu layers to obtain the ultimate output. This process can be expressed as:

$$\begin{array}{c}\hfill f_{\mathrm{img}}=\mathrm{reshape}\left(f_{{\mathrm{De}.\mathrm{ly}.}_{\mathrm{out}}}\right)\\ \hfill \mathrm{SR}=\mathrm{tail}\left(\mathrm{Bicubic}\left(f_{\mathrm{img}}\right)\right)\end{array}$$

(6)

2.2. Multi-Scale Feature Fusion and Enhance Module

2.2.1. Dilated Window Multi-Head Self-Attention

As shown in Figure 4c, in the computation of self-attention, the input features are first mapped into Query(Q), Key(K), and Value(V) matrices through three linear layers. In the scaled dot-product attention mechanism of the Transformer, the calculation of the pixel value $x_{i,j}$ at position $(i,j)$ in the output feature map $X_{\mathrm{out}}^{hw\times C}$ is performed as follows:

$$\begin{array}{cc}\hfill x_{i,j}& =\mathrm{Attention}(q_{i,j},K,V)\hfill \\ \hfill & =\mathrm{Softmax}\left(\frac{q_{i,j}{K}^T}{\sqrt{d}}\right)V,1\le i\le H,1\le j\le W\hfill \end{array}$$

(7)

where H and W denote the height and width of the feature map, respectively. $\sqrt{d}$ denotes the scaling factor, and the default value of d is equal to the hidden dimension of the K and the V matrix. The scaled dot-product attention calculates the attention score for each pixel by considering its relationship with all the other pixels. This operation becomes computationally expensive when the input feature map is large. To address this issue and introduce multi-scale information into the Transformer for enhancing the model’s capability of reconstructing SR images, we propose a multi-scale attention mechanism based on dilated convolution. This idea is inspired by studies such as [32,62,63,64]

As shown in Figure 4a, dilated convolution expands the receptive field and captures multi-scale information by utilizing different dilated rates. As shown in Figure 4b, we introduce this idea into Transformer and propose a new attention mechanism called dilated window self-attention (DWSA). DWSA uses window attention to replace global attention, which helps to improve the network computation speed, reduce resource overhead, and ensure the high-fidelity reconstruction of the remote sensing images. For any query pixel $q_{i,j}$ in the Q matrix, its attention is computed using a window centered at $(i,j)$ on the K and V matrix. The set of positions of the pixel points within the window $(i^{\prime },j^{\prime })$ can be expressed as:

$$\begin{array}{c}\hfill \left\{(i^{\prime },j^{\prime })\right|i^{\prime }=i+p\times r,j^{\prime }=j+q\times r\},-\frac{w}{2}\le p,q\le \frac{w}{2}\end{array}$$

(8)

where r denotes the dilated rate and w denotes the size of the window. For pixels at the edges of the feature map, we use mirrored edge filling to ensure that the feature map size does not change. The DWSA can be expressed as:

$$\begin{array}{cc}\hfill x_{i,j}& =\mathrm{Attention}(q_{i,j},K_r,V_r)\hfill \\ \hfill & =\mathrm{Softmax}\left(\frac{q_{i,j}{K}_r^T}{\sqrt{d}}\right)V_r,1\le i\le H,1\le j\le W\hfill \end{array}$$

(9)

where $K_r$ and $V_r$ denote the window with a dilated ratio r on the K and V matrices, respectively.

Considering an input feature map $X_{\mathrm{in}}^{hw\times C}$ and a window size of $M\times M$ for dilated convolutions, the computational complexity for DW-MHSA and MHSA can be defined as follows:

$$\begin{array}{cc}\hfill \mathsf{\Omega }\left(\mathrm{MHSA}\right)& =4hw{C}^2+2{\left(hw\right)}^{2}C\hfill \\ \hfill \mathsf{\Omega }\left(\mathrm{DW}-\mathrm{MHSA}\right)& =4hw{C}^2+2M^{2}hwC\hfill \end{array}$$

(10)

This is analogous to the windowed attention computation in the Swin Transformer [32]. However, it eliminates the need for intricate window-shifting operations to facilitate communication between different windows. Moreover, it enables the control of window size through the dilation rate r, thereby enabling the learning of multiscale information.

2.2.2. Transformer Encoder Layer

In this study, we set the number of Transformer encoder layers to 4, as shown in Figure 1. Each encoder layer has different dilation rates to extract the multiscale features and the number of encoder blocks is set to 1. The detailed structure of the encoder block is illustrated in Figure 3. In contrast to the approach proposed in [20], we replaced MHSA with DW-MHSA. Furthermore, we positioned a layer norm [65] before the attention and multi-layer perceptron (MLP) sub-layers within the encoder block. For the ith encoder layer, with input ${\mathrm{tokens}}_i$, where $f_0=\mathrm{tokens}$, the encoder layer can be represented as:

$$\begin{array}{cc}\hfill f_0& =\mathrm{tokens}\hfill \\ \hfill f_j^{\prime }& =\mathrm{DW}-\mathrm{MHSA}\left(\mathrm{LN}\left(f_{j-1}\right)\right)+f_{j-1},j=1,2,\dots ,n\hfill \\ \hfill f_j& =\mathrm{MLP}\left(\mathrm{LN}\left(f_j^{\prime }\right)\right)+f_j^{\prime },j=1,2,\dots ,n\hfill \end{array}$$

(11)

where j denotes the number of blocks in the encoder layer, and $f_j$ represents the output feature of the jth encoder block.

2.2.3. Transformer Decoder Layer

In our approach, we utilized decoder layers to fuse the multiscale features obtained from the encoder layers. Similarly, each decoder layer comprises several decoder blocks. Within the decoder block, we employed MHSA to extract the deep features at a global scale. Unlike the encoder block, the decoder block incorporates a hybrid attention module in addition to the MHSA module and MLP sub-network. As depicted in Figure 3, this module takes the multiscale features from the output of the encoder layer as the K, V matrices for computing attention, while the Q matrix is derived from the output of the previous decoder layer. In our work, the encoder layer produces features of different scales, which are gradually fused within the decoder layer through the mixed-MHSA module. Let $z_0$ denote the output of the encoder layer $[f_{E_1},f_{E_2},f_{E_3},\dots ,f_{E_n}]$. The process of the decoder layer can be represented as:

$$\begin{array}{cc}\hfill z_0& =[f_{E_1},f_{E_2},f_{E_3},\dots ,f_{E_n}]\hfill \\ \hfill z_j^{\prime }& =\mathrm{MHSA}\left(\mathrm{LN}\left(z_{j-1}\right)\right)+z_{j-1},j=1,2,\dots ,n\hfill \\ \hfill z_j^{″}& =\mathrm{Mixed}-\mathrm{MHSA}(\mathrm{LN}\left(z_{j-1}\right),\mathrm{LN}\left(z_0\right))+z_j^{\prime },j=1,2,\dots ,n\hfill \\ \hfill z_j& =\mathrm{MLP}\left(z_j^{\prime \prime }\right)+z_j^{\prime \prime },j=1,2,\dots ,n\hfill \end{array}$$

(12)

where $z_j$ denotes the output feature of the jth decoder block in the decoder layer.

2.3. The Overall Structure of the Discriminator

As illustrated in Figure 5, the traditional VGG-style discriminator was replaced with a PatchGAN discriminator using the U-Net structure, similar to [38]. This U-Net PatchGAN demonstrates superior discriminative ability for complex remote sensing images. It provides precise gradient feedback for local textures instead of distinguishing the global style of the generated image. The U-Net discriminator yields relative authenticity values for each pixel, offering detailed per-pixel feedback to the generator. However, the introduction of the U-Net structure also increased the instability of GAN training. To address this, spectral normalization regularization [66] was employed to stabilize the training process. Additionally, spectral normalization helps mitigate excessive sharpness and artifacts introduced by GAN training. The down-sampling in PatchGAN is achieved through convolution with a stride of 2, and up-sampling is implemented using bicubic interpolation.

3. Dataset and Experimental Setup

3.1. Dataset

For the first stage of the remote sensing image super-resolution task, our study utilized Landsat-8 Collection 2 Tier 1 and Sentinel-2 data as the experimental datasets. Both L8 and S2 provide multiple spectral bands, including visible and infrared bands, enhancing the accuracy of the LUCC task in the second stage. The focus of the first stage was on the remote sensing image SR, where we utilized a total of five L8 images and seven S2 images. The data used in the SR stage covered the northern regions of Henan Province and the western regions of Shandong Province. This area experiences a warm temperate monsoon climate with concurrent rain and heat. The earliest image was captured on 29 April 2023, during spring, and the most recent image was captured on 23 June 2023, during summer. The primary land types in this region include cultivated land and certain areas of the Taihang Mountains. The predominant vegetation consists of crops, coniferous forests, deciduous broadleaf forests, urban vegetation, and small wetlands and grasslands. In the second stage LUCC tasks, we utilized two images each from L8 and S2, captured on 1 May 2023, and 14 May 2023, respectively. These images were stitched and cropped to obtain remote sensing images of the Yongcheng City area. All the data used had cloud coverage controlled at 10% or below. Figure 6 displays the geographical locations of the selected data for the SR stage. To differentiate between Landsat-8 and Sentinel-2 data, L8 data is represented in true colors, while S2 data is displayed in false colors.

In the experiments, the 10 m resolution red (R), green (G), blue (B), and near-infrared (NIR) bands of the S2 data were used as the high-resolution target data. Corresponding bands with 30 m resolution from the L8 imagery were used as the LR input data. The acquired data were preprocessed with coordinate system alignment and atmospheric correction. The matched regions were then cropped into training samples. The L8 low-resolution samples were cropped into patches of size $80\times 80$ pixels with a stride of 70 pixels, while the corresponding high-resolution S2 samples were $240\times 240$ pixels with a cropping stride of 210 pixels. After manual screening to exclude data with noise, cloud cover, and other disturbances, a total of 12,800 patches were obtained. These patches were divided into training, validation, and test sets in a ratio of 8:1:1. Figure 7 illustrates some sample training images.

For the second stage of the LUCC task, we selected Yongcheng City in Shangqiu, China, as our study area. The image acquisition and preprocessing methods were consistent with the first stage. The processed images were stitched and cropped to obtain L8 and S2 images within the Yongcheng City area. The pre-trained DTGAN from the first stage was employed to generate SR images for the L8 imagery, as illustrated in Figure 8. The samples used for classification were the SR images of Yongcheng City. The land use/cover types in the Yongcheng City area were classified into five classes: barren land, building land, vegetation land, water, and mulching film.

3.2. Experimental Setup

This study employed the PyTorch framework [67,68] and was trained on an NVIDIA GeForce RTX 3090 graphics card. For the remote sensing image super-resolution task, a fixed global random seed of 0 was set during training. The Adam optimizer [69] was employed with an initial learning rate of $2\times {10}^{-4}$, a learning rate decay factor of $0.2$, decay epochs every 150 rounds, and a total of 500 epochs. The batch size was set to 64. Data augmentation techniques were employed during data loading, including random rotations of ${90}^{\circ }$, ${180}^{\circ }$, and ${270}^{\circ }$, as well as horizontal and vertical flips. The detailed configuration of the generator used in training is provided in

Table 1. The generator details of DTGAN.

Patch Size	CNN Backbone	P.E. in En.	P.E. in De.	En.Ly.	De.Ly.	En.ly.Depth	De.ly.Depth
$4\times 4$	RCAG	True	True	4	3	1	4

.

In the table, En.Ly. and De.Ly. represent the number of encoder layers and decoder layers, as shown in Figure 1. En.Ly. Depth and De.Ly. Depth indicate the number of encoder blocks and decoder blocks within a single encoder layer and decoder layer, respectively.

The pixel loss, perceptual loss, and spectral angle mapping (SAM) loss were utilized to train the generator, while the RaGAN loss [70] was employed to train the entire GAN network. We used the ${\mathcal{L}}_1$ loss to denote the pixel loss. Given the training dataset ${\{I_{\mathrm{LR}}^i,I_{\mathrm{HR}}^i\}}_{i=1}^N$, the ${\mathcal{L}}_1$ loss can be expressed as:

$${\mathcal{L}}_1=\frac{1}{N}\sum _{i=1}^N{\parallel I_{\mathrm{HR}}^i-G_{\omega }\left(I_{\mathrm{LR}}^i\right)\parallel }_1$$

(13)

where $G_w$ represents the generator with parameters w, $G_{\omega }\left(I_{\mathrm{LR}}^i\right)$ represents the super-resolution image $I_{\mathrm{SR}}^i$, and N denotes the total number of images used for training.

The perceptual loss utilizes a pretrained model as a feature extraction network to compare the feature representations of the generated images and the real images, thereby measuring their similarity. The perceptual loss aims to assess the image quality from a perceptual perspective by optimizing high-level semantic features to generate more visually realistic images. In the context of remote sensing image SR, we employed a ConvNet [71] model pretrained on the BigEarthNet [72,73] as our feature extraction network. The perceptual loss can be defined as:

$${\mathcal{L}}_p=\frac{1}{N}\sum _{i=1}^N{\parallel {\mathsf{\Phi }}_{\theta }\left(I_{\mathrm{HR}}^i\right)-{\mathsf{\Phi }}_{\theta }\left(G_{\omega }\left(I_{\mathrm{LR}}^i\right)\right)\parallel }_1$$

(14)

where ${\mathsf{\Phi }}_{\theta }$ represents the pretrained neural network with parameters $\theta$.

The Landsat-8 and Sentinel-2 images exhibit distinct spectral differences. To constrain the spectral changes during network training, we employed the SAM loss to optimize the spectral information of the SR images. SAM measures spectral similarity by calculating the angles between spectra. The SAM loss can be defined as:

$${\mathcal{L}}_s=\frac{1}{N}\sum _{i=1}^{N}arccos\left(\frac{{\left(I_{\mathrm{HR}}^i\right)}^T·G_{\omega }\left(I_{\mathrm{LR}}^i\right)}{\parallel {I_{\mathrm{HR}}^i\parallel }_2·{\parallel G_{\omega }\left(I_{\mathrm{LR}}^i\right)\parallel }_2+ϵ}\right)$$

(15)

where ${\left(I_{\mathrm{HR}}^i\right)}^T$ represents the transpose of $I_{\mathrm{HR}}^i$, and $ϵ$ denotes a small value used to prevent division by zero.

In contrast to the standard GAN network, which directly determines whether the input is real or fake, RaGAN evaluates the probability that the input data is more authentic than the real data. The optimization objective of RaGAN is to minimize the relative discrimination score of the generator and to maximize that of the discriminator. This enables the generator to better capture the overall data distribution, resulting in the generation of more realistic and diverse samples. The RaGAN loss can be expressed as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_G& =-\frac{1}{N}\sum _i^N{\mathbb{E}}_{I_{\mathrm{HR}}^i\sim p_{\mathrm{real}}}[log{D}_{\phi }\left(I_{\mathrm{HR}}^i\right)-log(1-D_{\phi }\left(G_{\theta }\left(I_{\mathrm{LR}}^i\right)\right))]\hfill \\ \hfill {\mathcal{L}}_D& =-\frac{1}{N}\sum _i^N{\mathbb{E}}_{G_{\theta }\left(I_{\mathrm{LR}}^i\right)\sim p_{\mathrm{fake}}}[log{D}_{\phi }\left(G_{\theta }\left(I_{\mathrm{LR}}^i\right)\right)-log(1-D_{\phi }\left(I_{\mathrm{HR}}^i\right))]\hfill \end{array}$$

(16)

where $p_{\mathrm{real}}$ and $p_{\mathrm{fake}}$ represent the distributions of the HR and SR data, respectively, and $D_{\phi }$ denotes the discriminator with parameters $\phi$. Hence, the total loss of the network can be expressed as:

$${\mathcal{L}}_{\mathrm{total}}={\alpha }_1{\mathcal{L}}_1+{\alpha }_2{\mathcal{L}}_p+{\alpha }_3{\mathcal{L}}_s+{\alpha }_4{\mathcal{L}}_{\mathrm{RaGAN}}$$

(17)

where ${\alpha }_{i,i=1,2,3,4}$ represents the weight coefficients for different loss functions. In our experiment, the different values of ${\alpha }_i$ are [1.0, 0.2, 0.05, 0.5].

For the LUCC task, we employed morphFormer [61] as the classifier. We partitioned a $5\times 5$ window around each pixel as the input for the network, and the output was the category of the central pixel. The model utilized the Adam optimizer [69], with an initial learning rate set to $5\times {10}^{-4}$ and a batch size of 256. The model was trained for a total of 50 batches.

3.3. Metrics

For the remote sensing SR task, we employed several quality evaluation metrics for the SR images, including the peak signal-to-noise ratio (PSNR), the structural similarity index measure (SSIM) [74], the spectral angle mapper (SAM), and the learned perceptual image patch similarity (LPIPS) [75].

PSNR is one of the most commonly used metrics for image quality evaluation. It measures the relationship between the maximum signal value and the noise in the image, and its unit is decibels (dB). Higher PSNR values indicate better image quality. The expression for PSNR is:

$$\begin{array}{cc}\hfill \mathrm{MSE}(G_{\omega }\left(I_{\mathrm{LR}}\right),I_{\mathrm{HR}})& =\parallel G_{\omega }\left(I_{\mathrm{LR}}\right)-I_{\mathrm{HR}}{\parallel }_2\hfill \\ \hfill \mathrm{PSNR}(G_{\omega }\left(I_{\mathrm{LR}}\right),I_{\mathrm{HR}})& =10\times {log}_{10}\left(\frac{\mathrm{MAX}\left(G_{\omega }\left(I_{\mathrm{LR}}\right)\right)}{\mathrm{MSE}(G_{\omega }\left(I_{\mathrm{LR}}\right),I_{\mathrm{HR}})}\right)\hfill \end{array}$$

(18)

where MSE represents the mean squared error between the SR image and the ground truth image.

SSIM measures the similarity between two images in terms of brightness, contrast, and structural information, with values ranging from 0 to 1. A value closer to 1 indicates a higher similarity between the two images. SSIM can be expressed as:

$$\begin{array}{c}\hfill \mathrm{SSIM}(G_{\omega }\left(I_{\mathrm{LR}}\right),I_{\mathrm{HR}})=\frac{(2{\mu }_{G_{\omega }\left(I_{\mathrm{LR}}\right)}{\mu }_{I_{\mathrm{HR}}}+C_1)(2{\sigma }_{G_{\omega }\left(I_{\mathrm{LR}}\right)}{I}_{\mathrm{HR}}+C_2)}{({\mu }_{G_{\omega }\left(I_{\mathrm{LR}}\right)}^2+{\mu }_{I_{\mathrm{HR}}}^2+C_1)({\sigma }_{G_{\omega }\left(I_{\mathrm{LR}}\right)}^2+{\sigma }_{I_{\mathrm{HR}}}^2+C_2)}\end{array}$$

(19)

where $\mu$ and $\sigma$ represent the mean and variance of the image, respectively. $C_1=k_{1}L$, $C_2=k_{2}L$ are constants, L represents the dynamic range of the image pixel values, and $k_1$ and $k_2$ are usually set to $0.01$ and $0.03$, respectively.

SAM measures the spectral similarity between a multispectral image and a reference image by calculating the angles between their spectra. A smaller angle indicates higher similarity, while a larger angle indicates lower similarity. SAM can be expressed as:

$$\mathrm{SAM}(G_{\omega }\left(I_{\mathrm{LR}}\right),I_{\mathrm{HR}})=arccos\left(\frac{{\left(G_{\omega }\left(I_{\mathrm{LR}}\right)\right)}^T·I_{\mathrm{HR}}}{\parallel G_{\omega }\left(I_{\mathrm{LR}}\right){\parallel }_2^{\frac{1}{2}}·{\parallel I_{\mathrm{HR}}\parallel }_2^{\frac{1}{2}}+ϵ}\right)$$

(20)

where ${\left(I_{\mathrm{SR}}\right)}^T$ represents the transpose of $I_{\mathrm{SR}}$.

LPIPS calculates the distance between the image activation maps using a pretrained network to measure the perceptual similarity between the images. A lower LPIPS score indicates a higher perceptual similarity between the images. LPIPS can be expressed as:

$$\mathrm{LPIPS}(G_{\omega }\left(I_{\mathrm{LR}}\right),I_{\mathrm{HR}})={\parallel {\mathsf{\Phi }}_{\theta }\left(I_{\mathrm{HR}}\right)-{\mathsf{\Phi }}_{\theta }\left(G_{\omega }\left(I_{\mathrm{LR}}\right)\right)\parallel }_1$$

(21)

For the LUCC task, we utilized precision, recall, and the F1-score to measure the accuracy of the classification results.

4. Experimental Results and Discussion

4.1. Experimental Comparison of the SR Model

To validate the effectiveness of our proposed DTGAN, we conducted comparisons with traditional interpolation methods as well as several state-of-the-art deep-learning-based image super-resolution techniques. The interpolation method employed bicubic interpolation. The deep learning methods included VDSR [10], DCMNet [76], LGCNet [15], RCAN [13], SwinIR [25], SRFormer [77], TransENet [22], and SRGAN [36]. Among these, VDSR [10], DCMNet [76], LGCNet [15], and RCAN [13] are CNN-based methods, while SwinIR [25], SRFormer [77], and TransENet [22] leverage the Transformer architecture for image super-resolution. SRGAN [36] is an advanced algorithm based on the GAN framework.

In

Table 2. Metrics of methods on test data.

Methods	PSNR	SSIM [74]	SAM	LPIPS [75]	Params	FLOPs
Bicubic	18.4568	0.5157	0.2578	2.3799	-	-
VDSR [10]	32.2639	0.8224	0.0625	0.1174	0.85M	49.10G
DCMNet [76]	32.3710	0.8248	0.0603	0.1097	2.26M	14.47G
LGCNet [15]	32.2520	0.8266	0.0612	0.1133	0.77M	44.28G
RCAN [13]	32.5094	0.8357	0.0588	0.1051	15.63M	99.32G
SwinIR [25]	32.2304	0.8320	0.0619	0.1100	3.13M	20.05G
SRFormer [77]	32.2275	0.8298	0.0617	0.1107	3.04M	19.35G
TransENet [22]	32.5410	0.8348	0.0589	0.1036	35.53M	58.31G
SRGAN [36]	31.8079	0.8274	0.0544	0.0773	1.32M	12.37G
DTGAN (Ours)	32.0343	0.8232	0.0491	0.0675	2.5M	10.39G

, the performance of different methods on the test dataset is displayed. The cells highlighted in red indicate the best accuracy, while the ones in blue represent the second-best accuracy. When compared to methods based on CNN and Transformer, GAN-based methods exhibit lower PSNR and SSIM scores but perform better in terms of the LPIPS perceptual metric. This suggests that images generated by GAN-based methods are more perceptually aligned with human vision. DTGAN achieved the best accuracy in the SAM and LPIPS metrics, indicating that our method’s images are closer to real images both in terms of their spectral and perceptual quality. Additionally, we compared the parameter count and computational complexity of different models. The parameter count signifies the model size, while the computational complexity indicates the model’s inference speed, with lower complexity leading to faster inference. For SRGAN and DTGAN, we only considered the parameter count and the computational complexity of the generator. Among the compared methods, LGCNet and VDSR have the fewest and second-fewest parameters, while DTGAN has the lowest computational complexity. DTGAN ranks fourth in terms of parameter count, but still outperforms several Transformer-based methods.

Figure 9 illustrates the visual results of the SR images produced by different methods, along with a comparison of the absolute errors with real images. From Figure 9, it can be observed that compared to GAN-based methods, the CNN and Transformer-based methods have higher PSNR and SSIM values but lower perceptual quality, appearing visually smoother and lacking detailed information. Hence, they exhibit larger absolute errors when compared with real images. The super-resolution results of SRGAN and DTGAN are visually similar, with smaller absolute errors when compared to HR images. However, DTGAN outperforms SRGAN in some texture details.

The spectral information of remote sensing images plays a crucial role in the accuracy of the LUCC task. We calculated the MSE between the SR images generated by different methods and the corresponding HR images for different bands on the test dataset. The results are illustrated in Figure 10. DTGAN achieved the smallest MSE in the blue and red bands, while SRGAN had the smallest MSE in the green and near-infrared bands. DTGAN exhibited the smallest total band MSE.

Due to differences in the radiometric values, the L8 and S2 images exhibit variations in the grayscale distribution. The histograms reflect the distribution of the grayscale values in the images. We calculated and compared the grayscale histograms of the LR, SR, and HR data for different bands in the Yongcheng City to assess our method’s capability to learn the grayscale distribution of the HR images. The results in Figure 11 demonstrate that, compared to the LR images, the grayscale distribution of the SR images is much closer to that of the HR images. In the blue and green bands, the grayscale distribution of the SR images is almost identical to that of the HR images. However, in the red and near-infrared (NIR) bands, the overall grayscale distribution of the SR images closely resembles that of the HR images, although there are significant differences in the peak values. We posit that this outcome arises from the wavelength disparities in the L8 and S2 imagery across these four spectral bands, coupled with the growth stages of the predominant winter wheat crops within the study area. Overall, DTGAN effectively learns the mapping relationship between the LR and HR image grayscale distributions, generating SR images that are visually, spectrally, and in terms of grayscale distribution, closer to the HR images.

4.2. Ablation Experimental of the SR Model

In this study, we conducted a series of ablation experiments to explore the impact of various components of the network on the reconstruction results of the SR images.

Table 3. The initial setup of the ablation experiment.

Patch Size	CNN Backbone	P.E. in En.	P.E. in De.	En.Layer	De.Layer	En.ly.Depth	De.ly.Depth
$4\times 4$	RCAG	True	True	4	3	1	1

presents the initial generator configuration used in the ablation experiments.

The ablation about patch size: Prior to feeding the shallow features extracted by the RCAG module into the encoder layer, the feature maps need to be divided into patches of size $P\times P$.

Table 4. The result of the ablation for the patch size.

Patch Size	PSNR	SSIM [74]	SAM	LPIPS [75]
$4\times 4$	31.7517	0.8347	0.0504	0.0846
$8\times 8$	31.4505	0.8191	0.0526	0.0732
$10\times 10$	31.0061	0.7961	0.0555	0.0782

demonstrates the influence of different patch sizes on the network performance in the SR task. When $P=4$, the network achieved the best results in terms of the PSNR, SSIM, and SAM metrics, while the perceptual metric LPIPS reached its optimum value at $P=8$. The process of partitioning the image into patches can be approximated as a down-sampling process. Larger patches imply a larger receptive field, aiding the model in capturing contextual information better. However, larger patches also mean more loss of local information. Smaller patches help the network handle local detailed features better but lead to the loss of global information, hindering the network in reconstructing the global structure of the SR images. The optimal value of P varies for different tasks and image resolutions. In our experiments, we ultimately set the value of P to 4.

The ablation about P.E.: Positional encoding is utilized for the network to learn the position information of the patches.

Table 5. The result of the ablation for the position encoding.

P.E. in Encoder	P.E. in Decoder	PSNR	SSIM [74]	SAM	LPIPS [75]
✓	✓	31.7517	0.8347	0.0504	0.0846
✗	✗	31.5830	0.8260	0.0515	0.0691
✓	✗	31.6678	0.8291	0.0515	0.0646
✗	✓	31.7199	0.8320	0.0507	0.0789

presents the results of ablation experiments on positional encoding. When both the encoder layer and decoder layer have positional encoding, the model achieves the best PSNR, SSIM, and SAM metrics, but obtains the lowest LPIPS value. This suggests that positional encoding is crucial for the SR task, improving the performance of the SR images according to traditional image evaluation metrics, but potentially reducing the perceptual quality of the SR images. Based on our experiments, we ultimately decided to retain positional encoding in both the encoder layer and the decoder layer.

The ablation about DW-MHSA: The extraction and fusion of the multiscale features represent the core of the proposed method.

Table 6. The result of the ablation for the dilated rate of DW-MHSA in encoder.

En.1	En.2	En.3	En.4	PSNR	SSIM [74]	SAM	LPIPS [75]
1	2	3	4	31.7517	0.8347	0.0504	0.0846
False	2	3	4	31.8997	0.8394	0.0499	0.0694
False	False	3	4	31.8213	0.8346	0.0505	0.0694
False	False	False	4	31.7480	0.8328	0.0505	0.0701
False	False	False	False	31.7813	0.8335	0.0504	0.0675

illustrates the impact of different scale attention mechanisms on the network performance. The numbers in the table represent the dilation rates used in calculating DW-MHSA in the corresponding encoder layer, and False indicates the use of MHSA. DW-MHSA calculates the local attention, focusing on the multiscale local details by adjusting the dilation rate r, but lacks the ability to capture the global contextual information. Therefore, when all the encoder layers use DW-MHSA, the network performance does not reach its optimum due to the absence of global information. Similarly, when all the encoder layers use MHSA, the network fails to achieve optimal performance due to the lack of multiscale local information. When the attention configuration is set to $[\mathrm{False},2,3,4]$, the model achieves the best performance across all the metrics. At this configuration, the encoder layers strike a balance between learning global information and multiscale local information.

The ablation about depths of En.Ly. and De.Ly.: The number of blocks in the encoder layer and the decoder layer affects the final quality of the generated SR images. A series of ablation experiments were conducted to investigate the depths of En.Ly. and De.Ly., and the results are shown in

Table 7. The result of the ablation for the encoder layer and the decoder layer depths.

Encoder	Decoder	PSNR	SSIM [74]	SAM	LPIPS [75]
1	1	31.8997	0.8394	0.0499	0.0694
1	2	31.6611	0.8302	0.0511	0.0771
1	4	32.0343	0.8232	0.0491	0.0675
1	8	31.5109	0.8252	0.0521	0.0811
2	1	31.5431	0.8259	0.0521	0.0794
4	1	31.7246	0.8316	0.0507	0.0746
8	1	31.6153	0.8272	0.0516	0.0739

. It can be observed that when the depth of the encoder layers is set to 1 and the depth of the decoder layers is set to 4, the model achieves the best accuracy in terms of PSNR, SAM, and LPIPS. This indicates that as the decoder layers become deeper, the model’s performance initially improves and then reaches a certain degree of overfitting.

4.3. Evaluation of SR Images for LUCC

The proposed DTGAN in this study achieved the best overall performance in the SR task. We applied it to the L8 image SR in the Yongcheng City area, and the resulting SR images are shown in Figure 8. We used the method proposed in [61] for the second-stage experiments. Land use classification was performed on the original LR, SR, and HR images. The macro average classification accuracy for images of different resolution is shown in

Table 8. The macro average score of the classification map for different spatial resolution.

	Precision	Recall	F1-Score
LR Classification Map	0.831	0.794	0.809
SR Classification Map	0.961	0.972	0.966
HR Classification Map	0.984	0.990	0.987

. It can be observed that compared to the LR classification map, the SR classification map achieved better performance in the various metrics.

Figure 12 and Figure 13 display the confusion matrix and the class accuracy of the classification results, respectively. It can be observed that for the vegetation category, the classification results at different spatial resolutions are very similar, as Yongcheng City has large areas of plain farmland. Vegetation has simple spatial characteristics and distinct spectral features compared to other land covers, making the classification results less dependent on spatial information. For land covers, such as bare land and building areas, with similar spectral but complex spatial features, higher spatial resolution provides richer spatial information, which helps improve the accuracy of classification.

Figure 14 provides a visual comparison of the classification results at various spatial resolutions. It can be observed that the LR classification map exhibits poor differentiation for buildings and bare land, with the finer linear roads being nearly indistinguishable in the LR image. For the regular rectangular-shaped pondwater bodies and mulching film land, the SR classification map has clearer edges due to the increased spatial resolution. However, it can be seen that the finer water bodies are also not well-classified in the SR map. This indicates that the SR images can furnish more detailed spatial information, thereby enhancing the accuracy of the LUCC. However, they are unable to retrieve all the information available in the LR images. As shown in Table 8 and Figure 13, compared to the LR classification map, the SR classification results show improvements in both overall accuracy and class accuracy. This enhancement is particularly notable for land categories, such as barren land and buildings, which possess similar spectral characteristics. The higher spatial resolution provided by the SR imaging captures more spatial details, aiding in the differentiation of land features with similar spectral properties but distinct spatial characteristics.

5. Discussion

In this study, we constructed a GAN based on a CNN and the Transformer architecture for Landsat-8 multispectral image super-resolution in real-world scenarios. The results indicate that our method outperforms other techniques in visual quality, as demonstrated by metrics such as LPIPS and SAM. The generated super-resolution data contribute significantly to enhancing the accuracy of the LUCC tasks. Compared to CNN, the GAN networks possess a more realistic image-generation capability. The GAN networks are designed to learn the distribution of real data, enabling them to generate high-resolution images that closely resemble real data. Moreover, the adversarial training approach of the GAN networks aids in recovering the edge and detail information from low-resolution images, resulting in SR images that perceptually resemble real HR images. On the other hand, the CNN-based super-resolution images typically exhibit higher PSNR values but may lack detailed features and sharp boundaries, providing a smoother effect on the fine details and the object edges. For remote sensing imagery, improved visual perception and the recovery of more detailed object boundaries are crucial. Enhanced visual perception leads to better visual interpretation and understanding of the surface information, aiding in a more intuitive analysis. Additionally, recovering more object boundaries and fine details provides richer surface information for various remote sensing applications, including LUCC, resulting in more refined land cover classification maps.

6. Conclusions

This study proposed a generative adversarial network that combines convolutional neural networks with Transformers to improve the spatial resolution of multispectral remote sensing images and to enhance the accuracy of LUCC maps. The research was conducted in two stages: image SR and LUCC. In the SR stage, we introduced Transformers into the generator to compensate for the local nature of CNNs and to enhance the model’s ability to learn local and global features. Additionally, we incorporated multi-scale information and dilated convolutions to improve the computational efficiency. In the LUCC stage, a pretrained model was used to generate HR images from L8 data. We demonstrated that using these SR images significantly improves the accuracy of LUCC maps. This study presents a novel approach for remote sensing image super-resolution, showing promising practical applications.