Single-Image Super Resolution of Remote Sensing Images with Real-World Degradation Modeling

Abstract

Limited resolution is one of the most important factors hindering the application of remote sensing images (RSIs). Single-image super resolution (SISR) is a technique to improve the spatial resolution of digital images and has attracted the attention of many researchers. In recent years, with the advancement of deep learning (DL) frameworks, many DL-based SISR models have been proposed and achieved state-of-the-art performance; however, most SISR models for RSIs use the bicubic downsampler to construct low-resolution (LR) and high-resolution (HR) training pairs. Considering that the quality of the actual RSIs depends on a variety of factors, such as illumination, atmosphere, imaging sensor responses, and signal processing, training on “ideal” datasets results in a dramatic drop in model performance on real RSIs. To address this issue, we propose to build a more realistic training dataset by modeling the degradation with blur kernels and imaging noises. We also design a novel residual balanced attention network (RBAN) as a generator to estimate super-resolution results from the LR inputs. To encourage RBAN to generate more realistic textures, we apply a UNet-shape discriminator for adversarial training. Both referenced evaluations on synthetic data and non-referenced evaluations on actual images were carried out. Experimental results validate the effectiveness of the proposed framework, and our model exhibits state-of-the-art performance in quantitative evaluation and visual quality. We believe that the proposed framework can facilitate super-resolution techniques from research to practical applications in RSIs processing.

1. Introduction

Remote sensing images (RSIs) captured using satellites or aircraft are essential in many applications such as environmental surveys [1,2], disaster monitoring [3,4], and resource censuses [5,6]. Spatial resolution is one of the most important metrics of RSIs that advanced satellites or aircraft researchers have been striving to improve; however, the high cost of high-performance imaging systems and harsh imaging conditions in space limit the resolution of RSIs to sub-meter levels. Single image super resolution (SISR) is a technique for reconstructing high-resolution (HR) images from low-resolution (LR) observations [7,8,9]. It is widely employed in a range of scenarios such as mobile devices, medical imaging, and remote sensing. It has been a research hotspot in the field of low-level image processing for decades.

SISR is an ill-posed problem since high-frequency information is lost during imaging. A single LR image may correspond to multiple HR solutions [7,10,11]. To address this problem, many practical frameworks have been proposed, which can be grouped into three categories: (i) interpolation-based methods, (ii) reconstruction-based methods, and (iii) learning-based methods.

Interpolation-based methods estimate missing pixel values with different interpolation functions such as bilinear, bicubic, and spline [12]. These methods are commonly used to scale images due to their high computational efficiency but often lead to over-smoothed results. Reconstruction-based methods attempt to investigate image degradation processes, such as out-of-focus, motion, aliasing, and noise, and build models to implement the inverse process. Reconstruction-based methods can recover sharp details with prior information of the degradation process, but suffer from heavy computation and reconstruction artifacts. Learning-based approaches learn the mapping between LR and HR images in the collected dataset and try to generalize to images outside the dataset. In recent years, convolutional neural networks (CNN) have received increasing attention due to their excellent nonlinear representation and feature extraction capabilities. Many CNN-based methods are proposed to solve the SISR problem and achieve state-of-the-art performance. Lu [13] proposed a multi-scale residual neural network (MRNN), which adopts the multi-scale properties to reconstruct high-frequency information accurately for the super resolution (SR) of RSIs. Xiong [14] modified the loss function and the structure of the super-resolution generative adversarial network (SRGAN) for SR of RSIs.

Deep learning (DL) methods have achieved promising results on SR of RSIs, but there are still some problems. Current datasets for the SR of RSIs, such as AID [15], UCMERCED [16], and RSIs-CB256 [17], are not designed for real-world situations. Since the HR observations for RSIs datasets are difficult to obtain, most RSIs datasets actually contain only LR images. To build input–target pairs to train the deep neural network, most studies [18,19,20] employ a simple strategy of generating lower resolution (abbreviated as LR’) images from LR images with bicubic interpolation. Networks trained on LR’–LR pairs can achieve satisfactory results on the test dataset since the test images are also LR’ images instead of the actual RSIs; however, the LR’–LR mapping is quite different from the LR–HR mapping. Due to the imperfect illumination, atmospheric propagation, lens imaging, sensor quantification, etc., the RSIs suffers from image degradation such as blur and noise. In addition, the image degradation can be mitigated through downsampling; therefore, the learned mapping comes from clean LR’ images to contaminated LR images instead of the expected mapping from contaminated LR images to clean HR images. This is the main reason why many previous studies can achieve excellent experimental results, but face performance degradation in practical scenarios.

To address this issue, inspired by recent advanced research [21,22,23], we apply a framework to construct realistic datasets by modeling real-world degradation in remote sensing images. As the realistic SISR task is more complicated than the ideal SISR task, we also propose a novel neural network with a balanced attention mechanism to achieve better performance. More specifically, we classify the degradation of RSIs into two categories, blur and noise. The blur simulates the diffraction limit of a lens, the disturbance of atmosphere, and the relative movement of image platforms to the Earth. The noise includes different kinds of imaging noise and JPEG compression noise. Blur kernels and noise patterns are collected and randomly used to synthesize realistic LR images and construct LR–HR training pairs. By training on realistic datasets, the performance of the model under real-world conditions can be significantly improved. As for the network, we propose a residual balanced attention network (RBAN) that integrates the residual in the residual structure and a balanced attention mechanism. A modified UNet model is adopted as a pixel-wise discriminator to achieve more realistic SISR results. Thorough experiments were conducted and the results demonstrate the effectiveness of the proposed framework and neural network.

The contributions of this study are highlighted as follows: (i) We propose a framework integrating blur kernel estimation and noise pattern extraction to model the degradation of remote sensing images; (ii) we construct a realistic RSIs dataset based on the open source Aerial Image Dataset (AID) for training and testing, which greatly enhances the performance of models on real RSIs; (iii) we propose a novel SISR model termed residual balanced attention network (RBAN) and apply a modified UNet model as the discriminator to improve perceptual performance. The proposed model achieves state-of-the-art performance on the realistic RSIs dataset.

The rest of this paper is organized as follows. In Section 2, related works about CNN-based SISR methods and RSIs SISR methods are summarized. In Section 3, the framework and model are proposed. Section 4 details the experimental results of the proposed method and compares it with the state-of-the-art methods. Conclusions are drawn in Section 5.

2. Releated Work

2.1. CNN-Based SISR Methods

Since the advent of CNN, it has outperformed traditional methods in a growing number of computer vision tasks, including SISR. To solve the SISR problem with CNN, Dong et al. [24] proposed a shallow super-resolution CNN (SRCNN), which can learn the mapping from interpolated LR images to HR images. SRCNN outperforms state-of-the-art traditional SISR methods such as K-SVD [25] and ANR [26], and attracts more researchers to study the CNN-based SISR methods. Later, Caballero et al. [27] proposed to use the pixel-shuffle layer to improve the spatial resolution of feature maps so that computations could be performed in LR space to reduce the computational effort. SRCNN contains only three convolutional layers, which limits its performance. To deepen the network and obtain stronger learning ability, Kim et al. [28] applied the residual learning strategy and proposed a 20-layer very deep super resolution (VDSR) model. Due to its larger receptive field, the reconstruction performance of VDSR is much better than that of SRCNN, and the residual learning strategy has become the standard configuration for CNN-based SISR since then. Inspired by VDSR, Ledig et al. [29] proposed SRResNet, which employs the residual learning strategy both globally and locally. Ledig also proposed the generative adversarial training framework SRGAN, which enables photo-realistic SISR. Since SRGAN, GAN-based methods have become the dominant framework for better perceptual results. By removing the batch normalization layer in SRResNet, Lim et al. [30] proposed an optimized residual block module and an enhanced deep super-resolution network (EDSR). Inspired by the channel attention mechanisms commonly used in high-level computer vision problems such as classification and detection, Zhang et al. [31] proposed a deep residual channel attention network (RCAN). Subsequently, Liang et al. [23] successfully adopted the shifted windows transformer module in SISR and proposed a state-of-the-art model SwinIR. Anwar [32] proposed a densely residual Laplacian network (DRLN), which employs cascading residual structure to allow low-frequency information flow to focus on learning high and mid-level features.

Nowadays, SISR has become one of the most important tasks in many computer vision challenges such as NTIRE [9] and AIM [8]. With the practical application of super-resolution technology, researchers began to realize the limitations of previous studies. The common practice for building super-resolution training datasets is to collect a large number of HR images and downsample them using bicubic interpolation; however, in real-world imaging, bicubic interpolation is insufficient to describe the degradation process; therefore, the focus of SISR research in recent years has gradually shifted to real-world SISR that takes into account degradations such as blur and noise. One of the representative strategies is to directly collect and align LR–HR pairs. Zhang et al. [33] captured images with a zoomable lens in different magnifications and carefully aligned the LR and HR images spatially. Trained on real-world LR–HR pairs, Zhang’s network achieved much better results on real-world images than networks trained on synthetic datasets. An alternative strategy is to model the degradation process and try to synthesize more realistic LR observations from HR images. Bell-Kligler et al. [21] proposed to use blur kernels to model the degradation and GANs to estimate the kernels. Another strategy is to train the network with unpaired real-world LR and HR images. Yuan et al. [34] proposed a cycle-in-cycle GAN, which treats the SISR as a domain translation task from LR space to HR space.

2.2. SISR of RSIs

Due to the huge demand for high spatial resolution in many remote sensing tasks such as scene classification, object detection, and instance segmentation, SISR has become a research hotspot in RSIs processing. Nguyen and Milanfar [35] first decomposed LR images with discrete wavelet transform (DWT), then used an interpolation algorithm to upsample the wavelet coefficients, and finally inverse transformed the coefficients to generate HR images. Based on maximum posterior probability, Li et al. [36] proposed a generalized hidden Markov tree model (MAP-uHMT) that uses a hybrid Gaussian model to represent the wavelet coefficients of an image and a hidden Markov tree to capture the dependencies between multiscale wavelet coefficients. Pan et al. [37] proposed a dictionary learning method for SISR of RSIs that combines compressed sensing and structural self-similarity.

In recent years, CNN-based methods have become the mainstream of SISR for RSIs due to their outstanding performance and extensive applicability. Lei et al. [38] proposed a SISR model named local–global combined networks (LGCNet) for RSIs based on CNN. LGCNet applied a novel ‘multifork’ structure to learn multilevel representations of RSIs including local details and global environmental priors. Jiang et al. [39] proposed a deep distillation recursive network (DDRN) effective for video satellite image SR. DDRN uses dense connections to create more linked nodes and a distillation compensation mechanism to compensate for high-frequency information to reconstruct more accurate SR images. Ma et al. [40] proposed a method that incorporates DWT to decompose LR images and recursive ResNet to predict high-frequency components; then, the reconstructed HR image can be obtained via inverse DWT. Zhang et al. [18] proposed a multiscale attention network (MSAN) to extract the multilevel features of RSIs. A scene-adaptive super-resolution strategy was also applied to more accurately describe the structural features of different scenes in MSAN. Guo et al. [41] designed a novel dense generative adversarial network (NDSRGAN) that integrates a multilevel dense network and a matrix mean discriminator for aerial imagery SR reconstruction.

3. Methodology

In this section, we introduce the proposed framework as shown in Figure 1. The framework is mainly divided into three stages. The first is to estimate blur kernels and noise patches from the real RSIs dataset, which is detailed in Section 3.2. The second is to generate a realistic synthetic training dataset with the real RSIs dataset and collected blur kernels and noise patches, which is detailed in Section 3.1. The third is to train a novel network based on the synthetic dataset, which is detailed in Section 3.3.

3.1. Realistic Degradation for SISR of RSIs

Previous SISR methods of RSIs model the imaging process as follows

$${\mathbf{I}}_{\mathrm{LR}}={\mathbf{I}}_{\mathrm{HR}}{\downarrow }_s,$$

(1)

where ${\mathbf{I}}_{\mathrm{LR}}$ and ${\mathbf{I}}_{\mathrm{HR}}$ denote the LR and HR remote sensing images, respectively, and s denotes the downsample scale factor. This makes it easy to generate LR images from real RSIs and train a network for reverse mapping; however, the model in Equation (1) is too simplistic and distorts the actual degradation process of RSIs. RSIs in real-world conditions contain blur and noise due to the imperfect illumination, atmospheric propagation, lens imaging, sensor quantification, $etc.$ In addition, the downsampling process can suppress blur and high-frequency noise; therefore, the network based on Equation (1) will learn a mapping from clean LR images to realistic HR images, which is inconsistent with the desired mapping from realistic LR images to clean HR images. This is why previous methods suffer from performance degradation when dealing with real-world RSIs.

To settle the aforementioned problem, we proposed to construct a realistic training dataset for SISR of RSIs using the following degradation model

$${\mathbf{I}}_{\mathrm{HR}}={\mathbf{I}}_{\mathrm{real}}{\downarrow }_c,$$

(2)

$${\mathbf{I}}_{\mathrm{LR}}=\left({\mathbf{I}}_{\mathrm{HR}}\otimes \mathbf{k}\right){\downarrow }_s+\mathbf{n},$$

(3)

where $\mathbf{k}$ and $\mathbf{n}$ represent the blur kernel and noise, respectively. ${\mathbf{I}}_{\mathrm{real}}$ indicates the real RSIs in the dataset, ${\mathbf{I}}_{\mathrm{HR}}$ indicates the clean HR image generated by downsampling the real RSIs and ${\mathbf{I}}_{\mathrm{LR}}$ indicates the synthetic realistic LR image. c and s are the downsampling scales of HR and LR images, respectively. The blur kernel and noise are collected from the dataset of real RSIs, so a more realistic dataset can be constructed by randomly combining the blur kernels, noise patches, and real RSIs. With the realistic dataset, a more robust CNN can be trained. The whole pipeline of constructing paired LR–HR images with real RSIs is shown in Algorithm 1.

Algorithm 1 Realistic data pairs generation

Input: Real RSIs set $\mathcal{X}$ Output: Realistic image pairs $\left\{{\mathbf{I}}_{\mathrm{LR}},{\mathbf{I}}_{\mathrm{HR}}\right\}$   Initialize kernel dataset $\mathcal{K}=\varnothing$ Initialize noise dataset $\mathcal{N}=\varnothing$ for all ${\mathbf{I}}_{\mathrm{real}}$ that ${\mathbf{I}}_{\mathrm{real}}\in \mathcal{X}$ do      Estimate $\mathbf{k}$ from ${\mathbf{I}}_{\mathrm{real}}$ by solving Equation (4) and add $\mathbf{k}$ to $\mathcal{K}$      Crop $\mathbf{n}$ from ${\mathbf{I}}_{\mathrm{real}}$      if $\mathbf{n}$ meets Equation (5) then         Add $\mathbf{n}$ to $\mathcal{N}$      end if end for for all ${\mathbf{I}}_{\mathrm{real}}$ that ${\mathbf{I}}_{\mathrm{real}}\in \mathcal{X}$ do      Generate ${\mathbf{I}}_{\mathrm{HR}}$ using Equation (2)      Randomly select ${\mathbf{k}}_i\in \mathcal{K}$, ${\mathbf{n}}_j\in \mathcal{N}$      Generate ${\mathbf{I}}_{\mathrm{LR}}$ using Equation (3) end for return$\left\{{\mathbf{I}}_{\mathrm{LR}},{\mathbf{I}}_{\mathrm{HR}}\right\}$

(1) Blur: Blur is a common degradation during the imaging of RSIs. We model the blur of RSI as a convolution with a blur kernel (filter). In earlier research, the blur kernel of a LR image was usually assumed to be the point spread function (PSF) of the camera; however, blur can be caused by a variety of reasons such as limited lens aperture, defocus, relative motions between image platforms, and the Earth. Thus, the blur kernel of a known imaging system may vary under different circumstances. It has been proved by Michaeli [42] that the correct SR blur kernels can be recovered directly from low-resolution images.

(2) Noise: Noise is another common degradation that can be divided into two main categories, imaging noise and processing artifacts. Imaging noises are caused by undesired responses of sensor during image capture. Imaging noises caused by different factors satisfy different statistical distributions, such as Gaussian distribution and Poisson distribution. Due to the limited download bandwidth of satellites or airplanes, RSIs is usually compressed with JPEG algorithm; however, the JPEG compression results in loss of high-frequency information and introduces undesired artifacts.

3.2. Estimation of Blur Kernel and Noise Patches

In this section, we detail the methods for estimating blur kernels and noise patches.

(1) Blur kernel estimation: There are mainly two strategies to build the blur kernel dataset. One is to model kernels with isotropic or anisotropic Gaussian distributions and generate kernels with random parameters within a preset range. Another strategy is to estimate and collect kernels from real images. We adopt the second strategy and apply a modified version of KernelGAN [21] presented by Ji et al. [22]. The authors hypothesized that the correct SR kernel maximizes the recurrence of patches between LR and HR images. As shown in Figure 2, KernelGAN is composed of a generator G and a discriminator D. G is a five-layer full convolutional neural network and is trained to produce downsampled patches of the input image. D is trained to distinguish downsampled image patches from original image patches. Then, the kernel can be explicitly extracted by convolving all the layers of G sequentially with stride 1.

The KernelGAN is designed to solve the following problem

$$\begin{array}{c}arg\underset{\mathbf{k}}{min}{∥\left({\mathbf{I}}_{\mathrm{real}}\otimes \mathbf{k}\right){\downarrow }_s-{\mathbf{I}}_{\mathrm{real}}{\downarrow }_s∥}_1+\left|1-\sum {\mathbf{k}}_{i,j}\right|\hfill \\ +\left|\sum {\mathbf{k}}_{i,j}·{\mathbf{m}}_{i,j}\right|+\left|1-\mathbf{D}\left(\left({\mathbf{I}}_{\mathrm{real}}\otimes \mathbf{k}\right){\downarrow }_s\right)\right|\hfill \end{array}.$$

(4)

The first term in the above equation encourages the downsampled image to preserve important low-frequency information of the source image. The second term constrains $\mathbf{k}$ to sum to 1 and the third term penalizes non-zero values near the boundaries with a constant mask. At last, the discriminator $\mathbf{D}\left(·\right)$ ensures the consistency of source domain.

(2) Noise estimation: Since the noise in the image is a mixture of a series of noise sources, such as thermal noise, read noise, signal disturbance, and JPEG compression. It is difficult to accurately calibrate the type and weight of each noise source; however, we find that image variances in flat scene regions, such as water surface or bare land, are mainly caused by noise. Based on this assumption, we can decouple noise and content by using a simple but effective rule:

$$\sigma \left({\mathbf{n}}_i\right)<v,$$

(5)

where $\sigma \left(·\right)$ denotes to calculate variance, v is the threshold of variance, and ${\mathbf{n}}_i$ is the image patch after subtracting the mean. The process of collecting noise patches is shown in Figure 3.

3.3. Residual Balanced Attention Network (RBAN)

(1) Network architectures: Let us denote ${\mathbf{I}}_{\mathrm{LR}}\in {\mathcal{R}}^{C\times H\times W}$ and ${\mathbf{I}}_{\mathrm{HR}}\in {\mathcal{R}}^{C\times sH\times sW}$ as LR and HR image, respectively, where C, H, and W are the size of channel, height, and width respectively; s is the scale factor. The goal of the network is to obtain the SR estimation ${\mathbf{I}}_{\mathrm{SR}}\in {\mathcal{R}}^{C\times sH\times sW}$ as close as the ground truth ${\mathbf{I}}_{\mathrm{HR}}$ for a given LR input ${\mathbf{I}}_{\mathrm{LR}}$. As shown in Figure 4, the proposed RBAN model mainly consists of three parts: shallow feature extraction, deep feature extraction, and reconstruction.

At the beginning of the network, there is a convolutional layer for extracting shallow features from the LR input image. The mapping function of this part can be written as

$${\mathbf{F}}_{\mathrm{SF}}=f_{\mathrm{conv}}^{k3s1}\left({\mathbf{I}}_{\mathrm{LR}}\right),$$

(6)

where $f_{\mathrm{conv}}^{k3s1}$ denotes the convolutional function with kernel size 3 and stride 1.

The deep feature extraction part is located after the shallow feature extraction part. It consists of multiple residual balanced attention groups (RBAG), one convolutional layer, and one balanced attention module (BAM). The deep feature extraction part has a skip connection to provide a shortcut from input to output, which is also known as the global residual strategy. In each RBAG, there are a group of basic residual blocks (RB), a convolutional layer and a BAM. There is also a skip connection from the input to the output of RBAG. The mapping function of the deep feature extraction part can be written as

$${\mathbf{F}}_{\mathrm{DF}}=M_{\mathrm{BAM}}\left(f_{\mathrm{conv}}^{k3s1}\left(M_{\mathrm{RBAG}}^n\left(\cdots M_{\mathrm{RBAG}}^1\left({\mathbf{F}}_{\mathrm{SF}}\right)\right)\right)\right)+{\mathbf{F}}_{\mathrm{SF}},$$

(7)

$$M_{\mathrm{RBAG}}\left(\mathbf{F}\right)=M_{\mathrm{BAM}}\left(f_{\mathrm{conv}}^{k3s1}\left(M_{\mathrm{RB}}^m\left(\cdots M_{\mathrm{RB}}^1\left(\mathbf{F}\right)\right)\right)\right)+\mathbf{F},$$

(8)

where $M_{\mathrm{RBAG}}$ is the mapping function of RBAG, $M_{\mathrm{BAM}}$ denotes the mapping function of BAM, and $f_{\mathrm{ReLU}}$ denotes the mapping function of ReLU activation layer. RB is the basic unit of SISR network, which usually consists of convolutional layers and activation layers, such as the batch normalization (BN) and the rectified linear unit (ReLU). In this work, we apply the RB structure that used in EDSR [30], which consists of a convolutional layer, a rectified linear unit (ReLU) and another convolutional layer in sequence. There is also a skip connection from the input to the output of RB. The mapping function of RB is formulated as

$$M_{\mathrm{RB}}\left(\mathbf{F}\right)=f_{\mathrm{conv}}^{k3s1}\left(f_{\mathrm{Re}\mathrm{LU}}\left(f_{\mathrm{conv}}^{k3s1}\left(\mathbf{F}\right)\right)\right).$$

(9)

In the end, an upsampler and a convolutional layer make up the reconstruction part. The upsampler is used to upscale the feature maps from LR space to HR space. The upsamplers are different for different scale factors. As shown in Figure 4, two pixel shuffle layers and two convolutional layers are interleaved in a ×4 upsampler. A convolutional layer located after the upsampler converts the feature map into the SR image output. The mapping function of reconstruction part can be written as

$${\mathbf{I}}_{\mathrm{SR}}=f_{\mathrm{conv}}^{k3s1}\left(M_{\mathrm{UP}}\left({\mathbf{F}}_{\mathrm{DF}}\right)\right),$$

(10)

$$M_{\mathrm{UP}}\left(\mathbf{F}\right)=f_{\mathrm{ps}}\left(f_{\mathrm{conv}}^{k3s1}\left(f_{\mathrm{ps}}\left(f_{\mathrm{conv}}^{k3s1}\left(\mathbf{F}\right)\right)\right)\right),$$

(11)

where $M_{\mathrm{UP}}$ is the mapping function of the upsampler, $f_{\mathrm{ps}}$ denotes the mapping function of the $\times 2$ pixel shuffle layer.

(2) Balanced attention module (BAM): Attention mechanism is an important strategy in SISR model, which emphasizes the different importance between channels or spatial locations by assigning weights to feature maps. To improve the performance of the model, we employ the balanced attention module (BAM) [43]. BAM models interdependence of different channels and spatial locations at the same time. The mapping functions of BAM can be written as

$$M_{\mathrm{BAM}}\left(\mathbf{F}\right)=\mathbf{F}·f_{\mathrm{sigmoid}}\left(f_{\mathrm{conv}}^{k7s1}\left(f_{\mathrm{MaxPool}}\left(\mathbf{F}\right)\right)\right)·f_{\mathrm{sigmoid}}\left(f_{\mathrm{conv}}^{k1s1}\left(f_{\mathrm{ReLU}}\left(f_{\mathrm{conv}}^{k1s1}\left(f_{\mathrm{AvePool}}\left(\mathbf{F}\right)\right)\right)\right)\right),$$

(12)

where $f_{\mathrm{MaxPool}}$ denotes the map function of maximum pooling layer, $f_{\mathrm{AvePool}}$ denotes the map function of average pooling layer, $f_{\mathrm{conv}}^{k7s1}$ denotes the convolutional function with kernel size 7 and stride 1, $f_{\mathrm{conv}}^{k1s1}$ is the convolutional function with kernel size 1 and stride 1, $f_{\mathrm{sigmoid}}$ is the sigmoid map function.

As shown in Figure 4, BAM contains two paths for channel attention and spatial attention, respectively. In the channel attention path, the input $C\times H\times W$ feature map is turned into a $C\times 1\times 1$ channel-wise feature map by the average pooling layer. Then, two $1\times 1$ convolutional layers and a sandwiched ReLU layer implement the ‘squeeze-and-excitation’ operation. Afterwards, the sigmoid layer turns the feature map into a $C\times 1\times 1$ channel-wise output weight vector. In the spatial attention path, the input $C\times H\times W$ feature map is turned into a $1\times H\times W$ spatial-wise feature map by the max pooling layer. Then, a $7\times 7$ convolutional layer and a sigmoid layer turn the feature map into a $1\times H\times W$ spatial-wise output weight vector. Finally, two output weight vectors are broadcast-multiplied with the input feature map to obtain the attention result.

(3) Discriminator Model: Adversarial learning is a strategy commonly used in recent models for realistic SISR results. These models are also known as generative adversarial networks (GAN). In standard GAN-based SR models, the generator model G is trained to produce HR estimations from input images such that the discriminator model D cannot distinguish them from ground-truth HR images.

In this article, we use the RBAN model as generator and employ a modified UNet discriminator as shown in Figure 4. The convolutional layer in the discriminator with a kernel size of 4 and a stride of 2 extracts features while downscaling the spatial resolution. Spectral normalization layers and leaky ReLU layers are used to ensure the training stability. ↑ denotes the upsample operation with bilinear interpolation. Different from the standard VGG-style discriminator that only estimates the realness of the whole image, the UNet-based discriminator estimates the realness of each pixel, which produces accurate gradient feedback for local textures.

We also enhance the discriminator with relativistic GAN (RGAN) [44] framework. Unlike D in standard GANs, which estimates the probability that an input image (${\mathbf{I}}_{\mathrm{HR}}$ or ${\mathbf{I}}_{\mathrm{SR}}$) is ground truth or estimated, D in RGAN predicts the probability that the ground-truth image ${\mathbf{I}}_{\mathrm{HR}}$ is relatively more realistic than the estimated one ${\mathbf{I}}_{\mathrm{SR}}$. Assuming the discriminator output of standard GAN as

$$D\left(\mathbf{I}\right)=f_{\mathrm{sigmoid}}\left(C\left(\mathbf{I}\right)\right),$$

(13)

where $f_{\mathrm{sigmoid}}$ is the sigmoid mapping function, $C\left(\mathbf{I}\right)$ is the non-transformed discriminator output. The discriminator output of RGAN is formulated as

$$D_{\mathrm{RGAN}}\left({\mathbf{I}}_{\mathrm{HR}},{\mathbf{I}}_{\mathrm{SR}}\right)=f_{\mathrm{sigmod}}\left(C\left({\mathbf{I}}_{\mathrm{HR}}\right)-E\left[C\left({\mathbf{I}}_{\mathrm{SR}}\right)\right]\right),$$

(14)

where $E\left[·\right]$ denotes taking average for all data in the mini-batch.

3.4. Loss Function

The objective of the SISR can be formulated as

$$\theta =\underset{\theta }{argmin}\frac{1}{N}\sum _{n=1}^N{L}_G\left(G_{\theta }\left({\mathbf{I}}_{\mathrm{SR}}^n\right),{\mathbf{I}}_{\mathrm{HR}}^n\right),$$

(15)

where G is the generator model parameterized by $\theta$, $L_G$ is the loss function of the generator, N is the number of training images, $n=1,2,\cdots ,N$.

Considering the realistic degradation process, more effective constraints should be imposed on the network during training. Thus, we combine pixel loss, perceptual loss, and adversarial loss with weighted average. The synthetic loss is formulated as

$$L_G=\alpha L_{\mathrm{pixel}}+\beta L_{\mathrm{perceptual}}+\gamma L_{\mathrm{adversarial}},$$

(16)

where $\alpha$, $\beta$ and $\gamma$ are the weight coefficients to balance three loss terms.

The pixel loss is to measure the pixel-wise difference between the ground-truth HR image and estimated SR image. It is the basic component of the loss function and plays an irreplaceable role in ensuring network convergence. We choose to use the $l_1$ function as the pixel loss, denoted as

$$L_{\mathrm{pixel}}=\frac{1}{CHW}\sum _{c,h,w}\left|G\left({\mathbf{I}}_{\mathrm{LR}}\right)-{\mathbf{I}}_{\mathrm{HR}}\right|,$$

(17)

where C, H, and W are the channel number, height, and width of the evaluated images, respectively.

The perceptual loss is to measure the differences between two images using features extracted with a pretrained 19-layer VGG network. As a commonly used image classification network, the pretrained VGG can extract high-level semantic information from input images, which helps to improve the quality of network output images. In this work, we use the VGG network truncated at the Conv5-4 layer to extract features and $l_1$ function to calculate the difference between feature maps

$$L_{\mathrm{perceptual}}=\frac{1}{CHW}\sum _{c,h,w}\left|f_{\mathrm{VGG}54}\left(G\left({\mathbf{I}}_{\mathrm{LR}}\right)\right)-f_{\mathrm{VGG}54}\left({\mathbf{I}}_{\mathrm{HR}}\right)\right|,$$

(18)

where $f_{\mathrm{VGG}54}$ is the map function of the truncated VGG network.

As for the adversarial loss, the loss functions are calculated under the RGAN framework, formulated as

$$\begin{array}{cc}\hfill L_{\mathrm{adversarial}}=& -E\left[log\left(1-D_{\mathrm{RGAN}}\left({\mathbf{I}}_{\mathrm{HR}},G\left({\mathbf{I}}_{\mathrm{LR}}\right)\right)\right)\right]\hfill \\ \hfill & -E\left[log\left(D_{\mathrm{RGAN}}\left(G\left({\mathbf{I}}_{\mathrm{LR}}\right),{\mathbf{I}}_{\mathrm{HR}}\right)\right)\right]\hfill \end{array},$$

(19)

$$\begin{array}{cc}\hfill L_D=& -E\left[log\left(D_{\mathrm{RGAN}}\left({\mathbf{I}}_{\mathrm{HR}},G\left({\mathbf{I}}_{\mathrm{LR}}\right)\right)\right)\right]\hfill \\ \hfill & -E\left[log\left(1-D_{\mathrm{RGAN}}\left(G\left({\mathbf{I}}_{\mathrm{LR}}\right),{\mathbf{I}}_{\mathrm{HR}}\right)\right)\right]\hfill \end{array},$$

(20)

where $L_D$ is the loss function of the discriminator.

4. Experiments and Analysis

4.1. Experimental Settings

(1) Datasets: In the experiments, we used three widely usedremote sensing datasets, including AID [15], UCMERCED [16], and RSIs-CB256 [17]. These datasets are all RGB image datasets originally used for aerial scene classification; therefore, these datasets contain a variety of scenes, such as oceans, lands mountains, and buildings, which are very suitable for evaluating the generalization ability of networks in practical conditions. AID contains 10,000 images of 600 × 600 pixels in 30 scenes, with resolutions ranging from 8 to 0.5 m. UCMERCED is a 21-class remote sensing image dataset. Each class consists of 100 images, each of which is of 256 × 256 pixels. RSIs-CB256 is a remote sensing image dataset with a spatial resolution of 0.3–3 m and a pixel size of 256 × 256. It contains 35 categories and more than 24,000 images.

We use AID to build the realistic training dataset considering that its image size is more suitable for the degradation modeling. At first, we use KernelGAN [21,22] to collect all possible blur kernels in AID to construct the blur kernel dataset $\mathcal{K}$. Secondly, we extract noise patches according to Equation (5) and construct the noise patch dataset $\mathcal{N}$. Then, we use AID, $\mathcal{K}$, and $\mathcal{N}$ to generate the realistic training dataset according to Equations (2) and (3). The downsampling scales c and s in the experiments are 2 and 4, respectively. For each image, $\mathbf{k}$ and $\mathbf{n}$ are randomly selected to increase the diversity of training samples. Finally, we randomly select 9500 image pairs as the training dataset and 500 image pairs as the test dataset for referenced evaluation. To compare the performance of networks trained with realistic dataset and ideal dataset, we also build an ideal AID training dataset with bicubic downsampling. In the ‘ideal’ dataset, the 4× bicubic downsampled HR images are labeled as LR images. As for UCMERCED and RSIs-CB256, to reduce time consumption, we randomly select 500 images from UCMERCED and RSIs-CB256, respectively, for non-referenced evaluation. In other words, the images in the UCMERCED and RSIs-CB256 dataset are not downsampled but directly used as network input to generate the corresponding SR images.

(2) Implementation details: During the training phase, the LR images were randomly cropped to 48 × 48 as inputs, the super-resolution scale was 4×, and the batch size was set as 16. Image flips and rotations were randomly used for data augmentation. Both the proposed model and other compared methods were trained for 2 $\times {10}^5$ iterations (337 epochs) using the ADAM optimizer. We set ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $\epsilon =1\times {10}^{-8}$ in ADAM. The learning rate was initialed as $1\times {10}^{-4}$ and halved every 2 $\times {10}^4$ iterations. The proposed model was implemented with PyTorch framework and trained on a NVIDIA GTX3090 GPU.

(3) Evaluation metrics: We employ 4 widely used referenced image quality assessment metrics including peak signal-to-noise ratio (PSNR) [45], structure similarity index (SSIM) [46], independent feature similarity (IFS) [47], and learned perceptual image patch similarity (LPIPS) [48] in this work. We also adopt 2 widely used non-referenced image quality assessment metrics including natural image quality evaluator (NIQE) [49] and entropy-based non-reference image quality assessment (ENIQA) [50] in this work. Since the main energy of images is concentrated in the low-frequency area pixel-wise referenced metrics such as PSNR and SSIM average differences per pixel, thereby focusing on the flat parts of the images. Feature-based metrics such as IFS and LPIPS calculate differences in the feature spaces, thereby paying more attention on the high-frequency components of images. Non-referenced metrics such as NIQE and ENIQA are also defined on feature spaces, thus they also focus on the edge parts of the images. To sum up, all these metrics are somewhat one-sided and require comprehensive consideration.

4.2. Experiments on Referenced Evaluation

(1) Setup: As there is no research that considers the degradation process for the SR of RSIs as far as we know, we compared the proposed method with 6 classic SISR networks, including SRCNN [24], VDSR [28], DDBPN [51], EDSR [30], SRGAN [29], and DRLN [32]. The bicubic interpolation was also used as a baseline. For a fair comparison, the non-GAN-based models were all trained with only the $l_1$ pixel loss, while SRGAN was trained with the same loss component weights mentioned in the original paper. Experimental results under 3 kinds of settings are provided to verify the effectiveness of the degradation modeling. The first setting was to train and test models on the ‘ideal’ AID dataset, where the LR images were downsampled with only bicubic interpolation, as denoted in Equation (1). The second setting was to train models on the ‘ideal’ AID dataset and test on the realistic AID dataset. The realistic dataset was built using the proposed degradation framework, as denoted in Equation (3). In the third setting, the models were trained and tested on the realistic AID dataset. In other words, in the first and second settings, the models were exactly the same but the test datasets were different. In the second and third settings, the test datasets were the same but the models were different.

(2) Results: The average quantitative evaluation metrics of 8 methods on the AID dataset are shown in

Table 1. Quantitative evaluation results of all methods on the AID dataset with three settings.

Train	Test	Methods	PSNR↑	SSIM↑	IFS↑	LPIPS↓	NIQE↓	ENIQA↓
Bicubic AID	Bicubic AID	Bicubic	25.73	0.7267	0.8416	0.5532	6.402	0.4534
		SRCNN [24]	26.54	0.7670	0.8629	0.4309	7.412	0.3288
		VDSR [28]	27.00	0.7861	0.8736	0.3805	7.140	0.2815
		DDBPN [51]	27.10	0.7906	0.8759	0.3741	6.854	0.2836
		EDSR [30]	27.24	0.7960	0.8794	0.3688	6.739	0.2862
		SRGAN [29]	26.10	0.7527	0.8605	0.3273	6.307	0.1720
		DRLN [32]	27.43	0.8034	0.8822	0.3499	6.706	0.2702
		RBAN-UNet	27.25	0.7944	0.8752	0.2710	5.318	0.2082
Bicubic AID	Realistic AID	Bicubic	23.65	0.6332	0.7417	0.7207	7.112	0.5602
		SRCNN [24]	23.93	0.6483	0.7556	0.6478	7.663	0.5053
		VDSR [28]	23.90	0.6479	0.7563	0.6493	7.425	0.5077
		DDBPN [51]	23.91	0.6484	0.7570	0.6486	7.207	0.5143
		EDSR [30]	23.91	0.6485	0.7576	0.6600	7.360	0.5217
		SRGAN [29]	23.64	0.6338	0.7576	0.6270	5.615	0.4188
		DRLN [32]	23.92	0.6488	0.7584	0.6672	7.265	0.5274
		RBAN-UNet	23.88	0.6468	0.7570	0.6552	6.683	0.5349
Realistic AID	Realistic AID	Bicubic	23.65	0.6332	0.7417	0.7207	7.112	0.5602
		SRCNN [24]	24.81	0.6926	0.7803	0.5607	7.398	0.4013
		VDSR [28]	25.14	0.7109	0.7936	0.4921	8.266	0.3299
		DDBPN [51]	25.29	0.7180	0.8009	0.4827	7.915	0.3337
		EDSR [30]	25.45	0.7250	0.8079	0.4750	7.404	0.3462
		SRGAN [29]	24.14	0.6649	0.7877	0.3749	4.547	0.1647
		DRLN [32]	24.83	0.7038	0.7840	0.5523	7.497	0.3814
		RBAN-UNet	25.68	0.7336	0.8160	0.3548	5.736	0.2462

The first and second best metrics are marked in red and green respectively. ↑ denotes the higher the better and ↓ denotes the lower the better.

, where the first and second best results are marked in red and green, respectively. Visual examples and corresponding evaluation metrics are provided in Figure 5. Since both bicubic and realistic AID datasets are synthesized, all referenced and non-referenced metrics are provided. The experimental results can be interpreted from two perspectives.

First, comparing the metrics for all 3 settings, all methods perform worse in the second setting than in the first. There is significant blurring in the visualization of the second setting. This indicates that models trained on bicubic dataset do not generalize well to the actual degradation process such as blur and noise. Without considering degradation, CNN-based methods are not qualified for practical use in SISR of remote sensing images. Furthermore, as can be seen from Table 1 and Figure 5, all methods perform better in the third setting than in the second setting. This indicates that the proposed degradation modeling framework is rational and the estimation of blur and noise is effective. Models trained with realistic dataset do have better generalization abilities for the super resolution of real remote sensing images.

Second, comparing all 8 methods (focus on the 3rd setting), RBAN-UNet achieves the best performance. The bicubic upsample method achieves the worst performance because it uses information from a very limited neighborhood area to estimates unknown pixel values, and its interpolation operation process does not vary with the image content. SRCNN performs much better because its multi-layer convolutional structure forms a much larger receptive field and the non-linear mapping enables SRCNN to achieve better computation for different image content. Based on SRCNN, the global residual structure of VDSR make it possible to stack more convolutional layers, which is beneficial to improve the performance. DDBPN adopts an iterative upscale and downscale strategy and dense connections in the network, which enables it to learn the interdependencies between LR images and HR images and achieves a better performance. EDSR performs better than DDBPN because its local residual strategy and efficient residual block design enable its super deep structure. DRLN achieves the best performance on bicubic dataset but performs much worse on realistic dataset because it lacks a targeted design to deal with degradation factors. SRGAN achieve worse PSNR, SSIM, and IFS results than EDSR, but better LPIPS, NIQE, and ENIQA results, because SRGAN is trained with perceptual loss and GAN framework. This proves a fundamental fact that introducing more details into SR images can increase the perceptual performance and may lead to pixel-wise performance degradation. Another fact is that non-referenced metrics prefer images with more details, but at the same time, these metrics have difficulty distinguishing between natural details and artifacts. The SR results of SRGAN do have more details, but many of them are artifacts and can not substantially improve the quality of SR images. To sum up, the perceptual loss and GAN framework are double-edged swords for improving SR performance. As for the proposed method, RBAN can serve as a good backbone by using the residual in residual structure and the residual balanced attention mechanism. Based on RBAN and SRGAN, RBAN-UNet applies pixel loss to balance perceptual loss and adversarial loss and uses a modified UNet as the pixel-wise discriminator in RGAN framework. RBAN-UNet achieves 4 optimal and 2 sub-optimal among 6 metrics under the 3rd setting, which is the best performance in all comparison methods. It can also be seen from the visualization that the results of RBAN-UNet contain sharper edges but do not introduce invalid textures. RBAN-UNet finds a good balance between details and cleanliness. The effectiveness of the proposed model is verified.

4.3. Experiments on Non-Referenced Evaluation

(1) Setup: In addition to the quantitative evaluation of all methods on the AID dataset using referenced metrics, we also evaluated all methods on the UCMERCED and RSIs-CB256 datasets. All methods were trained without any information from the UCMERCED and RSIs-CB256 datasets, which provides a good condition for evaluating the validity and generalization ability of all methods. Since the experiments were performed directly on the real images in UCMERCED and RSIs-CB256 datasets rather than the downsampled images, only non-referenced metrics are provided. The experiment results are provided with 2 kinds of settings: training on the ‘ideal’ AID dataset or realistic AID dataset.

(2) Results: The average quantitative evaluation metrics and visualization of 8 methods tested on the UCMERCED dataset are shown in

Table 2. Quantitative evaluation results of all eight methods on the UCMERCED dataset with two settings.

Train	Methods	NIQE↓	ENIQA↓
Bicubic AID	Bicubic	6.362	0.5368
	SRCNN [24]	7.431	0.4336
	VDSR [28]	7.337	0.4073
	DDBPN [51]	7.265	0.4064
	EDSR [30]	6.848	0.4209
	SRGAN [29]	5.719	0.2827
	DRLN [32]	6.400	0.4219
	RBAN-UNet	5.237	0.3827
Realistic AID	Bicubic	6.362	0.5368
	SRCNN [24]	7.940	0.3907
	VDSR [28]	6.295	0.3791
	DDBPN [51]	6.085	0.3646
	EDSR [30]	5.961	0.3909
	SRGAN [29]	4.329	0.1516
	DRLN [32]	7.033	0.4091
	RBAN-UNet	4.709	0.3169

The first and second best metrics are marked in red and green respectively. ↑ denotes the higher the better and ↓ denotes the lower the better.

and Figure 6, respectively. Models trained on the realistic AID dataset achieve higher metrics and better visualizations than models trained on bicubic AID dataset. This proves that the models do not overfit to the AID dataset and indeed gain generalization ability to deal with the degradation process. Horizontally comparing all 8 methods with the non-referenced metrics, SRGAN achieves the optimal metrics and RBAN-UNet achieves the sub-optimal metrics; however, as is mentioned above, NIQE and ENIQA are non-referenced feature-based image quality assessment metrics; therefore, images with more details, textures, and even artifacts are easy to achieve higher NIQE and ENIQA scores, which can be confirmed from the referenced evaluation experiments. From an intuitive point of view, SRGAN introduces lots of artifacts or invalid textures, which does not substantially improve the quality of reconstructed images; therefore, considering both visualization and non-reference metrics, it can be concluded that RBAN-UNet achieves the best performance on the UCMERCED dataset. The experiment results of RSIs-CB256 dataset shown in

Table 3. Quantitative evaluation results of all eight methods on the RSIs-CB256 dataset with two settings.

Train	Methods	NIQE↓	ENIQA↓
Bicubic AID	Bicubic	6.896	0.5670
	SRCNN [24]	6.424	0.4941
	VDSR [28]	6.169	0.4876
	DDBPN [51]	6.091	0.4908
	EDSR [30]	6.159	0.5011
	SRGAN [29]	5.453	0.4019
	DRLN [32]	6.133	0.5003
	RBAN-UNet	5.308	0.5393
Realistic AID	Bicubic	6.896	0.5670
	SRCNN [24]	6.780	0.4452
	VDSR [28]	6.194	0.4462
	DDBPN [51]	6.178	0.4475
	EDSR [30]	6.426	0.4847
	SRGAN [29]	3.979	0.2143
	DRLN [32]	6.123	0.4917
	RBAN-UNet	4.953	0.4369

The first and second best metrics are marked in red and green respectively. ↑ denotes the higher the better and ↓ denotes the lower the better.

and Figure 7 also support this conclusion. These experiments demonstrate that the proposed framework and model can effectively improve the performance of remote sensing image super resolution under practical conditions.

5. Discussion

In addition, we conducted an ablation study to further discuss the effectiveness of each proposed component. The quantitative and visual results are shown in

Table 4. Ablation results of RBAN-UNet on the realistic AID dataset.

Models	PSNR↑	SSIM↑	IFS↑	LPIPS↓	NIQE↓	ENIQA↓
Bicubic	23.652	0.6332	0.7417	0.7207	7.112	0.5602
RBAN-UNet (w/o degradation)	23.885	0.6468	0.7570	0.6552	6.683	0.5349
RBAN-UNet (w/o blur)	23.722	0.6411	0.7528	0.6316	6.649	0.5335
RBAN-UNet (w/o noise)	24.980	0.7053	0.7876	0.4403	5.239	0.2739
RBAN-UNet (w/o BAM)	25.610	0.7310	0.8137	0.3602	5.745	0.2487
RBAN-VGG	25.029	0.7186	0.8000	0.2722	4.877	0.1454
RBAN-UNet	25.676	0.7336	0.8160	0.3548	5.736	0.2462

The first and second best metrics are marked in red and green respectively. ↑ denotes the higher the better and ↓ denotes the lower the better.

and Figure 8, respectively.

5.1. Impact of Blur Kernel Estimation

As shown in Table 4 and Figure 8, RBAN-UNet (w/o degradation) achieves the worst performance among all discussed methods. RBAN-UNet (w/o degradation) denotes RBAN-UNet without any modeling of the blur and noise. It is equivalent to the model trained on bicubic dataset and tested on realistic dataset, as discussed in above subsections. RBAN-UNet (w/o noise) denotes RBAN-UNet without modeling the noise. It achieves much better metrics and sharper visual perception than RBAN-UNet (w/o degradation). Moreover, the complete RBAN-UNet outperforms RBAN-UNet (w/o blur) and the RBAN-UNet without modeling blur. These results confirm that the blur kernel estimation method is effective and the blur kernels help the networks to perform sharper super-resolution reconstructions.

5.2. Impact of Noise Patch Estimation

The complete RBAN-UNet model outperforms RBAN-UNet (w/o noise), which demonstrates the effectiveness of noise patch estimation and that the noise patches help the networks to perform better super-resolution reconstructions; however, there is no significant improvement in the results of RBAN-UNet (w/o blur) compared to RBAN-UNet (w/o degradation). We think the reason is that the influence of blur kernel exceeds that of noise patches.

5.3. Impact of BAM

The BAM that can be easily embedded in and removed from the model is a key component in the RBAN-UNet. As can be seen in Table 4 and Figure 8, removing BAMs from the model results in a significant drop in quantitative and perceptual performance when compared with the complete RBAN-UNet. This confirms the effectiveness and necessity of using BAM in network.

5.4. Impact of Discriminator

At last, we evaluated the function of the discriminator by replacing the discriminator in RBAN-UNet with VGG. Although RBAN-VGG achieves the best perceptual results, it introduces lots of invalid textures that are detrimental to understanding image content. RBAN-UNet achieves the best performance by comprehensively considering quantitative and perceptual results. It shows that the discriminator helps a lot to improve the perceptual performance, and the modified UNet achieves a better balance between quantitative and perceptual performances than VGG. The comparison confirms the effectiveness of the modified UNet discriminator.

6. Conclusions

In this article, a real-world degradation modeling framework and a residual balanced attention network with modified UNet discriminator (RBAN-UNet) have been proposed for remote sensing image super resolution. The quality of real RSIs is affected by a series of factors, such as illumination, atmosphere, imaging sensor responses, and signal processing, resulting in a gap in the performance of previous methods between laboratory conditions and actual conditions. To model the real-world degradation of RSIs, we propose to estimate the blur kernels and noise patches in the dataset separately. Then, the blur kernels and noise patches are used to construct a realistic dataset that follows the desired mapping function from realistic LR images to clean HR images. Moreover, we develop a novel CNN model to perform the SR reconstruction for RSIs. We use a residual in residual architecture as the backbone and embed balanced attention modules (BAM) to improve the performance. To generate more realistic results, a modified UNet pixel-wise discriminator is employed. Detailed experiments were carried out to compare the proposed model with classic SISR networks. Referenced experiments, non-referenced experiments, and ablation studies validate that the degradation modeling framework improves the performance of models dealing with real RSIs and the proposed RBAN-UNet model achieves a state-of-the-art performance in the real-world SISR problem for RSIs. In our future work, we will focus on decoupling the degradation and images inside the network instead of explicitly collecting the degradation datasets.