Pansharpening Remote Sensing Images Using Generative Adversarial Networks ^†

Abstract

Pansharpening is a remote sensing image fusion technique that combines a high-resolution (HR) panchromatic (PAN) image with a low-resolution (LR) multispectral (MS) image to produce an HR MS image. The primary challenge in pansharpening lies in preserving the spatial details of the PAN image while maintaining the spectral integrity of the MS image. To address this, this article presents a generative adversarial network (GAN)-based approach to pansharpening. The GAN discriminator facilitated matching the generated image’s intensity to the HR PAN image and preserving the spectral characteristics of the LR MS image. The performance in generating images was evaluated using the peak signal-to-noise ratio (PSNR). For the experiment, original LR MS and HR PAN satellite images were partitioned into smaller patches, and the GAN model was validated using an 80:20 training-to-testing data ratio. The results illustrated that the super-resolution images generated by the SRGAN model achieved a PSNR of 31 dB. These results demonstrated the developed model’s ability to reconstruct the geometric, textural, and spectral information from the images.

1. Introduction

Remote sensing technology has become increasingly popular across various fields, including business, agriculture, environmental monitoring, disaster management, land management, and natural resource utilization. However, it is costly, with the usage fees depending on the user’s data requirements, such as the shooting area, resolution, and cloud cover. In Refs. [1,2,3,4], the size and resolution of different cameras were reviewed, noting that high-resolution satellite images are more expensive and cover smaller areas.

Multispectral (MS) imagery, which typically includes different bands such as visible light and other spectra, has a lower resolution. Panchromatic (PAN) imagery is captured in a single band that encompasses all visible light bands, resulting in a higher resolution. PAN imagery is often combined with MS imagery to leverage the advantages of both. However, there is a fee difference between PAN and MS images, with PAN typically being more expensive, depending on the specifications. In pansharpening, a high-resolution single-band image is combined with a lower-resolution multiband image to improve the overall image quality. The high-resolution single-band imagery is used to upsample lower-resolution multiband images, improving the spatial details and visual interpretation [3,4].

With the advancement of artificial intelligence (AI) technology, super-resolution approaches are often used to generate high-resolution PAN images. Super-resolution techniques enhance image quality through upsampling, and there are many approaches to this available today. Many researchers have adopted generative adversarial networks (GANs) [1,2,3,4,5,6,7,8,9,10], a deep learning architecture that trains two neural networks to oppose each other. They utilize a deep residual network (ResNet) with a mean squared error (MSE) loss function as the base. An adversarial loss function is used for the discriminator, and content loss training is performed to extract the similarity of object features using visual geometry group (VGG) loss. Super-resolution ResNet (SRResNet) employs only the generator network architecture, while SRGAN utilizes both the generator and discriminator network architectures.

In the training stage, an HR image is used as the input and then downsampled to create an LR image. The original HR image and the downsampled LR image are used to train the model. In the execution stage, the original HR image is used again as the input. The process involves downsampling first and then upsampling using the trained SRGAN model. The resulting image is compared with the original HR image to validate the model. Figure 1 illustrates the results of applying this approach to satellite images. This article compares the differences in the amount of altered data and notes that these differences become apparent during the downsampling process and the subsequent upsampling process.

2. Methodology

2.1. The Ground Truth from Pansharpening

In this article, a pansharpening program was developed based on the theories from Refs. [10,12] to obtain HR color images as the ground truth. Most satellite sensors capture PAN and MS images, providing complementary information about the same scene. The goal of pansharpening is to fuse PAN and MS images to generate a high-quality image.

First, the LR color MS image is resized to match the dimensions of the HR black-and-white PAN image. The LR color multispectral (MS) image is converted from the RGB color space into the YC_rC_b color space based on (1)–(3).

$$Y=0.299R+0.587G+0.114B$$

(1)

$$C_b=-0.172R-0.339G+0.511B+128$$

(2)

$$C_r=0.511R-0.428G-0.083B+128,$$

(3)

where the Y channel represents the luminance component, the C_r channel represents the red-difference chroma component, and the C_b channel represents the blue-difference chroma component.

Next, the Y channel of the LR color image is replaced with the Y channel of the HR image. Finally, the LR color MS image is converted back from the YC_rC_b color space into RGB according to (4)–(6).

$$R=Y+1.371(C_r-128)$$

(4)

$$G=Y-0.698(C_r-128)-0.336(C_b-128)$$

(5)

$$B=Y+1.732(C_b-128),$$

(6)

where R, G, and B are the red, green, and blue values of each pixel in any area of the original image.

Using the above approach, the ground truth for this research was obtained. Figure 2 illustrates the original MS image, the original PAN image, and the pansharpened image, which combines the original MS image with the PAN image.

2.2. The Ground Truth from Adobe Photoshop

The ground truth was obtained using Adobe Photoshop [13]. First, the LR color image is resized to match the dimensions of the HR black-and-white image. The LR color image is placed in the upper layer while the HR black-and-white image is in the lower layer. Then, the upper and lower layers are decomposed separately from red, green, and blue (RGB) into hue (H), saturation (S), and luminance (L). Figure 3 illustrates the original MS image, the pansharpened image derived from the MS and PAN images, and the image processed using Adobe Photoshop. Next, the H and S are extracted from the upper layer, and L is extracted from the lower layer. Finally, the new HSL values are converted back into RGB to generate the HR color image.

2.3. The Network Architecture

Traditionally, when restoring HR images using upsampling approaches, the MSE loss is used as the loss function. Despite the better evaluation scores, it does not significantly enhance the visual quality from a human perspective, as it causes the loss of high-frequency signals in the image. To address this challenge, this article utilizes a new loss function, perceptual loss, which combines content loss and adversarial loss [7]. In the SRResNet model, it is adopted as the backbone and paired with a discriminator network to form the SRGAN architecture (Figures 1 and 2 in Ref. [5], Figure 4 in Ref. [6]). In the absence of an HR dataset, the pansharpening program is used [10,12]. The resolution of the synthesized HR dataset must match the scaling factor of the LR dataset, and any slight discrepancies must be adjusted to a 2:1 ratio. After importing the two datasets, the step of obtaining LR data through downsampling is skipped; instead, the existing LR data are used as the scaling factor. The LR data are then scaled to match the HR data using this scaling factor. The image size is adjusted based on the crop ratio, ensuring that both crops align with the scaling factor. This alignment ensures that the corresponding blocks in the image are correctly imported into the model for training.

2.4. The Loss Function

In GANs [9], the D_Loss function is a minimax process, where the objective function V(G,D) is defined as (7).

$$\begin{gathered} \underset{G}{min}\underset{D}{max}V(G,D)=E_{I^{HR}\sim P_{train}\left(I^{HR}\right)}\left\{\mathit{log}D\left(I^{HR}\right)\right\} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+E_{I^{HR}\sim P_G\left(I^{LR}\right)}\left\{\mathit{log}\left[1-D\left(G\left(I^{LR}\right)\right)\right]\right\}, \end{gathered}$$

(7)

where I^HR is the real image, and I^LR is the LR image. The discriminator D distinguishes between real HR images and the super-resolution images generated by the generator. Therefore, it is crucial to minimize the loss for distinguishing real images and the loss for distinguishing generator-generated super-resolution images. The discriminator loss (D_Loss) is the sum of the losses in (8), as shown below.

$$D_{Loss}=-\left\{y\cdot \log \left[D(x)\right]+\left(1-y\right)\cdot \log \left[1-D\left(G(x)\right)\right]\right\}$$

(8)

where y is the label for the real image (with a value between 0 and 1), D(x) is the predicted probability of the real image, and D(G(x)) is the predicted probability of the super-resolution image generated by the generator.

3. Results and Discussion

3.1. The Dataset

The training dataset was provided by the Center for Space and Remote Sensing Research (CSRSR), National Central University [11]. It consists of 12 MS satellite images and 12 PAN satellite images in JPG format with slight size differences. The MS satellite images range from 932 × 524 to 936 × 528 pixels, while the PAN images range from 1864 × 1048 to 1872 × 1056 pixels. In the initial acquisition of these image pairs, several of the images did not have an exact 2:1 size ratio.

For model training, two datasets are required: an HR dataset and an LR dataset. The HR dataset utilizes the true values generated by pansharpening, while the LR dataset consists of the original MS satellite images. The relative sizes of the images are adjusted based on the scaling factor, which is 2 in this study, meaning the image sizes must have a 2:1 ratio. However, the size ratio of the paired images is not always 2:1. For instance, an MS satellite image of 932 × 524 pixels might correspond to a PAN satellite image of 1872 × 1048 pixels. Failure to scale the images according to this scaling factor results in unsuccessful pansharpening of the ground truth images and leads to the model learning the incorrect parameters during training. Therefore, in this study, the MS satellite images are scaled to 932 × 524 pixels, and the PAN satellite images are scaled to 1864 × 1048 pixels.

Given the limited amount of data provided, data augmentation is applied to enhance the data’s richness. Specifically, different portions of the images are randomly cropped, and the brightness and contrast are adjusted to simulate various weather conditions. This allows the images to transition from day to night, winter to summer, sunny to rainy, etc., enabling the model to learn more diverse parameters. Additionally, the randomly cropped images must maintain the scaling factor ratio, and the crop positions must be adjusted accordingly. The data from eight pairs of images are decomposed into a total of 120,000 pairs for training, while another eight pairs of images are reserved for testing. The remaining four pairs of images are used as a test dataset to evaluate the model’s performance.

3.2. The Performance Evaluation

In the execution stage, the input image bypasses the downsampling process and undergoes super-resolution processing, and the resulting image is then compared with the ground truth image to calculate the peak signal-to-noise ratio (PSNR). Figure 4a illustrates the original MS satellite image resized to twice its size, Figure 4b presents the original MS image upsampled using SRResNet, and Figure 4c describes the original MS image upsampled using SRGAN. For comparison, certain areas such as houses and parking lots are highlighted with red boxes, and the red-boxed areas are zoomed in. Due to the limited data, an 8:4 training-to-testing split is used for validation.

In addition, an image histogram is created to record the number of pixels at each RGB level. The horizontal axis represents the intensity, ranging from 0 to 255, and the vertical axis represents the number of pixels at each intensity. In terms of the number of pixels at each level, the RGB channels are compared with the SRGAN-generated image shown in Figure 4c and with the pansharpening ground truth shown in Figure 2c. Figure 5 illustrates a comparison of the results, indicating that the color channels in both images are very similar.

Table 1. The performance in terms of the PSNR of SRResNet and SRGAN.

	Neural Network	SRResNet	SRGAN
Ground Truth
Pansharpened		28.44 dB	25.87 dB
Adobe Photoshop		17.10 dB	16.97 dB

presents a comparison of the images generated by SRResNet and SRGAN with the ground truth image produced by pansharpening and Adobe Photoshop. Table 1 and

Table 2. Execution times of SRResNet and SRGAN.

	Neural Network	SRResNet	SRGAN
Time
Execution Time		7.97 s	8.02 s

present the PSNR values for both images and the execution times of their respective algorithms. The results indicate that the resolution of the original MS satellite image is significantly improved, with its PSNR value increasing to 28 dB compared to that of the original image. Additionally, regarding the ground truth comparison of the PSNR values, the value for SRResNet is higher than that for SRGAN. This result confirms the findings of the SRGAN article [6] that SRResNet’’s higher PSNR scores make it favorable for machine interpretation. On the other hand, SRGAN tends to produce realistic results in terms of visual perception. In Ref. [6], the mean opinion score (MOS) is used as an alternative scoring metric, and SRGAN scores significantly higher in terms of its visual performance compared to that of SRResNet. The MOS methodology and corresponding data charts are thoroughly discussed in Reference [6], which demonstrates that SRGAN’s MOS scores more closely approximate those of the original high-resolution images.

4. Conclusions

This article employed GAN approaches to approximate pansharpening of remote sensing images. The data input approach for training was modified through the implementation of a deep ResNet with a perceptual loss function, utilizing adversarial and content losses for discriminator and generator training. Instead of using an HR dataset input and downsampling to obtain the LR dataset, simultaneous input of both the HR and LR datasets was performed. This approach enabled a clear definition of the initial and target states of the image, facilitating learning of the optimal parameters. To resolve any potential issues encountered, the differences between various ground truths were compared. A comparison of the super-resolution results for satellite imagery was conducted between models trained using the original SRGAN data input approach and those trained with the modified approach to ascertain which yielded superior results.