Deep Learning for Feature-Level Data Fusion: Higher Resolution Reconstruction of Historical Landsat Archive

Simple Summary

Existing freely available global land surface observations are limited by medium to coarse resolutions or short time spans. Here we developed a feature-level data fusion framework using a generative adversarial network (GAN), a deep learning technique, to leverage Landsat and Sentinel-2 observations during their over-lapping period from 2016 to 2019, and reconstruct 10 m Sentinel-2 like imagery from 30 m historical Landsat archives. Our tests with both simulated data and actual Landsat/Sentinel-2 acquisitions showed that the GAN-based super-resolution method could accurately reconstruct synthetic Landsat data at an effective resolution very close to that of the real Sentinel-2 observations. The promising results from our deep learning-based feature-level fusion method highlight the potential for reconstructing a 10 m Landsat series archive since 1984. It is expected to improve our capability of detecting and tracking finer scale land changes, identifying the drivers for and responses to global environmental changes.

Abstract

Long-term record of fine spatial resolution remote sensing datasets is critical for monitoring and understanding global environmental change, especially with regard to fine scale processes. However, existing freely available global land surface observations are limited by medium to coarse resolutions (e.g., 30 m Landsat) or short time spans (e.g., five years for 10 m Sentinel-2). Here we developed a feature-level data fusion framework using a generative adversarial network (GAN), a deep learning technique, to leverage the overlapping Landsat and Sentinel-2 observations during 2016–2019, and reconstruct 10 m Sentinel-2 like imagery from 30 m historical Landsat archives. Our tests with both simulated data and actual Landsat/Sentinel-2 imagery showed that the GAN-based fusion method could accurately reconstruct synthetic Landsat data at an effective resolution very close to that of the real Sentinel-2 observations. We applied the GAN-based model to two dynamic systems: (1) land over dynamics including phenology change, cropping rotation, and water inundation; and (2) human landscape changes such as airport construction, coastal expansion, and urbanization, via historical reconstruction of 10 m Landsat observations from 1985 to 2018. The resulting comparison further validated the robustness and efficiency of our proposed framework. Our pilot study demonstrated the promise of transforming 30 m historical Landsat data into a 10 m Sentinel-2-like archive with advanced data fusion. This will enhance Landsat and Sentinel-2 data science, facilitate higher resolution land cover and land use monitoring, and global change research.

1. Introduction

Many land changes occur at fine spatial scales ranging from meter to ~10 m [1,2], and thus remote sensing data at higher spatial resolution are needed to effectively detect and monitor these changes across landscapes. In order to understand the drivers associated with global environmental change, repeated observations over a long time period are also critical. The U.S. Geological Survey (USGS) Landsat series of satellites provides the longest temporal record of space-borne surface observations at 30 m [3]. It has been widely used for documenting how the Earth’s terrestrial ecosystem has changed over the past four decades [4,5], driven by the growing pressure of booming population, human modification, and climate change [6,7,8]. However, due to the mixed pixel problem [9,10], the 30 m spatial resolution of Landsat data still limits its applications for accurate quantification of biophysical properties and processes, especially over heterogeneous landscapes. For example, it is challenging to take full advantage of Landsat mission continuity to detect canopy-level flowering phenology [11], differentiate crop species [12,13], extract urban functional blocks [14], and map wetland and watershed interfaces [15].

The Sentinel-2 mission, initiated by the European Commission (EC) and the European Space Agency (ESA) constellation, on the other hand, aims to provide systematic global acquisition of fine-resolution multispectral imagery [16]. With the full operation of two identical satellites, Sentinel 2A/B has been able to provide an unprecedented observation of global land surfaces at a spatial resolution of 10–60 m with a five day revisiting cycle [16]. Given the relative fine spatial and temporal resolutions and free access to the full data archive, Sentinel-2 data has been used by an increasing number of end users in various research and applications, such as urban [14], agriculture [12], and forestry [17]. However, Sentinel-2 systems have had a relatively short five-year data record so far since the first launch of Sentinel-2A in 2015, thereby limiting its capability for long-term terrestrial change applications.

To take advantage of the complementary information from multiple sensors with varying spatial, temporal, and spectral characteristics, data fusion techniques have attracted tremendous interest within the remote sensing community over the past decades [18,19,20]. Many studies, for example, have focused on the spectral and spatial fusion via “pan-sharpening” [10,21,22,23]. Their main purpose is to generate high-resolution imagery at multiple spectral bands, by combining the multispectral characteristics at coarser spatial resolution with the fine spatial details from high spatial resolution panchromatic imagery. However, the outputs from pan-sharpening may suffer from spectral distortions, limiting their use in the field of quantitative remote sensing.

Spatio-temporal fusion has also been developed and applied to generate synthetic data at fine spatial resolution and with high temporal frequency, by blending remote sensing observations from multi-sensors with various spatial and temporal characteristics [6,19,20,24,25,26,27]. Among them, the spatial and temporal adaptive reflectance fusion model (STARFM) [19] is one of the most popular algorithms; it uses the concurrent image pairs of 30 m Landsat and 250 m MODIS to build the spatial relationship and adjusts the temporal dynamics with more frequent coarse-resolution MODIS observations [18,25]. However, the STARFM-like algorithms rely on at least one prior or posterior pair of fine and coarse spatial resolution images (e.g., Landsat and MODIS) acquired on the same day. This requirement makes the STARFM-like framework not suitable for fusing Landsat and Sentinel-2 images that often revisit at different overpassing days [28]. The temporal interpolation framework of STARFM-like algorithms make assumptions of no significant land cover changes occurring during the interpolated period [18,19]. This may cause large uncertainties in reconstructing historical Landsat imagery for denser time stacks. A fill-and-fit approach was also developed for land surface time series using alternative similar pixels for spatial filling and a harmonic model to gap fill the time series, based on a single year or growing season of Landsat [29]. These approaches are mostly designed for pixel-level observation fusion to generate continuous time series of the highest resolution available for tracking land surface dynamics. The feature perception information, however, has not been fully incorporated into fusion.

Several studies have been focused on the fusion of Landsat and Sentinel-2 images. For example, the NASA Harmonized Landsat and Sentinel-2 project creates a temporally denser Landsat-like surface reflectance datasets, by resampling 10 m Sentinel-2A and 2B data acquired every five days to match the 30 m resolution of Landsat. Although many efforts are made to ensure radiometric consistency, including atmospheric correction, cloud and cloud-shadow masking, spatial co-registration and common gridding, bidirectional reflectance distribution function normalization and spectral bandpass adjustment [30], finer spatial details from Sentinel-2 are not fully used for the fusion [28,31]. To overcome this issue, Wang et al. [31] proposed an area-to-point regression kriging (ATPRK)-based method to generate 10 m Landsat-8 like images, by fusing available 10 m Sentinel-2 and Landsat 8 panchromatic (PAN) bands. This geo-statistical fusion algorithm relies on the input covariates derived from the corresponding Sentinel-2 imagery to enhance spatial details of each lower-resolution imagery. However, ATPRK involves complex semi-variogram modelling from a co-kriging matrix, which is computationally expensive for large-scale super-resolution processing [28].

One possible cost-effective solution is to leverage Landsat and Sentinel-2 observations during their overlapping period, to reconstruct 10 m Sentinel-2-like imagery from historical Landsat imagery at 30 m before 2015. It is based on a transfer learning strategy enabled with deep learning algorithms in computer vision [32,33,34], e.g., applying the relationship learned from Landsat and Sentinel-2 over the most recent temporal period to historical pre-2015 Landsat data, to regenerate synthetic 10 m Landsat archive over the past four decades.

Convolution neural network (CNN)-based super-resolution (SR) image reconstruction has proven to be a promising approach in closing inherent resolution gaps of imaging sensors [33,35,36,37]. Here, we categorize existing deep learning CNN SR methods into two classes based on their main purpose. One class aims at optimizing distortion measures such as peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and mean square error (MSE) to yield more accurate images, and we named this group as PSNR-oriented methods. In order to achieve better quantitative measures, major solutions in this category consider pixel-wise loss functions and increase the network complexity of CNNs. For example, Ledig et al. [34] proposed SRResNet with 16 residual units to optimize MSE. Lim et al. [38] developed an enhanced deep super-resolution network (EDSR) to significantly improve the single image super-resolution performance by removing the batch normalization layers in SRResNet and increasing the number of residual blocks to 23.

Another group of image SR studies attempts to produce more photo-realistic images by including an extra perceptual loss function; these are superior in perceptual quality, thereafter referred as perceptual-oriented methods. For example, Johnson et al. [39] proposed perception loss to enhance the perceptual quality by minimizing the error in feature space instead of pixel space. A generative adversarial network (GAN) was also developed for image SR by applying both perceptual loss and adversarial loss to generate realistic textures during SR and achieves state-of-the-art performance [34]. Pouliot et al. [36] applied both shallow and deep convolution neural networks (CNNs) in Landsat image super-resolution enhancement, trained using Sentinel-2, in three study sites representing boreal forest, tundra, and cropland/woodland environments. An extended super-resolution convolutional neural network (ESRCNN) was developed [28], and it was suggested that the proposed deep learning models outperformed the ATPRK-based algorithm using experimental tests in two local study areas of North China’s Hebei province. However, these CNN approaches still fall into the category of non-GAN-based models without adding the adversarial discriminator to better tune the neural network. A similar non-GAN-based CNN model was used to super-resolve the lower-resolution bands (20 m and 60 m) of Sentinel 2 to 10 m spatial resolution, over 60 representative Sentinel-2 scenes across the globe [40].

Despite the promising results of these pilot studies, the evaluation and application of deep learning-based approaches in remote sensing data fusion of Landsat and Sentinel-2 is still limited. Firstly, most of existing studies working on the super-resolution of Landsat and Sentinel-2 rely on non-GAN-based models without fully coupling perceptual loss and adversarial loss into the CNN framework. Secondly, the training and tuning of deep learning algorithms in remote sensing data fusion is complex and sensitive to both spatial and temporal domains. Moreover, the majority of available studies are limited to local scales and short-temporal periods. Lastly, a comprehensive comparison of different algorithms, including statistical interpolation, geo-statistical modelling, non-GAN-based, and GAN-based deep learning models to super-resolve Landsat imagery from 30 m to 10 m, is needed to provide insights for practical applications. To our knowledge, there have been limited efforts so far on reconstructing 10 m Sentinel-2-like imagery from the historical Landsat archive using the state-of-the-art non-GAN and GAN-based deep learning approaches across different spatial and temporal scales.

Addressing these challenges, we aim to implement a feature-level data fusion framework for remote sensing SR image reconstruction. Specifically, we first developed the GAN-based fusion framework to leverage Landsat and Sentinel-2 observations during their overlapping period from 2016 to 2019, for reconstructing 10 m Sentinel-2-like imagery from historical 30 m Landsat archive. We then provided a comprehensive comparison of GAN-based deep learning models with baseline methods including statistical interpolation, geo-statistical modelling, non-GAN-based, to better inform SR model performance. We further investigated the potential of the GAN-based model in dynamic systems over different spatial and temporal domains. Finally we demonstrated the promise of transforming 30 m historical Landsat data into a 10 m Sentinel-2-like archive with this advanced data fusion.

2. Materials and Methods

2.1. Theoretical Basis

The aim of our study was to reconstruct an SR resolved image I_SR to the same spatial resolution as Sentinel-2 (I_HR), from a low spatial resolution Landsat image I_LR with a convolutional neural network (CNN). A non-GAN-based network (Section 2.1.1) and a GAN-based network (Section 2.1.2) were both used in this study for the purpose of comparing the performance of PSNR-oriented and perceptual-oriented deep learning methods in remote sensing image SR tasks.

As shown in Table S1, Landsat series (i.e., Landsat-5/7/8) have very close bandwidths to Sentinel-2A/B, particularly over blue, green, and red bands. This provided a theoretical basis of spectral comparability to build a band-over-band relationship between Landsat and Sentinel-2. For the experimental purpose in this study, we focused on using the true color composite of red–green–blue bands for both Landsat and Sentinel-2. The proposed methods below can be extended to other paired bands and high–low resolution image pairs as well.

2.1.1. Deep Convolutional Neural Networks

For a low spatial resolution image I_LR with W columns, H rows, and C bands, it can be expressed by a tensor of size $W\times H\times C$. Its corresponding I_HR and I_SR have the size of $rW\times rH\times C$, where r represents a scaling factor. In the training process, I_LR and I_HR image pairs were used as the input of a feed-forward CNN network $G_{\theta }$ parameterized by $\theta$. We adopted the SRResNet architecture [34] in this work, with the only difference in removing batch normalization (BN) layers in the residual block, because removing BN layers will help reduce computation costs and increase the model generalization ability [38]. As shown in Figure 1, SRResNet comprises 16 identical residual blocks using ParametricReLU as the activation function and downscaling with 2 trained sub-pixel convolution layers.

The objective function is:

$$\widehat{\theta }=argmi{n}_{\theta }{E}_n\left[l_{MSE}\left(G_{\theta }\left(I_{LR}^n\right), I_{HR}^n\right)\right], n = 1, 2, \dots , N$$

(1)

where $I_{LR}^n$ and $I_{HR}^n$ are corresponding low-high spatial resolution training image pairs, N is the total number of training image pairs, $l_{MSE}$ is the most widely used loss function for image SR, which calculates the pixel-level MSE between $I_{HR}^n$ and the generated image $I_{SR}^n$.

2.1.2. Generative Adversarial Networks

Generative adversarial networks (GAN), proposed by Goodfellow et al. [41] in 2014, have shown great breakthroughs in training unsupervised machine learning models and have gained popularity in computer vision fields. It consists of two networks; a generator G and a discriminator D. G is trained to generate output that is close to the real data. D is a multilayer perception model defined to distinguish whether a sample is from the generator or real data, represented as a simple binary classifier that outputs a single scalar.

For a better gradient performance, $E_n[-\log (D(G(x_n)))]$ is usually used as the G’s loss function. $E_{x~p_{data}(x)}[-\log (D(x))]+E_{x~p_g(x)}[-\log (1-D(G(x)))]$ is a common form of the D’s loss function. D and G are trained in conjunction to solve a two-player min–max problem. G tries to fool D into labeling its sample as real data by minimizing its loss function and generating sample approximating real data distribution. D tries to maximize the probability of assigning the correct label to sample both from generator’s distribution ($p_g$) and from data distribution ($p_{data}$).

A GAN-based model was constructed by taking the non-GAN-based model (Section 2.1.1) as the generator ($G_{\theta }$) and trained with a discriminator (Figure 2). The architecture of the discriminator we used closely followed ESRGAN [42], which is a relativistic average discriminator that instead of estimating if a generated image is real or fake, it estimates the realistic score of a generated image relative to real image.

2.1.3. Loss Function

The total loss of the generator $G_{\theta }$ is the weighted combination of adversarial loss and perceptual loss defined as

$$los{s}_{SR}=l_{VGG19-54}+w\cdot l_G$$

(2)

where w is the weight set to be 10⁻³, and $l_G$ is the generator’s loss $E_n[-\log (D(G(x_n)))]$. VGG loss, $l_{VGG19-54}$ is the perceptual loss, defined as the L₂-norm distance between the feature representations (i.e., feature obtained by the 4th convolution before the 5th maxpooling layer within the pre-trained 19-layer VGG network) of a generated image I_SR and the reference high spatial resolution image I_HR.

2.2. Implementation

The scaling factor r between I_SR and I_HR was 3 in all experiments, to be consistent with the spatial resolution difference between Landsat and Sentinel-2 dataset. The minimum batch size was set to be 16. Following the SRGAN, training images were cropped into 96 × 96 patches for each minimum batch.

For the simulated data experiment and real data experiment (Section 3.1 and Section 3.2), the modified SRResNet networks (referred as non-GAN-based network hereafter) were trained through 1 million iterations with the MSE loss, a learning rate of 10⁻⁴, and decayed by a factor of 2 at 50 k, 100 k, 200 k, and 300 k iterations.

When training the GAN-based model, we employed the trained non-GAN-based network as initialization for the generator to obtain better results [34]. GAN-based networks were trained with 5 × 10⁵ iterations at a learning rate of 10⁻⁴ and decayed by a factor of 2 every 2 × 10⁵ iterations. For optimization, we used the Adam method [43] with ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $\epsilon ={10}^{-8}$. The generator and discriminator were updated alternatively. We implemented these networks using Python 3.7.6 with the PyTorch 1.0.1 framework and trained them using NVIDIA 2080 Ti GPU.

2.3. Landsat and Sentinel-2 Datasets

We selected study areas spatially distributed across different landscapes of the U.S., to provide more diverse terrestrial land cover information for training and testing (Figure 3a). A total of 11 cloud-free Landsat and Sentinel-2 imagery pairs from 2016 to 2019 were collected and downloaded from the Google Earth Engine (https://earthengine.google.com/) data archive (

Table 1. Acquisition tiles and dates of paired Landsat and Sentinel-2 imagery for experimental tests.

	Landsat-8		Sentinel-2			Landsat-8		Sentinel-2
	Scene	Date	Scene	Date		Scene	Date	Scene	Date
#1	044033	2016-07-13	10SFH	2016-07-14	#7	046029	2018-08-18	10TDQ	2018-08-19
#2	044034	2016-07-13	10SEG	2016-07-14	#8	034032	2018-09-15	13TDE	2018-09-14
#3	041036	2017-12-18	11SLT	2017-12-15	#9	038032	2018-11-14	12TVL	2018-11-14
#4	021035	2018-04-29	16SEF	2018-04-28	#10	016041	2018-12-06	17RML	2018-12-05
#5	019036	2018-05-01	16SGD	2018-04-30	#11	027040	2019-01-01	14RNT	2019-01-05
#6	024030	2018-07-07	15TYH	2018-07-08

, Figure 3a). We identified Landsat and Sentinel-2 imagery pairs that both had less than 5% cloud cover and were less than 5 days apart.

We extracted the overlapping areas for each pair of Landsat and Sentinel-2 imagery, and further cropped them into patch-based Landsat and Sentinel-2 imagery pairs with a unified patch size (160 × 160 for Landsat and 480 × 480 for Sentinel-2 in this study). These cropped subsets of Landsat and Sentinel-2 imagery pairs (Figure 4) were then used as input of the CNN models.

We collected three additional groups of datasets for time-series super-resolution experiments in capturing intra-annual land cover dynamics with fine spatial resolutions (

Table 2. Acquisition of Landsat-8 imagery for time-series super-resolution experiments.

Massachusetts (Scene: 012031)			California (Scene: 043034)
#1	2014-03-18	#1	2014-01-22	#6	2014-06-15
#2	2014-04-03	#2	2014-02-23	#7	2014-07-01
#3	2014-05-21	#3	2014-03-11	#8	2014-08-18
#4	2014-08-25	#4	2014-04-28	#9	2014-09-03
#5	2014-09-26	#5	2014-05-14	#10	2014-10-05

). We acquired a series of Landsat-8 imagery in the states of Massachusetts and California to enhance their spatial resolutions to better capture phenology changes over vegetated area, cropping rotations over agricultural areas, and water dynamics over reservoirs.

To test the ultimate goal of transforming 30 m historical Landsat to 10 m Sentinel-like archive, we selected Schonefeld, Dubai, and Las Vegas as worldwide experimental sites (Figure 3b) to super-resolve annual Landsat imagery from 1985 to 2018.

3. Experimental Tests and Results

3.1. Tests with Simulated Data

In this experiment, cropped Sentinel-2 images with the size of 480 × 480 × 3 were used as I_HR images. The corresponding I_LR images (160 × 160 × 3) were obtained by upscaling Sentinel-2 images by a factor of three with the bicubic interpolation. A total of 1220 I_LR–I_HR image pairs were randomly split, with 90% of them used for training and 10% for validation. It should be noted that the validation dataset was used to provide an unbiased evaluation while tuning model hyperparameters. A selection of an additional 57 testing images were used to evaluate the final model performance.

We selected two sites out of the 57 testing images to visually compare the spatial details and spectral fidelity between the original Sentinel-2 image and reconstructed 10 m Sentinel-2-like images (Figure 5). Compared with the reference images (Figure 5a,i), the interpolation from the upscaling Sentinel-2 image (Figure 5b,j), non-GAN-based method (Figure 5c,k) and GAN-based method (Figure 5d,l) had very similar spectral characteristics. The scatterplots of the red–green–blue bands between the reference and reconstructed images at site A (Figure 6a) and site B (Figure 6b) quantitatively verified their high correlation in spectral signatures. Non-GAN derived images (r = 0.94 for site A and r = 0.96 for site B) achieved comparably higher correlation than both the interpolated (r = 0.91 for site A and r = 0.91 for site B) images and the GAN derived images (r = 0.90 for site A and r = 0.93 for site B). In contrast to the similar spectral characteristics, the reconstructed spatial details were considerably different among these methods. Compared with the true reference (Figure 5e,m), the interpolated images (Figure 5f,n) were very blurred, and it was very difficult to identify the exact road network and infrastructure edges. As shown in the zoomed-in subsets for both sites (Figure 5e–h,m–p). In contrast, the non-GAN derived images (Figure 5g,o) improved the spatial details to a certain degree, but blurring effects still existed. The GAN derived images (Figure 5h,p), on the other hand, retained the fine-resolution spatial details that were nearly identical to the original Sentinel-2 images. For example, the road network, building edges, and land parcel blocks identified from the original and reconstructed ones were quite similar through visual inspections.

We used a group of full-reference indices including quality index (QI) [44], PSNR, root-mean-square-error (RMSE), and ERGAS [45], and non-reference indices including NIQE [46], PIQE [47], and BRISQUE [46] to assess the model performance of the 57 testing images. Based on the quantitative measures (

Table 3. Model performance in the simulated data experiment assessed by full-reference and non-reference measures.

Measures	Bicubic	Non-GAN	GAN
QI	0.647 ± 0.060	0.775 ± 0.051	0.669 ± 0.066
PSNR	39.197 ± 3.600	41.977 ± 3.987	39.391 ± 3.856
RMSE	0.012 ± 0.005	0.009 ± 0.004	0.012 ± 0.005
ERGAS	12.212 ± 3.973	9.027 ± 3.294	11.903 ± 4.266
NIQE	5.995 ± 0.507	4.839 ± 0.776	2.911 ± 0.497
PIQE	78.008 ± 8.498	61.687 ± 9.048	24.509 ± 14.898
BRISQUE	52.079 ± 3.413	40.553 ± 5.194	25.257 ± 7.568

), non-GAN approach achieved a slightly better performance in retaining spectral fidelity than the traditional interpolation and GAN approach. In contrast, the non-reference indices revealed that the GAN derived results had much better spatial details than the interpolated and non-GAN derived results. For example, the score of the indicators such as NIQE, PIQE, and BRISQUE from GAN derived results were much lower than those from the interpolated and non-GAN derived results.

3.2. Tests with Real Data

Different from the simulated data experiment in Section 3.1, Landsat images were directly used as I_LR and Sentinel-2 images were used as I_HR in this experiment. Similarly, we used 90% out of 1220 Landsat and Sentinel-2 I_LR–I_HR image pairs for training, 10% for validation, and an additional 57 image pairs for testing the final model performance. In addition to the interpolated and non-GAN derived methods, we included another five methods as baselines to compare against, including the smoothing filter based intensity modulation (SFIM) [48], high pass filtering (HPF) [49], band-dependent spatial-detail (BDSD) [50], area-to-point regression kriging (ATPRK) [31,51], and target-adaptive CNN-based (TCNN) algorithms [52,53]. The selected five methods all required the guidance of 10 m Sentinel-2 imagery to downscale the corresponding 30 m Landsat imagery.

We again selected two sites out of the 57 testing images to visually compare the original Sentinel-2 image and reconstructed 10 m Sentinel-2-like images (Figure 7). The spectral signatures of the reference images, the interpolated Landsat, SFIM derived, HPF derived, BDSD derived, ATPRK derived, TCNN derived, non-GAN derived, and GAN derived 10-m Sentinel-2 images were similar in general (Figure 7). However, we could still identify the obvious spectral distortions from the SFIM, HPF, and BSSD derived results (Figure 7c–e,u–w). In contrast, the reconstructed images showed considerable differences in spatial details, which was reflected from the zoomed-in subsets in Figure 7j–r for site A and Figure 7ab–aj for site B. The derived images from the ATPRK, TCNN, and GAN methods retained the fine spatial details relatively well. The reconstructed images were almost identical to the original Sentinel-2 images; we could clearly identify the building blocks and roads.

We used the same set of quantitative measures to assess the model performance of the 57 testing images in a real-data experiment. The quantitative results (

Table 4. Model performance in the real data experiment assessed by full-reference and non-reference measures.

Measures	Bicubic	SFIM	HPF	BDSD	ATPRK	TCNN	Non-GAN	GAN
QI	0.34 ± 0.10	0.65 ± 0.19	0.63 ± 0.10	0.57 ± 0.15	0.85 ± 0.07	0.54 ± 0.10	0.65 ± 0.14	0.37 ± 0.15
PSNR	29.71 ± 1.72	30.23 ± 1.54	30.26 ± 1.53	30.43 ± 1.37	31.43 ± 1.38	29.80 ± 1.83	41.50 ± 6.03	37.14 ± 5.75
RMSE	0.04 ± 0.01	0.04 ± 0.01	0.03 ± 0.01	0.03 ± 0.01	0.03 ± 0.01	0.04 ± 0.01	0.01 ± 0.01	0.02 ± 0.02
ERGAS	76.97 ± 30.19	75.30 ± 30.90	74.83 ± 30.80	73.63 ± 31.33	71.06 ± 31.73	74.00 ± 27.24	12.03 ± 15.09	18.66 ± 14.99
NIQE	6.43 ± 0.81	4.45 ± 0.71	4.44 ± 0.66	4.50 ± 1.46	3.36 ± 0.75	3.79 ± 0.59	5.37 ± 0.86	3.40 ± 0.70
PIQE	87.11 ± 11.85	20.89 ± 11.03	30.36 ± 11.55	50.92 ± 24.69	29.85 ± 11.74	31.39 ± 11.84	64.68 ± 12.42	28.62 ± 11.37
BRISQUE	52.63 ± 3.46	30.45 ± 6.04	35.09 ± 6.32	42.71 ± 10.35	31.28 ± 6.43	31.70 ± 5.11	44.25 ± 4.37	30.27 ± 6.33

) revealed that the non-GAN-based model was better at retaining spectral fidelity, while the GAN-based model could construct more detailed spatial high frequency patterns. Noticeably, the ATPRK based model also achieved a comparable performance in persevering spatial and spectral information. However, it should be noted that the implementation of SFIM, HPF, BDSD and ATPRK models requires the input of corresponding Sentinel-2 images as the downscaling guidance, whereas the input of non-GAN and GAN-based models is only the 30 m Landsat imagery. On this basis, results shown in Figure 7 and Table 4 further reinforce the potential value of our proposed methods.

3.3. Tests with Time-Series Data

We further applied the GAN-based model trained from the real data in Section 3.2 to the time series of Landsat imagery across different landscapes, in order to test its performance of capturing inter-annual land cover dynamics, including phenology changes over vegetated area, cropping rotations over agricultural area, and water dynamics.

A selection of cloud-free Landsat-8 imagery was acquired in Massachusetts, where seasonal phenology changes are significant (Figure 8). The major land-cover types were forest, grass, water, bare land, and buildings for both sites A and B. Compared with the interpolated Landsat images (left panels of Figure 8a,b), the reconstructed Sentinel-2 like images (right panels of Figure 8a,b) greatly improved the spatial details while preserving spectral consistency.

A selection of monthly Landsat-8 imagery from January to October was acquired in California, where agricultural activities were dominant and cropping rotation was frequent (Figure 9). By applying the GAN-based model, we derived the corresponding 10 m Sentinel-2-like images (right panels of Figure 9. Results showed that the reconstructed images accurately captured crop phenology changes and depicted double cropping rotations (i.e., the first was from January to April, and second was from May to October). Given the relative homogeneity of land cover types in this region, the enhanced spatial details of reconstructed Sentinel-2-like images did not show considerable differences from those of the interpolated Landsat images through visual inspections. However, the intra-class difference in growing status and phenology changes could be easily identified from the reconstructed Senitnel-2-like images, whereas this was very difficult from the interpolated Landsat images.

An additional selection of monthly Landsat-8 imagery from January to October was acquired in California, where water bodies were dominant, with certain changes in spatial coverage and water colors (Figure 10). Compared with the interpolated Landsat images (left panels of Figure 10), the reconstructed Sentinel-2-like images (right panels of Figure 10) captured spatial details much more accurately without any blurring effects. For example, the inundation areas on August 18 and September 3 were depicted with fine spatial details in both intra-class variation and adjacent boundaries for Sentinel-2-like images, while the corresponding Landsat images only captured the general distribution of inundations with obvious blurs.

3.4. Reconstruction of 10 m Historical Landsat Archive

To address our ultimate goal of transforming 30 m historical Landsat images to a 10 m Sentinel-2-like archive, we acquired annual Landsat imagery from 1985 to 2018 in Schonefeld, Dubai, and Las Vegas (Figure 3b), where land cover changes were extreme (Figure 11). In Schonefeld, Germany, cropland was converted into artificial impervious areas for airport construction. In Dubai, United Arab Emirates, coastal expansion was tremendous, with the creation of Palm Islands and extreme urbanization. In Las Vegas, U.S.A., bare lands were rapidly converted to urbanized areas over the past decades. We applied the GAN-based models to generate the time-series 10 m Sentinel-2-like images over the selected three regions. Given the fact that there were no real 10 m Sentinel-2 images available before 2015, we visually compared the image quality in terms of spectral fidelity and spatial details between the interpolated Landsat and the reconstructed Sentinel-2-like images. As shown by the derived results with five-year intervals from 1985 to 2018 in Figure 11, the reconstructed images preserved consistent spectral fidelity and achieved plausible spatial details. We could also identify the very close spatial details between the predicted Sentinel-2-like imagery and the observed Sentinel-2 imagery in 2018. The zoomed-in comparison (Figure 12) further demonstrated that the improvement of spatial details using the GAN-based model was very obvious and robust over the long-term temporal period. For example, the road networks, building infrastructures, and street blocks were quite blurred in original Landsat images (left panels of Figure 12a–c), whereas they turned out to be quite clear in the reconstructed Sentinel-2-like images (right panels of Figure 12a–c).

4. Discussion

4.1. Training and Tuning of Deep Learning Models

Deep learning approaches are always complex in terms of neural network structures. They may be sensitive to the training and tuning process and the performance may vary on landscapes with varying spatial heterogeneity and temporal dynamics. These pose challenges for fusing remote sensing imagery with multispectral and multiscale observations. Here, we briefly discuss four important aspects of applying non-GAN and GAN-based deep learning SR models, including the training dataset, model generalization ability, potential benefit from transfer learning, and the impact of the number of iterations.

Firstly, training data are a key component to the machine learning algorithms that depend on the input reference and memorize the statistical and structured information for future predictions. Models usually perform better on the test set that have similar distribution as the training set. Therefore, the increase in the number and variability of training samples typically leads to better performance of the deep learning models. This calls for future efforts to expand and consolidate the regional to global training library that covers more comprehensive land use/cover type and changes.

Secondly, the model generalization ability describes a model’s ability to make accurate predictions on the new dataset that it has never encountered during the training. It is a key element of a successful and robust model. In this study, a number of Landsat and Sentinel-2 imagery pairs were collected across different landscapes in the United States. In Section 3.1 and Section 3.2, although training and testing datasets were randomly split without any overlapped information, they shared similar land cover types between adjacent locations. Therefore, similar patterns from training could be well transferred to the test dataset, making predictions over the testing data less challenging. However, the test dataset collected in the experiments using seasonal and long-term time-series imagery (Section 3.3 and Section 3.4) was totally independent from the training dataset; our model performed similarly and was able to accurately predict and capture the land-cover dynamics with high spectral fidelities and spatial details, demonstrating that our model generalizes very well. It further verified the possibility of leveraging limited samples of Landsat and Sentinel-2 observations during their overlapping period from 2016 to 2019 to transform 30 m historical Landsat images into a 10 m Sentinel-2-like archive.

Thirdly, transfer learning which makes use of all parameters of neural network pre-trained over large datasets has been proven to be quite helpful and efficient to solve a different but related problem [54]. It usually has the advantage of using fewer training samples and time while obtaining better results. To test whether remote sensing super-resolution could benefit from initial weights based on a model pre-trained from a natural image dataset, we conducted a group of experimental tests. In this experiment, the GAN-based network were first trained on DIV2K dataset [55], and then fine-tuned using remote sensing images at a smaller learning rate of 10⁻⁵, because the setting of a smaller learning rate performs better in fine-tuning. Comparative results revealed that the integration of natural images did not improve model performance (Figure S1, Table S2). This can be attributed to the difference in spectral wavelength and spatial details between remote sensing imagery and natural images.

Finally, the number of iterations is another critical issue that influences deep learning-based model accuracy. As shown in Figure 13a, the accuracy of the non-GAN based model was relatively much lower with 5000 to 200,000 iterations, increased rapidly with the increasing number of iterations, and then became stable with more than 400,000 iterations. Similar patterns were found for GAN-based models (Figure 13b). We noticed that the GAN-based model was less stable when compared to the non-GAN-based model, which reflects the fact that GAN is more sensitive to training than a normal deep CNN [32]. The sensitivity analysis provided some semi-empirical knowledge that could be very helpful in training and tuning the optimal deep learning models.

4.2. Future Prospect

There are so far no existing algorithms that can produce super-resolution images with both high quantitative accuracy and high perceptual quality at the same time. Non-GAN based methods, optimizing the pixel-wise reconstruction measure, tend to produce over-smoothed images that lack high-frequency textures, and have also been shown to correlate poorly with human perception [34,56]. In contrast, GAN-based methods achieve very good performance in reconstructing high-frequency spatial details but are often inferior in terms of distortion measure [34,39,56,57]. This leads to a common perception–distortion trade-off phenomenon. On one hand, the balance of quantitative accuracy and spatial details can be application-oriented depending on the specific purpose. On the other hand, although this trade-off cannot be addressed thoroughly [58], future work can be conducted to alleviate this dilemma. For example, in our model generator design, adding an L₁ regularization term to take the pixel level differences between the true and generated images into account forced the generated super-resolved images close to its corresponding $I_{HR}$. Therefore, the accuracy increased, while the perceptual quality was preserved. Designing more advanced loss functions is also another promising direction.

We applied the GAN-based super-resolution algorithm to leverage Landsat and Sentinel-2 observations during their overlapping period from 2016 to 2019 for transforming 30 m historical Landsat images into a 10 m Sentinel-2-like archive. Although the experimental results in both intra- and inter-annual scales showed robust performance of our proposed method (Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12), our methods and results should be interpreted in light of certain uncertainties. Firstly, the difference in sensors, observation geometries, and image quality may lead to potential biases in spatial details and spectral fidelity of the time-series reconstructed Landsat imagery. For example, the possible lower image quality of the input data, e.g., due to cloud/shadow contaminations and geometry distortions, may cause errors especially for long-term temporal reconstruction applications. In addition, although our proposed methods can be easily extended to other sensors such as National Agriculture Imagery Program (NAIP) imagery (Figure S2), RapidEye, and PlanetScope datasets, careful consideration should be made with regard to the scale discrepancy between two paired images. In this study, with a factor of three between the Landsat and Sentinel-2 pixel sizes, the reconstruction of super-resolution worked very well. However, if the scale difference is larger than eight, the single-image super-resolution may yield unrealistic and blurred results [59]. In this situation, we may use medium-resolution images between the coarse- and high-resolution images as the bridge to close the scale gap.

Despite these uncertainties, our study demonstrated that the fusion methods developed here and corresponding reconstructed Sentinel-2-like archive are very promising, and can be applied to a variety of other applications, such as urban, forest, wetland, wildfires, land cover classification and change detection. Potentially, all research and applications that traditionally rely on Landsat but require higher spatial details can be enhanced with the support of our feature-level fusion methods and resulting reconstructed Sentinel-2-like data archive.

5. Conclusions

Fine spatial resolution Earth observation images are desirable for much environmental change and land management research. However, the investigation of long-term terrestrial change has mostly relied on 30 m moderate-resolution Landsat imagery. This study sought to leverage Landsat and Sentinel-2 observations during their overlapping period from 2016 to more up-to-date images, to reconstruct 10 m Sentinel-2-like imagery from 30 m historical Landsat data since the Sentinel-2 era, and potentially over the past four decades. To achieve this goal, we presented a generative adversarial network (GAN) based data fusion framework that blended fine-resolution feature-level spatial information from Sentinel-2, to enhance the spatial resolution of the historical Landsat archive. We conducted a set of experimental tests using simulated data, actual Landsat/Sentinel-2 acquisitions, and time-series Landsat imagery. Our results showed Landsat data which were reconstructed to 10 m using a GAN-based super-resolution method, accurately captured the spatial features at an effective resolution close to Sentinel-2 observations. Historical reconstruction of 10 m Landsat observations from 1985 to 2018 in three different types of landscape changes further verified the robustness and efficiency of our proposed method for monitoring land changes. The promising results from our deep learning-based feature-level fusion method highlight the potential for reconstructing a 10 m Landsat series archive since 1984. The enhanced Landsat and Sentinel-2 data is expected to improve our capability of detecting and tracking finer scale land changes, identifying the drivers for and responses to global environmental changes, and designing better land management policies and strategies.