Fusing Sentinel-2 and Landsat 8 Satellite Images Using a Model-Based Method

Abstract

The Copernicus Sentinel-2 (S2) constellation comprises of two satellites in a sun-synchronous orbit. The S2 sensors have three spatial resolutions: 10, 20, and 60 m. The Landsat 8 (L8) satellite has sensors that provide seasonal coverage at spatial resolutions of 15, 30, and 60 m. Many remote sensing applications require the spatial resolutions of all data to be at the highest resolution possible, i.e., 10 m for S2. To address this demand, researchers have proposed various methods that exploit the spectral and spatial correlations within multispectral data to sharpen the S2 bands to 10 m. In this study, we combined S2 and L8 data. An S2 sharpening method called Sentinel-2 Sharpening (S2Sharp) was modified to include the 30 m and 15 m spectral bands from L8 and to sharpen all bands (S2 and L8) to the highest resolution of the data, which was 10 m. The method was evaluated using both real and simulated data.

1. Introduction

In 2013, the Landsat 8 satellite was launched, equipped with an Operational Land Imager (OLI) and thermal infrared (TIR) sensors. The L8 OLI provides seven spectral bands at a 30 m spatial resolution and one panchromatic band at a 15 m resolution. The revisit rate of the L8 is 16 days and, along with the fact that the area being imaged by the L8 often has cloud coverage, this means that time-series L8 data are often very sparse. In 2015 and 2017, the European Space Agency launched the Sentinel-2A and 2B satellites, respectively. The S2 multispectral imager (MSI) provides thirteen spectral bands: four bands at a spatial resolution of 10 m, six bands at 20 m, and three bands at 60 m.

In Figure 1 [1], the wavelengths and placements of the S2 MSI bands and the L8 OLI bands are shown. The spatial resolution of the S2 bands is higher (10 m or 20 m) than the resolution of the corresponding L8 bands (30 m). The facts that the S2 and L8 cover similar wavelengths and the data from both of these satellites share the same geographic coordinate system make them excellent candidates for data fusion. Many remote sensing applications require all data that are used to be at the same resolution. To address this requirement, considerable effort has been put into the subject of super-resolving S2 bands so that they all have a 10 m spatial resolution [2,3,4,5,6,7,8,9,10,11].

Together, the S2-A and S2-B satellites provide a revisit rate of 5 days [12] worldwide. This high revisit rate makes S2 data very useful for many applications, such as environmental monitoring and risk and disaster mapping.

Within the remote sensing community, there has been a long history of data fusion [13,14,15,16,17,18,19,20,21]. This is an important remote sensing topic and with the rapid development of remote sensing technologies and sensors, the importance of image fusion techniques has only increased. The need for methods that are capable of making use of data that were acquired from different sensors is very important and beneficial as they could facilitate new research and improve results. The wide coverage and frequent revisit rate of the S2 make it a good candidate for data fusion [22,23,24]. In [25], 5-day L8 OLI and harmonized surface reflectance data from the S2 MSI were combined and resampled to a 30 m resolution using the S2 tiling system.

In [26,27], S2 and Sentinel-1 (S1) data were combined. In [28], the comparative performance of decision- and pixel-level data fusion ensemble classifier maps using S2, L8, and Landsat 7 data was evaluated. A self-supervised framework for SAR–optical data fusion and land cover mapping tasks was also proposed, in which SAR and optical images were fused using a multiview contrastive loss at the image level and super-pixel level. In [29], the 10 m and 20 m S2 reflectance bands were sharpened using the 3 m Planetscope reflectance bands, providing a potential method for improved global monitoring. In the study, two well-established and computationally efficient sharpening methods were used. Additionally, both of the spectrally non-overlapping bands between the two sensors and the surface changes that could occur between acquisitions were examined. In [30], a methodological approach to fusing S2 and S1 data for the land cover mapping of the Lower Magdalena region in Colombia was proposed. First, the SEN2COR toolbox from ESA was used to preprocess the data, followed by land cover–land use classification using support vector machines. In [31], numerical regression was used to fuse spectral indices, based on high-resolution images from an unpiloted aerial vehicle and S2 images. In [32], an attentional super-resolution convolutional neural network was used to fuse S2 and L8 data to create a 10 m resolution NDVI (normalized difference vegetation index) time series, which it was evaluated using two heterogeneous areas. Inspired by deep learning, an extended super-resolution convolutional neural network framework was also developed in [33] to fuse S2 and L8 images.

The main contribution of this paper is the proposal of a method that jointly sharpens both S2 and L8 data to a resolution of 10 m. The method was developed by extending the S2Sharp (https://github.com/mou12/S2Sharp, accessed on 10 January 2022) method, which was proposed in [7]. Other contributions of this paper include the estimation process of the point spread function for L8 data, which is needed for the proposed method, and the way in which the 15 m L8 panchromatic band is used in the sharpening process.

2. The Method

The method for sharpening S2 data, that was presented in [7] (S2Sharp), was modified to sharpen both S2 and L8 data. S2Sharp sharpens both the 20 m and 60 m S2 bands to a spatial resolution of 10 m. The method belongs to a family of variational approaches that assume that multiresolution acquisitions can be represented using a linear acquisition model.

The method uses projections onto a low-dimensional subspace that are automatically estimated in the optimization process. The minimization problem that S2 solves is non-convex and cyclic descent is used for the optimization. The cost function that S2Sharp minimizes is:

$$J(\mathit{U},\mathit{Z})=\sum _{i=1}^{L_{S2}}\frac{1}{2}{‖{\mathit{y}}_i-{\mathit{M}}_i{\mathit{B}}_i\mathit{Z}{\mathit{u}}_{\left(i\right)}‖}^2+\sum _{j=1}^p{\lambda }_j{\varphi }_w\left({\mathit{z}}_j\right),$$

(1)

where $L_{S2}$ is the number of channels ($L_{S2}=12$, i.e., all S2 bands except for band number 10), ${\mathit{y}}_i$ is the (vectorized) S2 image observed at band i, $\mathit{Z}{\mathit{u}}_{\left(i\right)}$ is an estimate of the full-resolution images at band i, ${\mathit{M}}_i$ is a downsampling operator, ${\mathit{B}}_i$ is a blurring operator (according to the S2 sensor specifications), $\mathit{Z}$ is an $n\times p$ full-rank matrix ($p<L_{S2}<<n$), $\mathit{U}={\left[{\mathit{u}}_{\left(i\right)}^T\right]}_{i=1}^{L_{S2}}$ is an $L_{S2}\times p$ orthonormal matrix that spans a p-dimensional subspace in ${\mathbb{R}}^{L\times 1}$, and n is the number of pixels in the highest resolution bands. The regularization term ${\varphi }_w$ is a spatial regularization term, which is controlled by the tuning parameters ${\lambda }_j$.

To be able to sharpen both S2 and L8 data, we introduced a new method called Sentinel-2 Landsat 8 Sharpening (SLSharp), which was based on S2Sharp. To accomplish this, the following needs had to be addressed:

(1)

The cost function used by S2Sharp (1) needed to be modified to include the L8 bands;

(2)

The point spread function (PSF) of the L8 data needed to be estimated;

(3)

The 15 m panchromatic band of the L8 data needed to be resampled.

The first task was to modify the S2Sharp cost function (1) in the following way:

$$J_m(\mathit{U},\mathit{Z})=\sum _{i=1}^L\frac{1}{2}{‖{\mathit{y}}_i-{\mathit{M}}_i{\mathit{B}}_i\mathit{Z}{\mathit{u}}_{\left(i\right)}‖}^2+\sum _{j=1}^p{\lambda }_j{\varphi }_w\left({\mathit{z}}_j\right),$$

(2)

where $L=L_{S2}+8=20$ and ${\mathit{y}}_i,i>12$ are the L8 OLI bands. This extension of S2Sharp was similar to the extension that was carried out in [21], but it was not as straightforward. The second task was to estimate the PSF of each L8 band. S2Sharp assumes that the PSF is a symmetrical 2D Gaussian function and relies on the variance of the Gaussian function. For the S2 data, the variance is calculated [4] using a calibrated modulation transfer function (MTF), which is supplied by the S2 technical guide [34] for each band. The L8 OLI spatial response for each band is represented in terms of two separable along- and across-track line spread functions [35]. To estimate the PSF for the L8 data, the same assumption was made as for the S2 data, i.e., that the PSF was a symmetrical 2D Gaussian function. The average spatial response for each band was calculated by averaging the along- and across-track OLI spatial responses; then, the variance of the Gaussian function was found using a least squares estimation. In Figure 2, the OLI spatial response for band number 6 is shown, along with the estimated Gaussian function. The spatial response functions for the other L8 bands were similar.

Another issue with sharpening the L8 images was that the SLSharp method was only able to sharpen bands that had a spatial resolution that was an integer multiple of 10 m. The panchromatic band of the L8 data had a spatial resolution of 15 m, which is not an integer multiple of 10 m. The third task was to solve this issue. We downsampled this band using bicubic interpolation to a 20 m spatial resolution and then the method was able to sharpen the band to 10 m. This was not optimal as the downsampling operation threw out some information from the panchromatic band. To alleviate this issue, we also upsampled the panchromatic band using bicubic interpolation to achieve a resolution of 10 m. By doing these upsampling and downsampling operations, the method could sharpen the 15 m panchromatic band of the L8 data without discarding any information. The downside to this approach was that two bands with 10 m and 20 m resolutions were used instead of one 15 m panchromatic band, which slightly increased the required computations.

3. Evaluation

To evaluate SLSharp qualitatively, we used S2 and L8 images of southwest Iceland. Both images were acquired on 7 July 2019. The S2 image was acquired at 13:14 UTC and the L8 image was acquired at 12:46 UTC. A 288 × 288 pixel crop of the whole image was used. We also used S2 and L8 images of Glen Canyon Dam and its surroundings in Arizona, USA. The S2 image was acquired on 12 December 2021 at 18:24 UTC and the L8 image was acquired on 1 December 2021 at 18:03 UTC. A 540 × 540 pixel crop from the whole image was used. We refer to the images from Iceland and Arizona as IS and AZ images, respectively. These two sets of images were chosen because they were cloud-free, contained interesting details, and were heterogeneous. In Figure 3, the RGB images that were generated using bands 2–4 of the S2 and L8 images are shown.

The S2 RGB images were much sharper since the spatial resolution of bands 2–4 was 10 m compared to the 30 m resolution of the L8 bands. The regions of interest (ROIs) are within the yellow rectangles. Within the IS ROI is an urban area with clearly visible houses and a shoreline. Within the AZ ROI, the Glen Canyon Dam is clearly visible. To obtain the best results from SLSharp, a number of tuning parameters needed to be estimated and to do so, the method proposed in [36,37] was used. The parameters that needed to be estimated were p, the rank of the subspace, and the spatial regularization parameters ${\lambda }_j$ and $j=1,\dots ,p$.

The results obtained using SLSharp were compared to a modified version of SupReME, which is the S2 sharpening method presented in [4]. SupReME makes the same assumption as SLSharp, i.e., the PSF is a uniform 2D Gaussian function, so the variance in the L8 spectral response that was described in Section 2 could also be used to modify SupReME so that it was able to sharpen the L8 data. This modified version of SupReME, denoted here as the baseline method, served as a baseline for the performance of SLSharp. The results were also compared to the area-to-point regression kriging (ATPRK) [38] fusion method. The ATPRK fusion method was implemented by the authors of this paper as described in [38], using the code from the GitHub page of the ATPRK authors [39]. Figure 4 and Figure 5 show the sharpened results of the IS and AZ ROIs, respectively (S2 bands 2, 5, and 7; L8 bands 1, 5, and 8).

By examining the results within the ROI in Figure 4 and Figure 5, it could be seen that SLSharp outperformed the ATPRK and baseline methods in all bands. The images that were obtained using SLSharp had more detail and the outlines of the buildings were crisper and had fewer artifacts.

A quantitative evaluation of the relative dimensionless global error (ERGAS), root mean square error (RMSE), spectral angle mapper (SAM) in degrees, signal-to-reconstruction error (SRE) in dB, structural similarity index measure (SSIM), and universal image quality index (UIQI) metrics was performed using the same images at reduced scales, with Wald’s protocol for the IS and AZ images as well as a synthesized image of Escondido [40,41,42]. The images were first reduced by 2× and 6×. Bayesian optimization was then used to find the set of parameters that yielded the best results when resolving the 20 m and 60 m bands back to their original resolution. These parameters were then used to super-resolve the original full-scale images.

The Escondido image was simulated from a fine resolution AVIRIS image using methods that were similar to those in [7,8]. The original AVIRIS image of an area near San Diego, CA, USA, was acquired in November 2011. The image was 790 × 6796 pixels in 224 bands that spanned a spectral range of 366–2496 nm and covered the range of the S2 and L8 OLI sensors.

Reduced resolution versions of all bands were generated at 2× pixel size, 3× pixel size, and 6× pixel size. This was carried out by applying appropriate filters to simulate the PSF of each S2 and L8 OLI band before downgrading them by a factor of 2, 3, and 6, respectively.

The S2 and L8 versions were simulated by applying the S2 and L8 OLI spectral responses to a patch of the AVIRIS data, which depicted parts of the city of Escondido and its suburbs. Twenty bands at a spatial resolution of 5 m were filtered, subsampled and cropped to yield a 288 × 288 pixel ground truth for each band at a 10 m resolution. The bands were then filtered using the estimated Gaussian PSF of the S2 and L8 sensors and downgraded to yield four bands at a 10 m resolution, a single band at a 15 m resolution, six bands at a 20 m resolution, seven bands at a 30 m resolution, and two bands at a 60 m resolution. The 15 m panchromatic band of the L8 data was then upsampled and downsampled to a 10 m and 20 m resolution, as before, for use with SLSharp. For comparison, the baseline and ATPRK methods were also evaluated using the same data.

SLSharp performed better, in terms of sharpening most of the bands of the reduced scale images, in all metrics apart from UIQI, for which ATPRK had a slight advantage in the S2 bands and bands L1 and L7. However, the average UIQI that was obtained using SLSharp was lower. The simulated Escondido image results were more band-dependent. By examining the Escondido results in

Table 1. The ERGAS, RMSE, and SAM values for the reduced scale images of IS, AZ, and the simulated image of Escondido. Lower values are better. The columns are labeled A for ATPRK, B for baseline, and S for SLSharp. The best results are shown in bold.

		Escondido			AZ			IS
		A	B	S	A	B	S	A	B	S
ERGAS	20 m	7.57	7.31	7.75	7.64	6.17	5.99	19.36	12.42	12.10
	30 m	4.40	1.86	2.67	7.47	4.93	4.78	14.87	9.19	8.64
	60 m	1.28	0.69	1.48	4.03	3.30	2.73	7.07	3.97	3.18
RMSE	20 m	0.01	0.01	0.01	0.03	0.03	0.02	0.04	0.03	0.03
	30 m	0.01	0.00	0.00	0.06	0.04	0.04	0.05	0.03	0.03
	60 m	0.00	0.00	0.00	0.05	0.05	0.04	0.07	0.04	0.03
	All	0.01	0.01	0.01	0.05	0.04	0.03	0.05	0.03	0.03
SAM	20 m	8.24	8.00	7.91	4.93	4.73	4.81	8.53	7.48	8.00
	30 m	4.08	1.96	1.73	11.35	3.47	4.38	12.97	6.01	6.23

and

Table 2. The SRE, SSIM, and UIQI scores for the reduced scale images of IS, AZ, and the simulated image of Escondido. Higher values are better. The columns are labeled A for ATPRK, B for baseline, and S for SLSharp. The best results are shown in bold.

	SRE									SSIM									UIQI
	Escondido			AZ			IS			Escondido			AZ			IS			Escondido			AZ			IS
	A	B	S	A	B	S	A	B	S	A	B	S	A	B	S	A	B	S	A	B	S	A	B	S	A	B	S
B5	20.32	23.64	25.86	16.28	19.52	19.14	12.70	14.18	14.25	0.97	0.98	0.99	0.87	0.92	0.92	0.90	0.91	0.91	0.88	0.90	0.97	0.79	0.85	0.85	0.66	0.57	0.57
B6	20.09	20.52	25.75	16.90	19.42	19.39	14.65	15.74	16.20	0.96	0.95	0.99	0.88	0.91	0.92	0.90	0.90	0.92	0.88	0.83	0.96	0.80	0.84	0.85	0.65	0.58	0.57
B7	20.46	21.04	29.02	17.02	19.43	19.55	14.88	16.25	16.78	0.96	0.96	0.99	0.89	0.91	0.93	0.90	0.91	0.93	0.90	0.87	0.99	0.80	0.84	0.85	0.66	0.59	0.60
B8A	22.15	22.95	27.18	17.19	19.27	19.49	14.98	16.26	16.74	0.98	0.98	0.99	0.89	0.91	0.93	0.90	0.91	0.93	0.92	0.90	0.97	0.80	0.84	0.85	0.66	0.61	0.60
B11	31.19	25.66	16.72	19.25	20.21	20.26	16.47	16.45	16.48	1.00	1.00	0.98	0.91	0.92	0.91	0.89	0.89	0.90	0.79	0.57	0.35	0.84	0.86	0.85	0.69	0.59	0.55
B12	48.57	25.03	16.71	18.72	19.50	19.77	14.80	14.33	14.09	1.00	1.00	0.99	0.90	0.91	0.91	0.89	0.86	0.86	0.86	0.11	0.05	0.83	0.84	0.84	0.69	0.56	0.53
L8d	9.46	9.72	9.69	16.90	17.59	18.82	5.16	14.36	14.86	0.89	0.90	0.89	0.85	0.84	0.91	0.70	0.94	0.95	0.84	0.86	0.85	0.79	0.75	0.85	0.34	0.69	0.73
20 m	24.60	21.22	21.56	17.46	19.28	19.49	13.38	15.37	15.63	0.96	0.97	0.98	0.88	0.90	0.92	0.87	0.90	0.91	0.87	0.72	0.73	0.81	0.83	0.85	0.62	0.60	0.59
L1	19.65	29.04	28.15	12.77	15.02	14.91	8.54	12.35	13.36	0.96	0.99	0.99	0.81	0.84	0.88	0.85	0.89	0.92	0.77	0.91	0.91	0.75	0.75	0.81	0.65	0.61	0.65
L2	17.39	27.52	31.73	13.10	15.71	15.70	8.77	12.92	13.73	0.94	0.99	1.00	0.80	0.84	0.88	0.85	0.90	0.93	0.79	0.95	0.98	0.76	0.77	0.83	0.68	0.67	0.71
L3	15.82	25.56	30.06	14.15	17.88	18.26	9.14	14.35	15.03	0.94	0.99	1.00	0.80	0.85	0.90	0.85	0.92	0.94	0.82	0.95	0.99	0.78	0.80	0.87	0.68	0.69	0.76
L4	14.77	23.91	33.70	14.43	19.49	20.28	9.05	14.76	15.26	0.94	0.99	1.00	0.78	0.86	0.91	0.84	0.92	0.93	0.84	0.95	1.00	0.78	0.83	0.89	0.65	0.65	0.72
L5	16.22	22.98	27.12	14.52	18.96	19.31	11.96	17.15	17.34	0.89	0.98	0.99	0.75	0.82	0.88	0.77	0.90	0.92	0.57	0.90	0.97	0.75	0.79	0.86	0.53	0.66	0.70
L6	30.77	25.68	17.15	14.68	19.08	19.87	12.93	16.17	16.31	1.00	1.00	0.98	0.75	0.81	0.89	0.82	0.87	0.88	0.79	0.58	0.36	0.75	0.79	0.86	0.57	0.63	0.66
L7	49.64	25.03	17.05	14.35	19.17	19.79	13.23	14.19	14.01	1.00	1.00	0.99	0.73	0.82	0.88	0.86	0.86	0.87	0.90	0.11	0.04	0.74	0.79	0.86	0.65	0.59	0.62
30 m	23.47	25.68	26.42	14.00	17.90	18.30	10.52	14.56	15.01	0.95	0.99	0.99	0.77	0.83	0.89	0.83	0.89	0.91	0.78	0.76	0.75	0.76	0.79	0.85	0.63	0.64	0.69
B1	23.34	27.78	26.27	12.66	14.73	16.31	11.40	15.07	17.75	0.98	0.99	0.99	0.84	0.86	0.92	0.80	0.89	0.94	0.77	0.89	0.87	0.84	0.85	0.92	0.66	0.84	0.81
B9	21.54	27.64	18.73	14.41	15.63	17.40	9.68	15.70	17.01	0.99	1.00	0.99	0.82	0.80	0.89	0.54	0.82	0.83	0.47	0.72	0.47	0.80	0.77	0.87	0.45	0.62	0.66
60 m	22.44	27.71	22.50	13.54	15.18	16.85	10.54	15.38	17.38	0.99	1.00	0.99	0.83	0.83	0.90	0.67	0.86	0.89	0.62	0.81	0.67	0.82	0.81	0.90	0.55	0.73	0.73
All	23.84	23.98	23.81	15.46	18.16	18.64	11.77	15.02	15.58	0.96	0.98	0.98	0.83	0.86	0.90	0.83	0.89	0.91	0.80	0.75	0.73	0.79	0.81	0.86	0.62	0.63	0.65

, the following was noted:

• The ATPRK method obtained better SRE, SSIM, and UIQI scores in the simulated NIR bands B11, B12, L6, and L7;
• The baseline method obtained better SRE, SSIM, and UIQI scores in the simulated 60 m bands B1 and B9;
• Although SLSharp performed the best in most of the simulated bands, the baseline method obtained the best ERGAS, SRE, and SSIM scores and the ATPRK method obtained a better UIQI score when averaged across all bands.

Although the results were generated using equivalent procedures, there were significant differences between the performances for the reduced scale images and the simulated image. All methods had better average scores when using the simulated image.

SLSharp was tuned to maximize the average SRE across all bands. Evidently, this did not necessarily maximize all of the other unrelated metrics or the SRE scores of individual bands. The differences between the baseline method and SLSharp for a few isolated bands in the AZ and IS images were negligible, apart from SAM, which is a measure of spectral quality and would not be reflected in the visual quality of each band when viewed spatially.

The evaluated metrics for which the baseline method achieved better scores (ERGAS, SRE, and SSIM) were not measures of perceptual image quality but rather of distortion, which has long been known to be an unreliable indicator of perceptual image quality [41,43]. This could be confirmed by comparing the 30 m L8 bands in Figure 4 and Figure 5, which look much better when processed with SLSharp than the baseline method but have identical RMSE scores, as shown in Table 1. For SSIM, the differences were very small for the bands for which the baseline method performed better (a single band (L6) showed a 0.02 difference, whereas the others were 0.01 or less), while SLSharp outperformed the baseline method in the other bands by up to 0.05.

4. Conclusions

In this paper, we proposed the S2 L8 Sharpening (SLSharp) method to fuse Landsat 8 and Sentinel-2 satellite data. The S2Sharp algorithm was modified to include the 30 m bands of the L8 OLI, as well as the 15 m panchromatic image. The method was used to sharpen all Landsat 8 OLI and Sentinel-2 bands to a 10 m resolution. The method was evaluated visually, using real images, and quantitatively, using reduced scale and simulated images. Using these data, it was shown that SLSharp could improve the results compared to the baseline and ATPRK methods.