Sentinel-2 Pan Sharpening—Comparative Analysis ^†

Abstract

Pan Sharpening is an important part of the Remote Sensing science. Obtaining high spatial resolution data can be crucial in some studies. Sentinel-2 provides data of 10, 20 and 60 m, and it is a promising program for Earth observation studies. Although Sentinel-2 provides a high range of multispectral bands, the lack of panchromatic band disables the production of a set of fine-resolution (10 m) bands. However, few methods have been developed for increasing the spatial resolution of the 20 m bands up to 10 m. In this study, three different methods of producing panchromatic bands have been compared. The first method uses the closest higher spatial resolution band to the lowest spatial resolution band as a panchromatic band, the second method uses one single band as the panchromatic band produced as an average value out of all fine resolution bands, while the third method uses linear correlation for the selection of the panchromatic band. The 60 m bands have not been taken into consideration in this study. In order to compare these methods, three image fusion techniques from different fusion subsections (Component substitution—Intensity Hue Saturation IHS; Numerical method—High Pass Filter HPF; Hybrid Technique—Wavelet Principal Component WPC) have been applied on two Sentinel-2 images over the same study area, on different dates. For the accuracy assessment, both qualitative and quantitative analyses have been made. It has been concluded that using the average value of the visual and the near infrared bands can be accepted as a panchromatic band in the Sentinel-2 dataset.

1. Introduction

Image fusion is combining different images in order to obtain a new image with the best characteristics from both images. The aim in remote sensing image fusion is to combine the spatial information from the higher spatial resolution image (panchromatic image) and spectral information from a multispectral image with a lower spatial resolution, in order to obtain a higher spatial multispectral image [1]. Most Earth observation satellites, such as Worldview and Landsat, acquire a high-resolution panchromatic image and lower spatial resolution multispectral images [2], which is why it is required to fuse the multispectral images and the panchromatic images in order to produce a higher spatial resolution multispectral image.

Sentinel-2 is a fine spatial resolution satellite imaging mission created by the European Space Agency (ESA). It consists of two twin satellites, Sentinel-2A, launched 23 June 2015, and Sentinel-2B launched 7 March 2017. Sentinel-2 images cover 13 bands in the visible, near-infrared (NIR), and shortwave infrared (SWIR) wavelengths. Its spatial resolution contains 10, 20 and 60 m bands. The Red, Green, Blue and NIR bands have 10 m spatial resolution, four Vegetation Red Edge and two SWIR bands have 20 m spatial resolution, while the Coastal aerosol, Water vapour, and Cirrus bands have 60 m spatial resolution. Thus, Sentinel-2 does not offer panchromatic band with high resolution. However, taking an advantage of the four fine spectral resolution bands, panchromatic band can be produced and used in the Sentinel-2 image fusion for producing ten fine spatial resolution bands [3]. Wang et al. [4] researched fusion of Sentinel-2 lower spatial resolution bands using multispectral bands at a higher spatial resolution in the role of a panchromatic band, while Selva et al. [3] proposed averaging all four fine resolution bands in order to create a panchromatic band. Gasparovic and Jogun [5] used the near-infrared band as a panchromatic band in order to increase the spatial resolution of the 8 A Vegetation Red Edge, and both Short Wave Infrared bands, and used the average of the red and near-infrared band for the 5–7 Vegetation Red Edge bands.

Taking into consideration that the four 10-m bands are very different from each other, as well as from the Vegetation Red Edge and Short Wave Infrared bands, the mentioned methods for producing panchromatic band deserve some deeper investigation. For that purpose, these three different methods for producing panchromatic bands have been compared through different image fusion techniques from different fusion subsections (Component substitution—Intensity Hue Saturation IHS; Numerical method—High Pass Filter HPF; Statistical Image Fusion—Principal Component Analysis PC; Hybrid Technique—Wavelet Principal Component WPC). As all fusion techniques have advantages and limitations, a quality assessment is necessary before using them in different applications [6]. For evaluating the performance of the fused images, various quality metrics, such as quantitative and qualitative analysis have been proposed [7,8,9]. However, visual evaluation of satellite images is highly important where relevant elements for image interpretation are scale, patterns, contrast, color, tones, and shadows [10].

2. Data and Methods

In order to compare the mentioned panchromatic bands, in this paper, two datasets were analyzed. Both of the datasets cover the same study area (Anatolian Turkey) taken on different dates (30 August 2016 and 10 August 2017). The Sentinel-2 satellite images were downloaded from the Copernicus Open Access Hub. The images were converted to Unsinged-16 Bit tiff data type. The study area is covered mainly by a mixture of vegetation, wetlands, and small urban areas.

In order to increase the spatial resolution of the 20-m Sentinel-2 bands to 10-m, image fusion techniques should be performed. However, the main image fusion approaches, component substitution and multiresolution analysis, were originally developed for image fusion with a single fine band [4]. Sentinel-2 provides four 10-m bands that are highly correlated with the 20-m bands. In this paper, we compare three approaches for producing a single panchromatic band. The first approach produces the panchromatic band by averaging all fine multispectral bands [3,4], further referred as Pan1. The second method uses the near-infrared band as a panchromatic band in order to increase the spatial resolution of the 8A Vegetation Red Edge, and both Short Wave Infrared bands and used the average of the red and near-infrared band for the 5–7 Vegetation Red Edge bands further referred as Pan2. Additionally, Wang’s linear combination of the four fine bands was taken into consideration, as well as the 20-m bands, where the near-infrared band was used for fusion with 6, 7 and 8a bands, while bands 3 and 4 were used for fusing bands 5, 11 and 12, further referred to as Pan3. The last method was performed only on the image from 10 August 2017. As shown in the third method, it is imperative that Band 8a must be sharpened by the encompassing Band 8, while Band 11 and Band 12 should be sharpened by the red band rather than by Band 8. The selection of the sharpening band for bands 5, 6 and 7 raises an interesting question. Band 5 is more correlated to Band 4 then Band 8, and Band 7 is more correlated to Band 8. The selection of the sharpening band for Band 6 deserves more detailed investigation. Since Band 6 is in the middle of Band 4 and Band 8, a maximized combination of Band 4 and Band 8 is possible. This assumption was investigated and compared with the previously mentioned methods.

For the quantitative analyses, Wald’s protocol was followed which the most widely used one for validation of pan-sharpening methods [11]. For the comparison of used methods, three different image fusion techniques were performed: Component substitution—Intensity Hue Saturation IHS; Numerical method—High Pass Filter HPF; Hybrid Technique—Wavelet Principal Component WPC. For the quantitative analyses, four indices were used: correlation coefficient (CC) which provides correlation between the fused and the reference image, Universal Image Quality Index (UIQI) which uses covariance, variance, and means of fused and reference image [10], Relative Average Spectral Error (RASE) [12], and Spectral Angle Mapper (SAM), curtail for the case under concern.

3. Results

The quantitative analyses of the performed experiments are given in

Table 1. Quantitative analyses of the image fusion techniques for the 30.08.2016 image.

		CC	UIQI	RASE	SAM
Ideal		1	1	0	0
IHS	Pan1	0.935	0.942	2.17	0.019
	Pan2	0.947	0.943	4.40	0.019
HPF	Pan1	0.943	0.933	1.49	0.020
	Pan2	0.952	0.952	4.16	0.018
WPC	Pan1	0.971	0.923	1.36	0.001
	Pan2	0.987	0.983	1.81	0.009

,

Table 2. Quantitative analyses of the image fusion techniques for the 10.08.2017 image.

		CC	UIQI	RASE	SAM
Ideal		1	1	0	0
IHS	Pan1	0.992	0.990	2.23	0.029
	Pan2	0.968	0.959	2.75	0.030
	Pan3	0.989	0.979	2.38	0.040
HPF	Pan1	0.990	0.981	2.23	0.028
	Pan2	0.966	0.956	2.70	0.191
	Pan3	0.956	0.953	2.71	0.231
WPC	Pan1	0.966	0.956	2.70	0.026
	Pan2	0.996	0.987	1.75	0.018
	Pan3	0.998	0.989	1.64	0.017

and

Table 3. Quantitative analyses of the image fusion techniques for Band 6 of the 10.08.2017 image.

		Bias	CC	UIQI	SAM
Ideal	Pan	0	1	1	0
IHS	Avg	28.26	0.987	0.977	0.030
	(Red + NIR)/2	29.38	0.990	0.980	0.030
	NIR	31.12	0.989	0.980	0.039
HPF	Avg	1.013	0.966	0.954	0.193
	(Red + NIR)/2	0.97	0.967	0.953	0.028
	NIR	0.81	0.966	0.953	0.205
WPC	Avg	22.66	0.997	0.989	0.020
	(Red + NIR)/2	27.01	0.998	0.987	0.018
	NIR	23.25	0.998	0.990	0.015

and the results from the qualitative analyses can be observed from Figure 1. Figure 1 shows the downscaling results (from 20 m to 10 m) for a sub-area (Cifteler village) from the used image. It can be seen that the HPF results lead to spectral distortion (Figure 1e,h,k). The spectral distortion can be also noticed in Figure 1f,i,l where the urban features cannot be clearly observed. The IHS results tend to be superior over WPC and HPF in that order.

Four indices were used for the quantitative analyses, CC, UIQI, RASE, and SAM. CC, UIQI, and SAM were calculated for all bands and then their average value is presented in Table 1 for the first image, and Table 2 for the second image. Following Wald’s protocol, in order to compare the 20-m fused images with a 20-m reference image, first, the 20-m images were upscaled to 40-m and then downscaled to 20-m.

In Table 1 are presented the results from the first and the second method for producing panchromatic band mentioned in the Methods section. Both methods showed best results using the WPC fusion method.

In Table 2 the results from the three panchromatic bands are presented, fused using three different methods. The first panchromatic band is produced with averaging all four fine bands, the second is using the average value of Band 4 and Band 8 for fusing Band 5–7, and Band 8 for fusion Band 8a, 11 and 12. The third panchromatic band is using a correlation between the bands, or Band 4 for fusing Band 5, 11 and 12, and Band 8 for fusing Band 6, 7 and 8a. In two out of three cases, the first method gave the highest result in all four indices. However, the highest CC (0.998) and UIQI (0.989), and lowest RASE (1.64), and Sam (0.017) were achieved using the third panchromatic band, and the WPC fusion method.

Table 3 presents the comparison of the fusion methods and the used panchromatic bands for Band 6. Results show that in two out three cases (IHS and HPF), taking the average value of Band 4 and Band 8 as a panchromatic band give best results. It should be noted that in the third case (WPC), the results between the panchromatic band produced from Band 4 and Band 8 ((Red+NIR)/2), and Band 8 (NIR) are identical in the CC values, and the difference in the UIQI are 0.003 and 0.003 for SAM in favor of the NIR band, which is used as the panchromatic band.

4. Discussion

Fusing/Pan-sharpening Sentinel-2 bands cannot be done in the conventional methods due to lack of a panchromatic band. Sentinel-2 bands have a high spatial resolution of 10 m in the VNIR region and it offers 20 m of four Red Edge Vegetation bands and two shortwave infrared bands. High spatial resolution of ten Sentinel-2 bands could lead to complete data for Earth observation studies. In order to produce the missing panchromatic band in the Sentinel-2 dataset, several studies have been made. Wald compared the results from fusing the 20 m with a panchromatic band created as averaging all fine multispectral bands and panchromatic bands selected with a linear correlation [13]. It should be noted that the results from both methods were very close and in some cases, the first method performed better than the second.

Another study related to the same topic, uses the closest band as a panchromatic band, or it is using Band 8 (NIR) for fusing Band 8a (Vegetation Red Edge), Band 11 and Band 12 (Shortwave infrared bands), and uses the average value of NIR and Red band for fusing Band 5–7 (Vegetation Red Edge). The results were also compared with a panchromatic band produced by averaging all fine bands which gave higher results in the CC values.

From the high correlation between some bands, like Band 8 and Band 8a, it is only logical to use Band 8 for the pan sharpening of Band 8a. Also, it comes to mind that Band 11 and Band 12 should be pan-sharpened to Band 4 since all IR wavelengths except NIR are somewhat correlated to one another. However, taking in consideration the wave range of the panchromatic bands of Landsat 7 (0.520–0.900), Landsat 8 (0.503–0.676), or Worldview-2 (0.450–0.800), for fusing six Sentinel-2 bands, an average value of VNIR bands (0.490–0.842) can be accepted.

5. Conclusions

In this paper, a comparison of three different methods for producing a panchromatic band from Sentinel-2 bands was made. The first method of producing the panchromatic band was by averaging all 10-m bands (Band 2–4 and Band 8). The second method was by using the average of Band 4 and Band 8 (Red and NIR) as a panchromatic band for downscaling Band 5–7, and Band 8 as a panchromatic band for downscaling Band 8a, 11 and 12. The third method used linear correlation and used Band 4 as a panchromatic band for Band 5, 11 and 12, and Band 8 for Band 6, 7, and 8a. For Band 6, a deeper investigation was made. The comparison was made using three different frequently used image fusion methods (IHS, HPF, and WPC). The summary of the results of this study indicates that:

• All of the panchromatic bands are able to produce accurate results in downscaling Sentinel-2 20-m bands. In two out of three cases, the first method was superior, while in the third case, the results of the second and third methods were almost identical.
• Using a single panchromatic band is less time consuming and more practical.
• For the two images used in this paper, the superior fusion method was WPC with almost ideal CC (0.998) and SAM (0.001) values.
• Band 6 is best fused with a panchromatic band produced as an average value from Band 4 and Band 8.