Spatial Sharpening of KOMPSAT-3A MIR Images Using Optimal Scaling Factor

Abstract

This paper present efficient methods for merging KOMPSAT-3A (Korea Multi-Purpose Satellite) medium wave Infrared (MIR) and panchromatic (PAN) images. Spatial sharpening techniques have been developed to create an image with both high spatial and high spectral resolution by combining the desired qualities of a PAN image with high spatial and low spectral resolution and an MS/MIR image with low spatial and high spectral resolution. The proposed methods can extract an optimal scaling factor, and uses the tactics of appropriately controlling the balance between the spatial and spectral resolutions. KOMPSAT-3A PAN and MIR images were used to test and evaluate the performance of the proposed methods. A qualitative assessment were performed using the image quality index (Q4), the cross correlation index (CC) and the relative global dimensional synthesis error (Spectral/Spatial ERGAS). These tests indicate that the proposed methods preserve the spectral and spatial characteristics of the original MIR and PAN images. Visual analysis reveals that the spectral and spatial information derived from the proposed methods were well retained in the test images. A comparison of the results of the proposed methods with those obtained from applying existing ones such as the Multi Sensor Fusion (MSF) technique or the Guide Filter Based Fusion (GF) show the efficiency of the new fusion process to be superior to the one of the others. The results showed a significant improvement in fusion capability for KOMPSAT-3A MIR imagery.

1. Introduction

KOMPSAT (Korea Multi-Purpose Satellite)-3A is a very high-resolution optical satellite launched in March 2015 that provides panchromatic (PAN) images, Multi-Spectral (MS) images and Medium wave Infrared (MIR) images. In particular, KOMPSAT-3A MIR images have few counterparts of similar quality around the world. Therefore, KOMPSAT-3A MIR images are highly likely to be used in various fields such as the study of the thermal island phenomenon, drought monitoring, and analysis of unusual objects in urban areas where they are difficult to visually identify.

On the other hand, researchers in the field of remote sensing are currently studying opportunities and possibilities to mathematically combine thermal or MIR images with visible images. Such data fusion research is conducted to expand the utilization of thermal or MIR images taken from satellites or to provide various information simultaneously by combining two images that provide different information. For satellites that only record PAN images, such as EROS-A, EROS-B, and Worldview-1, or satellites that do not take PAN images, such as Rapid-Eye, data fusion technology can be applied to expand the scope of their utilization [1]. Other lines of research study replacing PAN images with high-resolution SAR images [1,2,3], for instance for classification tasks in urban areas [4] or to identify Antarctic ice [5]. In the case of infrared images that provide radiant energy information from the earth’s surface, the amount of radiation energy is less than that of visible wavelengths by the Planck function and Stefan-Boltzmann law. The ground sample distance (GSD) ratio between the PAN channel of Lansat-8 OLI/TIRS and the infrared channel is 1:6.7, but for KOMPSAT-3A it is 1:10. Therefore, studies have been conducted to provide both spatial details provided by PAN and surface temperature information provided by infrared channels through data fusion technology. In [6] an optimization method for fusion coefficients to improve spatial resolution of LANDSAT 8 TIR (Thermal Infrared) is proposed. A Thermal sharpening (TsHARP) algorithm using normalized difference vegetation index (NDVI) of multispectral channels was studied in [7,8]. Other studies have been carried out for injecting spatially detailed information of PAN into an infrared channel image mathematically [2] or for producing a fusion image using the technique of sparse representation [9,10]. Other authors worked on adjusting the injection ratio by applying the moving window technique to take into account the characteristics of features (urban, agricultural, forest) present in the images [11]. Recently, machine learning-based fusion methods have been studied in order to improve the resolution of infrared channel images by learning the relationship between multi-spectral based index maps (ex. NDVI, Enhanced Vegetation Index (EVI), normalized difference water index (NDWI), normalized difference moisture index (NDMI), and infrared channel images [12].

As such, various studies are continuously being conducted to improve the spatial resolution of TIR images. However, no research has been yet conducted on high-resolution optical images such as KOMPSAT-3A. In addition, existing methods have been applied to TIR images with wavelength band 10 μm or higher, and it is difficult to apply them directly to MIR images with a wavelength band between 3.3 to 5.2 μm.

In this paper, novel spatial sharpening techniques are proposed that is specifically designed to suit KOMPSAT-3A MIR images with the aim of fully exploiting the merging capabilities in KOMPSAT-3A. The proposed methods are similar to the Multi Sensor Fusion (MSF) [6] and Guide Filter Based Fusion (GF) [13] methods in that the high frequency component related mostly to spatial information is used for merging KOMPSAT-3A MIR images. It differs from the previous methods as optimal scaling factor parameters and the moment matching method are used. These methods adopt the extraction of the tactics of appropriately controlling the balance between spatial and spectral resolutions. The superior quality of the newly introduced methods in comparison with the currently known ones is proven by both visual analysis and several standard tests.

The remainder of the paper is organized as follows. Section 2 describes in detail the mathematical formulae for the new methods of merging KOMPSAT-3A MIR and PAN images and their derivation. In Section 3, the fusion results of the proposed methods using KOMPSAT-3A MIR and PAN images are presented and they are compared in Section 4 with those obtained using previous methods. Finally, conclusions are provided in Section 5.

2. Methods

The purpose of this study is to improve the resolution of KOMPSAT-3A MIR images to the resolution of PAN images through mathematical methods. To this end, this study aims to improve convergence performance by modifying existing techniques to suit KOMPSAT 3A MIR. Figure 1 summarizes the characteristics of proposed and previous methods.

Previous methods used in similar studies are MSF method [6] and GF method [13]. The difference between the methods proposed in this study and the previously used ones lies in the application of the moment matching and local scaling factor optimized for KOMPSAT 3A. The proposed methods in this paper are named MSF-P(Proposed) and GF-P, respectively.

The fused MIR image ($MI{R}^{FUS})$ can be defined by adding the high frequency component ($\mathsf{\delta }$) of a PAN image to the interpolated MIR image ($MI{R}^l)$ as follows:

$$MI{R}^{FUS}=MI{R}^l+\alpha ·\mathsf{\delta },$$

(1)

where $\alpha >0$ is the scaling factor to appropriately control the balance between the spatial and spectral resolutions. In addition, this parameter controls the amplitude of the high frequency(HF) PAN image applied to fused MIR image. On the other hand, $\mathsf{\delta }$ can be also defined using the low frequency(LF) PAN image as

$$\mathsf{\delta }=P^h-P^l,$$

(2)

Various fusion methods can be classified according to the extraction method of the HF PAN image ($\mathsf{\delta }$) and the scaling factor (α) applied.

2.1. Multi Sensor Fusion (MSF) Approach

In the MSF method [6] is developed for improving the spatial resolution of LANDSAT 8 TIR images. In particular, it was intended to estimate the optimal scaling factor by assessing the quality of the fused image with changes in the fusion coefficient. The following equation is a modified scaling factor for this study [6].

$$\mathsf{\alpha }=\frac{\sqrt{{\sum }_{i=1+w}^{N-w}{\sum }_{i=1+w}^{M-n}{\left\{{\overline{\sigma }}_{MI{R}^l}\left(i,j\right)\right\}}^2}}{\sqrt{{\sum }_{i=1+w}^{N-w}{\sum }_{i=1+w}^{M-n}{\left\{{\overline{\sigma }}_{\delta }\left(i,j\right)\right\}}^2}},$$

(3)

where N and M are the numbers of lines and pixels in the corresponding TIR image, W is the half of moving-window size and ${\overline{\sigma }}_{TIR}$ and ${\overline{\sigma }}_{\delta }$ are standard deviations of the samples extracted using moving-window in the HF PAN image and TIR image (${\overline{\sigma }}_{MIR},{\overline{\sigma }}_{\delta }$). Another feature of the MSF method is the use of modified MIR images in the fusion process. In the MSF method, the moment matching technique is applied to make the mean and standard deviation of MIR images similar to PAN. The proposed moment matching technique can be modified and expressed as follows [6]:

$${\acute{MIR}}^l=\frac{{\sigma }_{P^l}}{{\sigma }_{MIR}} ·\left(MI{R}^l- {\mu }_{MI{R}^l}\right)+ {\mu }_{P^l},$$

(4)

The purpose of MSF method is not simply to improve the spatial resolution of TIR images, but to derive new information through the effective fusion of PAN and TIR. Thus, if the MSF method is applied to KOMPSAT-3A MIR images, the fused image may exhibit characteristics closer to the PAN images than to the MIR ones.

2.2. Guided Fiter Based Fusion (GF) Approach

In one can find a fusion technique using guided filters to improve the spatial resolution of LANDSAT 8 TIR images [13]. If a fusion technique that improves the spatial resolution of a TIR image is applied using NDVI produced by visible near infrared (VNIR) bands, spectral distortion may occur significantly. Therefore, this study proposed a fusion technique that reduced spectral distortion by using shortwave infrared (SWIR) bands. The scaling factor proposed in the GF method can be modified and expressed as follows:

$$\mathsf{\alpha }=\frac{rang{e}_{MI{R}^l · }skewnes{s}_{MI{R}^l}}{rang{e}_{\delta · }skewnes{s}_{\delta }},$$

(5)

In this study, HF PAN images ($\mathsf{\delta }$) are extracted using a guide filter [13]. The guide filter generates a LF PAN image by using a pan image as an input image and an MIR image as a guide image. Meanwhile, in the GF method, a histogram matching technique is applied in order to change the statistics of the PAN image similarly to the TIR image. Therefore, when this technique is applied to KOMPAT-3A MIR, the fused image can exhibit characteristics closer to low-resolution MIR than to PAN.

2.3. Proposed Method

In this study, the scaling factor optimized for KOMPSAT-3A is proposed. The proposed scaling factor can effectively control the fusion of spatial information in the PAN image and spectral information in the MIR image. To determine the best scaling factor between spectral and spatial quality of the fused MIR image, the energy function can be written in the following form:

$$\mathrm{u}\left(\mathsf{\alpha }\right)= {\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{j=1}^M\left[\gamma {\left\{MI{R}^{FUS}\left(i,j\right)-MI{R}^l\left(i,j\right)\right\}}^2+{\left\{MI{R}^{FUS}\left(i,j\right)-{\acute{P}}^h\left(i,j\right)\right\}}^2 \right],$$

(6)

where $\gamma >0$ is the control parameter for controlling the spectral and spatial resolution of the fused MIR image, N and N are the total number of pixels and lines on PAN image coordinates, respectively, ${\acute{P}}^h$ is the image obtained by adjusting the mean and the standard deviation of the PAN image to the MIR images. In (6), the first term corresponds to the spectral ERGAS index [14], and the second term corresponds to the spatial ERGAS index [15]. The large $\gamma$ value reduces the spectral distortion of the fused MIR images, while the spatial resolution of the fused MIR images decreases. Otherwise, the small $\gamma$ value enlarges the spectral distortion, but it improves the spatial resolution. The energy function (6) can be rewritten using (1) as

$$\mathrm{u}\left(\mathsf{\alpha }\right)= {\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{j=1}^M\left[\gamma {\left\{\alpha ·\delta \left(i,j\right)\right\}}^2+{\left\{\alpha ·\delta \left(i,j\right)+MI{R}^l\left(i,j\right)-{\acute{P}}^h\left(i,j\right)\right\}}^2 \right],$$

(7)

The function $\mathrm{u}\left(\mathsf{\alpha }\right)$ reaches its global minimum when the first partial derivative $\frac{\partial \mathrm{u}\left(\mathsf{\alpha }\right)}{\partial \mathsf{\alpha }}$ is equal to zero, as given by

$$\frac{\partial \mathrm{u}\left(\mathsf{\alpha }\right)}{\partial \mathsf{\alpha }}={\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{j=1}^M\left[2\left(\gamma +1\right)\delta {\left(i,j\right)}^2\alpha +2\delta \left(i,j\right)\left\{MI{R}^l\left(i,j\right)-{\acute{P}}^h\left(i,j\right)\right\}\right]=0,$$

(8)

The optimal parameter $\mathsf{\alpha }$ can be determined from the following equation:

$$\mathsf{\alpha }=\frac{{{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^M \left[\delta \left(i,j\right)·\left\{{\acute{P}}^h\left(i,j\right)-MI{R}^l\left(i,j\right)\right\} \right] }{\left(1+\gamma \right){{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^M\delta {\left(i,j\right)}^2 },$$

(9)

This can be rewritten using (2), as given by

$$\mathsf{\alpha }=\frac{{{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^M \left[\left\{{\acute{P}}^h\left(i,j\right)-P^l\left(i,j\right)\right\}·\left\{{\acute{P}}^h\left(i,j\right)-MI{R}^l\left(i,j\right)\right\} \right] }{{{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^M {\left\{P^h\left(i,j\right)-P^l\left(i,j\right)\right\}}^2 }$$

$$= \frac{Cov\left({\acute{P}}^h\left(i,j\right)-P^l\left(i,j\right),{\acute{ P}}^h\left(i,j\right)-MI{R}^l\left(i,j\right)\right)}{\left(1+\gamma \right)·var\left({\acute{P}}^h\left(i,j\right)-P^l\left(i,j\right)\right)},$$

(10)

where Cov(X, Y) refers to the covariance of X and Y, and Var(X) indicates the variance of X. Consequently, the optimal parameter $\mathsf{\alpha }$ depends on the statistical properties of the high frequency component of the PAN image and the difference between the PAN and MIR images. On the other hand, when Equation (10) is applied to GSF-P, ${\acute{P}}^h\left(i,j\right)$ is modified to $P^h\left(i,j\right)$ and $MI{R}^l\left(i,j\right)$ is modified to ${\acute{MIR}}^l\left(i,j\right)$. If the regional characteristics in the test image are similar, it is not relevant to use the global scaling factor of Equation (10). However, if regions of different characteristics such as sea, tree, and urban areas are mixed, regional non-relationships in the images can cause color or spatial distortion in the fused image. Therefore, in this study, we tried to minimize regional distortion by extending the global scaling factor of Equation (10) to a window-based regional variable. When the local scaling factor corresponding to image coordinates $\left(i,j\right)$ is denoted by $\mathsf{\alpha }\left(i,j\right)$, its value is obtained through a $\mathrm{W}\times \mathrm{W}$ sized window operation centered on $\left(i,j\right)$ within the image. If Equation (10) is applied to all image coordinates $\left(i,j\right)$, a local scaling factor image of the same size as the PAN image can be obtained. Because the local scaling factor image by using Equation (10) is calculated as the value obtained from a window patch of size $\mathrm{W}\times \mathrm{W}$, unstable distortion can occur if the window size is too small or the luminance value is very uniform, which happens for instance in case of a river or a forest. In this study, a regional fusion coefficient was derived by applying a 15 × 15 window by an empirical method. On the other hand, the initial fused image produced by the proposed method applies the steps to adjust the mean and standard deviation similarly to the low-resolution MIR images (see Figure 1). These moment matching techniques can be applied by modifying expression (4).

2.4. Quality Assessment

A qualitative assessment of the proposed methods were achieved using the image quality index (Q4), the cross correlation index (CC), and the relative global dimensional synthesis error (ERGAS). The Q4 index was proposed in [16] for a quality measurement of the fused image. This index is a generalized version of the Q index based upon structural similarity [17] using the theory of quaternions [18]. The Q4 index takes real values in the range 0–1, where 1 indicates the ideal value of Q4. If the value of the Q4 index is close to 1, then the spectral quality of the fused image is high. The CC index is defined by the cross correlation between fused image and its reference image. It reflects the similarity of spectral information. If the two images are correlated, the CC is close to 1, which implies that the spectral character of the original image was preserved well.

The ERGAS index was proposed in [14] to evaluate the spectral quality of the fused image. This index is calculated from the root mean square errors (RMSE) between the original and the fused images. The lower the spectral distortion of the fused image, the lower is the ERGAS index. If the ERGAS is zero, it means that the original and the fused images are the same. Since this index can measure only the spectral distortion of the fused image, it is called spectral ERGAS. The spatial ERGAS was proposed in [15] to evaluate the spatial quality of the fused image. It is calculated from the RMSE between the histogram-adjusted PAN image and the fused image. If the spatial ERGAS index is low, it indicates that the fused image carries the detailed spatial information of the original PAN image.

There are other two validation methods, the synthesis property and consistency property, that are mainly used to evaluate the quality of the fused image (see Figure 2) [19,20]. The synthesis property considers that the fused image should be as close to the reference image as possible. However, at the full scale, the synthesis property can’t measure the differences between them, so the tested images need to be degraded by the scale ratio between the pixel sizes of the PAN and MIR images. By using degraded images, the degraded fused image is compared with the original observed MIR image as a reference. Next, the consistency property shows the degree to which the spatial degraded fused image is identical to the original observed MIR. MTF-filter method [21] and spline interpolation were applied for image degradation.

3. Results

3.1. Test Data

The KOMPSAT-3A system has enabled the generation of high-resolution images with a ground sample distance (GSD) of 0.55 m for a PAN image, 2.2 m for an MS image, and 5.5 m for an MIR image. The KOMPSAT-3A sensor has a single PAN spectral band between 450 and 900 nm, four MS spectral bands between 450 and 900 nm (B: 450–520 nm, G: 520–600 nm, R: 630–690 nm, and NIR: 760–900 nm), and an MIR band between 3300 and 5200 nm. The characteristics of the KOMPSAT-3A satellite system are shown in

Table 1. Characteristics of the tested KOMPSAT-3A image.

Characteristics	$\mathbf{P}\mathbf{A}\mathbf{N}$	MIR
Acquisition Data (UTC)	17.10.13
Satellite Altitude (km)	534.1
Roll/Pitch/Raw Angle (deg.)	−7.3/−19.1/0.5
GSD (Col./Row.) (m)	0.6/0.6	5.9/6.3
Size (Col./Row.) (pixel)	24,060/21,040	2406/2102
Center Coordinate (Latitude/Longitude)	34.6/127.2	33.1/128.3
Spectral Range ($\mathsf{\mu }\mathrm{m}$)	0.45–0.9	3.3–5.2

. The KOMPSAT-3A PAN and MIR images listed in Table 1 were used to validate the fusion performance of the proposed method.

The experimental data is the KOMPSAT-3A bundle imagery taken in Goheung, South Korea, on 13 October 2017. The Goheung region is located at the southernmost part of the Korean peninsula and consists of coastal, agricultural, and mountainous areas (see Figure 3). The processing level of the experimental data is 1R, which is the image without geometric correction. In the case of a general pan-sharpening technique, since PAN and MS images are captured at the same time, experiments can be performed with 1R images without additional geometric correction. However, for MIR images, there is a slight difference in acquisition time between PAN and MS (see Table 1). Therefore, for this experiment, precise orthogonal correction of experimental data or relative geometric correction between experimental data must be performed. For this experiment, precise orthogonal correction was performed for each experimental data with sub-pixel accuracy (UTM, GRS80, MSL). In addition, for convenience of experimentation, spatial resolutions in the row and column directions were re-interpolated to 5.5 m (MIR) and 0.55 m (PAN), respectively, and the image size (based on MIR) was cut into 1800 × 1800 (pixels) (see Figure 3).

3.2. Data Processing

The proposed methods are based on the existing MSF [6] and GF [13] methods. The differences between it and the two mentioned methods are compared and presented according to the progress of the image fusion (see Figure 1). The first difference is the modification of the PAN and MIR images. The MSF method modifies MIR based on PAN images, while in contrast, the GF method modifies the statistical characteristics of PAN images based on MIR images.

Figure 4 shows the PAN and MIR images before and after modification (See Figure 1). Figure 4a,c are original MIR image and PAN image. Figure 4b,d are the MIR image and the PAN image modified by the corresponding methods. In accordance with each method, high frequency component is injected into the MIR images before and after modification to produce the fused images. In other words, images fused into MSF will have characteristics similar to Figure 4b. Therefore, the statistical characteristics of the fused images will be closer to the PAN images than to the MIR ones.

The second difference between the new methods we propose and the existing ones can be noticed in the extraction procedure of high frequency components. In general, high frequency components ar expressed by highlighting a region with a major change in the surrounding pixel values. In other words, the extracted high frequency component is highlighted in areas where the difference in surface reflectivity to solar energy is significant. On the other hand, in MIR, regions with large differences in surface temperature as well as differences in surface reflectivity may be highlighted. This is because KOMPSAT-3A MIR, which has a wavelength band of 3.3–5.2 (μm), appears as a combination of solar radiation energy and earth radiation energy, unlike thermal infrared [22]. These features can increase the possibility of improving the spatial resolution of MIR images using PAN images. This is because the application of successful data fusion methods should be based on a basic assumption that there is a great similarity between the target image and the reference image [23]. For the extraction of high frequency components, MSF and GF methods apply different strategies. The MSF method applies a low-pass approach to high-resolution PAN images to extract a low frequency component ($P^l$) [6]. On the other hand, the GF method extracts low frequency components ($P^l$) through guided filtering of PAN and MIR images [13]. The high frequency component ($P^h$) is generated by subtracting the low frequency component ($P^l$) extracted by each methods from the PAN image. Figure 5 is an enlarged image of a partial region of the high frequency component generated by the two methods. Figure 5a,b show the difference at the boundary between buildings and roads. This characteristic is judged as a result of the difference in reflectance and emissivity of the PAN image and the MIR image. This is because the GF method reduces the part of the PAN image with a large difference in characteristics from the MIR image through guided filtering.

The third difference between the considered methods lies in the estimation method of the scaling factor, which determines the injection rate of the high frequency components. The fused image ($MI{R}^{FUS}$) is produced by the sum of interpolated MIR image ($MI{R}^l$) and high frequency components scaled by fusion factor. As already mentioned, the scaling factor is used to control the balance between spatial and spectral resolutions.

Table 2. Statistical characteristics of injection gains according to various scaling factors.

Scaling Factors		Degrade
		Average	Max. Value	Min. Value
MSF	Method by Equation (3)	1.16 (constant)
GF	Method by Equation (5)	−15.88 (constant)
MSF-P	Proposed method by Equation (8)	1.02	2.48	−0.70
GF-P		1.05	2.82	−0.50

shows the scaling factor estimated under both existing and newly proposed methods.

The MSF and GF methods use global scaling factors, with estimated values of 1.16 and −15.88, respectively. On the other hand, the proposed methods use local scaling factors, and the average of the estimated values is 1.02 (MSF-P) and 1.05 (MSF-P), respectively. Compared with other methods, the estimated value in GF method shows a very large difference. This reason is due to the difference in characteristics of the applied satellite sensor. This is because in the previous GF method, the scaling factor was empirically estimated according to the ratio of the range and skewness of the thermal infrared (TIR) and shortwave infrared (SWIR) images of LANDSAT 8 [13]. Therefore, if the scaling factor proposed in this study is directly applied to KOMPSAT 3A MIR image fusion, both spatial details and thermal information may be lost. Meanwhile, the proposed methods use local scaling factors estimated according to the size of a designated moving window. This is because the difference between the emissivity and reflectance of PAN and MIR may appear differently depending on the characteristics of the ground surface. Accordingly, in the proposed methods, local scaling factors ranging from −0.7 to 2.82 have been estimated (see Table 2). Finally, the proposed methods have the feature of matching the initial fused image with the mean and standard deviation of the MIR image. Figure 6 shows the images before (Figure 6a) and after (Figure 6b) the transformation. Through this process, distorted spectral information in the fused image can be minimized.

4. Discussion

In Figure 7, (a) and (b) represent images of the original MIR and Pan respectively. Figure 7c–f presents fused images produced by both the previous and the proposed methods. The upper left-hand corner of Figure 7 represents the mean, median, and standard deviation for the DN values of each image. The color scale of each image was expressed in the same range from 0 to 65,535. As mentioned earlier, the fused image with MSF method (see Figure 7c) shows spectral characteristics very similar to the modified MIR image (see Figure 7a). The image in Figure 7d is fused into GF method and shows a very large spectral distortion. This is the result of injection of inverted high frequency information due to underestimated scaling factor. Meanwhile, Figure 7e,f show patterns similar to those of low-resolution MIR images (see Figure 7a). However, they also highlight some differences in artificial regions such as buildings, paved roads, and solar panels. The reason for this is probably due to differences in the extraction method of high frequency components. This means that the guide filtering technique may be more effective if the purpose is to improve the resolution of MIR images. Figure 8 is an enlarged image of area A in Figure 7. In Figure 8, (a) and (b) represent images of the original MIR and Pan respectively. And (c–f) of Figure 8 represent images fused by MSF, GF, MSF_P and GF-P methods respectively.

In the image of Figure 8a, the area marked in red is the one where the solar panel is installed. On the other hand, in Figure 8b the upper part of the location where the solar panel is installed is displayed in red, but the lower part is expressed in a color close to green. That is, since the upper part is installed in front of the sun, the radiated solar energy is high, and thus, the brightness value is relatively high. On the other hand, Figure 8a did not show this difference because of the high radiation energy emitted by the earth. In other words, this phenomenon occurs because the MIR image detects both solar radiation energy and Earth radiation energy. Figure 8c–f represent fused images with the previous methods and the proposed methods. Among them, fused images by the proposed GF-P method (Figure 8f) showed spectral characteristics most similar to the original MIR image (Figure 8a). On the other hand, Figure 8c,e,f, excluding the Figure 8d image, showed significant improvements in spatial resolution. Compared with the other two methods, Figure 8c,f are judged to have a good degree of improvement in spatial resolution and preservation of spectral characteristics. However, Figure 8c was expressed relatively more similarly to the spectral characteristics of the original PAN image. This is because, as mentioned earlier, the fusion process involves the modification of the original MIR image similarly to the original PAN image (see Figure 4b.

Figure 9 is an enlarged representation of Figure 8 in Area B. Area B has a dock adjacent to the sea. The differences in each fusion methods are similar to Figure 8. In other words, it can be judged that the fused image by the proposed GF-P method (Figure 8f) are most appropriately satisfied with the improvement of spatial resolution and the preservation of spectral characteristics. The white circle in Figure 9a indicates the area where a ship is at anchor or in operation. Note that in Figure 9, the size of the ship is very small, so it is impossible to determine whether the ship is operating or not solely by the temperature of the ship. However, if the size of the ship is larger than the resolution of the MIR image, it will be possible to effectively determine through the fusion image the presence or absence of the ship and whether it is running or not.

In addition to the visual evaluation, qualitative assessments were conducted using four indices (

Table 3. Qualitative Results.

Algorithm	Synthesis Property				Consistency Property
	Q4 (ref. MIR)	CC (ref. MIR)	Spectral ERGAS (ref. MIR)	Spatial ERGAS (ref. PAN)	Q4 (ref. MIR)	CC (ref. MIR)	ERGAS (ref. MIR)	Spatial ERGAS (ref. PAN)
MSF	0.69	0.83	5.20	5.42	0.87	0.97	3.77	6.33
GF	0.07	0.22	11.99	14.04	0.55	0.66	6.14	7.21
MSF-P	0.76	0.84	4.08	9.66	0.96	0.97	1.95	10.34
GF-P	0.78	0.85	3.40	9.82	0.99	0.99	0.66	10.97

Bold numbers represent the highest ranking in each evaluation indices.

). There are a total of four evaluation indices used: Q4, CC, Spectral ERGAS, and Spatial ERGAS. Q4, CC, and Spectral ERGAS indicate the degree of preservation of spectral information, and the ideal values are 1, 1, and 0, respectively [15,16]. Spatial ERGAS represents the degree of conservation of spatial information, with an ideal value of zero [14]. Reference images used in Q4, CC, and Spectral ERGAS are MIR images. On the other hand, the reference image for the Spatial ERGAS is a PAN image. The evaluation methodology was divided into Synthesis Property and Consistency Property (see Figure 2). The experiments show consistent results in both evaluation methodologies and the proposed GF-P method, like in the visual evaluation results, was rated as having the best performance. On the other hand, the MSF method assigns the best performance to Spatial ERGAS. This is due to the differences in the reference images used by each evaluation index. In other words, Spatial ERGAS evaluates spatial resolution based on the PAN image. Thus, this is the result of not only improving spatial resolution but also due to the fact that similar to PAN images are evaluated together.

5. Conclusions

A novel efficient methods for the fusion of KOMPSAT-3A (which is the Korea Multi-Purpose Satellite) MIR and PAN images are presented. This method applies a strategy to improve spatial resolution, while preserving the spectral characteristics of MIR images. KOMPSAT-3A PAN and MIR images were used to validate the fusion performance of the proposed methods, which addresses various features such as mountains, plain, sea, and rural areas. The fused images were produced using some existing methods and the newly proposed ones, and visual and qualitative analyses were conducted, the latter by means of four evaluation indices, namely Q4, CC, Spectral ERGAS, and Spatial ERGAS. Evaluation methodologies for the qualitative analysis were divided into Synthesis Property and Consistency Property. Visual and qualitative evaluation results show that the best results were achieved via the proposed GF-P method. In future studies, it is necessary to more closely evaluate the usability of the proposed methodology by utilizing images with more diverse local characteristics.