Fusion of Hyperspectral and Multispectral Images with Radiance Extreme Area Compensation

Abstract

Although the fusion of multispectral (MS) and hyperspectral (HS) images in remote sensing has become relatively mature, and different types of fusion methods have their own characteristics in terms of fusion effect, data dependency, and computational efficiency, few studies have focused on the impact of radiance extreme areas, which widely exist in real remotely sensed scenes. To this end, this paper proposed a novel method called radiance extreme area compensation fusion (RECF). Based on the architecture of spectral unmixing fusion, our method uses the reconstruction of error map to construct local smoothing constraints during unmixing and utilizes the nearest-neighbor multispectral data to achieve optimal replacement compensation, thereby eliminating the impact of overexposed and underexposed areas in hyperspectral data on the fusion effect. We compared the RECF method with 11 previous published methods on three sets of airborne hyperspectral datasets and HJ2 satellite hyperspectral data and quantitatively evaluated them using 5 metrics, including PSNR and SAM. On the test dataset with extreme radiance interference, the proposed RECF method achieved well in the overall evaluation results; for instance, the PSNR metric reached 47.6076 and SAM reached 0.5964 on the Xiong’an dataset. In addition, the result shows that our method also achieved better visual effects on both simulation and real datasets.

1. Introduction

Hyperspectral (HS) remote sensing can obtain rich spectral information of ground objects and have broad application prospects in the fields of object detection [1], fine classification [2], and character attribute inversion [3]. However, due to limitations inherent in its imaging mechanism, hyperspectral remote sensing does not achieve the same level of spatial resolution as multispectral remote sensing, resulting in the loss of spatial texture information. The most common solution is to fuse hyperspectral data with multispectral (MS) data of a higher spatial resolution [4]. In recent years, hyperspectral remote sensing has developed rapidly, bringing a large number of hyperspectral remote sensing data sources and operationalization requirements for massive hyperspectral data fusion tasks [5]. Recently, Italy launched the PRecursore IperSpettrale della Missione Applicativa (PRISMA) earth observation satellite in March 2019 [6]; Germany launched the Environmental Mapping and Analysis Program (EnMAP) hyperspectral satellite in April 2022 [7,8]; and China launched GF-5 [9], ZY-1 02D [10], and HJ-2 A/B [11] satellites from 2018 to 2020 which carry different types of optical remote sensing payloads, including a hyperspectral camera. Simultaneous observation data from multiple sources in single satellite mission bring benefits to hyperspectral data fusion tasks with high spatial resolution data, which avoid the effects of atmospheric radiation transmission and changes in ground objects over time.

The main methods for the fusion of HS and MS images (his and MSI) can be divided into three categories: pan-sharpening-based methods, machine learning-based methods, and deep learning-based methods. The pan-sharpening methods extend the fusion approach of panchromatic images and MS images to the fusion of hyperspectral and multispectral data. They mainly include component substitution (CS)-based methods [12] and multi-resolution analysis (MRA)-based methods [13]. CS-based methods decompose the spatial and spectral information of hyperspectral and multispectral data and then replace the low-resolution (LR) spatial information of HS data with that of MS data to fuse into high-resolution (HR) hyperspectral data, as exemplified by the methods used in [14,15]. MRA-based methods use multiscale decomposition to obtain MS spatial detail information, which is then fused into the corresponding spectral bands of HS to obtain enhanced hyperspectral resolution, as seen in the methods used in [16,17]. Pan-sharpening methods, while able to achieve good spatial information results, suffer from more severe spectral distortion.

The machine learning-based method approach is to construct a mathematical model for HS and MS data fusion-based on machine learning theory, which mainly includes methods based on Bayesian theory, matrix decomposition, and tensor representation. Bayesian theory methods uses prior knowledge to establish a posterior distribution, providing an intuitive explanation for the fusion process [18,19,20]. Wei et al. adopted a Bayesian model based on sparse coding and dictionary learning to address fusion problems by introducing a maximum a posteriori (MAP) estimator [21]. Bayesian fusion methods achieve regularization through the prior distributions added by fusing scene data, such as l2-Gaussian priors, total variation priors, sparse priors, and low-rank priors [22,23,24]. Matrix decomposition methods utilize the spectral unmixing theory to decompose HS and MS data into endmember matrices and abundance matrices. High-resolution hyperspectral fusion is achieved by recombining the endmember matrix of HS data with the abundance matrix of MS data [25,26,27,28,29]. Yokoya et al. used coupled non-negative matrix factorization (CNMF) to extract the endmember matrices and abundance matrices of HS and MS data for image fusion [25]. Nezhad et al. combined sparse representation with image unmixing theory to acquire information reorganization between images [30]. Tensor representation methods treats hyper-spectral data as tensors, which can better preserve the spatial–spectral structure of image data compared to matrix decomposition methods [31,32,33]. Dian et al. [31] proposed a non-local sparse tensor decomposition method for HS data super-resolution. This method decomposes HSIs into sparse core tensors and dictionary estimates, where the dictionary and core tensors can be learned from LR-HSI and HR-MSI. Zhang et al. [34] proposed a low-rank Tucker decomposition model combined with two graphs to fuse HSI and MSI. Li et al. [35] treated HR-HSI as a three-dimensional tensor and proposed to solve the fusion problem by estimating the core tensor and dictionary along three modes. The machine learning-based methods make full use of the prior factors of images, such as sparsity, low rank, and global similarity, and have good interpretability and theoretical basis. They are commonly used in engineering practice. However, their ability to describe nonlinear factors in the fusion process is poor, and the fusion effect of abnormal areas in the image also needs to be improved.

Due to their advantages in solving nonlinear problems, deep learning-based methods are widely used in remote sensing applications such as target recognition, terrain classification, and attribute inversion [36,37,38]. Increasingly, researchers have begun to utilize deep learning for HS data fusion studies. Palsson et al. [39] introduced a three-dimensional convolutional neural network (3D-CNN) fusion approach to compensate for the potential loss of spatial information when processing three-dimensional HS data in two dimensions. Yang et al. [40] proposed a two-branch convolutional neural network (Two-CNN) method to extract spectral information from HS images and spatial information from MS images, fusing the features through a fully connected layer. Zhou et al. [41] developed a CNN-based pyramid fusion model consisting of an encoder and a pyramid fusion subnet. Xue et al. [42] proposed a variational fusion network, where the degradation model and data priors are implicitly represented by deep learning networks and jointly learned from training data. Su et al. [43] proposed a novel unsupervised Generative Adversarial Network (GAN) to address the HSI and MSI fusion problem with arbitrary point spread function (PSF) and spectral response function (SRF). Although deep learning methods can achieve better fusion results, they require a large amount of training data and suffer from issues such as overfitting and limited generalization ability [44].

Although existing hyperspectral and multispectral fusion algorithms have accomplished a good performance on well-preprocessed publicly available HS datasets. We found that extreme radiance areas widely exist in real hyperspectral remote sensing data, resulting in severe distortion of spectral information, which will have a significant impact on fusion results, as shown in Figure 1. These abnormal regions can be classified into two categories: overexposed regions and underexposed regions (low radiance). Both are unavoidable due to the imaging principles of hyperspectral cameras. For the high quantification requirements, hyperspectral cameras usually adopt fixed-gain imaging. Even if the gain setting ensures that the brightness of most ground objects is within the dynamic range, there will still be a small number of targets with specular reflection and other overexposure due to factors such as the surface material of the ground objects and the observation angle, which is reflected in the hyperspectral image as spike noise in certain bands. For low-reflectance ground objects such as water bodies or shadowed areas, the detector receives low energy from the ground objects, leading to a decrease in the signal-to-noise ratio when the detector’s background noise remains unchanged. Despite the fact that the abnormal regions only occupy a small portion of the hyperspectral data in terms of area, they are often the focus of attention for hyperspectral applications such as target recognition, change detection, and anomaly detection.

To enhance the effectiveness of hyperspectral fusion in radiance extreme areas, this paper proposes a hyperspectral and multispectral fusion algorithm architecture called RECF, which is based on radiance extreme compensation. This idea is built upon the architecture of spectral unmixing fusion methods. Based on the global iterative unmixing of hyperspectral and multispectral data in existing methods, the approach proposes utilizing local information constraints to process underexposed areas and utilizing multispectral data for replacement compensation in overexposed areas. Firstly, the overexposed regions are extracted (based on distinct characteristics) for subsequent processing. The Piecewise Smooth Non-Negative Matrix Factorization (PSNMF) is adopted as the data unmixing method, which can effectively suppress noise in both spatial and spectral dimensions. Then, during the iterative unmixing process, the HS data are reconstructed using the endmember and abundance matrices. By comparing the original HS data, error maps based on Spectral Angle Mapper (SAM) and Root Mean Square Error (RMSE) are derived to extract regions with a low signal-to-noise ratio (SNR). The constraint term of PSNMF is updated for regions with low SNR and regions with low average spectral band energy. Finally, the hyperspectral endmembers and multispectral abundances are combined to form the fused high-resolution hyperspectral data. For overexposed regions, compensation is achieved by replacing the multispectral abundances with those from non-overexposed regions with similar SAM values. The main contributions of this paper are as follows:

(1)

The novel RECF method is the first to focus on the impact of radiance extreme regions on hyperspectral fusion and explicitly proposes a solution framework.

(2)

To address the underexposed areas, effective region extraction is achieved through the reconstruction error map. By utilizing piecewise constrained non-negative matrix factorization, the influence of spectral noise caused by underexposed areas on fusion is effectively suppressed.

(3)

For overexposed regions, optimal replacement compensation is calculated with the help of multispectral data.

(4)

RECF is an unsupervised method that can be applied to the fusion of HS and MS remote sensing data in real-world applications. Through experiments with simulated and real data, its fusion effect on overexposed and underexposed regions is significantly better compared to the current advanced methods.

The remainder of this paper is organized as follows. Section 2 describes the spectral unmixing fusion model, constrained non-negative matrix factorization and the impact of targets in extreme brightness regions on fusion. Section 3 formulates the proposed fusion algorithm and optimization strategy. Section 4 elaborates the experimental results from both simulated and real data and provides an analysis of these results. Finally, the conclusion is given in Section 5.

2. Materials and Methods

2.1. Unmixing-Based Fusion Model

The hyperspectral and multispectral fusion task can be regarded as the recovery and reconstruction of high-spectral-resolution hyperspectral data (HH ∈ ℝ^Lh×Nm) from low-spatial-resolution hyperspectral remote sensing observation data (HS ∈ ℝ^Lh×Nh) and low-spectral-resolution multispectral remote sensing observation data (MS ∈ ℝ^Lm×Nm). L_h and L_m represent the number of HS and MS bands, and N_h and N_m represent the number of pixels of the whole HS and MS data. The relationship between the observed HS and HH can be expressed as Equation (1), where S is the point spread function (PSF), and n_h represents the noise and model error, including Truncation noise from overexposure and spectral noise from low response. The relationship between the observed MS and HH can be expressed as Equation (2), where R is the spectral response function (SRF), and n_m represents the noise and model error. In practical applications, S is obtained through the registration estimation of the two types of data to be fused or approximated using an interpolation model, while R is derived from the radiometric calibration of the two types of data or estimated approximately, such as the method adopted in [45].

$$HS=HH\cdot S+n_h$$

(1)

$$MS=R\cdot HH+n_m$$

(2)

According to the Linear Mixing Model (LMM), each pixel in an HS image is composed of a linear combination of spectra from several endmembers [46]. For an HS image, all the endmember spectra constitute its endmember matrix, E = (E₁, E₂, ..., E_K) ∈ L × K, where L is the number of spectral bands, and K is the number of endmembers. The linear combination coefficients of all pixels form the abundance matrix, A = (A₁, A₂, ..., A_N) ∈ K × N, where K corresponds to the number of endmembers in the endmember matrix, and N is the total number of pixels. Therefore, the original hyperspectral data HS can be approximately represented as the product of its endmember matrix, E_h, and abundance matrix, A_h. Similarly, the original multispectral data MS can be approximately represented as the product of its endmember matrix, E_m, and abundance matrix, A_m. The fused high-resolution hyperspectral data can be approximately represented as the product of E_h and A_m, as shown in Equation (3). Combining Equations (1) and (2), A_h can be seen as the convolution of A_m with S, and E_m can be seen as the convolution of E_h with R, as shown in Equations (4) and (5).

$$HH\approx E_h\cdot A_m$$

(3)

$$HS\approx HH\cdot S=E_h\cdot A_m\cdot S=E_h\cdot A_h$$

(4)

$$MS\approx R\cdot HH=R\cdot E_h\cdot A_m=E_m\cdot A_m$$

(5)

Non-negative matrix factorization (NMF) is a commonly used spectral unmixing algorithm, characterized by the fact that all matrix elements after decomposition are non-negative [47,48]. The objective of fusing HS and MS data based on NMF is to minimize the expression shown in Equation (6), where E_h and A_m are the targets that need to be iteratively optimized, and ‖·‖_F represents the Frobenius norm. In addition to constraining E_h and A_m to be non-negative, it is usually required that the sum of the column vectors in the abundance matrix, A_m, is 1.

$$\begin{array}{c}\underset{E_h,A_m}{\arg \min }\frac{1}{2}{‖E_h\cdot A_h-HS‖}_F^2+{‖E_m\cdot A_m-MS‖}_F^2,\\ s.t. E_h\ge 0,A_m\ge 0,{\mathbf{1}}_K^T{A}_m={\mathbf{1}}_N^T\end{array}$$

(6)

2.2. Piecewise Smoothness Constrained Non-Negative Matrix Factorization

Due to the inherent non-convexity of the NMF algorithm, there exist infinitely many solutions for non-negative matrix factorization, and it is not easy to converge to the global optimum. Therefore, apart from the non-negativity and sum-to-one constraints, other constraints on the endmember matrix and abundance matrix are also widely used. Among them, Piecewise Smoothness Constrained Non-Negative Matrix Factorization (PSNMF) incorporates piecewise smoothness constraints into NMF [49]. Smoothness is an inherent property of natural objects, describing their continuity and homogeneity in spatial and spectral dimensions. Therefore, smoothness is a commonly used constraint in spectral unmixing. Piecewise smoothness, as opposed to overall smoothness, allows for abrupt changes in certain areas, providing an advantage in preserving the spatial texture and spectral features of objects. This algorithm imposes smoothness constraints on both the endmember and abundance matrices, with the objective function as Equation (7).

$$\underset{E,A}{\arg \min }\frac{1}{2}{‖E\cdot A-HS‖}_F^2+{\alpha }_E〈\mathsf{g}\left(E-E_N\right)〉+{\alpha }_A〈\mathsf{g}\left(A-A_N\right)〉$$

(7)

where E_N and A_N represent the local neighborhoods of the endmember matrix, E, and abundance matrix, A, respectively; and α_E and α_A are the corresponding regularization coefficients. The function g() is a smoothing function both acting on E and A that adopts a discontinuous adaptive Markov Random Field model [49], as shown in Equations (8) and (9). E_i is an endmember spectrum in E, and N_i = (I − 1, i + 1) is its fixed wavelength range neighborhood. A_pk represents the abundance located at position (p, k) in A, and N_ijp = ((I − 1)j, (i + 1)j, i(j − 1), i(j + 1))) is its neighborhood. The positive parameter γ determines the shape of the function g(), and the γ for the spectral correlation (denoted as γ_E) and spatial correlation (denoted as γ_A) should have different values.

$$\mathsf{g}\left(E_i-E_{N_i}\right)={\displaystyle \sum _{i’\in N_i}g\left(E_i-E_{i’}\right)}={\displaystyle \sum _{i’\in N_i}\left(-e^{-{(E_i-E_{i’})}^2/{\gamma }_E}+1\right)}$$

(8)

$$\mathsf{g}\left(A_{pk}-A_{N_{pk}}\right)={\displaystyle \sum _{i’\mathrm{j}’\in N_{ijp}}g\left({\underset{\_}{A}}_{ijp}-{\underset{\_}{A}}_{i’\mathrm{j}’p}\right)}={\displaystyle \sum _{i’\mathrm{j}’\in N_{ijp}}\left(-e^{-{({\underset{\_}{A}}_{ijp}-{\underset{\_}{A}}_{i’\mathrm{j}’p})}^2/{\gamma }_A}+1\right)}$$

(9)

2.3. The Impact of Radiance Extreme Areas on Fusion

During the real-world process of hyperspectral remote sensing imaging via satellites, due to factors such as the performance of the hyperspectral camera itself, observation angles, atmospheric radiation transmission, and the reflection characteristics of ground objects, the acquired hyperspectral remote sensing data often suffer from quality degradation. In hyperspectral remote sensing images, this manifests as the presence of overexposed or underexposed regional targets, as shown in Figure 1.

The overexposed regions in hyperspectral remote sensing images are usually caused by specular reflections from highly reflective ground objects or due to special observation angles. Because of the need for quantitative remote sensing, hyperspectral remote sensing cameras typically use a fixed gain during imaging, and it is difficult to ensure that, within a wide spectral range, ground objects do not experience overexposure in any spectral band. In contrast, multispectral remote sensing has fewer spectral bands and more flexible gain settings, so overexposure is less common. The overexposed regions in hyperspectral remote sensing images have obvious characteristics: the full spectrum or partial spectral bands reach the maximum or zero value of the DN (Digital Number) quantization, leading to distortion of the hyperspectral curve.

The overly dark regions in hyperspectral remote sensing images are usually caused by low-reflectance ground objects, objects located in shadowed areas, or low solar altitude angles during imaging. Due to the low incident energy on the regional target, and the fact that hyperspectral cameras distribute this energy across various spectral bands, the effective signal energy for a single spectral band of the regional target is low. In contrast, multispectral remote sensing has fewer spectral bands but a wider spectral range for each band, so the same regional target has higher effective signal energy. The overly dark regions in hyperspectral remote sensing images manifest as increased noise, leading to false peak and valley features in the spectral curve of the regional target. Both overexposed and overly dark regions, if not specifically addressed during hyperspectral fusion, can degrade the fusion effect in those areas, particularly affecting the fusion of small targets.

3. Proposed Method

3.1. Idea of RECF

For the fusion of HS and MS data in real remote sensing scenarios, effectively dealing with extreme radiance areas of overexposure and underexposure is a practical problem to be solved. Based on the above motivation, we propose the RECF hyperspectral fusion algorithm with extreme radiance compensation, as shown in Figure 2. E_h and A_h represent the endmember and abundance matrices of HS data, E_m and A_m represent the endmember and abundance matrices of MS data, P_OE represents the overexposed pixels in hyperspectral data, and HH is the fused high-spatial-resolution hyperspectral data.

Based on the traditional unmixing-based HS/MS fusion architecture, RECF innovatively adds three processing steps specifically for extreme radiance areas.

Firstly, during the iterative unmixing process of hyperspectral data, a reconstruction error map characterizing the effect of spectral fusion is calculated. Typically, regions with low-hyperspectral signal-to-noise ratios have larger reconstruction errors, allowing for the extraction of overly dark regions during the iterative unmixing process through the error map.

Secondly, we use non-negative matrix factorization with local smoothing constraints for the iterative unmixing of HS data. The local smoothing constraints are applied separately to the hyperspectral endmember matrix and the hyperspectral abundance matrix. The local neighborhood range for the endmember matrix is determined using the energy distribution of each spectral band in the HS data, while the local neighborhood range for the abundance matrix is determined using the mask from the error map.

Lastly, for overexposed regions in the HS image, their spectral information is already distorted and cannot be used for fusion, requiring separate processing. After iterative unmixing and fusion in non-overexposed regions, optimal replacement compensation for the fused data in the overexposed regions is achieved using MS data.

3.2. Error Map

The error map represents the difference between the reconstructed HS data and the original HS data during the iterative unmixing process, specifically involving the endmember matrix and abundance matrix. In this paper, a combined configuration error map is constructed using the Spectral Angle Mapper (SAM) and Root Mean Square Error (RMSE), as shown in Equation (10). HS_org represents the original HS data, while E_h and A_h are the HS endmember matrix and HS abundance matrix obtained through iterative unmixing, respectively.

$$ErrorMap=\rho \mathrm{SAM}(H{S}_{org},E_h{A}_h)+(1-\rho )\mathrm{RMSE}(H{S}_{org},E_h{A}_h)$$

(10)

As shown in Figure 3, the SAM error characterizes differences in spectral curve features. Typically, regions with low signal-to-noise ratios or small target areas can lead to increased SAM errors in reconstructed HS data. The RMSE error characterizes differences in spectral energy intensity. Usually, regions with abnormal distributions of radiance energy intensity can cause an increase in RMSE errors (Figure 3). The construction of the error map should comprehensively consider both the SAM and RMSE. Based on the different focal objectives of fusion compensation, the weight parameters, ρ, can be empirically set. By binarizing the error map using methods such as Otsu’s method [50], regions with large reconstruction errors can be extracted through a binary mask, providing a local neighborhood range for subsequent smoothing constraints on the abundance matrix.

3.3. Local Smoothing Constraint for the Abundance Matrix

The smoothness of abundance originates from the spatial correlation of remote sensing ground objects. Adding a smoothness constraint to the abundance matrix during hyperspectral unmixing essentially utilizes the spatial correlation of adjacent ground objects to reduce the impact of HS data noise on fusion. The PSNMF [49] method uses a fixed template to calculate the abundance matrix local neighborhood, A_N, in Section 2.2, which is represented as N_ijp = ((I − 1)j, (i + 1)j, i(j − 1), i(j + 1))) in Equation (9). While fixed templates are computationally simple, they can introduce interference from different types of ground objects when processing the edges of similar ground objects, thereby reducing the spectral accuracy of the fused data at the edges.

Therefore, in this paper, the result of the convolving binary error map with a fixed template is used as the local neighborhood, N_ijp, for the current position i, j of the abundance matrix, as shown in Figure 4. Here, the fixed template adopts a 3 × 3 pixel template, represented as N_ijp = ((i − 1)(j − 1), (i − 1)j, (i − 1)(j + 1); i(j − 1), ij, i(j + 1); (i + 1)(j − 1), (i + 1)j, (i + 1)(j + 1)). By utilizing the error map to construct the local neighborhood of the abundance matrix, this approach achieves two main benefits. Firstly, it applies smoothing only to regions with large reconstruction errors, avoiding the impact of nearby ground objects on the original spectral features of other regions. Secondly, compared to a fixed local neighborhood template, this method enables fusion at the edges of similar ground objects to be free from the influence of different types of nearby ground objects.

3.4. Local Smoothing Constraint for the Endmember Matrix

The smoothness of endmember spectra originates from the correlation between spectral bands of ground objects. Adding a smoothness constraint to the endmember matrix during hyperspectral unmixing essentially utilizes the similarity between adjacent spectral bands in the spectral dimension to reduce the impact of hyperspectral data noise on fusion. The PSNMF [49] method also uses a fixed endmember matrix local neighborhood, E_N, in Section 2.2, which is represented as N_i = (i − 1, i + 1) in Equation (8). While imposing a smoothing constraint on the endmember spectra can achieve noise suppression in the spectral dimension, it also weakens spectral features to some extent. Therefore, the local neighborhood range for smoothing should be carefully set.

The HS data noise is proportional to signal energy, and the size of the local neighborhood range is inversely proportional to the average energy of each band in each endmember spectrum, i.e., N_i = (i − n_i, i + n_i). In this paper, a simple linear model is adopted, where n_i = N_max × (1 − S_i), with N_max being the maximum neighborhood range value, and S_i being the normalized average energy of i-th band of HS data, as shown in Figure 5. This approach allows for a more dynamic and adaptive smoothing process that takes into account the varying noise levels and spectral characteristics of the hyperspectral data.

3.5. The Iterative Update Rule for PSNMF Fusion

We adopt the multiplicative update rules method used in [25] to iteratively update the HS and MS endmember and abundance matrices E_h, A_h, E_m, and A_m during the fusion process. For updating E_h and A_h, we use the iterative update rules of PSNMF [49], with the addition of a local smoothness constraint regularization term, as shown in Equations (11)–(14).

$$\begin{array}{c}{E}_h\leftarrow E_h.\ast \left(HS\cdot {A_h}^T+{\alpha }_E\left(E_h.\ast h\left(E_h-E_{\mathit{hN}}\right)-g’\left(E_h-E_{\mathit{hN}}\right)\right)\right)./\hfill \\ \left(E_h{A}_h{A_h}^T+{\alpha }_E{E}_h.\ast h\left(E_h-E_{\mathit{hN}}\right)\right)\end{array}$$

(11)

$$\begin{array}{c}{A}_h\leftarrow A_h.\ast \left({E_h}^T\cdot HS+{\alpha }_A\left(A_h.\ast h\left(A_h-A_{\mathit{hN}}\right)-g’\left(A_h-A_{\mathit{hN}}\right)\right)\right)./\\ \left({E_h}^T{E}_h{A}_h+{\alpha }_A{A}_h.\ast h\left(A_h-A_{\mathit{hN}}\right)\right)\hfill \end{array}$$

(12)

$$E_m\leftarrow E_m.\ast MS\cdot {A_m}^T./\left(E_m{A}_m{A_m}^T\right)$$

(13)

$$A_m\leftarrow A_m.\ast {E_m}^T\cdot MS./\left({E_m}^T{E}_m{A}_m\right)$$

(14)

3.6. Compensation for Overexposed Areas

For the overexposed areas in the original HS data, prominent features of the overexposed data are extracted and excluded from the aforementioned unmixing and fusion process to prevent distorted spectral information from affecting the iterative optimization of the endmember and abundance matrices. After the unmixing and fusion steps, the hyperspectral endmember matrix, E_h, and the multispectral abundance matrix, A_m, constitute the fused high-resolution hyperspectral data, HH. The location R_OE_MS corresponding to the overexposed area R_OE_HS of the original hyperspectral data is found in A_m. Assuming that the multispectral data are not overexposed and that similar remote sensing objects also exist outside the overexposed area, the point (i′, j′) with the closest spectrum to the current overexposed area R_OE_MS pixel position (i, j) is searched for in the normally illuminated area R_MS. The spectral information at position (i, j) in the fused HH is then compensated for using the hyperspectral endmember matrix, E_h, and the abundance information of point (i′,j′), as shown in Equations (13) and (14). Here, r_i,j represents the radiance intensity difference factor between objects (i, j) and (i′, j′), and it is calculated using Equation (17).

$$H{H}_{i,j}=r_{i,j}{E}_h{A}_{m,i’,j’}$$

(15)

$$\begin{array}{c}\underset{i’,j’}{\arg \min }\mathrm{SAM}\left(E_m{A}_{m,i,j},E_m{A}_{m,i’,j’}\right)\\ i,j\in R\_MS\_OE\\ i’,j’\in R\_MS\end{array}$$

(16)

$$r_{i,j}={\displaystyle \sum _{i=1}^{L_m}{E}_m{A}_{m,i,j}}/{\displaystyle \sum _{i=1}^{L_m}{E}_m{A}_{m,i’,j’}}$$

(17)

Furthermore, the details of algorithm flowchart are described in

Table 1. RECF flowchart.

Algorithm Flowchart: Radiance Extreme Area Compensation Fusion

Input: HS, MS, Algorithm parameters αE, αA, γE, γA, ρ

Output: HH

Step1: Identify and extract overexposed areas from HS as R_OE_HS

Step2: Fusion radiance normal light region based on PSNMF  (1) Initialize Eh by VCA, and update Ah by (12), with Eh fixed.  (2) Calculate error-map between HS and Eh·Ah  (3) Update Em and Am by (13) and (14). while not converge do   (4) Initialize Em by (5), and update Am by (14), with Em fixed.   (5) Update Em and Am by (13) and (14).   (6) Initialize Ah by (4), and update Eh by (11), with Ah fixed.   (7) Calculate error-map between HS and Eh·Ah   (8) Update Eh and Ah by (11) and (12). end  (9) HH = Eh·Am.

Step3: Compensate overexposed areas of fused HH by (15) and (16)

. The value range of α_E, α_A, γ_E, γ_A, and ρ is from 0 to 1; they are selected as 0.5, 0.1, 0.01, 0.5, and 0.8 in this article, respectively.

4. Experiments

4.1. Experimental Data

To validate the effectiveness of the RECF method, we conducted controlled experiments using three airborne hyperspectral datasets from Pavia University, Chikusei, and Xiong’an, as well as hyperspectral and multispectral remote sensing images obtained from the HJ-2 satellite, as shown in

Table 2. Main parameters of experimental data.

Datasets	Platform	Bands	Resolutions (m)	Image Size	Scene Size
Pavia University	Airborne	224	1.3	610 × 340	320 × 320
Chikusei	Airborne	128	2.5	2517 × 2335	320 × 320
Xiong’an	Airborne	250	-	3750 × 1580	320 × 320
HJ-2 HSI	Satellite	100	48	2048 × 2048	180 × 180
HJ-2 MSI	Satellite	5	16	6144 × 6144	540 × 540

.

(1)

Pavia University Datasets:

The Pavia University database contains a portion of hyperspectral data captured by the German airborne Reflective Optics System Imaging Spectrometer (ROSIS-03) over the city of Pavia, Italy, in 2003. The spectrometer continuously imaged 115 bands within the wavelength range of 0.43–0.86 μm, resulting in images with a spatial resolution of 1.3 m. The dataset measures 610 × 340 pixels and includes 9 classes of ground objects, such as trees, asphalt roads, bricks, and meadows, among others.

(2)

Chikusei Datasets:

The Chikusei database was captured by the Headwall Hyperspec-VNIR-C sensor in Chikusei, Japan, on 29 July 2014. This dataset contains 128 bands ranging from 343 to 1018 nanometers, with a size of 2517 × 2335 and a spatial resolution of 2.5 m. There are a total of 19 categories of ground objects, including urban and rural areas. This dataset was created and made publicly available by Dr. Naoto Yokoya and Prof. Akira Iwasaki from the University of Tokyo.

(3)

Xiong’an Datasets:

The Xiong’an dataset was collected by the full-spectrum multi-modal imaging spectrometer of the high-resolution aerial system developed by the Shanghai Institute of Technical Physics, Chinese Academy of Sciences. The spectral range covers 400–1000 nm with 250 bands, and the image size is 3750 × 1580 pixels. There are a total of 19 categories of ground objects, including rice stubble, grassland, elm, ash tree, etc.

(4)

HJ-2 satellite hyperspectral and multispectral data

The Environmental Disaster Reduction Satellites 2A and 2B are optical remote sensing satellites jointly developed and constructed by China’s Ministry of Ecology and Environment and Ministry of Emergency Management. They were launched on 27 September 2020 and have continuously produced remote sensing data products since then. They are equipped with four types of optical payloads: multispectral cameras, hyperspectral imagers, infrared cameras, and atmospheric correction instruments. Among them, the multispectral camera can collect multispectral data from five spectral bands, ranging from blue to near-infrared, with a spatial resolution of 16 m and a swath width of 800 km. The hyperspectral imager can collect visible HS data from 100 spectral bands with a spectral range of 450–900 nm, a spatial resolution of 48 m, and a swath width of 100 km. Since the collected HS and MS data are simultaneous and from the same perspective, there is no need to consider the impact of changes in the spectral and atmospheric lighting conditions of the ground objects due to time. At the same time, hyperspectral and multispectral data can also achieve high geometric registration accuracy.

To compare the methods more intuitively and effectively, we performed uniform preprocessing operations on the airborne HS data. From each of the three HS datasets, we extracted a 320 × 320 pixel area with rich ground object types as the hyperspectral reference data. The hyperspectral reference data were downsampled by a factor of 4, using Gaussian filtering to simulate the HS data used in the fusion experiment, as shown in Figure 6b. The hyperspectral reference data were also convolved with the spectral response function of the multispectral data to simulate the MS data used in the fusion experiment, as shown in Figure 6a. Here, the spectral response function selected was that of the Sentinel-2 satellite’s multispectral blue, green, red, and near-infrared bands, so the simulated MS data had four bands. To verify the fusion effect of the algorithm on dark areas, Gaussian noise was added to the HS data to reduce its signal-to-noise ratio. To validate the fusion effect of the algorithm on overexposed data, overexposed pixels were manually added to the HS data based on the ground objects that were prone to overexposure, as indicated by the red markers in Figure 6c.

For the real in-orbit HS and MS remote sensing imagery of the HJ-2 satellite, we selected three different types of ground object scene data and performed the same preprocessing operations. Both HS and MS data used L1A-level remote sensing data products that had undergone relative radiometric correction and geometric rough correction. Due to the difference in spatial resolution between the two types of data, which is a factor of 3, the HS data were cropped to an experimental data size of 180 × 180 pixels, while the MS data were cropped to an experimental data size of 540 × 540 pixels. To verify the algorithm’s effectiveness in handling abnormal radiance areas, the selection of the experimental area included both overexposed and underexposed regions, as shown in Figure 7. Additionally, the measured spectral response functions of both HS and MS data were utilized as known parameters by the algorithm.

4.2. Compared Methods

In order to compare the performance of various methods for hyperspectral fusion under extreme radiance conditions, we selected the following methods as comparison algorithms: Gram–Schmidt algorithm (GS) [15], modulation transfer function generalized Laplacian pyramid (GLP) [17], smoothing filter-based intensity modulation (SFIM) [13], subspace-based regularization (HySure) [51], coupled non-negative matrix factorization (CNMF) [25], fast fusion of multi-band Images (FUSE) [52], coupled spectral unmixing (Lanaras) [27], coupled sparse tensor factorization (CSTF) [35], non-local sparse tensor factorization (NLSTF) [31], spatial and spectral fusion with CNN (SSFCNN) [53], and spatial–spectral reconstruction network (SSRNET) [54].

The comparison methods are all derived from the original author’s publicly available code, and the parameter values for each method were set according to the recommendations in the original papers or the default values in the public code. Methods GS, GLP, SFIM, HySure, CNMF, FUSE, and Lanaras are from HSMS Fusion Toolbox at https://openremotesensing.net/knowledgebase/hyperspectral-and-multi-spectral-data-fusion/hsmsfusiontoolbox/, accessed on 1 March 2023. Codes of CSTF and NLSTF are available at https://github.com/renweidian/CSTF, accessed on 1 March 2023 and https://github.com/renweidian/NLSTF, accessed on 1 March 2023. Codes of SSFCNN and SSRNET are available at https://github.com/hw2hwei/SSRNET, accessed on 1 March 2023.

4.3. Evaluation Metrics

To more comprehensively evaluate the effectiveness of different algorithms, we adopted five reference-based fusion quality evaluation metrics, namely PSNR, SAM, ERGAS, Q2n, and CC [4]. All of these evaluation metrics were used in the experiment on airborne hyperspectral data. For the fusion of satellite HJ-2 HS and MS remote sensing data, since there is no true high-resolution HS data as an evaluation reference, no-reference evaluation metrics are usually selected, such as QNR [12]. However, the no-reference evaluation metric is not applicable here because it uses the original HS data, which are interfered with by extreme radiance, as a reference in the spectral dimension. So, only the true-color composite images and spectral curves of fused data are presented which can already visually demonstrate the significant effectiveness of the method proposed in this paper. The definitions of each fusion evaluation metric are as follows:

(1)

PSNR

The peak signal-to-noise rate (PSNR) for each band of HSI is defined as

$$\mathrm{PSNR}=\frac{1}{L_h}{\displaystyle \sum _{i=1}^{L_h}10 {\log }_{10}\left(\frac{\max \left({‖H\mathrm{Re}{f}_i‖}_F^2\right)}{{‖H\mathrm{Re}{f}_i-H{H}_i‖}_F^2}\right)}$$

(18)

(2)

SAM

Spectral angle mapper (SAM) evaluates the spectral distortion and is defined as

$$\mathrm{SAM}=\frac{1}{N_m}{\displaystyle \sum _{j=1}^{N_m}\arccos \left(\frac{H{H_j}^{T}H\mathrm{Re}{f}_j}{{‖H{H_j}^T‖}_2{‖H\mathrm{Re}{f}_j‖}_2}\right)}$$

(19)

(3)

ERGAS

The relative global dimensional synthesis error (ERGAS) reflects the global quality of the fused results and is defined as follows (where d is spatial downsampling factor, and μk is the mean of HH):

$$\mathrm{ERGAS}=\frac{100}{d}\sqrt{\frac{1}{L_h}{\displaystyle \sum _{i=1}^{L_h}\frac{{‖H\mathrm{Re}{f}_i-H{H}_i‖}_F^2}{{\mu }_k^2}}}$$

(20)

(4)

Q2n

Q2n is a generalization of the universal image quality index (UIQI) and an extension of the Q4 index to HS images based on hypercomplex numbers.

$$Q{2}_{N\times N}^n=\frac{{\sigma }_{z,v}}{{\sigma }_z{\sigma }_v}\cdot \frac{2\overline{z}\overline{v}}{{\left(\overline{z}\right)}^2+{\left(\overline{v}\right)}^2}\cdot \frac{2{\sigma }_z{\sigma }_v}{{\sigma }_z^2+{\sigma }_v^2}$$

(21)

(5)

CC

The correlation coefficient (CC) is computed as follows, where ρ is the Pearson correlation coefficient:

$$CC=\frac{1}{L_h}{\displaystyle \sum _{i=1}^{L_h}\rho \left(H{\mathrm{Ref}}_i,\mathrm{H}{\mathrm{H}}_i\right)}$$

(22)

4.4. Experimental Results

(1)

Results on the airborne HS dataset

The evaluation metric results of various methods on the airborne HS dataset are shown in

Table 3. Evaluation metrics on the Pavia University dataset.

Method	Pavia University
	PSNR	SAM	ERGAS	Q2n	CC
GS	39.1229	2.9641	2.0209	0.9810	0.9903
GLP	35.9758	2.2076	2.9290	0.9830	0.9858
SFIM	34.5101	2.2498	3.4474	0.9800	0.9806
HySure	37.1035	3.4303	2.6331	0.9776	0.9886
CNMF	38.7076	2.2799	2.2632	0.9833	0.9921
FUSE	37.5293	3.0178	2.6681	0.9803	0.9869
Lanaras	37.6645	2.7155	2.3391	0.9806	0.9895
CSTF	37.6332	2.6852	2.3087	0.8012	0.9878
NLSTF	44.0807	2.3732	1.4578	0.9075	0.9945
SSFCNN	39.9397	2.7074	1.8341	0.9816	0.9922
SSRNET	41.2645	2.5188	1.7124	0.9844	0.9935
RECF	44.6691	2.2393	1.4514	0.9866	0.9949

,

Table 4. Evaluation metrics on the Chikusei dataset.

Method	Chikusei
	PSNR	SAM	ERGAS	Q2n	CC
GS	43.6715	2.0144	2.8178	0.9772	0.9897
GLP	41.3536	1.2231	3.4620	0.9811	0.9868
SFIM	40.4918	1.2953	3.6851	0.9794	0.9852
HySure	42.2294	2.4063	3.7914	0.9524	0.9852
CNMF	43.2031	1.5914	3.2515	0.9739	0.9884
FUSE	41.7854	2.1761	3.4878	0.9698	0.9876
Lanaras	42.8317	1.8481	3.4735	0.9698	0.9854
CSTF	40.1310	2.0358	3.7386	0.9971	0.9832
NLSTF	49.6090	1.6227	2.2853	0.9982	0.9916
SSFCNN	39.0689	2.1279	3.1738	0.9654	0.9874
SSRNET	40.1855	1.9724	3.0532	0.9613	0.9855
RECF	49.9857	1.3416	2.2541	0.9792	0.9926

and

Table 5. Evaluation metrics on the Xiong’an dataset.

Method	Xiong’an
	PSNR	SAM	ERGAS	Q2n	CC
GS	38.6550	0.8677	1.0265	0.9481	0.9905
GLP	36.6875	0.6300	1.1409	0.9652	0.9878
SFIM	36.0686	0.6725	1.2092	0.9646	0.9862
HySure	37.0925	1.4416	1.2198	0.9267	0.9870
CNMF	37.2273	0.8035	1.1261	0.9667	0.9881
FUSE	37.6000	0.8757	1.1778	0.9622	0.9872
Lanaras	37.7233	0.9458	0.9180	0.9543	0.9921
CSTF	39.2185	0.8638	0.8334	0.8629	0.9926
NLSTF	46.4314	0.6905	0.4492	0.9529	0.9975
SSFpecCNN	33.9311	1.7599	1.7292	0.9824	0.9844
SSRNET	33.3463	1.6577	1.9545	0.9209	0.98197
RECF	47.6076	0.5964	0.3739	0.9844	0.9985

. It can be seen that our RECF method has clear advantages in the three metrics of PSNR, ERGAS, and CC, and it also performs well in the two metrics of SAM and Q2n. In the Pavia University dataset, the RECF method achieved optimal values for PSNR, ERGAS, Q2n, and CC metrics, with scores of 44.6691, 1.4514, 0.9866, and 0.9949, respectively, while achieving a second-best value for the SAM metric, with a score of 2.2393. In the Chikusei dataset, the RECF method obtained optimal values for PSNR, ERGAS, and CC metrics with scores of 49.9857, 2.2541, and 0.9926, respectively, and once again, a second-best value for the SAM metric, with a score of 2.2393. The Xiong’an dataset simulated severe extreme radiance interference, where the RECF method demonstrated remarkable effectiveness, significantly outperforming other methods, with its PSNR, SAM, ERGAS, Q2n, and CC metrics achieving scores of 47.6076, 0.5964, 0.3739, 0.9844, and 0.9985, respectively.

In addition, among the other comparison algorithms, the GLP, NLSTF, and SSRNET methods also achieve good results. The GLP algorithm performs well in the SAM metric, NLSTF has a better effect in the PSNR metric, and the deep learning-based SSRNET method also achieves overall good results in the comprehensive metrics.

The performance of various methods on the true color band composite images and SAM-maps of the airborne HS dataset is shown in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13. Through the true color band composite images, it can be seen that our proposed RECF method achieves the most visually impressive results for overexposed areas. The overexposure phenomenon in the experimental HS data is effectively eliminated in the fused high-resolution hyperspectral data. In the true color images of the fusion results from the three datasets, this conclusion can be clearly seen through the locally enlarged overexposed buildings in the yellow box in Figure 8, Figure 10 and Figure 12. In addition, by comparing with the reference data, the color of the overexposed areas processed by the RECF algorithm is effectively preserved, indicating that the RECF method achieves good spectral fidelity.

Through the SAM-maps, it can be seen that, for low-radiance areas such as water bodies or shadows, our proposed RECF method also achieves the best visual results, with SAM errors remaining low in these areas. The conclusion can be clearly observed from the river areas in the SAM-maps of the fusion results from the Chikusei dataset (Figure 11) and the Xiong’an dataset (Figure 13). A higher SAM value represents a greater spectral error, while the RECF method maintains the lowest SAM values in the river areas.

Among other methods, NLSTF and SSRNet have achieved good visual effects on true color band composition images and SAM-maps.

Overall, on the airborne HS dataset with superimposed overexposure and underexposure treatments, the RECF algorithm achieves the most comprehensive and excellent metric results and stands out in terms of visual effects.

(2)

Results on HJ-2 satellite HS remote sensing images

The performance of various methods on real on-orbit HS and MS remote sensing data from the HJ-2 satellite is shown in Figure 14, Figure 15 and Figure 16 and spectral curves of fused HH data are shown in Figure 17. Three typical scenes were selected for the experiment, including manmade buildings, airports, vegetation, water bodies, and other ground features. From the original LR-HS images, it can be observed that overexposure is a common issue in real HS remote sensing images. Furthermore, due to the imaging mechanism of the camera, the spectrum of overexposed areas may be chaotic, manifesting as color distortion in true-color images, such as the red color on the roof of the airport terminal in Scene 2 (Figure 15).

Due to the absence of reference data and the lack of no-reference evaluation metrics suitable for hyperspectral fusion affected by extreme radiance interference, a subjective evaluation was conducted here solely based on the visual effects of the true-color composite images of the fused data. It can be seen that the RECF algorithm proposed in this paper achieves better results on this metric. In the fused HR HS true color composite images, the RECF algorithm outperforms other algorithms, especially in overexposed areas where its impact is almost visually undetectable, and the fused data quality is close to that of the multispectral true color band composite images.

It should be noted that the deep learning method SSRNet, which we selected for processing real HS remote sensing data fusion tasks, exhibits significant distortion in both color information and texture details. This is because SSRNet is a supervised learning method, and we trained the model using the same experimental approach as for the airborne HS dataset by constructing training data through downsampling of HS data. However, the modal differences between actual MS and HS data differ from those between HS data and its own downsampled version, making this approach unsuitable for HS data fusion tasks in real-world applications.

4.5. Discussion

To validate the effectiveness of the proposed RECF method for hyperspectral fusion under extreme radiance conditions, including overexposed and underexposed areas, this paper conducted controlled experiments using airborne hyperspectral data and real on-orbit HS and MS remote sensing data from the HJ-2 satellite.

In the airborne HS data experiment, we selected and extracted specific a scene from the dataset that contains both high-radiance and low-radiance regions, artificially constructed overexposed pixels, and added noise to simulate the low-SNR characteristic of the underexposed areas. The degree of preprocessing we apply to the experimental scene data determines the evaluation metric results. By adding more overexposed areas and greater noise, the proposed method achieves more significant effects.

In the HJ-2 satellite HS/MS data experiment, we evaluate the proposed method only based on the visual effects of the fused images and spectral curves. We do not employ a reduced resolution of the original HS data for reference-based evaluation, nor do we use unsupervised fusion evaluation metrics. This is because both approaches consider the spectral information of the original HS data as ground truth reference, which does not align with the actual problem that the proposed method intends to solve.

We also observed that supervised learning and data-driven deep learning methods can suffer from significant distortion in fusion outcomes during practical tasks. This is due to the lack of HR HS data for supervised learning in real-world scenarios. Alternative approaches, such as training deep models through the downsampling of HS data, are limited in effectiveness due to differences in data modalities. Therefore, we believe that the focus of hyperspectral fusion research for practical applications should be on unsupervised learning methods.

5. Conclusions

This paper proposes a hyperspectral and multispectral data fusion algorithm called RECF (radiance extreme area compensation fusion) to address the severe spectral distortion issues. Based on the traditional unmixing-based fusion architecture, RECF adds the following steps, including constructing an error map during iterative unmixing, adding local smoothing constraints to the endmember and abundance matrices, and utilizing multispectral data for overexposure compensation.

Based on three airborne hyperspectral datasets from Pavia University, Chikusei, and Xiong’an, we simulated hyperspectral data affected by extreme radiance interference and conducted a reference-based evaluation. RECF achieved the best results in the overall evaluation; for instance, its PSNR reached 47.6076 and SAM reached 0.5964 on the Xiong’an dataset. Additionally, we conducted experiments on HJ-2 satellite hyperspectral and multispectral data. RECF also achieved optimal visual effects on both the fused images and spectral curves, thus demonstrating that our method is particularly well-suited for handling unsupervised fusion tasks of real hyperspectral and multispectral remote sensing data.

The fusion method-based on radiance extreme compensation essentially proposes the idea of utilizing the local similarity of remote sensing ground object spectra to achieve fusion compensation for underexposed areas, and leveraging the imaging mechanisms of multispectral data that are less prone to overexposure to achieve replacement compensation for overexposed areas after fusion. In this paper, we combined it with the spectral unmixing method of non-negative mean decomposition and achieved significant results. In the future, we can also explore combining it with other fusion methods, such as tensor decomposition and unsupervised deep learning.