A Spectral Enhancement Method Based on Remote-Sensing Images for High-Speed Railways

Abstract

This paper proposes a pansharpening model in order to obtain remote-sensing images with high spatial resolution and high spectral resolution. Based on a generic component substitution (CS) fusion framework, the model utilizes the difference between the high-frequency component of the panchromatic (PAN) image and the high-frequency component of the luminance (L) image to express the missing spatial detail information of the ideal high-resolution multispectral (HRMS) image. A rolling guidance filter (RGF) is used in this framework to achieve the effective extraction of high-frequency information from remote-sensing images while reducing the spectral distortion of subsequent operations. The modulation transfer function (MTF) values of the sensor are also applied to the selection of adaptive weighting coefficients to further improve the spectral fidelity of the fused images. At the same time, the choice of suitable interpolation and gain coefficients improves the generalizability of the model while reducing spectral and spatial distortions. Finally, the use of a guided filter (GF) also greatly improves the quality of the fused image. The experimental results show that the model can effectively improve the spatial resolution for foreign objects at the perimeter of high-speed railways, while also ensuring the color fidelity of foreign objects such as colored steel tiles.

1. Introduction

China’s railway network is dense, and it is situated in a complex environment. With the growing development and popularity of the high-speed railway network, the safety of high-speed railway operations has become a major concern. In recent years, there have been many cases of high-speed railway delays or accidents caused by the intrusion of foreign objects (such as colored steel tiles, billboards, etc.) or natural disasters such as mudslides, so the development of early warning and detection technologies suitable for sensing foreign objects around high-speed railways has become an important research direction. The wide-area surveillance characteristics of remote-sensing satellites, especially with the increase in the number of satellite return cycles and ground resolution, make their application in wide-area monitoring and early warning along railway lines even more advantageous. The images obtained from remote-sensing satellites are usually panchromatic (PAN), multispectral (MS), and hyperspectral (HS) images; among these, the PAN images have higher spatial resolution but lack spectral information, while MS images contain spectral information of multiple bands, but they have lower spatial resolution than PAN images. As the spectral information of foreign bodies around high-speed railways is unique and this is equivalent to its own label, it is the key to the early warning of foreign bodies around high-speed railways. The integrity of the spatial information of foreign objects around the high-speed railway not only affects the comfort of our human eyes to watch the image, but also makes it easier for us to make sample identification labels. Therefore, remote-sensing early warning of foreign objects around high-speed railways requires both sufficient ground resolution and accurate multispectral information. In summary, combining the advantages of PAN and MS images, and obtaining high-spatial-resolution multispectral (HRMS) images using image processing methods, is of great significance for improving the surveillance and early warning capability for foreign bodies on the ground around high-speed railways.

2. Related Works

To date, a large number of pansharpening methods have been developed. According to the underlying logic of the image processing, pansharpening can be broadly classified into three categories [1]: methods based on multi-resolution analysis (MRA), methods based on component substitution (CS), and methods based on variational optimization (VO).

MRA-based methods consider that MS images lose a lot of spatial detail information compared to HRMS images. The basic idea of the MRA-based methods is to add the spatial detail information of PAN images to MS images to achieve sharpening enhancement of MS images [2]. In extracting the detail information of PAN images, the principle of multi-scale decomposition of remote-sensing images is normally used, with methods such as undecimated wavelet transform (UDWT) [3], nonsubsampled contourlet transform (NSCT) [4], generalized Laplacian pyramid (GLP) [5], “à trous“ wavelet transform [6], edge-preserving filters [7], etc. In terms of injecting detailed information, common models include the context-based decision (CBD) model [8], space data mining (SDM) model [9], high-pass modulation (HPM) model [10], enhanced context-based (ECB) model [5], etc. As MRA-based methods are based on MS images, the color fidelity of multispectral fused images is higher while also improving spatial resolution, which is conducive to effective identification of ground markers, but the spatial quality of the fusion often depends on the multi-scale decomposition method and the choice of injection model [11].

CS-based methods start with the idea that PAN images can be considered as high-frequency features in HRMS images [12], so they use PAN images to replace spatial information features in the MS image feature space to achieve sharpening enhancement of multispectral images. Because the sharpened image contains all the high-frequency features of the PAN image, its spatial resolution is high, and the visual effect of the image edge features is good. Due to their simple operation and easy implementation, many of these algorithms are widely used in various types of remote-sensing image processing software, and they can be divided into intensity–hue–saturation (IHS) [13], principal component analysis (PCA) [14], Gram–Schmidt (GS) [15], and so on, depending on the feature space formed by the MS image. Since CS-based methods replace high-frequency features of MS images, resulting in the degree of correlation between the replaced components and PAN images often determining the final sharpening effect, most subsequent research improvements have also focused on how to increase that correlation. Examples of this are the partial replacement adaptive CS (PRACS) algorithm [16], designed by simulating the spectral response through applying a linear regression strategy that applies regression coefficients obtained at low-spatial-resolution scales to weighted parameters; the Brovey algorithm [17], designed by using adaptive gain coefficients; and the band-dependent spatial detail (BDSD) algorithm [18], designed by minimizing the mean square error of the degraded model. In recent years, many scholars have combined the CS-based framework with many new models to make further innovations. For example, in 2016, Lari et al. combined the IHS algorithm in the CS-based model with the “à trous” wavelet (ATW) transform to design the IHS-ATW algorithm [19], which uses the I image generated by the IHS transform and the panchromatic image for ATW fusion to generate a new intensity component, reducing the spectral distortion of the IHS algorithm. In 2017, Zhong et al. combined the MRA-based model with the BDSD algorithm in the CS-based model, which designed the generalized BDSD (GBDSD) algorithm [20] by considering the low-pass weighted components of panchromatic images into the generalized expression of the CS-based model, improving the spectral resolution of the fused images of the BDSD algorithm. In 2019, Liu et al. applied the transform-based gradient transfer model to the synthesis of replacement components of panchromatic images to design pansharpening with the transform-based gradient transferring (PTBGT) algorithm [21], which improves the retention of spectral information in multispectral images. In 2021, Dong et al. used the binary partition tree (BPT) image segmentation algorithm to further estimate the calculation of the replacement component and the injection gain of the panchromatic image in the CS-based model [22], reducing the spectral distortion by improving the spectral similarity between the panchromatic image and its replacement component. However, due to the different spectral response of remote-sensing satellite PAN image detectors and MS image detectors, spectral distortion when using CS-based methods is inevitable, so how to reduce this distortion is one of the main directions of research.

VO-based methods focus on the application of constrained optimization to the solution of the system of disequations between PAN images, MS images, and HRMS images [23]. It formulates the process of panchromatic sharpening as an inverse problem of mapping high-resolution multispectral images to low-resolution multispectral images using reasonable assumptions and a priori knowledge. For the solution of a morbid problem, regularization is usually required to obtain the best results. Many subsequent improvements to this class of methods have been based on improvements to regularization methods. For example, Li and Leung [24] adaptively adjusted the weight of panchromatic images in the sharpening process by constraining the regularization parameters, and Zhang [25] adaptively adjusted the regularization parameters using the gradient descent method. Typically, VO-class-based methods require the input of multiple panchromatic and multispectral images from the same scene. As a result, the fusion results of this method are poor when large spatial and spectral errors are inherent in the input data. Benefiting from the development of sparse representation theory in signal science, its theory has also been applied in the field of remote-sensing image fusion. Unlike the rest of the VO class of methods, the sparse representation approach considers that HRMS images can be obtained by constructing an overcomplete dictionary of high-resolution multispectral images and their sparse coefficients [26]. Most of the subsequent studies have focused on how to construct overcomplete dictionaries. This class of methods often requires a large amount of runtime and runspace, and small improvements in fusion quality require significantly higher running costs. Nowadays, VO-based methods cannot yet be applied to engineering projects.

In order to improve the fusion image effect and enhance the surveillance and early warning capability for foreign objects on the ground around the railway, this paper investigates an edge-preserving adaptive CS algorithm (EPACS). The original CS-based method has been optimized by solving two problems: (1) how to increase the correlation between the PAN image and each channel of the MS image to reduce the spectral distortion of the fused image caused by the global variability of the relative spectral response functions and low correlation, and (2) how to increase the detail injection of the PAN image to improve the spatial resolution of the MS image.

3. Edge-Preserving Adaptive CS Algorithm

Since HRMS images of actual ground scenes are not directly available, there is no direct criterion for evaluating the quality of reconstructed HRMS images. Theoretically, it would be desirable for the HRMS image to have the same rich spectral information as the MS image, for which the upsampled image ($\widetilde{\mathrm{MS}}$ image) of the MS image can be considered to replace the low-frequency part of the HRMS image; at the same time, it would be desirable for the HRMS image to have the same spatial detail features as the PAN image, and the difference between the amount of detail in each channel of the HRMS image should be similar to that of the MS image, so the high-frequency part of the HRMS image can be considered to be replaced by the relationship between the high-frequency part of the PAN image $I_{PAN}$ and the MS image $I_{MS}$.

The HRMS image $I_{HRMS}$ is represented as

$$I_{HRMS}^n=I_{HRM{S}_{high}}^n+I_{HRM{S}_{low}}^n:=w(I_{PAN},I_{M{S}_H}^n)+I_{\widetilde{MS}}^n$$

(1)

where “:=” is the defining symbol (indicating that it does not exist by convention in nature); $I_{HRM{S}_{high}}^n$ and $I_{HRM{S}_{low}}^n$ are the high- and low-frequency portions of the nth channel of the HRMS image, respectively; $I_{\widetilde{MS}}^n$ is the nth channel of the $\widetilde{\mathrm{MS}}$ image; and $w(I_{PAN},I_{M{S}_H}^n)$ is the high-frequency information extracted between the PAN image and the high-frequency part $I_{M{S}_H}^n$ of the nth channel of the MS image.

As shown in Figure 1, the CS-based fusion framework is often defined as

$$w(I_{PAN},I_{M{S}_{High}}^n)=g^n(I_{PAN}-I_L)$$

(2)

$$I_L=w^n{I}_{\widetilde{MS}}^n$$

(3)

where $I_L$ is a luminance image similar to the PAN image synthesized using information from each band of the $\widetilde{\mathrm{MS}}$ image; gⁿ is the injection gain of the nth channel band detail quantity; and wⁿ is the weighting factor for the nth channel of the L image.

The EPACS framework studied in this paper is shown in Figure 2. To solve problem 1 in Section 1, the $I_{PAN}-I_L$ in the traditional CS framework is replaced by the difference image ($I_{PA{N}_H}-I_{L_H}$) between the high-frequency component $I_{PA{N}_H}$ of the PAN image and the high-frequency component $I_{L_H}$ of the L image (as shown in Figure 2, ①), and the linear regression strategy for solving the L image is optimized (as shown in Figure 2, ②), while an adaptive gain coefficient g is chosen. To solve problem 2, the model details image (D) is adaptively enhanced using a guided filter (GF) (as shown in Figure 2, ③), while the use of a rolling guidance filter (RGF) also enhances the representation of spatial information in PAN images (as shown in Figure 2). The EPACS algorithm is described in detail below.

3.1. Fused Image Detail Information Extraction

Analyzing the physical meaning of Equations (1) and (2) together, it is found that $g(I_{PAN}-I_L)$ expresses the high-frequency component of the HRMS image, i.e., the detailed features such as contours and edges of the image. The L image can be seen as an approximate PAN image synthesized using an MS image, i.e., a synthetic approximation to the PAN image on the $\widetilde{\mathrm{MS}}$ image scale, which ideally has the same spectral response as the PAN image. If the HRMS image is considered to have the same amount of detail as a panchromatic image, $I_{PAN}-I_L$ represents the amount of detail D that is missing from the MS image relative to the HRMS image; $g^n(I_{PAN}-I_L)+I_{MS}^n$ represents how this relatively missing detail is added to each MS image channel.

Since the actual spectral response relationships in each band are different for each sensor panchromatic image sensor and multispectral image sensor, the synthetic L image is only an estimate and is not equivalent to a panchromatic image on the MS image scale. That is, $I_{PAN}-I_L$ may introduce an error which causes instability in the spectral information, or even a lack of spatial information, affecting the quality of the reconstructed image. To reduce this part of the error, the process of solving for detail D is changed to $I_{PA{N}_H}-I_{L_H}$ using the edge-preserving filter. The representation of the spectral and spatial information of the fused image is increased by reducing the interference of detail-independent pixels, i.e., by reducing the error due to the subtraction of low-frequency components between images. Problem 1 in Section 1 is thus transformed into a question of how to obtain $I_{PA{N}_H}$ and $I_{L_H}$.

3.1.1. $I_{PA{N}_H}$ Acquisition

For the acquisition of $I_{PA{N}_H}$, this paper uses the RGF to process images. Its advantage over other multi-scale decomposition methods is that it can more accurately separate spatial information at large scales, edge information at medium scales, and texture information at fine scales, allowing it to blur images while preserving edge texture features. This feature helps to reduce the ringing and blending effects during the fusion process, thus making the fusion effect more in line with human visual perception.

The filtering process of the RGF consists of two main steps:

• Acquisition of the guided image using a Gaussian filter.

Using a Gaussian filter to eliminate fine structures in the original image T with a structure size smaller than $l_s$, the expression for the operation on the central pixel a of the input image is as follows:

$$G(a)=({\displaystyle \sum _{b\in S}\exp (}-\frac{\left|\right|a-b|{|}^2}{2l_s^2})\ast T_{(b)})/H$$

(4)

$$H={\displaystyle \sum _{b\in S}\exp (}-\frac{\left|\right|a-b|{|}^2}{2l_s^2})$$

(5)

where S is the set of surrounding pixels of the central pixel a; b is an element in S; and the structural scale parameter $l_s$ can be regarded as the minimum standard deviation of the Gaussian kernel.

2.

Edge recovery using a joint bilateral filter.

The previously obtained filtered image G(k¹) is used as the guide image, and joint bilateral filtering is performed for the input image to obtain the filtered image k². Then, the filtered image k² is regarded as the new guide image, and the previous step operation is repeated; the output image O(kⁱ⁺¹) after the ith iteration operation can be expressed as

$$O_{(a)}=k_{(a)}^{i+1}=\frac{1}{{\displaystyle \sum _{b\in S}U}}{\displaystyle \sum _{b\in S}U{T}_{(b)}}$$

(6)

$$U=\exp (-\frac{\left|\right|a-b|{|}^2}{2{\sigma }_s^2})-(\frac{\left|\right|k_{(a)}^i-k_{(b)}^i|{|}^2}{2{\sigma }_r^2})$$

(7)

where T is the input image; O is the output image; S is the set of surrounding pixels of the central pixel a; b is an element of S; ${\sigma }_s$ and ${\sigma }_r$ are two hyperparameters that control the structural scale weight and range weight, respectively; and kⁱ⁺¹ is the result of the ith iteration.

In summary, the acquisition of PAN image detail information $I_{PA{N}_H}$ can be formulated as

$$I_{PA{N}_H}=I_{PAN}-RGF(K,I_{PAN},{\sigma }_s,{\sigma }_r)$$

(8)

where K is the number of iterations.

3.1.2. $I_{L_H}$ Acquisition

The $I_{L_H}$ is estimated using the linear regression strategy employed by PRACS, which assumes that there is some linear relationship between the HRMS image to be derived and the PAN image, i.e., the PAN image can be synthesized with information from each channel of the HRMS image information.

$$I_{PAN}={\displaystyle \sum _{i=1}^N{w}_i{I}_{HRMS}^i}+c_i$$

(9)

From Equation (2), the image upsampled to the PAN image scale can be interpreted as the low-pass image of the HRMS image, and then the low-pass filtering of both sides of Equation (9) yields

$$I_{P_L}={\displaystyle \sum _{i=1}^N{w}_i{I}_{\widetilde{MS}}^i}+d_i$$

(10)

where $I_{P_L}$ is the low-pass images of PAN images, and c and d are the linear regression parameters.

Equation (10) is solved using the least squares method:

$$\underset{w_1,w_2\dots w_n}{\min }||I_{P_L}-{\displaystyle \sum _{i=1}^n{w}_i{I}_{\widetilde{MS}}^i|{|}^2}$$

(11)

The weight parameter w_i can be obtained using Equation (11). Up to this point, w_i, $I_{P_L}$, and $I_{\widetilde{MS}}^i$ are known. Then, solve for di in Equation (10) using the pending coefficient method to find the synthetic approximation L of the PAN image on the $\widetilde{\mathrm{MS}}$ image scale.

$$I_L={\displaystyle \sum _{i=1}^n{w}_i{I}_{\widetilde{MS}}^i}+d_i$$

(12)

The fusion process does not introduce spectral distortion if the low-pass image of the PAN image happens to be perfectly linear in relation to the channels of the MS image. However, in practice, the spectral response ranges of panchromatic and multispectral imaging from remote-sensing satellites do not coincide, making the L image not representative of a PAN image on the $\widetilde{\mathrm{MS}}$ image scale, so $I_{PA{N}_H}-I_{L_H}$ is used instead of $I_{PAN}-I_L$.

As shown in Figure 3, for the GF-2 remote-sensing satellite, the wavelength range of the PAN image is larger than the overall wavelength range of the MS image (red band, green band, blue band, near-infrared band), and when viewed in the frequency domain, it is the PAN image and the MS image that have different frequencies of light. As shown in Figure 4, the MTF curves [27] of each sensor indicate the passable frequency range of the signal. In order to make Equation (10) better satisfy the linear relationship and let both sides of it have the same frequency range, we designed a low-pass filter for PAN images based on the MTF curves of the MS image sensors, i.e., by letting both sides of Equation (10) have the same frequency range, so as to obtain a better spectral weight w to improve the quality of the fused image. Since the MTF values for the different spectral channel cut-off frequencies of the $\widetilde{\mathrm{MS}}$ images $I_{\widetilde{MS}}$ vary slightly, but the $I_{P_L}$ images are single channel, the average MTF output for each spectral channel of the $\widetilde{\mathrm{MS}}$ images is used.

$$I_{P_L}=G_{MTF(\widetilde{MS})}(I_{PAN})$$

(13)

Here, $G_{MTF\left(\widetilde{MS}\right)}$ denotes the low-pass filter designed using the average of the deadline frequency of the MTF curves of each channel of $\widetilde{\mathrm{MS}}$. The cutoff normalized Nyquist frequency of the MTF curve can be obtained by the edged edge method [28]. The MTF curve can be fitted using a Gaussian curve.

In summary, we first use the RGF filter to obtain the detailed image $I_{M{S}_H}$ of the MS image.

$$I_{{\widetilde{MS}}_H}^i=I_{M{S}^i}-RGF(K,I_{\widetilde{MS}}^i,{\sigma }_s,{\sigma }_r)i=1,2\dots N$$

(14)

Then, using the MTF filter $G_{MTF({\widetilde{MS}}_H)}$ designed through the upsampled image $I_{{\widetilde{MS}}_H}$ of the $I_{M{S}_H}$, low-pass filtering is performed for the $I_{PA{N}_H}$ image to obtain $I_{PA{N}_{HL}}$. Finally, $I_{L_H}$ is obtained by the linear regression strategy of PRACS.

$$I_{L_H}={\displaystyle \sum _{i=1}^n{w}_i{I}_{{\widetilde{MS}}_H}^i}+d_i$$

(15)

3.2. Gain Factor Selection

As described in [13], the injected gain g_i should satisfy $({\sum }_k{g}_i{)}^{-1}=1$ if the effect of differences in the spectral response functions of different sensors is ignored. However, the relationship between the panchromatic band range and the multispectral image band range is often not fully linear, and in order to increase this nonlinear relationship and to make the algorithm more adaptive for different images and different channels, the gain coefficients g₁,g₂…g_n are defined as

$$g_i=\frac{\mathrm{cov}(I_{{\widetilde{MS}}_H}^i,I_{L_H})}{\mathrm{cov}(I_{L_H},I_{L_H})}i=1,2\dots N$$

(16)

where cov(A, B) denotes the covariance between the images A and B.

The amount of detail injected into the different spectral channels of a multispectral image is

$$D_1^i=g_i(I_{PA{N}_H}-I_{L_H})$$

(17)

3.3. Enhancement of Fusion Image Detail Information

To solve problem 2 in Section 1, the spatial detail information is enhanced using the structure transfer properties of the GF [29]. The GF is an edge-preserving filter that filters the input image p by the guided image T so that the final output image q is largely similar to the input image p, but the texture part is similar to the guided image T. When the input image of the guided filter and the guided image are the same image, the guided filter will smooth the input image. When the input image and the guided image are different images, the guided filter will extract the structural information from the guided image and merge the structural information into the input image.

The expression for the GF is

$$q_i={\displaystyle \sum _j{W}_{ij}(T)p_j}$$

(18)

where i, j are the indexes of the pixel points in the image, respectively, and $W_{ij}\left(T\right)$ is the polynomial associated with the guided image T.

The guided filter theory assumes that the output image q and the guided image T are locally a linear model. In the local window $M_k$, the linear relationship can be expressed as

$$q_i=a_k{T}_i+b_k\forall i\in M_k$$

(19)

where a_k, b_k are linear model parameters.

The cost function is designed so that the mean square error between the output image q and the input image p is minimized; its cost function in the window $M_k$ is

$$E(a_k,b_k)={{\displaystyle \sum _{i\in M_k}((a_k{T}_i+b_k-p_i)}}^2+\epsilon a_k^2)$$

(20)

where $\epsilon$ is the regularization coefficient of the L2 norm.

Let N be the number of pixels in the window $M_k$. According to the condition that the partial derivative is 0 at the minimum value, Equation (20) for a_k, b_k for the partial derivative has

$$\frac{\partial \mathrm{E}}{\partial a_k}=0,\frac{\partial \mathrm{E}}{\partial b_k}=0$$

(21)

Solving Equation (21) yields

$$a_k=\frac{1}{\left|w\right|}\frac{{\displaystyle {\sum }_{i\in w_k}{T}_i{p}_i-{\mu }_k\overline{p_k}}}{{\sigma }_k^2+\epsilon }$$

(22)

$$b_k={\overline{p}}_k-a_k{\mu }_k$$

(23)

where w is the number of pixels in the window $M_k$; ${\mu }_k$ is the mean value of the guided image T over the window $M_k$; ${\sigma }_k$ is the variance of the guided image T over the range of the window $M_k$; and $\overline{p_k}$ is the mean value of the input image p over the window $M_k$.

In summary, the process of guided filtering can be expressed as

$$q=GF(T,p,r,\epsilon )$$

(24)

where r is the radius of the guided filter window.

The structure-shifting nature of the guided filter allows it to extract features that are present in the guided image but not in the input image, similar to our idea of extracting missing details from MS images. Therefore, the $I_{PA{N}_H}$ is used as the guide image, and each channel of the $I_{{\widetilde{MS}}_H}$ is used as the input image to further extract the amount of detail D₂ in order to increase the representation of the spatial information of the image as well as the spectral information, the expression of which is shown below.

$$D_2^i=I_{PA{N}_H}-GF(I_{PA{N}_H},I_{{\widetilde{MS}}_H}^i,r,\epsilon )i=1,2\dots N$$

(25)

It follows that for different spectral channels of the $I_{{\widetilde{MS}}_H}$ image, we will have different degrees of information enhancement using guided filtering. This corresponds to the fact that, in practice, the ability to express detail between different channels of an image should vary. Thus, the use of the guided filter not only enhances the representation of image detail but also improves the adaptivity between different channels.

3.4. HRMS Image Acquisition

In summary, the amount of detail we injected into each channel of the HRMS image is

$$D^i=D_1^i+D_2^i$$

(26)

Finally, the HRMS images are synthesized using a generic CS-based fusion framework as

$$I_{HRMS}^i=D^i+I_{\widetilde{MS}}^i$$

(27)

4. Experimental Results and Analysis

4.1. Selection of Data and Evaluation Indicators

Two sets of images from different satellites were used to test the validity of the EPACS method in this paper. Figure 5, Figure 6, Figure 7 and Figure 8 show those from the DEIMOS-2 remote-sensing satellite (containing common features such as trees and houses). The PAN image has a spatial resolution of 1 m and a size of 100 × 128; the multispectral image contains four channels of R, G, B, and NIR with a spatial resolution of 4 m and a size of 25 × 32 × 4. Figure 9 was derived from our remote-sensing image dataset of the high-speed railway perimeter (including the railway, nearby colorful steel tiles, and other foreign objects) based on GF-2 satellite remote-sensing images. The PAN image has a spatial resolution of 1 m and a size of 512 × 512; the multispectral image contains four channels of R, G, B, and NIR with a spatial resolution of 4 m and a size of 128 × 128 × 4.

For better visual assessment, only the B, G, and R channels were used to display the images in this paper. In order to objectively assess the imaging quality of remote-sensing images, in addition to the visual assessment of the fusion results, a quantitative assessment of the images obtained by panchromatic sharpening was also required. Since the HRMS reference image could not be obtained directly from the sensor, we used the Wald criterion to make an assessment at the degradation scale (using the fused image of the four-fold downsampled PAN image and the MS image for comparison with the initial MS image).

The evaluation metrics included ERGAS [30] (REF = 0), SAM [31] (REF = 0), RASE [32] (REF = 0), RMSE [32] (REF = 0), and UIQI [33] (REF = 1). Furthermore, in order to be able to quantitatively characterize the spectral and spatial distortions produced during the synthesis process, we assessed them on the original image scale using the QNR criterion. The metrics evaluated included D_S (REF = 0), $D_{\lambda }$ (REF = 0), and QNR (REF = 1), which are defined in the article [34].

Because there is no uniform evaluation index for the quality of remote-sensing image fusion, and the spectral response functions of different satellite multispectral image sensors and panchromatic image sensors are not exactly the same, this leads to a certain amount of subjectivity in the choice of hyperparameters. In this paper, three parameter variations of EPGAS, SAM, and QNR were used as the basis for selecting hyperparameters for different sensor images; the EPGAS and SAM represented the integrated image quality and spectral quality, respectively, under the Wald criterion, and the QNR represented the integrated image quality under the QNR criterion. Based on a number of comparative experiments, the following choices were made for hyperparameters: the hyperparameters of this framework in the DEIMOS-2 remote-sensing image fusion process were K = 4, ${\sigma }_s$ = 3, ${\sigma }_r$ = 0.8, r = 3, and $\epsilon$ = 0.1; the hyperparameters of this framework in the GF-2 remote-sensing image fusion process were K = 3, ${\sigma }_s$ = 3, ${\sigma }_r$ = 0.8, r = 2, and $\epsilon$ = 0.3. The adjustment range for each hyperparameter is as follows: the number of iterations K ranges from two to six, with a modification step of 2; RGF scale weights ${\sigma }_s$ ranges from two to six, with a modification step of 1; RGF range weights ${\sigma }_r$ ranges from 0.2 to 0.8, with a modification step of 0.2; GF window radius r ranges from two to eight, with a modification step of 2; GF regularization factor ε ranges from 0.1 to 0.4, with a modification step of 0.1. In order to demonstrate the effect of variations in the hyperparameters on image quality, some data are provided in Appendix A for easy reference.

Meaning of evaluation indicators:

• Root Mean Square Error (RMSE):

$$RMSE=\sqrt{E[{(I_{HRMS}-I_{\widetilde{MS}})}^2]}$$

(28)

where E is the mathematical expectation of the image. The lower the RMSE value, the more similar the multispectral data are to each other.

• Relative Average Spectral Error (RASE):

$$RASE=\frac{100}{{\displaystyle \sum _{i=1}^N{\mu }_i}}\sqrt{N{\displaystyle \sum _{i=1}^{N}E[{(I_{HRMS}^i-I_{\widetilde{MS}}^i)}^2]}}$$

(29)

where ${\mu }_i$ is the average value of the ith band. The lower the RASE value, the more similar the multispectral data are to each other.

• Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS):

$$ERGAS=100\frac{d_h}{d_l}\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{(\frac{RMSE(i)}{\mu (i)})}^2}}$$

(30)

$$RMSE=\sqrt{E[{(I_{HRMS}-I_{\widetilde{MS}})}^2]}$$

(31)

where d_h/d_l is the ratio of the spatial resolution of the PAN and MS images; $\mu \left(i\right)$ is the mean value of band i of the reference multispectral image; N is the number of bands of the reference MS image; and RMSE(i) is the mean square error of the reference multispectral image and the fused image at band i. ERGAS is a global error metric improved by the RASE. The lower the ERGAS value, the more similar the multispectral data are to each other.

• Spectral Angle Mapper (SAM):

$$SAM(V_{\widetilde{MS}(i)},V_{HRMS(i)})=\arccos (\frac{〈V_{\widetilde{MS}(i)},V_{HRMS(i)}〉}{‖V_{\widetilde{MS}(i)}‖\cdot ‖V_{HRMS(i)}‖})$$

(32)

where $V_{\widetilde{MS}(i)}$ and $V_{HRMS(i)}$ are the vectors constructed by concatenating the bands of the $\widetilde{\mathrm{MS}}$ image at position i and the vectors constructed by concatenating the bands of the HRMS image at position i, respectively. <...> denotes a dot product operation; ||...|| denotes a vector two-norm operation; and SAM is the absolute value of the spectral angle of two vectors and reflects the degree of spectral distortion. If the spectra of $\widetilde{\mathrm{MS}}$ and HRMS are the same, then the SAM value is 0.

• Universal Image Quality Index (UIQI)

The UIQI reflects the correlation, average brightness, and contrast between the two images and takes values in the range [−1, 1], with a value of 1 when the fused image is identical to the reference image.

$$UIQI=\frac{{\sigma }_{A,B}}{{\sigma }_A\cdot {\sigma }_B}\frac{2\overline{A}\cdot \overline{B}}{[{(\overline{A})}^2+{(\overline{B})}^2]}\frac{2{\sigma }_A\cdot {\sigma }_B}{({\sigma }_A^2+{\sigma }_B^2)}$$

(33)

Here, $\overline{A}$ and $\overline{B}$ are the pixel means of the reference image and the fused image, respectively; ${\sigma }_A$ and ${\sigma }_B$ are the standard deviations of the reference image and the fused image, respectively; and ${\sigma }_{A,B}$ is the covariance of the reference image and the fused image.

• Quality with No Reference (QNR):

This criterion consists of two metrics, spectral distortion and spatial distortion, which are combined to produce a unified quality metric, QNR; the higher the value, the better the result of the fusion.

$$QNR={(1-D_{\lambda })}^{\alpha }{(1-D_s)}^{\beta }$$

(34)

The values of $\alpha$ and $\beta$ are usually set to 1.

$$D_{\lambda }=\sqrt[p]{\frac{1}{N(N-1)}{\displaystyle \sum _{i=1,r=1}^N{\displaystyle \sum _{i\ne r}^N|UIQI(I_{\widetilde{MS}}^i,I_{\widetilde{MS}}^r)-UIQI(I_{HRMS}^i,I_{HRMS}^r){|}^p}}}$$

(35)

$$D_s=\sqrt[q]{\frac{1}{N}{\displaystyle \sum _{i=1}^N|UIQI(I_{\widetilde{MS}}^i,I_{PAN})-UIQI(I_{HRMS}^i,I_{PAN}){|}^q}}$$

(36)

Here, N is the number of multispectral image channels; i and r are the number of channels in the image; p and q are image deflation factors with a default value of 1; and $D_{\lambda }$ and D_S are the spectral distortion and spatial distortion, respectively, which are calculated using the difference in UIQI values between the original image and the fused image.

4.2. Comparison of Fusion Effects

In order to verify the effectiveness of the improved framework of the ERGAS algorithm used in this paper, algorithms were compared for ①, ②, ③ of Figure 2, and various excellent remote-sensing image fusion algorithms were selected for a cross-sectional comparison of fusion algorithms.

4.2.1. Comparison of the Improvement Effects of Different Fusion Image Methods

(1)

Comparison of the effects of $I_{PA{N}_H}-I_{L_H}$ versus $I_{PAN}-I_L$

To verify that $I_{PA{N}_H}-I_{L_H}$ can increase the fused image spectral information, the IHS algorithm was used as an example. First, the RGF filter was used to obtain the high-frequency images $I_{{\widetilde{MS}}_H}$ and $I_{PA{N}_H}$ of the $\widetilde{\mathrm{MS}}$ image and the PAN image, respectively, and then the IHS transform was performed for $I_{{\widetilde{MS}}_H}$ to obtain the image $I_{L_H}$, after which the subsequent steps of the IHS algorithm were performed. A comparison of the reconstructed images with the original IHS algorithm is given in Figure 5, and a comparison of the corresponding evaluation parameters is given in

Table 1. Objective evaluation of the improved IHS algorithm using $I_{PA{N}_H}-I_{L_H}$ and objective evaluation of the original IHS algorithm, where bolded indicates optimal.

	ERGAS	SAM	${\mathit{D}}_{\mathit{\lambda }}$	DS
Improved IHS	3.8855	4.0969	0.0930	0.1943
Original IHS	3.0250	4.9747	0.1692	0.3700

. It can be seen that the improved IHS method can effectively reduce the spectral distortion in the green part of the fused image, and the spectral-distortion-related indexes SAM and $D_{\lambda }$ are greatly reduced; however, the performance in the comprehensive evaluation index ERGAS was not as good as the IHS algorithm, which was due to the fact that some fine spatial structures were not shown (e.g., the segmentation line of the roof in the lower left corner). Firstly, because the L component of the IHS transform decomposition of the multispectral image was different from that of the PAN image, the fine structure of the image was not well expressed by $I_{PA{N}_H}-I_{L_H}$. Secondly, because the aim of any algorithm is a trade-off between spectral resolution and spatial resolution, the original IHS algorithm focused on the spatial information of the image, leaving the color information poorly represented. The improved algorithm does not enhance the spatial information of the image, but it does make a better trade-off between the two.

(2)

Comparison of MTF filter optimization status

To verify the performance of the low-pass filter that was designed based on the MTF of MS images in this framework, for purposes of comparison, a low-pass filter was designed with four-fold downsampling followed by four-fold upsampling of panchromatic images using bilinear interpolation, as is commonly used in the MRA-based method, and the rest of the steps were the same as in the framework of this paper.

A comparison of the reconstructed Images with the bilinear interpolation is given in Figure 6, and a comparison of the corresponding evaluation parameters is given in

Table 2. Objective evaluation of different low-pass filtering methods, where bolded indicates optimal.

	ERGAS	SAM	${\mathit{D}}_{\mathit{\lambda }}$	DS
Bilinear interpolation filtering	2.6051	5.0796	0.0930	0.1943
Low-pass filtering based on MTF	2.3356	3.5350	0.0793	0.2961

. It can be seen that the low-pass filter designed based on the MTF has a somewhat higher green saturation compared to the bilinear interpolation, but the other differences are not significant; the MTF-based filter has greater improvement in both the spectral distortion index SAM and $D_{\lambda }$; the comprehensive evaluation index ERGAS and the spatial distortion index D_S also get a small improvement, which verifies the relevant theory in Section 3.1.2.

(3)

Comparison of image information with and without guided filtering enhancement

To verify that guided filtering enhances the representation of frame image information, the frames in this paper were compared with and without GF, and the remaining steps were identical. Figure 7 gives a comparison of the reconstructed images and

Table 3. Objective evaluation of image information with or without GF, where bolded indicates optimal.

	ERGAS	SAM	RASE	${\mathit{D}}_{\mathit{\lambda }}$
GF is not used	3.0447	3.8588	23.7845	0.0865
GF is used	2.3356	3.5350	18.4927	0.0793

gives a comparison of the corresponding evaluation parameters. It can be seen that the reconstructed image performed better on tiny features (e.g., on the division line of the roof in the lower left corner) after using the guided filter to enhance the image information. The guided filter reconstructed images had large improvements in the metric RASE, which describes spatial distortion, and in the composite evaluation metric ERGAS; small improvements were also obtained in the metrics SAM and $D_{\lambda }$, which describe spectral distortion, indicating that the use of guided filtering can enhance the representation of detailed information, especially image detail, in the images framed in this paper.

4.2.2. Comparison of the Effects of the Complete Algorithm

To verify the overall effectiveness of the EPACS algorithm used in this paper, the GIHS algorithm, PCA algorithm, GS algorithm, Brovey algorithm, BDSD algorithm PRACS algorithm, IHS-ATW algorithm and PTBGT algorithm in the CS-based algorithm, and the HPF algorithm, MTF-GLP (GLP [5] with MTF [10] with unitary injection model) algorithm, MTF-GLP-CBD (MTF-GLP with CBD) algorithm [8] and MTF-GLP-ECB (MTF-GLP with ECB) [5] algorithm in the MRA-based algorithm were selected to compare the fusion effects.

Figure 9 gives a comparison of the fusion effects of these methods on the GF-2 satellite images.

Table 4. Comparison of objective evaluation indicators for each fusion method used on GF-2 images, where bolded and underlined indicates best, underlined indicates second best.

Underlying Logic	Reference	ERGAS	SAM	RMSE	RASE	UIQI	${\mathit{D}}_{\mathit{\lambda }}$	Ds	QNR
CS	GIHS	0.7712	1.6332	0.0393	7.0554	0.9643	0.1801	0.3283	0.5507
	PCA	0.7843	1.8182	0.0411	7.3727	0.9634	0.1322	0.2879	0.6180
	GS	0.7792	1.7442	0.0403	7.2331	0.9624	0.1276	0.2908	0.6188
	Brovey	0.9216	1.6728	0.0403	7.2386	0.9623	0.1634	0.3137	0.5741
	BDSD	0.5939	1.7495	0.0307	5.4830	0.9810	0.0957	0.2545	0.6741
	PRACS	0.5297	1.6732	0.0386	5.1288	0.9820	0.0908	0.2512	0.6809
	IHS-ATW	0.5847	1.7356	0.0312	5.2748	0.9785	0.0966	0.2478	0.6795
	PTBGT	0.5798	1.7421	0.0347	5.4716	0.9736	0.0975	0.2316	0.6954
MRA	HPF	0.5908	1.5210	0.0311	5.5842	0.9787	0.0967	0.1644	0.7548
	MTF-GLP	0.5776	1.5759	0.0307	5.3676	0.9810	0.1220	0.2348	0.6724
	MTF-GLP-CBD	0.6792	1.6201	0.0356	6.3719	0.9740	0.1218	0.2139	0.6904
	MTF-GLP-ECB	0.5941	1.5925	0.0315	5.6428	0.9790	0.0677	0.1403	0.8015
EPACS		0.5927	1.7676	0.0306	5.5018	0.9826	0.0601	0.1183	0.8287

gives objective evaluation indicators for each fusion method used on the GF-2 satellite images, where underlined numbers indicate they are sub-optimal, and bolded and underlined numbers indicate they are optimal. In terms of visual results, the algorithms performed well in expressing the edge features of simple foreign objects such as colorful steel tiles around the railway. However, the GIHS, PCA, GS, and Brovey algorithms showed significant color distortion in expressing the color of the tiles, especially for white tiles, which was mainly due to the inconsistent spectral response of the different sensors, leading to the offsetting of the injected radiation during the fusion process. Both the IHS-ATW algorithm and the PTBGT algorithm improve the spectral fidelity of the CS-based algorithms, but the edge detail is significantly under-injected; in addition, the MTF-GLP-CBD algorithm and the MTF-GLP-ECB algorithm showed intermittent local edge features when describing white-colored steel tiles at railway perimeters, indicating that this type of algorithm has different sensitivities for different regions of the image. On the other hand, the EPACS, PRACS, and MTF-GLP algorithms were visually better in this paper. In terms of the evaluation metrics, the EPACS algorithm achieved optimal performance in terms of RMSE, UIQI, $D_{\lambda }$, D_s, QNR, and many other metrics. Compared with the traditional CS algorithm, the EPACS algorithm showed stronger spectral fidelity, and the fused image could more accurately describe the color information of dangerous features such as colorful steel tiles near the railway, while improving the spatial resolution compared to the original upsampled multispectral image.

Figure 8 gives a comparison of the fusion effects of each algorithm for the DEIMOS-2 satellite images.

Table 5. Comparison of objective evaluation indicators for each fusion method used on DEIMOS-2 images, where bolded and underlined indicates best, underlined indicates second best.

Underlying Logic	Reference	ERGAS	SAM	RMSE	RASE	UIQI	${\mathit{D}}_{\mathit{\lambda }}$	Ds	QNR
CS	GIHS	3.0250	4.9747	0.0211	25.3484	0.9635	0.1692	0.3700	0.5234
	PCA	2.9820	5.2747	0.0201	24.1421	0.9665	0.1665	0.3651	0.5292
	GS	3.0448	5.0617	0.0208	24.8984	0.9644	0.1632	0.3664	0.5302
	Brovey	2.8966	3.8172	0.0202	24.2460	0.9667	0.1104	0.3192	0.6056
	BDSD	2.5411	4.3736	0.0170	20.2631	0.9753	0.0224	0.1808	0.8008
	PRACS	2.8964	4.2900	0.0176	21.2283	0.9768	0.0830	0.2488	0.6888
	IHS-ATW	2.4487	4.3846	0.0165	20.4300	0.9766	0.0754	0.2356	0.7067
	PTBGT	2.4803	4.3500	0.0171	21.6530	0.9758	0.0633	0.2433	0.7088
MRA	HPF	2.8766	3.1819	0.0189	22.6973	0.9721	0.1327	0.2426	0.6569
	MTF-GLP	2.6046	4.0451	0.0169	20.2425	0.9787	0.1510	0.3103	0.5856
	MTF-GLP-CBD	2.8059	4.0782	0.0186	22.1715	0.9753	0.1527	0.2884	0.6029
	MTF-GLP-ECB	2.6230	3.8263	0.0170	20.3708	0.9785	0.0775	0.1876	0.7550
EPACS		2.3356	3.5350	0.0154	18.4927	0.9822	0.0222	0.1867	0.7952

gives objective evaluation metrics for each of the fusion algorithms used on the DEIMOS-2 satellite images. From the visual effect, the GIHS, GS, and PCA algorithms in the CS class showed different degrees of spectral distortion when expressing the green region in the lower right corner; the Brovey, the PRACS, and the IHS-ATW algorithms in the CS class offered some improvement for the spectral distortion phenomenon, but compared with the fusion results of the EPACS algorithm used in this paper, both the color saturation and brightness of the lower-right-corner region were far inferior. The HPF, MTF-GLP, and MTF-GLP-CBD algorithms in the MRA-based algorithms produced a grainy feel to the edges of the image, although there was no significant distortion in the green area in the lower right corner. This was mainly due to the insufficient detail injection in the panchromatic images. The MTF-GLP-ECB, the BDSD, and the PTBGT algorithms all performed well in terms of color and detail information in local areas, but the green area in the upper left corner of the MTF-GLP-ECB algorithm appeared locally blurred, and the red area in the upper right corner of the BDSD algorithm was significantly too bright and blurred at the edges, indicating that both algorithms exhibited good local adaptation but not good global adaptation. In contrast, the EPACS algorithm had no significant spectral distortion and showed good edge features for complex features, indicating that the algorithm can effectively reduce spectral distortion while ensuring effective extraction of detailed information.

As can be seen from the table, the CS algorithms such as GIHS, PCA, and GS had much higher global error indicators such as ERGAS, SAM, RMSE, and RASE than the MRA methods, while the UIQI was much lower than for the MRA methods, indicating that the images synthesized by these methods were less similar to the MS image spectral data. The root cause of this was that the global differences and low correlation of the relative spectral response functions were not considered in the fusion process, resulting in the addition of some redundant information to the MS images. In contrast, the Brovey algorithm with the design of adaptive gain coefficients, the PRACS algorithm with its adaptive weight coefficients, the IHS-ATW algorithm with its combining the ATW transform with the IHS algorithm, and the PTBGT algorithm with its design of a variational gradient transfer model all reduced the impact of the low correlation between the images of the CS-based algorithms, but this single improvement still did not allow them to outperform the MRA-based algorithms. The MRA-based algorithms, due to the design of the underlying logic and the MTF of the sensor, resulted in more similarity between the fusion results and the target multispectral data, and outperformed the CS class in terms of the global evaluation metrics.

In this paper, the EPACS algorithm achieved optimal performance on the metrics ERGAS, RMSE, RASE, and UIQI evaluated using the Wald criterion; it performed second best on SAM and was not far from the optimal, and it had better improvement compared to the other CS-based algorithms. The EPACS algorithm performed best on the metric $D_{\lambda }$ evaluated using the QNR criterion; on the metric Ds, the EPACS algorithm performed second best and the BDSD algorithm performed best. In summary, the experimental results showed that the adaptive weighting coefficients, the choice of gain coefficients, and the adaptive enhancement of model details by guided filtering through the MTF optimization of the EPACS algorithm greatly reduced the effect of the inevitable spectral distortion caused by the global variability and low correlation in the relative spectral response function, so that the fused image showed a spectral resolution no weaker than that of MRA-based images. Furthermore, in this design, we used $I_{PA{N}_H}-I_{L_H}$ to replace the $I_{PAN}-I_L$ of the generic CS framework, and the use of the GF filter as well as the RGF filter also enhanced the representation of the spatial information of the PAN images, effectively improving the spatial resolution for foreign objects such as color steel tiles at the perimeter of the high-speed railway, while also ensuring the color fidelity of the foreign objects.

5. Discussion

For the remote-sensing image enhancement of high-speed railways, this paper focuses on improving the traditional CS-based algorithm model from the following two perspectives. (1) Improving the spectral resolution of CS-based algorithms. To solve the problem, this paper first shows that the $I_{PAN}-I_L$ in the generic CS framework brings about severe spectral distortion due to the different spectral response of panchromatic and multispectral image sensors, for which the $I_{PA{N}_H}-I_{L_H}$ design is used instead. In Figure 5 and Table 1, it is clear that the improved image color fidelity has been improved considerably. In order to improve the similarity between PAN images and L images, this paper takes the multispectral image MTF curve into account in the CS-based fusion framework, designs a panchromatic image low-pass filter based on the multispectral image MTF curve, and applies it to the synthesis of L images. In Figure 6 and Table 2, it can be found that the degree of improvement is not as obvious as that of the $I_{PA{N}_H}-I_{L_H}$ design, but it is effective in the saturation of green and red of the image as well as in the evaluation index. This also shows that MTF filters not only improve the effectiveness of MRA-based methods [6] but also work in CS-based frameworks. (2) Improving the spatial resolution of CS-based methods. To solve this problem, the RGF filter is used in this paper in the improved design of $I_{PA{N}_H}-I_{L_H}$. The amount of detail added by any sharpening algorithm will have an impact on the fusion results: adding too much detail may lead to a decrease in spectral resolution; adding too little detail may lead to a decrease in spatial resolution. The RGF filter allows manual control of the scale of the high-frequency component of the panchromatic image by controlling the number of iterations K, thus enabling manual control of the amount of detail D. This allows us to increase the amount of detail as much as possible while maintaining the color information. In Table 1, we can find that $I_{PA{N}_H}-I_{L_H}$ has not only improved in the spectral performance index, but also in the spatial performance index. In this paper, GF is used to further enhance the information of each channel. In Figure 7 and Table 3, we can find that the image is effectively enhanced in terms of effective representation of minute feature details after the GF enhancement. Finally, the fusion effect is compared on GF-2 and DEIMOS-2 images, respectively, in this paper, proving that the proposed framework in this paper can well preserve the colour features and edge detail features of objects, which is crucial to improve the visualization and recognition of foreign objects around high-speed railways. The framework proposed in this paper also can be applied in the fields of storm detection, sand and dust detection, and post-earthquake assessment due to the many adaptive components designed into it.

6. Conclusions

In this paper, an EPACS algorithm is proposed that uses $I_{PA{N}_H}-I_{L_H}$ to replace the design of $I_{PAN}-I_L$ in the generic CS framework, increasing the representation of the spatial and spectral information of the fused images; some adaptive details that depend on data spatial variations were also designed, such as MTF low-pass filter construction, adaptive weighting coefficients and adaptive gain coefficients, and adaptive enhancement of guided filtering for different channel details. These adaptations allow the EPACS algorithm to produce a good fusion effect for images from different sensors, different scales, and features with different spectral characteristics. The EPACS algorithm fuses images clearly in subjective vision, improving the spatial resolution of the MS image of foreign object features on the railway perimeter, while also maintaining their color characteristics well. In terms of objective evaluation metrics, the EPACS algorithm is significantly improved compared to the other CS-based algorithms, and it demonstrates a spectral fidelity that is not inferior to those of the MRA-based algorithms.

The use of high-spatial-resolution means for earth observation began in the military field and is gradually being adopted in the civilian sector. With the growing application of high-speed railway lines, it is of great practical significance to accurately detect and recognize the special features around a railway that might affect the safety of high-speed railway operation, and to do so in a timely and effective manner. In order to better improve the visual interpretation of remote-sensing images, it is essential to synthesize remote-sensing images with good edge characteristics and color features. A new design trend in the development of remote-sensing image fusion technology has emerged in recent years, namely, the use of constrained optimization algorithms to solve the panchromatic sharpening problem. For the time being, such improved algorithms based on the idea of constraint optimization tend to add a significant amount of computational complexity; however, it is believed that with the continuing development of computer science, their time complexity can be significantly optimized. Therefore, the incorporation of constrained optimization ideas into the present framework will be a major research direction to further improve the quality of the fused images. We have thus used the EPACS algorithm as a pre-processing link for eventual use in high-speed rail perimeter foreign object intrusion warning applications.