A Novel Method Based on GPU for Real-Time Anomaly Detection in Airborne Push-Broom Hyperspectral Sensors

Abstract

The airborne hyperspectral remote sensing systems (AHRSSs) acquire images with high spectral resolution, high spatial resolution, and high temporal dimension. While the AHRSS captures more detailed information from the terrain objects, the computational complexity of data processing is greatly increased. As an important application technology in the hyperspectral domain, anomaly detection (AD) processing must be real-time and high-precision in many cases, such as post-disaster rescue, military battlefield search, and natural disaster detection. In this paper, the real-time AD technology for the push-broom AHRSS is studied, the mathematical model is established, and a novel implementation framework is proposed. Firstly, the optimized kernel minimum noise fraction (OP-KMNF) transformation is employed to extract informative and discriminative features between the background and anomalies. Secondly, the Nyström method is introduced to reduce the computational complexity of OP-KMNF transformation by decomposing and extrapolating the sub-kernel matrix to estimate the eigenvector of the entire kernel matrix. Thirdly, the extracted features are transferred to hard disks for data storage. Then, taking the extracted features as input data, the background separation model-based CEM anomaly detector (BSM-CEMAD) is imported to detect anomalies. Finally, graphics processing unit (GPU) parallel computing is utilized in the Nyström-based OP-KMNF (NOP-KMNF) transformation and the BSM-CEMAD to improve the execution efficiency, and the real-time AD for the push-broom AHRSS could be realized. To test the feasibility of the implementation framework proposed in this paper, the experiment is carried out with the Airborne Multi-Modular Imaging Spectrometer (AMMIS) developed by the Shanghai Institute of Technical Physics as the data acquisition platform. The experimental results show that the proposed method outperforms many other state-of-the-art AD methods in anomalies detection and background suppression. Moreover, under the condition that the downlink data could retain most of the hyperspectral data information, the proposed method achieves real-time detection of pixel-level anomalies, with the initial delay not exceeding 1 s, the false alarm rate (FAR) less than 5%, and the true positive rate (TPR) close to 98%.

1. Introduction

With the development of remote sensing technology, the acquired images have undergone a development process from panchromatic to multispectral to hyperspectral [1]. Compared with panchromatic and multispectral images, hyperspectral images provide richer spectral information, which helps to identify the subtle features of the terrain objects [2]. Recently, hyperspectral remote sensing has been widely used in various fields, such as military reconnaissance, urban planning, disaster assessment, resource exploration, and environmental monitoring [3,4,5,6,7]. According to the platforms hyperspectral sensors carried, they can be divided into ground-based hyperspectral remote sensing systems (GHRSSs), spaceborne hyperspectral remote sensing systems (SHRSSs), and airborne hyperspectral remote sensing systems (AHRSSs). In practice, the payloads of GHRSSs could be replaced according to the requirement for data acquisition, while the GHRSSs is challenging to be widely used because of the limited data acquisition areas when they are worked. The SHRSSs could obtain data in a large area, while their payloads are tough to replace and maintain. Additionally, the area of data acquisition is limited by the satellite transit situation, and it is almost impossible to collect data in a timely manner in areas that face emergencies. The AHRSS makes up for the shortcomings of the GHRSS and the SHRSS. The payloads of AHRSSs could be replaced according to the demand in practice, allowing for capturing data rapidly in areas where the human resources are difficult to reach or involve. Moreover, the flight altitude setting of the airborne platforms is flexible, and multi-scale remote sensing data could be collected according to the requirements [8]. The basic information of the representative AHRSS is shown in

Table 1. The basic information of the representative AHRSS.

The AHRSS Name	Place of Development	Producing Year	Scanning Mode
AIS	America	1983	push-broom
AVIRIS	America	1987	whisk-broom
CASI/SASI/MASI/TASI	Canada	1988	push-broom
AISA	Finland	1992	push-broom
HyMap	Australia	1997	whisk-broom
PHI	China	1997	push-broom
AVIS	Germany	1997	push-broom
OMIS	China	2000	whisk-broom
AVIS-2	Germany	2001	push-broom
APEX	Switzerland/Belgium	2009	push-broom
Hyper-Cam	Canada	2009	push-broom
MAKO	America	2010	push-broom
Sieleters	France	2013	push-broom
AISA-OWL	Finland	2014	push-broom
AMMIS	China	2016	push-broom
HyTES	America	2016	push-broom
AVIRIS-NG	America	2019	push-broom

, and it can be seen that push-broom scanning represents the mainstream of the AHRSS.

While the AHRSS captures more detailed information from the terrain objects, the computational complexity of data processing is greatly increased [6]. As an important application technology in the hyperspectral domain, anomaly detection (AD) processing must be real-time and high-precision in many cases, such as post-disaster rescue, military battlefield search, and natural disaster detection [9]. Methods based on deep learning, by mining the deep features of images, have emerged as a research hotspot in recent years for achieving anomaly target detection. Depending on whether annotated samples are needed, deep learning-based anomaly detection methods can be categorized into supervised and unsupervised anomaly detection. Current anomaly detection methods based on deep learning are depicted in

Table 2. Anomaly detection methods based on deep learning.

Category	Representative Methods
Unsupervised deep learning-based anomaly detection methods	Sparse decomposition and auto-encoder (AE) [15]; stacked AE [16,17,18]; spectral constraint adversarial AE network [19,20,21]; generative adversarial network (GAN) combined with AE [22]; exploiting embedding manifold of autoencoders [23]; and semi-supervised spectral learning with GAN [24].
Supervised deep learning-based anomaly detection methods	Convolutional neural network (CNN) [25]; online CNN [26]; and CNN combined with low-rank representation (LRR) [27].

. Over the past few years, researchers have developed numerous methods for hyperspectral anomaly detection. However, several critical technical challenges still persist in practical applications: (1) parameter tuning, (2) algorithm generality, and (3) real-time capability [10]. Furthermore, with the advancement of automated applications, conducting real-time anomaly detection and transmitting results to the ground for intelligent decision making has also become a pressing technological challenge that needs to be overcome. As mentioned above, push-broom scanning represents the mainstream of the AHRSS. Many related methods on the real-time AD for the push-broom AHRSS are proposed, mainly focusing on fast matrix computation, hardware design, and parallel computation [10]. For matrix inverse adaptive calculation and optimization, some algorithms, such as the real-time recursive causality [11], the linear algebra libraries and multicore platforms [12], and the Cholesky decomposition and linear algebra [11], are proposed to improve the computational efficiency of anomaly detection methods effectively. In terms of hardware design, a two-module field-programmable gate array (FPGA) [13] based on the Reed-Xiaoli (RX) algorithm and the constrained energy minimization (CEM) algorithm is designed to transform those two algorithms according to requirements. In parallel computation, the parallel computing version of the RX detector is studied on the multicore platform [12] and graphics processing unit (GPU) [14], which achieve a remarkable acceleration effect in execution time.

Although these studies have made some achievements, the real-time anomaly detection for the push-broom AHRSS still faces the following difficulties: (1) the nonlinear characteristics and high dimensionality of hyperspectral images are significant, and it is difficult to extract features effectively; (2) constructing a precise representation of anomalies and background information for complex scenes is challenging, and as a result, the target detection accuracy for AHRSS is limited recently; (3) the hyperspectral image has many bands, leading to high data processing calculation and low processing efficiency; (4) the AHRSS captures more detailed information from the terrain objects, and the images have high research value. Ensuring that the downlink data retain most of the hyperspectral data information while performing real-time processing is also a problem to be considered.

In this paper, the real-time AD technology for the push-broom AHRSS is studied, the mathematical model is established, and a novel implementation framework is proposed. To test the feasibility of the implementation framework, this paper designs an experiment carried out with the Airborne Multi-Modular Imaging Spectrometer (AMMIS) developed by the Shanghai Institute of Technical Physics as the data acquisition platform. The experimental results show that the proposed method outperforms many other AD methods (including RX detector [12], low-probability detector (LPD) [28], abundance- and dictionary-based low-rank decomposition (ADLR) [29], collaborative representation detector with background purification and saliency weight (CRDBPSW) [30], anomaly detection via integration of feature extraction and background purification (FEBPAD) [31], and kernel isolation forest-based hyperspectral anomaly detection method (KIFD) [32]) in anomalies detection and background suppression. Moreover, under the condition that the downlink data could retain most of the hyperspectral data information, the proposed method achieves real-time detection of pixel-level anomalies, with the initial delay not exceeding 1 s, the false alarm rate (FAR) less than 5%, and the true positive rate (TPR) close to 98%.

The main contributions of this paper are introduced as follows:

(1)

Considering the data acquisition, data transmission/storage, and data processing of the push-broom AHRSS, the real-time AD technology for the push-broom AHRSS is studied in this paper, the mathematical model of it is established, and the critical problems that need to be solved can be explained by mathematical theoretical analysis, which provides a theoretical reference for the difficulties in the push-broom AHRSS real-time AD technology.

(2)

A novel AHRSS real-time AD implementation framework is proposed based on the mathematical model. Firstly, the optimized kernel minimum noise fraction (OP-KMNF) transformation is employed to extract informative and discriminative features between the background and anomalies. Secondly, the Nyström method is introduced to reduce the computational complexity of OP-KMNF transformation by decomposing and extrapolating the sub-kernel matrix to estimate the eigenvector of the entire kernel matrix. Thirdly, the extracted features are transferred to hard disks for data storage. Then, taking the extracted features as input data, the background separation model-based CEM anomaly detector (BSM-CEMAD) is imported to detect anomalies. Finally, graphics processing unit (GPU) parallel computing is utilized in the Nyström-based OP-KMNF (NOP-KMNF) transformation and the BSM-CEMAD to improve the execution efficiency, and the real-time AD for the push-broom AHRSS could be realized.

(3)

To test the feasibility of the implementation framework, this paper designs an experiment carried out with the Airborne Multi-Modular Imaging Spectrometer (AMMIS) developed by the Shanghai Institute of Technical Physics as the data acquisition platform. The experimental results show that the proposed method outperforms many other AD methods (including RX detector, LPD, ADLR, CRDBPSW, FEBPAD, and KIFD) in anomalies detection and background suppression. Moreover, under the condition that the downlink data could retain most of the hyperspectral data information, the proposed method achieves real-time detection of pixel-level anomalies, with the initial delay not exceeding 1 s, the false alarm rate (FAR) less than 5%, and the true positive rate (TPR) close to 98%.

The remainder of the manuscript is organized as follows: In Section 2, the proposed methods are described in detail. The experimental results are shown in Section 3. Section 4 analyzes and discusses the results, and conclusions are presented in Section 5.

2. Proposed Methods

As mentioned above, the real-time AD for the push-broom AHRSS still faces many difficulties. Considering the data acquisition, data transmission/storage, and data processing of the push-broom AHRSS, the real-time AD technology based on the graphics processing unit (GPU) for the push-broom AHRSS is studied in this paper. In this section, the mathematical model of the real-time AD technology for the push-broom AHRSS is established, and the GPU-based parallel computing is described.

2.1. Mathematical Model Analysis

The realization of the push-broom AHRSS real-time AD could be abstracted as the mapping function:

$$y=f_y\left(x;\eta \right),$$

(1)

where $y$ represents the expected output of the push-broom AHRSS real-time AD, $x$ represents the image data captured by the push-broom AHRSS, $\eta$ represents the parameter to be determined in the mapping function, and $f_y$ represents the abstracted mapping function dependent on the parameter $\eta$.

To solve the difficulties in the real-time AD for the push-broom AHRSS, a novel AHRSS real-time AD implementation framework is proposed in this paper. The implementation flowchart is shown in Figure 1. It can be seen that the factors affecting the expected output $y$ include the effectiveness of the features extracted by feature extraction, the efficiency and reliability of data transmission/storage, the detection capability of the anomaly detection method, and the execution efficiency of the data processing. The determination of the parameter $\eta$ in the mapping function is affected by these factors, which can be abstracted as the mapping function:

$$\eta =\epsilon \ast d_t\ast f_{\eta }(d_r;a_d),$$

(2)

where $d_r$, $a_d$, $\epsilon$, and $d_t$ represent the effects of the effectiveness of the features extracted by feature extraction, the detection capability of the anomaly detection method, the execution efficiency of the data processing and transmission/storage, and the reliability of data transmission/storage in the determination of the parameter $\eta$, respectively.

In the mathematical model, the influence of the execution efficiency of the data processing and transmission/storage and the reliability of data transmission/storage in the determination of the parameter $\eta$ is represented by the perturbation factors $\epsilon$ and $d_t$ in the mapping function, which plays a decisive role in the whole mapping relationship. The execution efficiency of the data processing and transmission/storage, $\epsilon$, is affected by two factors—(a) the execution efficiency of the data processing ${\epsilon }_{dr-ad}$ and (b) the execution efficiency of data transmission/storage, ${\epsilon }_{dt}$— and the mathematical relationship could be expressed as:

$$\epsilon ={\epsilon }_{dr-ad}+{\epsilon }_{dt}$$

(3)

After the mathematical model of the real-time AD technology based on the GPU for the push-broom AHRSS is established, the critical problems that need to be solved can be explained by mathematical theoretical analysis. Improving the reliability of data transmission/storage and the execution efficiency of the data processing and transmission/storage can be described as the process of optimizing the perturbation factors $\epsilon$ and $d_t$, while improving the feature extraction performance and AD accuracy can be described as the process of optimizing the parameters $d_r\mathrm{a}\mathrm{n}\mathrm{d}{a}_d$ in the mapping function.

In previous works, feature extraction, anomaly detection, and algorithm complexity optimization are studied [33,34,35]. The optimized kernel minimum noise fraction (OP-KMNF) transformation for feature extraction [33], the background separation model-based CEM anomaly detector (BSM-CEMAD) for anomaly detection [34], and the Nyström-based OP-KMNF (NOP-KMNF) transformation for algorithm complexity optimization [35] are proposed. On the problem of the inaccurate noise estimation in kernel minimum noise fraction (KMNF) transformation, a mixed noise estimation model (MNEM) combining the Gaussian prior denoising model, the Sobel operator, and the median filter is proposed for OP-KMNF transformation. The MNEM is more robust and effective, retains more edge features and details, and is more suitable for noise estimation during KMNF transformation. Experiments using various hyperspectral datasets with different spatial and spectral resolutions were conducted. The results show that the OP-KMNF transformation overperforms other feature extraction methods (including linear discriminant analysis (LDA) [36], principal component analysis (PCA) [37], minimum noise fraction (MNF) transformation [38], optimized MNF (OMNF) [39], factor analysis (FA) [40], kernel PCA (KPCA) [41], KMNF transformation [42], optimized KMNF (OKMNF) transformation [43], and local preserving projections (LPPs) [44]) for feature extraction and has better adaptability to the changes in spatial resolution and spectral resolution [33]. To reduce the computational complexity of the KMNF transformation, a Nyström-based KMNF transformation (NKMNF) feature extraction method is proposed. The entire kernel matrix of the feature vector is estimated by decomposition and extrapolation of the sub-kernel matrix. The experimental results demonstrate that the NKMNF transformation has lower computational complexity and a better feature extraction performance than KMNF [34]. Similarly, the Nyström method is also applicable to the OP-KMNF transformation to reduce the computational complexity and improve the execution efficiency. Aiming to obtain accurate background and abnormal pixel sets, a background separation model (BSM) combining outlier removal, an iterative strategy, and an RX detector is proposed. The BSM-CEMAD takes the background and abnormal pixel sets as the input of CEMAD to improve the anomaly detection capability. The experimental results show that the BSM-CEMAD has better anti-noise performance, anomalies detection, and background suppression ability, and has better adaptability to the changes in spatial resolution and spectral resolution [35].

To overcome the difficulties in the real-time AD for the push-broom AHRSS, a novel AHRSS real-time AD implementation framework is proposed based on the mathematical model. The technical realization flowchart is shown in Figure 2. Firstly, the OP-KMNF transformation is employed to extract informative and discriminative features between the background and anomalies. Secondly, the Nyström method is introduced to reduce the computational complexity of OP-KMNF transformation by estimating the eigenvector of the entire kernel matrix by decomposing and extrapolating the sub-kernel matrix. Thirdly, the extracted features are transferred to hard disks for data storage. Then, taking the extracted features as input data, the BSM-CEMAD is imported to detect anomalies. Finally, graphics processing unit (GPU) parallel computing is utilized in the Nyström-based OP-KMNF (NOP-KMNF) transformation and the BSM-CEMAD to improve the execution efficiency, and the real-time AD for the push-broom AHRSS could be realized.

2.2. GPU-Based Parallel Computing

According to the type of hardware platforms, hyperspectral image real-time processing technology can be divided into three categories: (1) real-time hyperspectral image processing technology based on FPGA; (2) real-time hyperspectral image processing technology based on cloud computing; (3) real-time hyperspectral image processing technology based on GPU. In practice, GPU and FPGA cannot process large-scale hyperspectral images because of the limitation of their memory capacity, while the cloud computing has no such limitations. The low power consumption makes FPGA more suitable for real-time processing of AHRSS without a data downlink, while its acceleration performance is not superior to GPU. Compared to FPGA and cloud computing, GPU has lower cost and better acceleration performance, which is the most cost-effective hardware platform in high-performance computing [45]. The comparison of different hardware platforms is shown in

Table 3. Comparison of different hardware platforms [37].

	High Speedup Performance	Low Power Consumption	Low Cost	Large Memory Capacity
FPGA		$\surd$
Cloud computing			$\surd$	$\surd$
GPU	$\surd$		$\surd$

. Recently, GPU-related technologies have developed rapidly. Taking NVIDIA GPU products as an example, its organizational structure has been upgraded from Fermi architecture to Ampere architecture, and the processing power has been significantly improved. The development of NVIDIA GPU architecture is shown in Figure 3.

Basic linear algebra subroutines (BLASs) provide a series of interface standards for basic linear algebra operations and are widely used to develop various high-quality linear algebra software, such as LAPACK and FLAME [46,47]. To perform vector and matrix operations effectively, NVIDIA computing unified device architecture (CUDA) provides a BLAS library (CUBLAS) based on the GPU parallel computing platform. Some relevant research demonstrates that CUBLAS has superior performance in processing vector and matrix operations [48,49]. An analysis of NOP-KMNF transformation and BSM-CEMAD show that voluminous vector and matrix operations exist in their implementation program. These operations mainly encompass vector multiplication, matrix multiplication, and matrix eigenvalue computation. The overall algorithmic time complexity of the processing workflow could be represented as $O\left(N^2\right)$. In this paper, the NVIDIA CUBLAS is utilized to develop the GPU parallel computing-based NOP-KMNF (GNOP-KMNF) transformation and BSM-CEMAD (GBSM-CEMAD). The computational efficiency of GNOP-KMNF transformation and GBSM-CEMAD are evaluated in the hardware composed of an NVIDIA GeForce RTX2060 GPU card and an Intel (R) Core (TM) i7-10750H CPU. The NVIDIA GeForce RTX2060 GPU hardware specifications are shown in

Table 4. The NVIDIA GeForce RTX2060 GPU hardware specifications.

Parameters	Performance
CUDA Driver Version/Runtime Version	11.4/10.0
CUDA Cores	1920
Total amount of global memory	6144 MBytes
Memory clock rate	5501 MHz
Max clock rate	1200 MHz
Memory bus width	192 bits
Warp size	32
Total number of registers available per block	65,536
Total amount of shared memory per block	49,152 bytes
Total amount of constant memory	65,536 bytes

.

3. Results

In this section, the experimental results of the method proposed in this paper are described. In Section 3.1, the data acquisition platform AMMIS VNIR hyperspectral push-broom sensor is introduced. In Section 3.2, the real hyperspectral dataset named Vehicle dataset captured by AMMIS VNIR hyperspectral push-broom sensor and conducted in the experiments is presented. In Section 3.3, the evaluation criteria of the proposed methods are described. Two experiments are designed to assess the feasibility of the implementation framework in real-time anomaly detection for the push-broom AHRSS. In order to evaluate the effectiveness of the real-time processing, the Vehicle dataset is used as the captured image, and the runtimes of (1) feature extraction, (2) data transmission/storage, and (3) anomaly detection with different data sizes along the flight direction are tested. To ensure the reliability of the experimental results, each experiment is conducted five times, and the average values are used to analyze the results, which are shown in Section 3.4. The second experiment is designed to assess the AD capability of the proposed method, and the results in the Vehicle dataset are shown in Section 3.5.

3.1. Data Acquisition Platform

The AMMIS is an AHRSS developed by the Shanghai Institute of Technical Physics, Chinese Academy of Sciences. It is designed for various applications, such as land management, urban planning, natural disaster monitoring and assessment, agricultural and forestry surveys, water resources utilization, water quality monitoring, and so on. It can be configured in different modules according to the application requirements [50]. In this paper, the AMMIS visible and near-infrared range (VNIR) hyperspectral push-broom sensor is utilized as the data acquisition platform. The AMMIS VNIR hyperspectral push-broom sensor specifications are shown in

Table 5. AMMIS VNIR hyperspectral push-broom sensor specifications.

Parameters	Performance
Spectral range ($\mathsf{\mu }\mathrm{m}$)	0.40~0.95
Instantaneous field of view ($\mathrm{m}\mathrm{r}\mathrm{a}\mathrm{d}$)	0.25
Signal-to-Noise Ratio	$\ge$500
Across-track pixels	2048
Frame frequency (Hz)	160
Effective spectral bands	250

, and its imaging diagram is shown in Figure 4.

3.2. Input Data

The Vehicle dataset used in this paper is collected by the AMMIS VNIR hyperspectral push-broom sensor in Xiong’an New Area, Hebei Province, China. The dataset consists of 2048 pixels along the across-track direction and 15,000 pixels along the flight direction. The wavelength range is 0.40~0.95 $\mathsf{\mu }\mathrm{m}$, and the spatial resolution is 0.5 m. The pseudo-color and the reference map are shown in Figure 5.

3.3. Evaluation Criteria

To assess the performance of the anomaly detection capability of the proposed method, the 3D receiver operating characteristic (3D ROC) curve, the 2D ROC curve of (${\mathrm{P}}_{\mathrm{D}},{\mathrm{P}}_{\mathrm{F}}$), the 2D ROC curve of (${\mathrm{P}}_{\mathrm{D}},\mathrm{t}$), and the 2D ROC curve of (${\mathrm{P}}_{\mathrm{F}},\mathrm{t}$) are visualized in this paper. Furthermore, the area under the 2D ROC curve of (${\mathrm{P}}_{\mathrm{D}},{\mathrm{P}}_{\mathrm{F}}$) ($AUC(D,F)$), the area under the 2D ROC curve of (${\mathrm{P}}_{\mathrm{D}},\mathrm{t}$) ($AUC(D,t)$), the area under the 2D ROC curve of (${\mathrm{P}}_{\mathrm{F}},\mathrm{t}$) ($AUC(F,t)$), the AUC value of target detectability (${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{T}\mathrm{D}}$), the AUC value of background suppressibility (${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{B}\mathrm{S}}$), the AUC value of target detection in the background (${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{T}\mathrm{D}-\mathrm{B}\mathrm{S}}$), the AUC value of the overall detection probability (${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{O}\mathrm{D}\mathrm{P}}$), the AUC value of overall detection (${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{O}\mathrm{D}}$), and the AUC value of the signal-to-noise probability ratio (${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{S}\mathrm{N}\mathrm{P}\mathrm{R}}$) are used as evaluation indexes to quantitatively evaluate the anomaly detection capability [51]. The calculation formulas of these evaluation criteria are shown in

Table 6. The calculation formulas of the anomaly detection evaluation criteria.

Evaluation Criteria	Calculation Formulas
$AUC(D,F)$	the area under the 2D ROC curve of (${\mathrm{P}}_{\mathrm{D}},{\mathrm{P}}_{\mathrm{F}}$)
$AUC(D,t)$	the area under the 2D ROC curve of (${\mathrm{P}}_{\mathrm{D}},\mathrm{t}$)
$AUC(F,t)$	the area under the 2D ROC curve of (${\mathrm{P}}_{\mathrm{F}},\mathrm{t}$)
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{T}\mathrm{D}}$	$AUC\left(D,F\right)+AUC(D,t)$
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{B}\mathrm{S}}$	$AUC\left(D,F\right)-AUC(F,t)$
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{T}\mathrm{D}-\mathrm{B}\mathrm{S}}$	$AUC\left(D,t\right)-AUC(F,t)$
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{O}\mathrm{D}\mathrm{P}}$	$AUC\left(D,t\right)+(1-AUC\left(F,t\right))$
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{O}\mathrm{D}}$	$AUC\left(D,F\right)+AUC(D,t)-AUC(F,t)$
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{S}\mathrm{N}\mathrm{P}\mathrm{R}}$	$AUC(D,t)/AUC(F,t)$

.

In practical applications, the specific false alarm rate (FAR) and true positive rate (TPR) give the operator a more intuitive understanding of the performance of the methods. As a matter of common knowledge, the lower the false alarm rate, the higher the detection rate, the better the method’s performance is. In this paper, a novel evaluation criterion based on the FAR and TPR in the 3D ROC curve is proposed in this section.

$$\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}=\mathrm{T}\mathrm{P}\mathrm{R}\ast \left(1-\mathrm{F}\mathrm{A}\mathrm{R}\right).$$

(4)

As shown in Formula (4), the $\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}$ values are calculated based on the FAR and TPR corresponding to each coordinate point on the 3D ROC curve. The larger the $\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}$ value, the better the detection results are. Taking the ${\mathrm{F}\mathrm{A}\mathrm{R}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and the ${\mathrm{T}\mathrm{P}\mathrm{R}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ at the maximum $\mathrm{Index}$ value together as the evaluation criterion of the anomaly detection methods, the results can be assessed more intuitively.

3.4. Real-Time Processing Analysis

Experiments on hyperspectral images with different data sizes are conducted in this section. To ensure the reliability of the experimental results, each experiment is conducted five times, and the average values are used to analyze the results. Figure 6 shows the computational cost comparisons for NOP-KMNF-Ratio transformation and GNOP-KMNF-Ratio transformation with different data sizes, and the computational cost comparisons for BSM-CEMAD and GBSM-CEMAD with different data sizes are shown in Figure 7.

The results shown in Figure 6 and Figure 7 indicate that (1) with the increase in data sizes, the acceleration effect of GNOP-KMNF transformation rather than that of NOP-KMNF-Ratio transformation and GBSM-CEMAD rather than that of BSM-CEMAD are more and more significant; (2) when the data size is 288 × 288 × 250, the execution efficiency of GNOP-KMNF transformation is about 336 times that of NOP-KMNF transformation, which shows the superior performance of the GNOP-KMNF transformation feature extraction algorithm in execution efficiency; (3) when the data size is 128 × 128 × 250, the execution efficiency of GBSM-CEMAD is about 223 times that of BSM-CEMAD, which indicates the superior performance of GBSM-CEMAD in execution efficiency.

As mentioned above, the data processing includes three parts: (1) feature extraction, (2) data transmission/storage, and (3) anomaly detection. The Vehicle dataset is used as the captured image, and the runtimes of these three data processing parts with different data sizes along the flight direction are tested. To ensure the reliability of the experimental results, each experiment is conducted five times, and the average values are used to analyze the results. The execution efficiency of GNOP-KMNF transformation, data transmission/storage, and GBSM-CEMAD with different data sizes are shown in Figure 8, Figure 9, and Figure 10, respectively. To quantitatively show the experimental results, the runtimes of GNOP-KMNF transformation, data transmission/storage, and GBSM-CEMAD with different data sizes are shown in

Table 7. The runtimes of GNOP-KMNF transformation, data transmission/storage, and GBSM-CEMAD with different data sizes.

Data Sizes	$\mathbf{Runtimes}\left(\mathbf{m}\mathbf{s}\right)$
	GNOP-KMNF	Data Transmission/Storage	GBSM-CEMAD
2048 × 250 × 10	68.58	18.20	33.01
2048 × 250 × 20	96.42	18.40	62.43
2048 × 250 × 30	127.68	18.80	91.42
2048 × 250 × 40	154.74	19.10	120.49
2048 × 250 × 50	183.40	19.20	149.86
2048 × 250 × 60	215.78	19.50	179.39
2048 × 250 × 70	242.96	19.70	206.98
2048 × 250 × 80	272.24	19.80	240.27
2048 × 250 × 90	302.06	20.10	262.98
2048 × 250 × 100	334.30	20.30	291.50
2048 × 250 × 110	362.30	20.40	320.75
2048 × 250 × 120	392.62	20.50	352.40
2048 × 250 × 130	423.18	20.90	383.11
2048 × 250 × 140	454.68	21.10	411.90
2048 × 250 × 150	482.86	21.40	441.82
2048 × 250 × 160	506.92	21.60	463.09

.

Analyzing the data in Table 6, it can be seen that when the data size is 2048 × 250 × 160, the whole processing time of GNOP-KMNF transformation, data transmission/storage, and GBSM-CEMAD is 991.61 ms. The frame frequency of the AMMIS VNIR hyperspectral sensor is 160 Hz, the data acquisition rate is 313 MB/s, and the processing speed of the proposed method is 315.65 MB/s. In real-time processing, the hyperspectral image data of 2048 × 250 × 160 are used as a hyperspectral image cube to be processed. With the initial delay not exceeding 1 s, the real-time AD for the AMMIS VNIR hyperspectral sensor can be realized. The real-time AD processing for the AMMIS VNIR hyperspectral sensor is shown in Figure 11, and the anomaly detection results of the proposed method for the Vehicle dataset is shown in Figure 12.

3.5. Anomaly Detection Capability Evaluation

In this section, the anomaly detection capability of the proposed method and other state-of-the-art algorithms (including RX detector, LPD, ADLR, CRDBPSW, FEBPAD, and KIFD) is evaluated. Because the size of the Vehicle dataset is too large, part of the anomaly detection results using different detectors is shown in Figure 13 to display the detection results better.

Part of the anomaly detection results using different detectors for the Vehicle dataset are shown in Figure 13. By observing these result images, it can be seen that the proposed method suppresses more of the background and highlights the anomalies from the background. Compared with RX, ADLR, CRDBPSW, and FEBPAD, LPD and KIFD detect more anomaly pixels and obtain an acceptable background suppression.

To quantitatively assess the results, the 3D ROC curve and its generated three 2D ROC curves of the Vehicle dataset are demonstrated in Figure 14, and the AUC values calculated from three 2D ROC curves for the Vehicle dataset are shown in

Table 8. The AUC values calculated from three 2D ROC curves for the Vehicle dataset.

Evaluation Criteria	Proposed	RX	LPD	ADLR	CRDBPSW	FEBPAD	KIFD
$\mathrm{A}\mathrm{U}\mathrm{C}(\mathrm{D},\mathrm{F})$	0.982476	0.866253	0.826230	0.689321	0.835342	0.849385	0.944433
$\mathrm{A}\mathrm{U}\mathrm{C}(\mathrm{D},\mathrm{t})$	0.499693	0.073389	0.296374	0.759743	0.155015	0.118904	0.471660
$\mathrm{A}\mathrm{U}\mathrm{C}(\mathrm{F},\mathrm{t})$	0.043713	0.015264	0.124230	0.630917	0.018577	0.012252	0.062557
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{T}\mathrm{D}}$	1.482169	0.939642	1.122604	1.449064	0.990357	0.968289	1.416093
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{B}\mathrm{S}}$	0.938763	0.850989	0.702000	0.058404	0.816765	0.837133	0.881876
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{T}\mathrm{D}-\mathrm{B}\mathrm{S}}$	0.455980	0.058125	0.172144	0.128826	0.136438	0.106652	0.409103
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{O}\mathrm{D}\mathrm{P}}$	1.455980	1.058125	1.172144	1.128826	1.136438	1.106652	1.409103
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{O}\mathrm{D}}$	1.438456	0.924378	0.998374	0.818147	0.971780	0.956037	1.353536
${\mathrm{A}\mathrm{U}\mathrm{C}}_{\mathrm{S}\mathrm{N}\mathrm{P}\mathrm{R}}$	11.43122	4.807980	2.385688	1.204189	8.344458	9.704865	7.539684

.

As shown in Figure 14, the proposed method shows better performance in the ROC curves. For the 2D ROC curves of (${\mathrm{P}}_{\mathrm{F}},\mathrm{t}$), the curve of the proposed method is closest to the bottom-right corner. Moreover, the 2D ROC curves of (${\mathrm{P}}_{\mathrm{D}},{\mathrm{P}}_{\mathrm{F}}$) and (${\mathrm{P}}_{\mathrm{D}},\mathrm{t}$) of the proposed method are closest to the top-left corner. The results demonstrate the significant performance of the proposed method.

For the AUC values calculated from three 2D ROC curves for the Vehicle dataset shown in Table 8, the $AUC(D,t)$ of ADLR is better than that of the proposed method, while the $AUC\left(F,t\right)$ and other evaluation criteria values of ADLR are much worse than that of the proposed method. Additionally, the $AUC\left(F,t\right)$ of FEBPAD outperforms that of the proposed method, while the $AUC(D,t)$ and other evaluation criteria values of FEBPAD are much worse than that of the proposed method. In summary, the results show that the proposed method has a better anomalies detection and background suppression performance.

The $\mathrm{Index}$ values based on the FAR and TPR in the 3D ROC curves for different detectors are calculated, which are shown in

Table 9. The Index values based on the FAR and TPR in the 3D ROC curves for the Vehicle dataset.

Methods	$\mathbf{I}\mathbf{n}\mathbf{d}\mathbf{e}\mathbf{x}$	${\mathbf{F}\mathbf{A}\mathbf{R}}_{\mathbf{m}\mathbf{i}\mathbf{n}}$	${\mathbf{T}\mathbf{P}\mathbf{R}}_{\mathbf{m}\mathbf{a}\mathbf{x}}$
RX	0.617582	0.258901	0.833333
LPD	0.565461	0.272979	0.777778
ADLR	0.739566	0.260434	1.000000
CRDBPSW	0.625933	0.216223	0.798611
FEBPAD	0.578459	0.199056	0.722222
KIFD	0.809781	0.036294	0.840278
Proposed	0.934588	0.045527	0.979167

. When the $\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}$ value is the largest, the ${\mathrm{F}\mathrm{A}\mathrm{R}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ of the proposed method is 0.045527, and the ${\mathrm{T}\mathrm{P}\mathrm{R}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ of the proposed method is 0.979167. The results demonstrate that the proposed method can achieve real-time detection of pixel-level anomalies, with the initial delay not exceeding 1 s, the FAR of less than 5%, and the TPR close to 98%.

4. Discussion

In this paper, two experiments are designed to assess the effectiveness of the implementation framework in real-time anomaly detection for the push-broom AHRSS.

In Section 3.4, the computational cost comparisons for NOP-KMNF-Ratio transformation and GNOP-KMNF-Ratio transformation with different data sizes are tested. The results indicate that (1) with the increase in data sizes, the acceleration effect of GNOP-KMNF transformation rather than that of NOP-KMNF-Ratio transformation and GBSM-CEMAD rather than that of BSM-CEMAD are more and more significant; (2) when the data size is 288 × 288 × 250, the execution efficiency of GNOP-KMNF transformation is about 336 times that of NOP-KMNF transformation, which shows the superior performance of the GNOP-KMNF transformation feature extraction algorithm in execution efficiency; (3) when the data size is 128 × 128 × 250, the execution efficiency of GBSM-CEMAD is about 223 times that of BSM-CEMAD, which indicates the superior performance of GBSM-CEMAD in execution efficiency. In order to evaluate the effectiveness of the real-time processing, the Vehicle dataset is used as the captured image, and the runtimes of three data processing parts with different data sizes along the flight direction are tested. It can be seen that when the data size is 2048 × 250 × 160, the whole processing time of GNOP-KMNF transformation, data transmission/storage, and GBSM-CEMAD is 991.61 ms. The frame frequency of the AMMIS VNIR hyperspectral sensor is 160 Hz, the data acquisition rate is 313 MB/s, and the processing speed of the proposed method is 315.65 MB/s. In real-time processing, the hyperspectral image data of 2048 × 250 × 160 are used as a hyperspectral image cube to be processed. With the initial delay not exceeding 1 s, the real-time AD for the AMMIS VNIR hyperspectral sensor could be realized. In Section 3.5, the experiment is designed to assess the anomaly detection capability of the proposed methods. The experimental results show that the proposed method outperforms many other AD methods (including RX detector, LPD, ADLR, CRDBPSW, FEBPAD, and KIFD) in anomalies detection and background suppression. Moreover, under the condition that the downlink data could retain most of the hyperspectral data information, the proposed method achieves real-time detection of pixel-level anomalies, with the initial delay not exceeding 1 s, the FAR less than 5%, and the TPR close to 98%.

In this paper, the real-time AD technology for the push-broom AHRSS is studied, and a novel implementation framework is proposed. In order to achieve high-precision real-time anomaly detection for the push-broom AHRSS, the whole implementation process of hyperspectral image acquisition, transmission, and processing is analyzed, and the mathematical model of the real-time AD technology for the push-broom AHRSS is established. On this basis, the solutions to the current difficulties of this research are given from the mathematical point of view, which lays a theoretical foundation for the subsequent implementation. The processing power of GPUs is measured by the ability to perform floating-point operations per second (FLOPS). In practical applications, the processing power of GPUs depends on many factors, such as hardware architecture, number of cores, frequency, and memory bandwidth. Figure 15 shows the thermal design power of these NVIDIA desktop GPUs, and the processing power of NVIDIA desktop GPUs since 2015 are shown in Figure 16.

Compared with the NVIDIA GeForce RTX 2060 used in the experiments, the processing power of the most powerful NVIDIA GeForce RTX 4090 on single-precision floating-point operations has increased by 12.80 times, and the processing power on double-precision floating-point operations has increased by 6.40 times. With the continuous development of GPU parallel computing technologies, the execution efficiency of the algorithm will be further improved, and the initial delay time will be effectively shortened. While for the thermal design power, the most powerful NVIDIA GeForce RTX 4090 has a 2.81-fold increase compared with the NVIDIA GeForce RTX 2060 used in the experiments, how to select the hardware platform based on the careful consideration of execution efficiency and power consumption in actual flight missions still needs further discussion.

Moreover, the implementation framework designed in this paper on the real-time AD technology based on the GPU for the push-broom AHRSS is guided by practical application. In practice, the data acquisition speed by the push-broom AHRSS is affected not only by the frame frequency of the sensors, but also by the flight speed of the airborne platform. Carrying out simulation experiments only considering the push-broom AHRSS specifications is limited. It is necessary to design and conduct experiments on real-time anomaly detection for the push-broom AHRSS in combination with actual flight missions.

5. Conclusions

In this paper, the real-time AD technology for the push-broom AHRSS is studied, the mathematical model is established, and a novel implementation framework is proposed. To test the feasibility of the implementation framework proposed in this paper, the experiment is carried out with the AMMIS developed by the Shanghai Institute of Technical Physics as the data acquisition platform. The experimental results show that the proposed method outperforms many other AD methods (including RX detector, LPD, ADLR, CRDBPSW, FEBPAD, and KIFD) in anomalies detection and background suppression. Moreover, under the condition that the downlink data could retain most of the hyperspectral data information, the proposed method achieves real-time detection of pixel-level anomalies, with the initial delay not exceeding 1 s, the FAR less than 5%, and the TPR close to 98%. Through the analysis of the critical issues involved in this research, a relatively complete implementation framework with an algorithm design and data processing for the real-time AD technology on the push-broom AHRSS is proposed in this paper. Furthermore, the mathematical model of the real-time AD technology for the push-broom AHRSS is constructed, the algorithm design and verification are carried out, and the test experiment validation is conducted. In this paper, the AMMIS developed by Shanghai Institute of Technical Physics, Chinese Academy of Sciences, is utilized as the data acquisition sensor to carry out test experiments. While the experiments in this paper are conducted on the obtained data, we are interested in designing and conducting the experiment under the real airborne imaging environment in the future.