Bayes-Optimized Adaptive Growing Neural Gas Method for Online Anomaly Detection of Industrial Streaming Data

Abstract

Online anomaly detection is critical for industrial safety and security monitoring but is facing challenges due to the complexity of evolving data streams from working conditions and performance degradation. Unfortunately, existing approaches fall short of such challenges, and these models may be disabled, suffering from the evolving data distribution. The paper presents a framework for online anomaly detection of data streams, of which the baseline algorithm is the incremental learning method of Growing Neural Gas (GNG). It handles complex and evolving data streams via the proposed model Bayes-Optimized Adaptive Growing Neural Gas (BOA-GNG). Firstly, novel learning rate adjustment and neuron addition strategies are designed to enhance the model convergence and data presentation capability. Then, the Bayesian algorithm is adopted to realize the fine-grained search of BOA-GNG-based hyperparameters. Finally, comprehensive studies with six data sets verify the superiority of BOA-GNG in terms of detection accuracy and computational efficiency.

1. Introduction

Anomalies can be identified as abnormal points that have different characteristics from the majority of the data, and such novel observations may result from a specific failure, unexpected noise, etc., in industrial facilities monitoring. Online anomaly detection of industrial data streams can support operation and maintenance staff in identifying and locating potential equipment failures in a timely and accurate manner, avoiding serious faults and accidents.

Today, a complex data stream is common from Distributed Control System (DCS) and IoT devices in many industry fields, where multivariate data of infrastructures with complex correlations and time-varying characteristics are continuously arriving. The complexity brings an insurmountable challenge to anomaly detection, and the evolving data streams may make the preprocessing or trained models quickly outdated.

This paper concerns the computational method to deal with complex evolving data streams. Incremental learning-based anomaly detection provides an effective way to address the challenge. In particular, a competitive neural network has been widely used, as it is appropriate for an unsupervised setting that is natural for anomaly detection with rare labels. Existing state-of-the-art competitive neural network models, such as Growing Neural Gas (GNG) [1,2], Self-Organizing Neural Network (SONN) [3,4], Adaptive Resonance Theory (ART) [5], however, are designed for static offline data and cannot effectively cope with evolving data streams. There are a few competitive neural network-based methods proposed for time series anomaly detection [6,7,8]. However, they focus on learning the time-varying characteristic via adding adaptive learning strategies, ignoring the evaluation of the computational overhead of improved algorithms for evolving data streams.

The goal of this paper, thus, is to provide a novel framework for online anomaly detection that adaptively adopts the baseline GNG algorithm in an online setting, effectively dealing with the challenges of complex and evolving data streams.

2. Related Work

2.1. Incremental Learning

Undoubtedly, it is either too ineffective or too inefficient to handle a complex evolving data stream using a pre-trained fixed model or repeatedly creating a new model [9]. A common method is to build an initial model and incrementally update the model over the arriving data. Thus, Incremental Learning (IL) adapts a model to only the latest data, regardless of how data streams evolve. According to the different learning factors, the IL method can be categorized into three types, Sample Incremental Learning (SIL), Class Incremental Learning (CIL), and Feature Incremental Learning (FIL) [10]. SIL aims to continuously learn internal attributes to maintain model presentation ability for dynamic data streams and extract new knowledge to enhance model accuracy. CIL attempts to learn a new class of arriving data and add the new class into the historical class set so the classification performance can be improved. For the FIL method, such an algorithm is devoted to adding new features of evolving data to construct a new representation space to achieve the goal of improving classification accuracy. Due to the scarcity of anomalies in the anomaly detection tasks, it is difficult to define the boundary of normal and abnormal samples, and data labels are lacking for building a supervised model. Therefore, the CIL approach has difficulty performing well in anomaly detection of data streams. Additionally, excellent and distinguishable features in anomaly detection tasks are challenging to extract dynamically, especially in the modeling task of data streams [11]. Thus, SIL is better suited to constructing unsupervised models for the anomaly detection of evolving data streams.

2.2. Incremental Anomaly Detection

The time-varying characteristics of data streams require an anomaly detection model to have incremental learning ability. Existing state-of-the-art SIL-based anomaly detection approaches mainly include improved classical machine learning methods and competitive neural networks. Andrew [12] proposed a model integrating autoencoder and incremental clustering, providing a reliable machine learning-based monitoring method for electrical applications with varying power cycle patterns. Bigdeli [13] designed a two-layer cluster-based anomaly detection, which is fast, noise-resilient, and incremental, to lower the false alarm rate of real-time anomaly detection for dynamic data. Kaan [14] designed a two-stage filtering and hedging algorithm for sequential anomaly detection, where an incremental decision tree is used to construct a multimodal probability density function, and an adaptive thresholding scheme is used to detect anomalies. Furthermore, deep learning is also generally selected as the baseline algorithm for incremental anomaly detection, and its advantage is its spatiotemporal feature-catching ability. Nawaratne [15] proposes the incremental spatiotemporal learner for real-time video surveillance, and the learner is an unsupervised deep-learning module to continuously update and distinguish between new anomalies and normality over time. Agarwal [16] develops an LSTM-autoencoder-based incremental anomaly detection model to detect machine chatter, not only capturing changes in system dynamics over time but also incrementally improving the detection accuracy via transfer learning. The existing incremental anomaly detection approaches based on improved classical machine learning or deep learning algorithms always aim to design a continuous learning strategy for detected data via an offline setting, paying little attention to evolving data streams. Furthermore, many deep learning-based incremental anomaly detection models focus on learning temporal relationships in local sequences and incrementally updating the initial model, which is not suitable for handling arbitrarily evolving data streams. On the other hand, the optimization of computational overhead for deep anomaly detection models is another challenge for real-time streaming anomaly detection.

Competitive neural network-based incremental anomaly detection methods are always shallow neural networks, which can not dynamically learn the data stream characteristics and also have lower computation complexity. Typical algorithms of such methods are the Self-Organizing Incremental Neural Network (SOINN), Growing Neural Gas (GNG), Adaptive Resonance Theory (ART), and their variants. Fahn [17] designed a SOINN-based abnormal trajectory detection for efficient video condensation; the feature extraction module can compress the original video size to 10%, while the detection accuracy is maintained at 95%. Hu [18] developed a novel algorithm for fault diagnosis of redundant inertial measurement units. The method introduces SOINN and PCA to achieve high accuracy of tiny faults with low computational complexity. Based on the original GNG algorithm, Mahmoudabadi [19] integrates a fuzzy inference system with GNG for online anomaly detection of data streams; the selection trick of winning neurons in GNG is improved, and the algorithm shows better accuracy on public datasets over existing clustering models. Song [20] optimizes the adjustment of the learning rate and neuron addition and deletion strategies; the accuracy and computational efficiency of GNG are both improved in online anomaly detection. Therefore, in comparison with improved machine learning models, the competitive neural network-based incremental anomaly detection approach has the disadvantages of good precision and timeliness, which is more suitable for streaming anomaly detection. It is worth noting that promoting the ability to present complex data streams and optimize the adjustment strategy of the network structure is challenging. The motivation of this paper is to further optimize our work in [20].

2.3. Highlights

The main idea in this paper is to use the GNG algorithm as a baseline model, which can process evolving data streams via an adaptive learning mechanism. On the basis of our previous work in [20], we propose a novel framework (BOA-GNG) for online anomaly detection of data streams, and the contributions of this work are briefly summarized as follows.

1. An improved GNG algorithm is proposed to better model the complex and evolving data streams for online anomaly detection. Therein, a novel learning rate adjustment of GNG is designed to obtain better model convergence and stability; the neuron addition strategy is improved for better data representation ability.

2. In terms of hyperparameter settings for GNG, a Bayesian algorithm is introduced for fine-grained search of network hyperparameters instead of brute force search, optimizing the computational efficiency of online detection.

3. Extensive experiments on six data sets verify the superiority of BOA-GNG in terms of detection accuracy and computational efficiency, and further ablation studies are conducted to show the effectiveness of the improvement strategies.

3. Methodology

3.1. Algorithm of Original GNG

The base model of the proposed method is the GNG algorithm, and the theory and process of the GNG are described briefly. Then, the vital improvements for the original GNG are detailed and demonstrated in this section.

The original GNG, which combines the neural gas [21] and competitive Hebbian learning [22], is a topological graph. It aims to represent the characteristics of multivariate data by allowing the number of neurons to increase and taking into account the neighborhood relations of neurons. The learning process of the original GNG includes the adjustment of neuron parameters, adjustment of relationships between neurons, insertion of neurons, and deletion of neurons, of which the process is as follows in Algorithm 1, and the relevant parameters are shown in Table 1.

Table 1. Parameters of the Growing Neural Gas (GNG) algorithm.

Parameter	Parameter Meaning
$A$	The set of all neurons
$C$	The set of edge-connecting neurons
$\omega$	The weight of neuron
${\epsilon }_1$	The learning rate of winning neuron
${\epsilon }_2$	The learning rate of neuron in the neighborhood of winning neuron
$e$	The cumulative error of neuron
$n(s)$	The set of neurons that connect to the neuron $s$
$a_{\max }$	The maximum of edge’s age
$t$	The current time
$\lambda$	The step size of inserting neurons

Algorithm 1: The original Growing Neural Gas (GNG)

1.  Input: $\lambda$$,{\epsilon }_1$$,{\epsilon }_2$$,a_{max}$; 2.  $t$ → 0; 3.  Initialize the network $A$ with at least 2 neurons and $C=\varnothing$; 4.  For each new instance $x$ from the data stream: 5.  $t$ → $t+1$; 6.  Find the two neurons closest to $x$ from $A$, which are $s$ and $z$, by using the Euclidean distance: $s=\arg {\min }_{n\in A}‖x-{\omega }_n‖$, $z=\arg {\min }_{n\in A/\left\{s\right\}}‖x-{\omega }_n‖$; 7.  Update cumulative error of winning neuron $s$$:e_s=e_s+{‖x-{\omega }_s‖}^2$; 8.  Increase the age of the edges associated with $s$; 9.  Update the weights of the winning neuron and the neurons connected to it: ${\omega }_s={\omega }_s+{\epsilon }_1\times (x-{\omega }_s)$$,\forall v\in n(s):{\omega }_v={\omega }_v+{\epsilon }_2\times (x-{\omega }_v)$; 10. If $s$ and $z$ are connected by edges, reset the edge’s age to 0; 11. Else link the $s$ and $z$ with an edge of age 0: $c=c\cup \{(s,z)\}$ 12. Remove old edges with age > $a_{max}$ and neurons that become isolated; 13. If $t$ is multiple of $\lambda$, then let $q=\arg {\max }_{n\in A}{e}_n$ and let $f=\arg {\max }_{l\in n(q)}{e}_l$ 14. Create a new neuron $r$ between $q$ and $f$, which: ${\omega }_r=0.5\times ({\omega }_q+{\omega }_f)$$,e_r=0.5\times e_q$ 15. Exponentially decrease the representation error of all neurons: $\forall n\in A:e_n=0.9\times e_n$

As shown in the original GNG algorithm, a minimum number of neurons is initially created (lines 1~3), and then, new neurons and new neighborhood connections (edges) are added between them during learning, according to the input instances. For each new instance x from the data stream (lines 4~5), the two closest neurons $s$ and $z$ are found via Euclidean distance (line 6). A local representation error $e_s$ is increased for the winning neuron (line 7), and the age of the edges connected to this neuron is updated (line 8). The winning neuron $s$ and the neighboring neurons (linked to $s$ by an edge) are adapted according to the learning rate ${\epsilon }_1$ and ${\epsilon }_2$ (line 9). Moreover, the two neurons $s$ and $z$ are linked by a new edge (age is 0) (lines 10~11). When the edges reach a maximum age $a_{max}$ without being reset, they will be deleted. If any neuron belonging to the graph becomes isolated, it is also deleted (line 12). With the input of data streams, the graph periodically creates a new neuron between the two neighboring neurons that have accumulated the largest representation error (lines 13~14). Finally, the representation error of all neurons is subject to an exponential decay (line 15) in order to emphasize the importance of the recently measured errors.

As stated above, the GNG model can dynamically represent the distribution characteristics of streaming data, as shown in Figure 1. However, it also suffers from several drawbacks. First, the original GNG organizes neurons to represent the streaming data using fixed learning rates, affecting the model convergence and stability. Second, inserting new neurons into the model every k step cannot guarantee the necessity of the new neurons, and it may also result in model redundancy. Third, the hyperparameter optimization of GNG via grid search is time-consuming and imprecise, which makes it difficult to obtain the optimal solution.

Figure 1. The dynamic topology learning of Growing Neural Gas (GNG) algorithm. The blue bullets are neurons of GNG. The red bullet represents newly inserted neuron as the topology grows.

Several variants of GNG have been proposed to solve the previous problems, such as GNG-U [23], GWR [24], and Online GNG [25]. GNG-U defines a utility measure that removes neurons located in low-density regions and inserts them in regions of high density. However, we may need to create new neurons without the necessity of removing others. GWR judges whether new neurons need to be added according to the activity and firing thresholds of the winning neurons, but it does not consider optimizing the learning rate. The model of Online GNG can estimate the learning efficiency and adjust the network size to automatically fit the changing data space. However, neighbor-related strategies treat all neurons equally, which may result in some common parameters amplified by the sparsest or densest. In this paper, a novel GNG-based anomaly detection method of data streaming is proposed on the basis of our previous work in [20], and detailed improvements are presented as the description of highlights in Section 2.3.

3.2. Insertion Strategy of New Neurons

Representing a large amount of data with a few neurons is an essential idea of GNG models. As described in Algorithm 1, the original GNG creates a new neuron periodically, the strategy of which cannot adapt to a sudden change in the distribution of streaming data with a new data topology. If there has been no new pattern for a long time, the algorithm will create a lot of redundant neurons. On the other hand, a threshold $\mu$ is defined in our previous work (GNG-I) to determine whether an input is distinctive enough to insert new neurons, and thus, the fixed threshold $\mu$ may limit its ability to adapt to dynamically changing data streams. In this paper, we further optimized the threshold-setting strategy.

When a new instance $x$ of the data stream is coming, we find the winner $s$ by following line 6 of Algorithm 1. If $s$ has directly connected neighbors, then the new threshold $T_s$ is calculated using the maximum distance between $s$ and its neighbors following Formula (1).

$$T_s=\underset{n\in n(s)}{\max ‖{\omega }_s-{\omega }_n‖}$$

(1)

where ${\omega }_s$ is the weight of $s$, ${\omega }_n$ is the weight of $n$, $n(s)$ is the set of neurons that connect to the neuron $s$, and $‖{\omega }_s-{\omega }_n‖$ is the Euclidean distance between neuron $s$ and neuron $n$. If $s$ has no neighbor, the threshold $T_s$ is calculated using the minimum distance between $s$ and other neurons:

$$T_s=\underset{n\in A\backslash \left\{s\right\}}{\min ‖{\omega }_s-{\omega }_n‖}$$

(2)

where $A$ is the set of all neurons. During the learning process, if the distance between a new instance $x$ and its winner neuron $s$ is larger than the threshold, the algorithm adaptively inserts new neurons to support the new topology. The original GNG inserts the new neuron between the two neurons with the largest and second-largest cumulative error (line 14 of Algorithm 1), overlooking the role of the new neuron. However, the new neuron describes a new inputting pattern for the model. Therefore, the insertion location in this paper is between $x$ and $s$, taking into account both the winner neuron and the new neuron, and the weights of the new neuron can be calculated as follows.

$${\omega }_i=0.5\times ({\omega }_s+x)$$

(3)

The purpose of optimizing the neuron insertion is to improve the model’s adaptability in online anomaly detection scenarios for streaming data. When the data distribution is stationary, fewer neurons are created, and it is necessary to create neurons in a timely manner in order to handle changes.

3.3. Adjustment Strategy of Learning Rate

As shown in Table 1, to learn the inputting pattern, the winner and its neighbors will change their weights by multiplying fixed rates ${\epsilon }_1$ and ${\epsilon }_2$. Previous works [26,27] indicate that using a dynamic learning rate is helpful for modeling streaming data. In this paper, the adaptive learning rate is designed, and the calculation equation can be calculated as follows.

$$\left\{\begin{array}{l}{\epsilon }_1=e^{-\frac{‖x-{\omega }_s‖}{c_s\times T_s}}\\ {\epsilon }_2=e^{-\frac{‖x-{\omega }_n‖}{10(c_n+1)\times T_n}},\forall n\in n(s)\end{array}\right.$$

(4)

where $s$ is the winner neuron and $n$ is the neighbor neuron of $s$, $n(s)$ is the set of neurons that connect to the neuron $s$, $‖x-{\omega }_i‖$ is the Euclidean distance between instance $x$ and neuron $i$, $c_i$ is the winning times of the neuron $i$, $T_i$ has been defined in Equations (1) and (2). There are two main advantages of such an adaptive learning rate.

First, neurons with more winning times are considered to be more important, and these relatively vital neurons only need small adjustments after continuous incremental learning. The relatively larger learning rate enables quick updates to the weights at the beginning, while the gradual attenuation of the learning rate enables the neurons with higher winning frequency to converge gradually at the end of model training.

Second, the adaptive learning rate enables the model to adjust its neurons at appropriate step sizes. For example, if the distance between the input and its winner is large, the winner should adjust the weights fast. Otherwise, if the winner is mature by winning enough patterns, this acceleration can be decreased.

3.4. Optimization of Hyperparameters

The choice of GNG-based hyperparameters is very important for its performance. In this paper, the Bayesian optimization method is adopted to select optimal hyperparameters for the proposed BOA-GNG model. Bayesian optimization, which mainly consists of the probabilistic surrogate model and acquisition function, is an efficient global optimization algorithm [28,29]. Its objective function is designed as follows.

$$f_p(x)=\frac{1}{N_v}{\displaystyle \underset{i=1 }{\overset{N_v }{{\sum }_{x\in v}}}{e}^{-‖x-{\omega }_x‖}}$$

(5)

where $p$ is the set of parameters to be optimized, $v$ is the validation set, $x$ is the instance from validation set, $N_v$ is the total number of validation set, and ${\omega }_x$ is the neuron nearest to $x$. We hope to find the optimal parameter $p$, which minimizes the average distance from $x$ to the existing neurons and maximizes the value of the objective function at the same time. Taking the objective function as the optimization goal, the process of Bayesian optimization can be described below.

1. Initialization: Initialize the search scope of parameter set $p$. Start by randomly selecting a small set of initial sampling points from the search space. Evaluate the objective function to obtain initial observations.

2. Build a Probabilistic Surrogate Model: Based on the initial observations, construct a probabilistic surrogate model (defined as a Gaussian process in this paper) that approximates the objective function. This model captures both the mean prediction and the uncertainty of the objective function.

3. Optimize the Acquisition Function: Define an acquisition function (probability of improvement is selected as the acquisition function in this paper) that quantifies the utility of sampling at a given point based on the current surrogate model. Optimize this acquisition function to find the next point to sample.

4. Sample the Next Point: Evaluate the objective function at the point identified by optimizing the acquisition function. This new observation is used to update the surrogate model.

5. Iterate: Repeat steps 3 and 4, continuously updating the surrogate model and optimizing the acquisition function until a stopping criterion is met (e.g., a maximum number of iterations is reached or the objective function value improves below a certain threshold).

6. Return the Best Solution: At the end of the optimization process, return the point in the search space that corresponds to the best objective function value observed.

By balancing exploration and exploitation, it can efficiently navigate the search space to find high-quality solutions.

4. BOA-GNG-Based Anomaly Detection of Streaming Data

Anomaly detection methods learn a model from a reference set of regular (or normal) data and classify the unexpected data as irregular (or abnormal) [30]. However, if the reference data comes as a stream and its distribution is subject to change over time, the model trained over historical data may lose efficacy as the data distribution of the streaming data is changing. The proposed BOA-GNG can adapt to evolving data streams, and the representational topology structure is adaptively changing during the online anomaly detection of streaming data. In the detecting process, the distance-based method is used to estimate the anomaly state of input data, and the threshold $T_d$ can be calculated as follows.

$$T_d=\frac{1}{N_C}{\displaystyle \sum _{({\omega }_i,{\omega }_j)\in C}‖({\omega }_i,{\omega }_j)‖}$$

(6)

where $N_C$ is the current number of edges in the BOA-GNG model, and $({\omega }_i,{\omega }_j)$ is an edge connecting neurons. $C$ is the set of edge-connecting neurons, $‖{\omega }_i,{\omega }_j‖$ is the Euclidean distance of $({\omega }_i,{\omega }_j)$. Manually choosing a convenient value for the decision parameter $T_d$ is hard because it not only depends on the dataset but also on the number of neurons in the model, which varies over time. Therefore, we heuristically set $T_d$ equal to the expected distance between neighboring neurons in the model. In other words, $T_d$ at any time is defined as the average length of edges at that time.

Obviously, each neuron in the BOA-GNG model can be considered as the center of a hyper-sphere, and the model covers the space that represents regular data at any time. If a new instance $x$ from the data streams comes in, and the distance between it and the winner neuron is larger than $T_d$, it means that such data are not part of the existing topology of the model, and $x$ is considered an anomaly. The anomaly judge rule is defined as follows.

$${‖x-{\omega }_s‖}_{s\in A}>T_d$$

(7)

where $A$ the set of all neurons, $x$ is the current inputting data, ${\omega }_s$ is the weight of the winning neuron $s$, and $‖x-{\omega }_s‖$ is the Euclidean distance between $x$ and $s$. $T_d$ is calculated using Equation (6). As in Equation (7), the Euclidean distance of each $x$ and its winner neuron is calculated for online anomaly detection. The whole process of the BOA-GNG can be described as the pseudocode in Algorithms 2 and 3, and the variants of BOA-GNG are shown in Table 2.

Table 2. The notations of Bayes-Optimized Adaptive Growing Neural Gas (BOA-GNG).

Parameters	The Meaning of the Parameters
$N_m$	Limitations of the number of nodes
$N_w$	The threshold for the deletion of isolated neurons
$c_i$	The winning times of neuron $i$
$GP$	Gaussian process
$p$	The search scope of hyperparameters
$T\_best$	The optimal hyperparameters set
$T\_next$	The process hyperparameters set
$f_p$	The optimal objective function
$f_p\_next$	The process objective function
$D$	The set of ($T\_next,f_p\_next$)

Algorithm 2: The learning process of BOA-GNG

1.   Input: $N_m$, $N_w$, $a_{max}$; 2.   $t$ → 0; 3.   Initialize the network $A$ with at least 2 neurons and $c_i=0(i=1,2)$, $C=\varnothing$; 4.   For each new instance $x$ from the data stream; 5.   $t$ → $t+1$; 6.   Find the two neurons closest to $x$ from $A$, which are $s$ and $z$, by using the Euclidean distance: $s=\arg {\min }_{n\in A}‖x-{\omega }_n‖$$,z=\arg {\min }_{n\in A/\left\{s\right\}}‖x-{\omega }_n‖$; 7.   Update the winning times of neuron $s$:$c_s=c_s+1$; 8.   Increase the age of the edges associated with $s$; 9.   Calculate the learning rate ${\epsilon }_1$ and ${\epsilon }_2$ by Equation (4); 10. Update the weights of the winning neuron and the neurons connected to it: ${\omega }_s={\omega }_s+{\epsilon }_1\times (x-{\omega }_s)$$,\forall v\in n(s):{\omega }_v={\omega }_v+{\epsilon }_2\times (x-{\omega }_v)$; 11. If $s$ and $z$ are connected by edges, reset the edge’s age to 0; 12. Else link the $s$ and $z$ with an edge of age 0: $c=c\cup \{(s,z)\}$ 13. Remove old edges with age $>a_{max}$; 14. If the total number of neurons larger than $N_m$: Delete the neurons, which become isolated and $c<N_w$; 15 If $‖x-{\omega }_s‖>N_w$, then create a new neuron $r$ between $s$ and $x$, which: ${\omega }_r=0.5\times ({\omega }_s+x)$

Algorithm 3: The hyperparameter optimization of BOA-GNG via Bayesian

Initialize $p$, $T\_best$ as a random value, $f_p$ to negative infinity and $D=\varnothing$; Select the next set of hyperparameters based on the $GP$: $T\_next=GP.(T\_best,f_p)$ 3. Configure GNG proposed with the new hyperparameters and train the model: BOA-GNG. train (train set); 4. Evaluate the performance of $f_p\_next$ as describe in Equation (5) by using the validation set; 5. Update $D=(T\_next,f_p\_next)$ 6. $GP.\mathrm{update}(D)$ 7. Update the current $T\_best$ and $f_p$, if $f_p\_next>f_p$: $T\_best=T\_next$, $f_p=f_p\_next$ 8. Repeat steps 2~7 until the maximum number of iterations or $f_p$ stabilizes; 9. Output $f_p$ and the BOA-GNG model.

5. Validation of the Proposed Method

In this section, we validate the proposed method. Firstly, we describe the experimental datasets. Secondly, the evaluation indicators of the experiments are introduced. Thirdly, the effect of improvements is discussed. Finally, comparison and ablation studies are presented, and the results are discussed.

5.1. Datasets

The experimental datasets include five publicly available datasets and one real engineering dataset from the Payload of China’s aerospace satellite. Table 3 gives a brief summary of the six datasets.

Table 3. Summary of the six experimental datasets.

Dataset	Train Set Samples	Test Set Samples	Dimension
Shuttle	44,686	1778	9
KDD-CUP99 HTTP	80,000	2100	27
Satellite	4100	1000	36
SMAP	70,000	7000	26
MSL	15,000	1500	56
Payload Dataset	80,000	16,662	65

Shuttle dataset: This dataset was utilized to delineate the position of radiators on NASA space shuttles, primarily for classification purposes. Initially comprising 58,000 samples, 80% were in the first category. The version used in this study is post-processed data, where samples belonging to the first category are considered normal, while other categories are considered anomalous. In the experiments, the test set consisted of 1778 samples evenly distributed between normal and anomalous data.

KDD-CUP99 HTTP dataset: This dataset was collected over a period of 9 weeks from a simulated Air Force network and includes network connection and system audit data, which are commonly used to validate the performance of intrusion detection algorithms. To meet unsupervised or semi-supervised requirements, this study ultimately adopted a simplified version of KDD99, known as KDD-CUP99 HTTP. This subset used only HTTP traffic data from the original dataset, which consists of 620,000 samples, including 1053 anomalous samples, accounting for 0.17%. In the experiment, 80,000 normal samples were used for training, and 2100 samples were used for testing.

Satellite Dataset: This dataset was generated by the Australian Centre for Remote Sensing from NASA data. This dataset contains 36 parameters and serves as a multi-class classification dataset. Throughout the experiment, all normal data were considered the positive class, while all anomalous data were considered the negative class. A selection of 4100 normal samples was used for training, and 1000 samples were reserved for testing, of which 925 were normal and 75 were anomalous.

SMAP dataset: This dataset was obtained from a NASA spacecraft and contains real-world data. It consists of 55 telemetry channels, 429,735 telemetry values, and 69 anomalous sequences. Anomalies within the dataset are divided into two categories: point anomalies and contextual anomalies. For the experiment, 70,000 normal samples from channels A1 to A9 were selected for training, and 7000 samples were selected for testing, with 10% of the testing samples being anomalous.

MSL dataset: This dataset is a real-world dataset derived from NASA’s spacecraft via the Mars Science Laboratory (MSL). It comprises anomalous data stemming from Incident, Surprise, and Anomaly (ISA) reports from a spacecraft monitoring system. The data set consists of 27 telemetry channels, 66,709 telemetry values, and 36 anomalous sequences. To perform the experiments, 15,000 normal samples were selected for training, and 1500 samples were selected for testing, with 10% of the testing samples being anomalous.

Payload Dataset: This dataset consists of operation data collected from the Payload of China’s aerospace satellite. It includes variables such as current, voltage, temperature, and command parameters, totaling 66 dimensions. To represent a typical anomaly of a complex mode type during its operational phase, a subset of 96,662 samples was extracted following preprocessing steps, including noise reduction. Upon analyzing the data samples, a change in current was observed during abnormal operational stages, although it remained within the valid threshold values. Figure 2 shows that under normal conditions, the payload operates with a current of approximately 1.44 A within the South Atlantic Anomaly (SAA) and around 1.33 A outside the SAA. However, an anomaly occurred with a current surge to approximately 1.49 A. To reduce the influence of significant variations in current dimensionality during experimentation, we reduced the dataset to retain the remaining 65 dimensions and then selected 80,000 normal samples as the training set and the remaining 16,662 samples as the test set. The test set comprised 3083 anomalous samples and 13,580 normal samples.

Figure 2. The current curve before and after fault.

5.2. Performance Metrics

Some metrics were chosen to evaluate the performance of the proposed method during the experiments in this section, including precision $p$, recall $r$, F1 score, and the processing speed $v$.

$$p=\frac{TP}{TP+FP}$$

(8)

$$r=\frac{TP}{TP+FN}$$

(9)

$$f=\frac{2\times p\times r}{p+r}$$

(10)

$$v=\frac{N_d}{t_d}$$

(11)

where $TP$ (True Positive) indicates the number of correctly detected anomalies, $FP$ (False Positive) is the number of incorrectly detected anomalies, $FN$ (False Negative) denotes the number of missed detected anomalies, $N_d$ is the total number of samples, and $t_d$ is the time used to process the samples.

5.3. The Improvement Effect of BOA-GNG

In this section, we demonstrate the detailed effect of improvement for BOA-GNG. As described in Section 3.3, BOA-GNG adopts the adaptive learning rate of neurons, which is related to the Euclidean distance of the input data and the winner neuron, the maximum Euclidean distance between the winner neuron and its neighbors, and the winning times of each neuron. Figure 3a indicates the changing curve of the learning rate and neuron winning times of one neuron on the Payload dataset. Although the adaptive learning rate fluctuates due to the calculated Euclidean distance of the input data and winning neuron during the learning process, we can see that the improved learning rate adjustment strategy can obtain a faster convergence speed.

Figure 3. (a) The changing curve of learning rate and neuron winning times. (b) Value range setting of Bayes-Optimized Adaptive Growing Neural Gas (BOA-GNG) hyperparameters.

Here, we discuss the detailed hyperparameter optimization of BOA-GNG based on the Payload dataset. The hyperparameters of the proposed method described in line 1 of Table 3 are the maximum edge’s age $a_{max}$, the maximum of the neurons $N_m$, and the threshold of isolated neuron deletion $N_w$. The parameter set $p=(a_{max},N_m,N_w)$, and the objective function is defined in Equation (5). We selected samples from the training set at a ratio of 4:1 as the validation set, randomly selected 320 sets of $p$, and then calculated the value of the objective function. In order to determine the appropriate search scope of the parameter set $p$, the control variable method was used to analyze the relations of optimization loss and hyperparameters. As shown in Figure 3b, as the value of $a_{max}$ is greater than 120, the value of the objective function decreases. If the value of $N_m$ is greater than 260, the value of the objective function remains almost unchanged. The value of the objective function decreases when the value of $N_w$ is greater than 150. Therefore, initialize the search scope of the parameter set $p$ is as follows:

$$p=\text{{}{a}_{max}\in [20,120];N_m\in [20,260];N_w\in [20,150]\text{}}$$

(12)

On the basis of the range setting of hyperparameters, the Bayesian optimization steps in Algorithm 3 are conducted to search the optimal parameter setting of BOA-GNG, and the best parameter combination $p$ is obtained, where $p=(32,160,88)$. Figure 4 shows the iteration loss of BOA-GNG under different hyperparameter settings via grid search and Bayesian optimization, which demonstrates that the latter can obtain a better convergence value with a faster convergence speed.

Figure 4. The iteration loss of Bayes-Optimized Adaptive Growing Neural Gas (BOA-GNG) under different hyperparameter settings.

5.4. Comparison Experiments

In the comparison experiments, the proposed model was compared with the original GNG, GWR, GNG-I, SOIIN, and K-Means. The experiments were conducted on a computer equipped with an Intel(R) Core (TM) i7-10700 CPU at 2.90 GHz and 16 GB of RAM. According to the results of the test set, the values of the corresponding evaluation criteria are shown in Table 4. For each dataset, the result of the best-performing method is highlighted in bold, and we can come to the conclusion that BOA-GNG achieves good results on all datasets.

Table 4. The comparison results of Bayes-Optimized Adaptive Growing Neural Gas (BOA-GNG) and other methods on six datasets.

Datasets	Criteria	GNG [1]	GWR [24]	GNG-I [20]	SOINN [17]	K-Means	Proposed Method
Shuttle dataset	p	0.9762	0.9976	0.9988	0.9836	0.9976	0.9988
	r	0.9818	0.9613	0.9601	0.9692	0.9612	0.9901
	f	0.9790	0.9791	0.9791	0.9700	0.9791	0.9944
HTTP dataset	p	0.9918	0.8072	0.9944	0.7100	0.8400	0.9959
	r	0.9990	0.7608	0.9979	0.9189	0.7197	0.9990
	f	0.9954	0.7833	0.9962	0.8011	0.7752	0.9972
Satellite dataset	p	0.9789	0.9597	0.9687	0.9800	0.8734	0.9802
	r	0.9514	0.9005	0.9708	0.7935	0.5070	0.9632
	f	0.9649	0.9292	0.9698	0.8769	0.6416	0.9716
SMAP dataset	p	0.9852	0.7924	1.0000	0.8483	0.8160	1.0000
	r	0.9912	0.9683	0.9939	1.0000	0.8394	0.9942
	f	0.9877	0.8716	0.9969	0.9179	0.8275	0.9971
MSL dataset	p	0.8662	1.0000	0.8662	0.6033	0.9200	1.0000
	r	0.9706	0.9529	0.9706	0.9592	0.9918	0.9807
	f	0.9769	0.9751	0.9769	0.7407	0.9545	0.9902
Payload dataset	p	0.8907	0.8149	0.9571	0.8150	0.9996	0.9997
	r	0.7594	0.9996	0.9987	1.0000	0.7744	0.9949
	f	0.8198	0.8979	0.9775	0.8981	0.8727	0.9973

Given the sequence nature of streaming data, divide the test set into three equal parts in order (part 1, part 2, part 3). To simulate the continuous arrival of streaming data, define that Stream 1 includes part 1, Stream 2 includes part 1 and part 2, and Stream 3 includes part 1, part 2, and part 3. For each dataset, the results of the rolling test of BOA-GNG are shown in Table 5. As the data flow continues to expand, the values of the corresponding evaluation criteria of BOA-GNG remain at a relatively high level.

Table 5. The results of Bayes-Optimized Adaptive Growing Neural Gas (BOA-GNG) on the rolling tests.

Datasets	Data Stream	p	r	f
Shuttle dataset	Stream 1	1.0000	1.0000	1.0000
	Stream 2	1.0000	1.0000	1.0000
	Stream 3	1.0000	0.9978	0.9989
HTTP dataset	Stream 1	1.0000	1.0000	1.0000
	Stream 2	1.0000	0.9104	0.9531
	Stream 3	1.0000	0.9538	0.9764
Satellite dataset	Stream 1	1.0000	0.9970	0.9985
	Stream 2	1.0000	0.9955	0.9977
	Stream 3	0.9613	0.9946	0.9777
SMAP dataset	Stream 1	1.0000	0.9858	0.9928
	Stream 2	1.0000	0.9858	0.9928
	Stream 3	1.0000	0.9846	0.9922
MSL dataset	Stream 1	1.0000	0.9911	0.9955
	Stream 2	1.0000	0.9673	0.9834
	Stream 3	1.0000	0.9673	0.9834
Payload dataset	Stream 1	1.0000	0.9984	0.9992
	Stream 2	1.0000	0.9940	0.9970
	Stream 3	0.9973	0.9946	0.9959

For online abnormal detection scenarios, computational efficiency is important as it is the high accuracy of anomaly detection, which includes learning time and detecting time for incremental learning techniques. As a result, the experiments compare the computational efficiency of BOA-GNG and other GNG-based models. The results are shown in Table 6, and the bold fonts denote the highest computational efficiency of the methods on each dataset. $v_1$, $v_2$, respectively, represent the average velocity of the learning and detecting processes. We show that BOA-GNG outperforms other models on most datasets in terms of learning and velocity detection.

Table 6. Comparison of online learning rates on each dataset.

Datasets	GNG [1]		GWR [24]		GNG-I [20]		Proposed Method
	v1 (dot/s)	v2 (dot/s)	v1 (dot/s)	v2 (dot/s)	v1 (dot/s)	v2 (dot/s)	v1 (dot/s)	v2 (dot/s)
Shuttle	497.40	19,772.57	357.49	15,303.42	3796.60	186,191.67	3822.58	203,118.18
KDD-CUP99	453.26	21,276.60	1079.77	6467.26	479.47	22,727.27	459.61	22,222.22
Satellite	2204.30	9761.90	334.97	3504.27	1822.22	9318.18	2469.88	12,058.82
SMAP	2987.62	4888.26	315.71	3564.15	2687.14	5728.31	3111.11	5988.02
MSL	2504.17	4213.48	304.63	3170.53	2008.03	4065.04	2369.67	4043.13
Payload dataset	276.72	2002.50	143.63	1714.53	482.83	4212.74	535.98	4364.43

5.5. Ablation Experiments

Ablation experiments are conducted to verify the effectiveness of the optimization strategy in Section 3.2 and Section 3.3 We define the model with a fixed learning rate as BOA-GNG_FLR, with StepLR as BOA-GNG_SLR, and with the fixed insertion step size as BOA-GNG_FIS. The results of anomaly detection performance are shown in Table 7 and the result of the best-performing model is highlighted in bold. It can be seen that the adaptive learning rate and new insertion strategy have a positive effect on improving the GNG-based anomaly detection performance.

Table 7. Results of ablation experiments on the payload dataset.

Algorithms	p	r	f
BOA-GNGFLR	0.8159	0.9951	0.8966
BOA-GNGSLR	0.8667	0.9951	0.9264
BOA-GNGFIS	0.9997	0.9834	0.9915
BOA-GNG	0.9997	0.9949	0.9973

6. Conclusions

In this paper, we propose a novel incremental learning model named BOA-GNG for online anomaly detection of streaming data. The proposed approach adopts our previous model [22] as a baseline model, of which the learning rate, neuron insertion strategy, and network optimization strategy are made for better dynamic data learning ability and online detection performance.

We demonstrate that the adaptive learning rate can obtain faster convergence ability compared to the previous linear learning rate. Then, the new neuron insertion method can improve the model’s adaptability for evolving data streams and make it more flexible to adapt to changes in data distributions. Finally, Bayesian optimization is introduced for fast and fine-grained hyperparameter setting instead of grid search.

Five open datasets and a real engineering dataset from the aerospace satellite are taken as experimental cases to verify the effectiveness and superiority of the proposed model. Results indicate that BOA-GNG can improve precision, recall, F1 score, and computing efficiency of online anomaly detection compared with classical GNG-based models, and both the precision and recall of BOA-GNG reach more than 95% on six datasets. Additionally, the lowest learning and detecting velocity can reach nearly 500 dot/s and 4000 dot/s. Further studies demonstrate that an adaptive learning rate and new neuron insertion strategy are effective in improving the baseline model.