Robust Anomaly Detection of Multivariate Time Series Data via Adversarial Graph Attention BiGRU

Abstract

Multivariate time series data (MTSD) anomaly detection due to complex spatio-temporal dependencies among sensors and pervasive environmental noise. The existing methods struggle to balance anomaly detection accuracy with robustness against data contamination. Hence, this paper proposes a robust multivariate temporal data anomaly detection method based on graph attention for training convolutional neural networks (PGAT-BiGRU-NRA). Firstly, the parallel graph attention (PGAT) mechanism extracts the time-dependent and spatially related features of MTSD to realize the MTSD fusion. Then, a bidirectional gate recurrent unit (BiGRU) is utilized to extract the contextual information of the data to avoid information loss. In addition, reconstructing the noise for adversarial training aims to achieve a more robust anomaly detection of MTSD. The experiments conducted on real industrial equipment datasets evaluate the effectiveness of the method in the task of MTSD, and the comparative experiments verify that the proposed method outperforms the mainstream baseline model. The proposed method achieves anomaly detection and robust performance in noise interference, which provides feasible technical support for the stable operation of industrial equipment in complex environments.

1. Introduction

In recent years, the rapid development of emerging technologies, such as the Internet of Things (IoT), cloud computing, and 6G, has led to explosive growth in data size and highly complex data patterns in healthcare [1], aerospace [2], manufacturing [3], and cybersecurity [4]. The sensor samples and records form a time series of data (TSD) according to time correlation, using the progression of the variable of interest over time or at irregular intervals. Timing data reflect the real-time state of the system due to the complexity of the sensor operating environment, the inherent characteristics of the sensor itself, and the imbalance of system resources and other interferences [5]. Thus, there are irregular fluctuations in the real-time collection of data that are not anomalies. Consequently, there has been growing attention on developing data processing methods that can adequately extract time series data letter features and effectively improve the accuracy and real-time performance of identifying abnormal behaviors in recent years [6].

TSD relies on time-lapse, which contains a large amount of information with many features that are difficult to achieve through manual labeling. Moreover, unsupervised feature learning is an effective way to analyze time series data. For example, support vector regression (SVR) [7] employs time-lag coordinate autoregression and phase space reconstruction to extract temporal data features for prediction and to verify the prediction accuracy with wind speed data. Time series forest (TSF) [8] constructs a collection of tree systems for time series data to randomly extract sampling features and linear relationships at each tree node and combines entropy gain and distance metrics to evaluate the effectiveness of the model. Given the complex and highly dynamic characteristics of physical equipment systems in the era of artificial intelligence (AI), relying solely on single-dimensional time series data may result in vulnerability to noise interference, leading to insufficient model robustness. Hence, multiple sensors are required to provide more comprehensive information, monitor equipment anomalies, and predict anomalies to prevent uncontrollable events.

To avoid incorrectly identifying normal fluctuations in timing data as anomalies, the correlation between different variables of the anomaly detection of multivariate time series data (MTSD) is used to satisfy the detection performance in multidimensional timing data patterns. Deep learning methods [9] are extremely capable of handling complex correlations of MTSD with nonlinear properties to mine high-dimensional data features. Furthermore, unsupervised reconstruction methods [10] are developed for MTSD anomaly detection in practical applications. However, existing research mainly focuses on specific scenarios, lacking a comprehensive approach to satisfy the data pattern complexity and detection accuracy robustness of MTSD. Therefore, there is an urgent need for an unsupervised deep learning-based MTSD processing method that fully exploits temporal dependencies and spatial correlations of the nodes, improves detection accuracy, and maintains the robustness of the detection.

Currently, MTSD processing is limited by the uncertainty and complexity of industrial data. The issues are summarized as follows:

(1)

Existing MTSD typically use dimension shifting and scaling to eliminate data size, increasing training speed. However, due to the inconsistent influence of different dimensional data on the results, failure to fully consider the spatial correlation characteristics between different dimensional data leads to inadequate model performance [11]. However, due to the complex change patterns of MTSD and the inclusion of both continuous and discrete variables, it is difficult to make the model compatible with historical data and the current task, while adequately extracting spatially relevant features. Thus, the optimal scheme for compatible hybrid temporal and spatial feature extraction has yet to be investigated.

(2)

Extracting potential representations from MTSD is susceptible to noise interference and fluctuations in the training set. As shown in Figure 1 (green indicates fluctuation, red indicates abnormality), the collected data contain a significant amount of noise, visualizing considerable noise in a real dataset, which is a common phenomenon in the real world. Existing methods lack the robustness to deal with MTSD in the face of noise and anomalous contamination. Hence, it is difficult for the model to achieve the expected results due to large errors caused by noise and interference during data reconstruction.

In summary, although MTSD can provide more comprehensive information, the existing MTSD anomaly detection is highly susceptible to noise and anomaly interference. Moreover, no research has been focused on the highly robust processing of MTSD, according to the survey. This paper proposes a processing method compatible with extracting MTSD temporal and spatial features to realize data fusion and add noise against training to improve robustness. The contributions of this paper are summarized as follows:

(1)

A parallel improved graph attention (PGAT) is proposed to analyze temporal dependence and spatial correlation in MTSD without any prior knowledge, thereby avoiding misclassification of normal fluctuations as anomalies.

(2)

Integrating deep learning technology, BiGRU model is selected to learn the spatio-temporal features of dimensional time series data. In addition, the context-dependent features are obtained to prevent information loss and enhance the robust performance of the model.

(3)

The advantages of predictive and reconstructive models are combined by introducing joint optimization. The reconstructed noise samples are used for adversarial training to improve the robustness of the model in the face of contaminated data. The performance and explanatory power of the model are tested on three public datasets, proving that the proposed method outperforms eight typical baseline models. Finally, the proposed model is applied to anomaly detection in laboratory equipment, validating the effectiveness of the method.

The rest of the paper is organized as follows: Section 2 reviews the related work on multidimensional time series data fusion and robust anomaly detection processing. Section 3 introduces the proposed PGAT-BiGRU-NRA model and provides a detailed description of the main modules. Section 4 conducts experiments and provides experimental results to evaluate the model. Finally, Section 5 concludes this article.

2. Related Work

2.1. MTSD Fusion

Multivariate data can provide more comprehensive information than univariate data, eliminating the effects of instability and susceptibility to noise interference of univariate data by complementing each other’s support. This robust characterization enhances the accuracy of information extraction [12]. Data fusion technology effectively improves the processing capability and utilization rate of time series big data. Moreover, it integrates, correlates, and analyzes noisy massive high-dimensional multivariate data and provides reliable and high-quality data support for subsequent anomaly detection prediction and decision-making [13]. Currently, commonly used data fusion methods can be categorized into probability-based fusion methods, D-S evidence theory-based fusion methods, knowledge-based fusion methods, and attention mechanism-based fusion methods.

Table 1. Summary of data fusion methods based on probability, D-S evidence theory, and knowledge.

Methods	Models	Implementation	Limitations
Probability	Bayesian [14]	Circulant matrix derivation	Difficulty in obtaining a priori probabilities and modeling data when dealing with high-dimensional complex data.
	Kalman filtering [15]	Attenuation factor perturbs weights
	Markov model [16]	Singular value decomposition norm
D-S Evidence Theory	D-SBNs [17]	Feature fuzzy prior probability	The quality function is difficult to define, and the model is not robust enough.
	BRDD-S [18]	Renyi divergence belief entropy
Knowledge	PCASVM [19]	Principal component analysis	Sensitive to noise points and performs poorly in the face of unbalanced datasets.
	C-clustering [20]	Multiple rounds of clustering
	CNN [21]	Convert to RGB image

summarizes data fusion methods based on probability, D-S evidence theory, and knowledge.

The attention mechanism is first applied to the vision domain [22] based on the idea of weight allocation, which assigns weights according to the importance of the observation features to focus on the important information and enhance the data processing capability of the model. The multi-head self-attention mechanism [23] learns multiple caveats through various heads. Each head can sense contextual information to extract external discriminative features, improving the self-attention mechanism. However, it comes with computational complexity and the risk of overfitting. The graph attention network [24] mines the interdependence between nodes when dealing with multidimensional data. By introducing a multi-head attention mechanism in the graph network, it efficiently calculates and assigns the weights of different nodes in the neighborhood, thereby aggregating the information from neighboring nodes to map to a certain node to improve the model fusion processing effect. MST-GAT [25] is based on the multimodal graph attention mechanism and temporal convolutional network to capture spatio-temporal correlations in multidimensional time series data and enhance the interpretable performance for detecting anomalies, which lays the foundation for our research.

2.2. Anomaly Detection

The prediction-based model trains the prediction model using historical time series data and detects anomalies by comparing the difference between the model’s predicted values and the actual sampled values. Long Short-Term Memory with Autoencoder (LSTM-AE) [26] combines multi-source data for hyperparameter tuning, enhancing the detection performance of the prediction method by interpreting anomalies in the LSTM prediction results utilizing an autoencoder. A real-time anomaly detection algorithm based on Hierarchical Temporal Memory and Bayesian Network (RADM) [27], validates its effectiveness in a cloud platform sensing system. While prediction-based models excel at capturing periodicity and trends in timing data, they often struggle with complex timing patterns and sudden anomalies.

The reconstruction-based model detects anomalies by learning normal patterns and comparing the differences between the reconstructed input data and the input data. MTSD offer rich information to distinguish between anomalous performance problems and normal fluctuations, which is crucial for further system troubleshooting and performance repair. Existing research typically has developed unsupervised processing methods combined with deep learning applicable to multidimensional unlabeled data. Contrastive autoencoder for anomaly detection (CAE-AD) [28] introduces a multi-granularity comparison method to extract normal data patterns, adds a projection layer to learn the context, and transforms the time-frequency domain view through data augmentation to address the normal pattern modeling of multidimensional data without proper constraints. OmniAnomaly [29] improves anomaly interpretation and model robustness by learning multidimensional time series with continuous regularization of random variables to capture normal patterns in the data while providing time series reconstruction probabilities, enhancing anomaly interpretation and model robustness. Unsupervised Anomaly Detection (USAD) [30] integrates an adversarial training architecture for fast training and isolating anomalies to infer normal and abnormal behavior with unsupervised learning, addressing the slow and error-prone diagnosis of traditional methods in the face of surging data complexity. Modified auxiliary classifier generative adversarial networks (MACGAN) [31] introduce the Wasserstein distance loss function to solve defects like modal collapse and gradient disappearance in the generated samples from multidimensional data. It also adds an independent classifier to improve the model’s anomaly interpretation ability, making it compatible with classification and discrimination. It is worth noting that reconstruction-based models are robust in handling complex models and sudden anomalies but are sensitive to noise and the quality of training data. AMBi-GAN [32] constructs an anomaly detection model based on generative adversarial networks with long- and short-term memories, which fully captures the time dependence of temporal data and improves the robustness of the model through adversarial training, providing solutions for our research.

Existing studies have demonstrated that reconstruction-based methods generate samples during computation and add random noise to create antagonistic samples, improving the robustness of the model. However, the existing methods are not compatible with extracting spatio-temporal correlation information and require integration with multidimensional data fusion to correctly extract the temporal dependence of the data. This method is designed to efficiently differentiate the robust performance of normal fluctuations and anomalies, thereby enhancing the anomaly interpretation capability of the model to further provide reliable data support for system troubleshooting and repair.

3. PGAT-BiGRU-NRA Framework

This section describes the unsupervised MTSD anomaly detection problem. In response to the fact that existing MTSD processing methods are sensitive to noise and anomalies in the training data, which limits the further improvement of the detection accuracy, a noise adversarial training MTSD processing model compatible with time dependence and spatial correlation for robust anomaly detection of MTSD, as well as a detailed description of the key modules, is provided.

3.1. Description of the Problem

The observed data are defined as $\Sigma =\left\{X_1,X_2,...{,X}_n\right\}$, where $n$ represents the dimension of the multivariate data. $X_m$ is the ordered univariate observed over time in the m dimension, denoted by $X_m=\left\{x_m^1,x_m^2,...,x_m^T\right\}$, where $T$ indicates the data time step limit within the time window. $\Sigma \in {\mathbb{R}}^{n\times T}$ in MTSD means data processing aims at fully extracting its temporal and spatial features, calculating the anomaly scores of the test data within a time step, and treating the data corresponding to a time step that exceeds the threshold score as anomalies.

In addition, it is considered that time series acquisition usually contains multiple time nodes, and data between time nodes are captured through a time window using a window size $T$ and a step size $l$. As shown in Figure 2, the rectangle represents a time window, denoted as a vector-dimensioned time window sliding on the time axis with a step size $l$.

3.2. Overall Framework

The overall framework of the proposed model is shown in Figure 3, in which GATv2 is used to introduce dynamic attention, spatial correlation GAT is used for the extraction of spatial correlation features, and temporal correlation GAT is used for the extraction of time-dependent features of multidimensional time series data. In addition, BiGRU is able to consider both past and future information of temporal data, capturing contextual information through processing inputs in both forward and backward directions. Finally, integrating the advantages of predictive and reconstructive models for anomaly detection tasks, introducing robust joint adversarial optimization, and adding noise countermeasures to the reconstructive model give the whole model a higher robust performance. The implementation details of the architecture will be systematically described in the key technologies in this chapter.

The methodology outlined in the paper comprises the following key steps:

(1)

Data Preprocessing: data preprocessing is performed for MTSD to select normalization methods according to the requirements, and 1D-CNN is applied to extract high-level local features of each time series in the preprocessed data;

(2)

Fusion of Multidimensional Data: parallel GATv2 is used to learn data dependencies from both the temporal dimension and the spatial feature dimension, emphasizing the fusion of multiple features with different weights;

(3)

Learning-Fused Feature Patterns: the BiGRU layer connects the outputs of 1D-CNN and parallel GATv2 to extract contextual information from MTSD and capture time-dependent sequential pattern features;

(4)

Robust Anomaly Detection and Interpretation: VAE reconstructs the model and adds noise countermeasures to improve the robustness of the fully connected network’s joint predictive modeling and obtain the result.

3.3. Key Technologies

3.3.1. Data Preprocessing

The original data are normalized using min-max to make the data of the same size. Local features of the data are identified and extracted using 1D-CNN with a kernel size of 7. $\mathit{min}\left(X_{train}\right)$ is the minimum value of the training set, and $\mathit{max}\left(X_{train}\right)$ is the maximum value of the training set.

$$x_i^{\prime }={\displaystyle \frac{x_i-\mathit{min}\left(X_{train}\right)}{\mathit{max}\left(X_{train}\right)-\mathit{min}\left(X_{train}\right)}}$$

(1)

3.3.2. Parallel Improvement of Graph Attention Mechanisms

As shown in Figure 4, the graph structure based on GAT considers the interdependencies between nodes to obtain a new representation of the nodes. Define the input feature vector denoted as $\overrightarrow{h_i}=\left\{\overrightarrow{h_1},\overrightarrow{h_2},\cdots ,\overrightarrow{h_N}\right\},\overrightarrow{h_i}ϵ{\mathbb{R}}^F$, $N$ represents the number of nodes, and $F$ denotes the feature dimension. The output of each node is calculated through the weighting strategy of GAT as follows:

$${\overrightarrow{h}}_i^{\prime }=\sigma \left({\alpha }_{i,i}W\overrightarrow{h_i}+{\sum }_{j\in N\left(i\right)}{\alpha }_{i,i}W\overrightarrow{h_j}\right)$$

(2)

where ${\overrightarrow{h}}_i^{\prime }$ denotes the output of node $i$, $\sigma \left(·\right)$ represents the sigmoid activation function, $\overrightarrow{h_j}$ indicates one of the neighboring nodes of node $i$, and $W\in {\mathbb{R}}^{F\times F^{\prime }}$ means the weight matrix reflecting the relationship between the input and output features.

The degree of association between the nodes is defined as follows:

$$e_{i,j}={\overrightarrow{a}}^{T}LeakyReLU\left(W\left[{\overrightarrow{h}}_i\left|\right|\overrightarrow{h_j}\right]\right)$$

(3)

where $\overrightarrow{a}$ is the attention learning parameter for the feature dimension, which $\mathit{exp}\left(·\right)$ represents the exponential function of the natural constant $e$. Therefore, the attention score by mapping nodes is expressed as follows:

$${\alpha }_{i,j}={\displaystyle \frac{\mathit{exp}\left(e_{i,j}\right)}{{\Sigma }_{k\in N\left(\dot{i}\right)\cup \left\{i\right\}}\mathit{exp}\left(e_{i,k}\right)}}$$

(4)

We use two parallel GATv2 oriented toward spatial and temporal features, respectively, as follows:

The structure of the spatially oriented graph attention layer is designed to detect correlations between multidimensional variables and enable spatial feature extraction without a priori knowledge. MTSD, each node, and each edge are considered as a complete graph, a feature of one dimension, and a correlation between features, respectively, and the inputs are defined as $\overrightarrow{x_i}=\left\{{\overrightarrow{x}}_i^t|i\in \left[0,k\right),t\in \left[0,n\right)\right\}$, where $k$ denotes the total number of dimensions of multivariate features, and $n$ represents the total number of timestamps in the sliding window, which is used as an input to GAT to extract the spatial features of MTSD.

The time-oriented graph attention layer is structured to capture temporal dependencies in a time series by considering the tiling of a sliding window as a complete graph, each node as a feature vector of the sliding window, and each bar as a correlation between different time windows. Similarly to spatial-oriented GAT, its output is also a k × n vector matrix, and the temporal and spatial features of the parallel GAT output are connected with the local features of the preprocessing output to form an $n\times 3k$ dimensional feature vector, incorporating the comprehensive information of MTSD.

3.3.3. Bi-Directional Gated Recurrent Unit

The anomaly detection of multivariate time series data (MTSD) needs to capture long-term trend anomalies and sudden transient anomalies at the same time. The problem of gradient disappearance and gradient explosion exists in RNN-based data processing tasks. BiGRU can solve this problem and better learn the long-term and short-term temporal dependencies of time series data. The forward GRU captures the long-term cumulative effect of the equipment state, and the reverse GRU verifies the chain reaction of the sudden anomalies. The spatio-temporal features extracted from PGAT are further refined by the contextual modeling of BiGRU, to avoid the loss of features. The GRU unit is shown in Figure 5.

GRU unit contains two key gating mechanisms: the reset gate and the update gate. The reset gate $r_t=\sigma \left(W_j{x}_t+U_j{h}_{t-1}\right)$ captures short-term dependencies, where $W_j$ and $U_j$ are the weights of the reset gates, and the update gate $z_t=\sigma \left(W_i{x}_t+U_i{h}_{t-1}\right)$ is used to capture long-term dependencies, where $W_i$ and $U_i$ are the weights of the update gates. The aggregated hidden state of inputs and outputs of the previous layer is $h_t^{~}=tanh\left(r_t{U}_j{h}_{t-1}+W_h{x}_t\right)$. It follows that the native output of the GRU unit is $h_t=\left(1-z_t\right){·h}_{t-1}+z_t{h}_t^{~}$. In time step $t$, the forward layer produces the forward hidden layer output to be passed to the next time step for further computation as $h_t^F=tanh\left(h_{t-1},x_t\right)$. Similarly, the output of the reverse hidden layer at time step t is passed in reverse to time step $t-1$ as $h_t^B=tanh\left(h_{t+1},x_t\right)$. The combination of the forward and reverse layers at each time step is $Y_t=tanh\left(h_t^F,h_t^B\right)$, which fully considers the historical data and future data time characteristics.

3.3.4. Robust Joint Adversarial Optimization

Noise injection into the BiGRU layer followed by the production of reconstructed samples allows the model to learn generalization capabilities against perturbations during training and to dynamically adjust the perturbation magnitude through backpropagation. By superimposing isotropic Gaussian noise, the perturbation direction introduces stochastic deviations from the gradient sign dominance. This noise term compels the model to learn generalization capabilities against neighborhood perturbations during training while preserving the explicit attack directionality of adversarial perturbations to improve robust performance. The perturbation generation formula is expressed as follows:

$$z=ϵ·sign\left({\mathsf{\nabla }}_{x}L\left(f_{\theta }\left(x\right),y\right)\right)+N(0,{\sigma }^2)$$

(5)

The reconstruction model partially learns the distribution pattern of the potential representation of the data. The reconstruction of the sample is to use the game process to set the objective function as follows:

$$\begin{gathered} f^1\left(x\right)=E_{x_r~P_r\left(·\right)}\log F\left(x\right)+E_{z~P_g\left(·\right)}\log \left[1-F\left(R\left(z\right)\right)\right] \\ f\left(x\right)=mi{n}_{\theta }{max}_{\phi }\left\{f^1\left(x\right)\right\} \end{gathered}$$

(6)

where $P_r\left(·\right)$ represents the distribution of the real data, $P_g\left(·\right)$ indicates the low-dimensional spatial distribution of the noise, and $\theta \mathrm{a}\mathrm{n}\mathrm{d} \phi$ denote the reconstruction with the predicted parameters, with the encoder probabilistically selecting between reconstructing the noisy data and the real data to generate the reconstruction of the sample model for robust joint adversarial optimization, as shown in Figure 6.

In the anomaly detection tasks, the proposed model leverages adversarial networks, a reconstruction-based model that captures the data features of the entire time series to add noise for reconstructing the samples, as well as a prediction-based model for predicting future values, improving the robustness of the model through adversarial training of the reconstructed model and the predicted model. The parameters of the two models are updated simultaneously during the training process. The loss function is defined as follows:

$$Loss=\begin{array}{c}\lambda {Loss}_{For}+\left(1-\lambda \right)\end{array}{Loss}_{Rec}$$

(7)

where $\lambda$ denotes the probabilistic hyperparameters of the reconstruction loss and the prediction loss. The prediction model is partially passed through the fully connected layer after BiGRU and stacked hidden layers for feature extraction. The root mean square error of the final fully connected output layer is defined as the loss function:

$${Loss}_{For}=\sqrt{{\Sigma }_{\dot{i}=1}^n(f_F\left(x_{n+1,i}\right)-f_F\left({\widehat{x}}_{n+1,i}\right){)}^2}$$

(8)

where $f_F\left(x_{n+1,i}\right)$ denotes the value of the prediction model at the next timestamp of the BiGRU output layer for the current input.

3.3.5. Anomaly Detection

The error between the predicted value and the actual value obtained after training the model is defined as an anomaly score using (9). The threshold is based on the 99th percentile of the test set anomaly scores. If the anomaly score exceeds the threshold value, the timestamp is identified as an anomaly. After anomaly detection, the features that cause the timestamp anomaly are determined to speed up the troubleshooting rate. Then, the detected anomalies $x_t$ are ranked according to ${\widehat{x}}_t^i$, and the features of the $i$ dimension are finally selected to rank the top $k$ features as the root cause of the anomaly to achieve an explanation of the anomaly.

$$score={\Sigma }_{i=1}^m\sqrt{{(x}_{n+1,i}-{\widehat{x}}_{n+1,i}{)}^2}$$

(9)

The proposed PGAT-BiGRU-NRA is implemented as Algorithm 1.

$\mathbf{A}\mathbf{l}\mathbf{g}\mathbf{o}\mathbf{r}\mathbf{i}\mathbf{t}\mathbf{h}\mathbf{m} \mathbf{1} \mathrm{P}\mathrm{G}\mathrm{A}\mathrm{T}-\mathrm{B}\mathrm{i}\mathrm{G}\mathrm{R}\mathrm{U}-\mathrm{N}\mathrm{R}\mathrm{A} \mathrm{f}\mathrm{o}\mathrm{r} \mathrm{M}\mathrm{T}\mathrm{D}\mathrm{S} \mathrm{a}\mathrm{n}\mathrm{o}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{y} \mathrm{d}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$

$\mathbf{Input}: \mathrm{A} \mathrm{data} \mathrm{matrix} \mathrm{X}, \mathrm{Queries} \mathrm{labels} \mathrm{matrix} \mathrm{Y}, \mathrm{the} \mathrm{sliding} \mathrm{window} \mathrm{size} \mathrm{w},$           $\mathrm{the} \mathrm{sliding} \mathrm{step} \mathrm{is} \mathrm{l}, \mathrm{Iteration} \mathrm{steps} \mathrm{is}\mathrm{epochs};$ $\mathbf{Output}: \mathrm{Queries}/\mathrm{Classified} \mathrm{labels} \widehat{\mathrm{y}}=\{\widehat{{\mathrm{y}}_1},\widehat{{\mathrm{y}}_2},...,\widehat{{\mathrm{y}}_{\mathrm{t}}}\};$ 1. $\mathit{T}\mathbf{=}\mathbf{[}\mathbf{1}\mathbf{,}\mathbf{2}\mathbf{,}\mathbf{\dots }\mathbf{,}\mathit{t}\mathbf{]};//T\mathrm{h}e lengt\mathrm{h} of t\mathrm{h}e time series;$ 2. $\mathit{N}\mathbf{=}\mathbf{[}\mathbf{1}\mathbf{,}\mathbf{2}\mathbf{,}\mathbf{\dots }\mathbf{,}\mathit{n}\mathbf{]};//Dimensionality of time series;$ 3. $\mathit{I}\mathit{n}\mathit{i}\mathit{t}\mathit{i}\mathit{a}\mathit{l}\mathit{i}\mathit{z}\mathit{e} \mathit{t}\mathbf{h}\mathit{e} \mathit{p}\mathit{r}\mathit{o}\mathit{p}\mathit{o}\mathit{s}\mathit{e}\mathit{d} \mathit{n}\mathit{e}\mathit{t}\mathit{w}\mathit{o}\mathit{r}\mathit{k} \mathit{m}\mathit{o}\mathit{d}\mathit{e}\mathit{l}.$ 4. $\mathit{f}\mathit{o}\mathit{r} (i=0;i<t-w;i+=l) \mathit{d}\mathit{o}$ 5.    $X_1\leftarrow 1D-CNN;//feature extraction$ 6.    $X_2\leftarrow Time\_fature\_GAT\left(X_1\right);$ 7.    $X_3\leftarrow Spac{e}_{f}atur{e}_{G}AT\left(X_1\right);$ 8.    $X_4\leftarrow BiGRU(X_1,X_2,X_3);$ 9.    $Noise\leftarrow 1/\sigma \sqrt{2\pi }·exp\left(-{\left(X_1-\mu \right)}^2/2{\sigma }^2\right);$ 10.   $Recon\leftarrow Refactoring(X_4,Noise);$ 11.   $Forecasting\leftarrow Adversarial\_training\_Prediction(X_4,Recon);$ 12. $\mathit{e}\mathit{n}\mathit{d} \mathit{f}\mathit{o}\mathit{r}$ 13. $\mathit{f}\mathit{o}\mathit{r} (epoc\mathrm{h}\leftarrow 1 to epoc\mathrm{h}s) \mathit{d}\mathit{o}$ 14.   $Forecasting\leftarrow model(train\_set);$ 15.   $optimizer\leftarrow Adam(model.parameters,lr\leftarrow 0.01);$ 16.  $\mathit{i}\mathit{f} (Forecasting is not fully data\_leng\mathrm{h}\wedge \left|x_i\right|<\sqrt{T}) \mathit{t}\mathit{h}\mathit{e}\mathit{n}$  17.     $loss =\sqrt{{\Sigma }_{\dot{i}=1}^n(f_F\left(x_{n+1,i}\right)-f_F\left({\widehat{x}}_{n+1,i}\right){)}^2};//Classify loss$ 18.     $Use loss backwark and optimizer step updating model;$ 19.   $\mathit{e}\mathit{n}\mathit{d} \mathit{i}\mathit{f}$ 20.  $\mathit{e}\mathit{n}\mathit{d} \mathit{f}\mathit{o}\mathit{r}$ 21.  ${\widehat{x}}_{n+1,i}\leftarrow model(test\_set);$ 22. $\mathit{w}\mathit{h}\mathit{i}\mathit{l}\mathit{e} (\left|\widehat{y}\right| is not fully labels) \mathit{d}\mathit{o}$ 23.    $score ={\Sigma }_{i=1}^m\sqrt{{(x}_{n+1,i}-{\widehat{x}}_{n+1,i}{)}^2};$ 24.    $\mathit{i}\mathit{f} (score>T\mathrm{h}res\mathrm{h}old value) \mathit{t}\mathit{h}\mathit{e}\mathit{n}$ 25.       $Classify \widehat{y_i}=1;$ 26.    $\mathit{e}\mathit{n}\mathit{d} \mathit{i}\mathit{f}$ 27.     $\mathit{i}\mathit{f} (score<=T\mathrm{h}res\mathrm{h}old value) \mathit{t}\mathit{h}\mathit{e}\mathit{n}$ 28.       $Classify \widehat{y_i}=1;$ 29.    $\mathit{e}\mathit{n}\mathit{d} \mathit{i}\mathit{f}$ 30. $\mathit{e}\mathit{n}\mathit{d} \mathit{w}\mathit{h}\mathit{i}\mathit{l}\mathit{e}$ 31. $\mathit{r}\mathit{e}\mathit{t}\mathit{u}\mathit{r}\mathit{n} \widehat{\mathit{y}}=\left\{\widehat{{\mathit{y}}_1},\widehat{{\mathit{y}}_2},...,\widehat{{\mathit{y}}_{\mathit{t}}}\right\};$

4. Experimental Results and Analysis

4.1. Description of Datasets

Experiments are conducted using three public datasets and datasets collected by laboratory equipment. SMD collects real-time data for a period of 5 weeks on 28 different cloud platform servers, with each machine containing 38 dimensions of data (each dimension is a metric of the machine). SMAP and MSL, respectively, are soil moisture active–passive satellite and Mars science laboratory rover datasets provided by NASA. The experimental equipment is a gearbox platform, and the vibration data of normal and abnormal states are collected and fused as the anomaly detection dataset. To verify the ability of the proposed model to better distinguish between normal fluctuations and anomalies, as well as to assess its migration performance for MTSD, comparative tests are conducted on all the machines in the three datasets, and its efficiency is verified on the laboratory equipment dataset. The statistical characteristics of the dataset are shown in

Table 2. Statistical characterization of the datasets.

Datasets	SMD	SMAP	MSL	platform
Features	38	25	55	4
Training size	23,696	135,183	58,317	500,240
Testing size	23,696	427,617	73,729	500,131
Abnormal Rate (%)	4.16	13.13	10.27	15.3

.

4.2. Experimental Setup

4.2.1. Evaluation Metrics

Precision $P$, recall $R$, $F1$ scores, and $AUC$ scores are metrics to evaluate the performance of the model against the baseline model. The analysis reveals that most of the state-of-the-art methods regarding $AUC$ scores have been empirically more than 0.97, including the proposed model. Thus, $P$, $R$, and $F1$ scores are chosen as the evaluation indexes, and they are calculated as follows:

$$\begin{gathered} P=TP/TP+FP \\ R=\mathrm{T}\mathrm{P}/TP+FN \\ F1=2\times P·R/P+R \end{gathered}$$

(10)

where $TP$ denotes a true positive example predicted as positive and actually positive, FP indicates a false positive example predicted as positive and actually negative, and $FN$ symbolizes a false negative example predicted as negative and actually positive. To obtain the best $F1$ score, it is searched by enumerating all possible thresholds denoted as $best-F1$.

4.2.2. Comparative Methods

In this paper, the proposed method is compared with eight classical baseline models, which are described as follows:

(1)

OC-SVM [33] maps the samples through the kernel function to a higher dimensional space to delineate normal and abnormal boundaries;

(2)

LOF [34] detects anomalies using the local density deviation of the target data relative to the neighborhood;

(3)

BeatGAN [35] is based on an adversarial self-encoder and adds a discriminator to improve the formality of the reconstructed samples;

(4)

CAE-AD [28] enhances data feature representation using a self-encoder combined with contextual information for comparative learning;

(5)

IForest [36] isolates anomalies in the data by modeling the integration of randomly selected features on the division of observations;

(6)

OmniAnomaly [29] considers time-dependent, potentially spatially stochastic modeling to capture complex data distributions;

(7)

DeepSVDD [37] employs a co-training network to learn single-classified target data features for spherical mapping to normal data;

(8)

MAD-GAN [38] is based on recurrent neural networks to capture multivariate temporal distribution features while considering discriminative and generative loss detection anomalies.

4.2.3. Experimental Realization Details

In this paper, PGAT-BiGRU-NRA is implemented with GPU. A 1D-CNN with the same sliding window size w set to 100, a sliding step size l set to 8, and kernel size 7 is used, and the raw training data are divided according to an 8:2 ratio. The amount of neuron data h for BiGRU is set at 150, and the number of layers is set to 1. To train the model, the epoch, batch size, learning rate, and dropout are set to 30, 256, 0.001, and 0.4, respectively, and noise intensity α is set to 0.1. Training is performed with the Adam Optimizer.

4.3. Experimental Results

The performance and explanatory power of PGAT-BiGRU-NRA are tested using three public datasets. The experiment proves that the proposed method outperforms eight typical baseline models. Finally, the proposed model is used for anomaly detection in laboratory equipment and random noise is added to the dataset to prove the effectiveness and robustness of the method.

4.3.1. Analysis of Experimental Results

Table 3. Performance comparison between the proposed model and the baseline model.

Method	SMD			SMAP			MSL
	P	R	F1	P	R	F1	P	R	F1
OC-SVM	0.432	0.362	0.415	0.452	0.236	0.761	0.524	0.423	0.520
LOF	0.600	0.761	0.671	0.435	0.999	0.607	0.854	0.925	0.886
BeatGAN	0.694	0.712	0.806	0.832	0.926	0.724	0.903	0.853	0.921
CAE-AD	0.639	0.626	0.741	0.821	0.812	0.746	0.756	0.645	0.861
IForest	0.602	0.505	0.549	0.302	0.999	0.465	0.668	0.931	0.775
OmniAnomaly	0.701	0.731	0.696	0.758	0.976	0.853	0.883	0.900	0.901
DeepSVDD	0.778	0.334	0.479	0.456	0.276	0.376	0.981	0.742	0.837
MAD-GAN	0.731	0.409	0.527	0.608	0.999	0.765	0.935	0.944	0.940
PGAT-BiGRU-NRA	0.764	0.883	0.887	0.980	0.997	0.988	0.905	0.982	0.942

shows that the anomaly detection in the SMD dataset performs poorly in the baseline model. By analyzing the dataset, there are 28 machines in SMD, among which the raw data collected by machines 1–3, 1–8, 2–6, and 3–5 have lower detection accuracy due to the scarcity of anomaly samples and obvious noise interference, which is not enough to train the anomaly detection model and inevitably misclassify normal fluctuations as anomalies. On the other hand, the SMAP dataset has a score of 0.988 because the distribution of anomalies is concentrated, the correlation is stable, and the model is able to effectively capture the characteristic patterns of the data and counteract the noise interference.

The proposed model also shows excellent performance in SMD after adding the noise countermeasure module. Although there is a trade-off in comparing the other baseline models P and R, as shown in Figure 7, the visualization shows that the F1-score of the proposed model outperforms the baseline model in all aspects, which proves that the proposed model has higher robustness and is especially suitable for the complex noise and dynamic anomaly detection needs in industrial scenarios.

The baseline method exhibits different performance across different data. OC-SVM, a lightweight model, delineates anomaly boundaries through kernel function mapping, showing good performance with a small number of anomalous samples but struggles with large-scale datasets and noisy disturbances in the training data. LOF operates based on a clustering model that clusters the target data and performs density estimation to identify anomalies. However, it identifies anomalies when the data points deviate significantly from most of the data. IForest defines anomalies as sparse points that stray from the high-density group, making the anomalous data close to the root while the normal data are far from it by tree modeling. Due to the modeling performance of LOF and IForest, they show superior results on data with low outlier density. DeepSVDD models effectively distinguish anomalies in the high-dimensional space by learning normal category samples, which to some extent can deal with noise in the data with a certain degree of robustness; however, they perform poorly when there is a large difference between the distribution of normal data and anomalous data. BeatGAN, CAE-AD, OmniAnomaly, and MAD-GAN perform relatively well in models based on deep learning, where BeatGAN introduces a discriminator in the encoder model to discriminate the truthfulness of the reconstructed samples and avoid overfitting the model due to reconstruction. CAE-AD introduces comparative learning for nonlinear data representations and constructs clear boundaries for normal data to deal with complex distributions. OmniAnomaly uses a multi-scale attention mechanism to capture anomalies in a time series and emphasizes the importance of global context to better characterize anomaly patterns across the time series. MAD-GAN utilizes a generative adversarial network idea to set up generators and discriminators to learn data distributions through a gaming process, improving the robustness of the model.

In summary, considering the noise and anomalies that can exist in time series data, paying full attention to the spatio-temporal characteristics of the data, and extracting the contextual information to map the overall trend of the data are essential for improving the data processing robustness and the anomaly detection performance in MTSD, which is precisely the reason for the better performance of the proposed model.

4.3.2. Sensitivity Experiments on PGAT-BiGRU-NRA Parameters

Experiments are conducted on the dataset to explore the effect of each parameter on the performance of the model, and the experimental results are displayed in Figure 8. The hyperparameters of the model include the size of the sliding window $w$, the potential spatial dimensionality $h$, the negative slope $\alpha$, and the intensity of the reconstruction noise $\gamma$. The model is then used as an example of a model with a sliding window.

Figure 8a shows that in the SMD dataset, the larger the window size, the slightly lower the F1-score. The reason for the analysis may be that the anomalous dependence of the SMD dataset on the short-term contextual features of the time series, where the temporal dependency is prominent in contributing to the results, does not show a significant advantage over the other datasets. Therefore, considering the average results, the default setting of $w=100$ is used in the setup, and in the actual application, it is adjusted appropriately considering the periodic characteristics of MTSD.

The latent space dimension $h=\{50, 100, 150, 200, 250\}$ retains the multidimensional sample information within the latent space. The larger the $h,$ the richer the information that can be retained for better reconstruction. However, too large $h$ leads to overfitting. Figure 8b depicts that it is not sensitive to h on the SMD dataset and SMAP dataset, and it scores lower on the MSL dataset with $h$ = 50, which may be due to the high data complexity of the MSL dataset and complex correlations between multidimensional data. Thus, $h=150$ is chosen as the default setting based on the average.

The negative slope of the graph attention structure $\alpha =\{0,0.2,0.6,0.8,1\}$ introduces nonlinear properties that enable the network to learn more complex node relationships. However, a slope that is too large negatively leads to gradient explosion and reduced training stability. Figure 8c illustrates that the reason for lower scores on the MSL dataset when $\alpha$ is 0 may be that the model is not sufficiently extracting the nonlinear properties between the data. However, the model is unstable when $\alpha$ is greater than 0.2. Thus, the average $\alpha =0.2$ is selected as the default setting.

The reconstruction noise intensity is set to $\gamma =\{0.01, 0.05,0.1,0.5,1\}$. When performing adversarial training of noise reconstruction samples, a smaller $\mathsf{\gamma }$ implies less noise feature component learned by the model. Consequently, the processing of test samples with added random noise will misclassify the noise interference as anomalies, leading to a decrease in the model precision. However, a larger $\mathsf{\gamma }$ may lead to learning complex data features, which can decrease the distance between anomalies and anomaly scores of the normal data, resulting in a decrease in the anomaly recognition ability and a reduction in the recall rate. As shown in Figure 8d, $\gamma =0.1$ is defined as the default setting performing best in the results averaged across the datasets.

4.3.3. Ablation Experiment

The validity of the key modules in the PGAT-BiGRU-NRA model is investigated by ablation experiments, and four variant models of the PGAT-BiGRU-NRA model are constructed, including deletion of Time-GAT, deletion of Space-GAT, BiGRU-NRA, PGAT-NRA, and PGAT-BiGRU.

Table 4. Results of the proposed model and its ablated version model.

Method	SMD			SMAP			MSL
	P	R	F1	P	R	F1	P	R	F1
Delete Time-GAT	0.763	0.889	0.876	0.953	0.972	0.956	0.916	0.947	0.932
Delete Space-GAT	0.759	0.872	0.865	0.983	0.976	0.961	0.909	0.953	0.919
BiGRU-NRA	0.696	0.743	0.739	0.937	0.944	0.963	0.856	0.966	0.908
PGAT-NRA	0.732	0.865	0.884	0.974	0.982	0.961	0.879	0.952	0.914
PGAT-BiGRU	0.702	0.814	0.793	0.936	0.892	0.829	0.883	0.894	0.889
PGAT-BiGRU-NRA	0.764	0.883	0.887	0.980	0.997	0.988	0.905	0.982	0.942

compares the five variant models with the proposed model, which shows that the proposed model obtains the highest average scores on all three metrics and performs best.

The analysis of the results shows that the performance of the proposed model decreases when the key modules of the proposed model are removed or replaced, and the best results are obtained by the model that incorporates the three proposed main modules. Analyzing the results obtained from the models of the single-graph attention module and no-graph attention module, it can be concluded that MTSD show better results in fusion processing by fully extracting time-dependent features and spatially relevant features and retaining the original characteristics of the data, proving that the parallel graph attention network module is crucial for data fusion processing. It can be concluded from analyzing the results of the replaced BiGRU module that ensuring important information is not lost during the long-term propagation training process is essential. However, the GRU unit can consider the previous data to hide the features. The gating structure of BiGRU can take into account both historical and future data to retain the data features more comprehensively, allowing the module to ensure that the data features are not lost in data processing to improve model performance. The reconstruction module increases the gap between model anomalies and normal data for anomaly scores by reconstructing noisy samples for adversarial training to improve the robust performance of data processing. By analyzing the results of removing the PGAT-BiGRU model and using the model in test samples with added random noise, the lack of model robustness leads to a sharp decrease in performance. The results of the ablation experiments highlight the importance of each module of the proposed method.

4.4. Model Performance Analysis

Experiments are conducted on the dataset collected by the testbed to verify the effectiveness of the proposed model. In order to verify the robust performance of the model, random noise interference, excluding anomalies, was added to the sample data. The precision of the experimental results is 87.6%, which shows that the model achieves better results. To show the results of anomaly detection more intuitively, the anomaly scores and thresholds are visualized with anomaly labels on some of the timestamps of the dataset, as shown in Figure 9. In Figure 9a, the red line indicates the anomaly score, the black line represents the threshold value, the yellow line indicates the anomaly obtained from the model prediction, and the blue line shows the true anomaly value. When the score exceeds the threshold value, it is an anomaly. The values that are wrongly predicted are analyzed in the purple boxes. As shown in Figure 9b, the purple box indicates misjudged anomalies. By analyzing the reason for the error, we see that there are normal fluctuations in the real data in this part, and the sensitivity of the reconstructed part to the fluctuations makes it misjudged as an anomaly. However, from the overall point of view, the model achieves satisfactory results.

5. Conclusions

This paper proposed the PGAT-BiGRU-NRA model, which considered the susceptibility of the existing model to noise and added noise-counteracting reconstruction samples to improve the robust performance of the model. We will conduct experiments on more real datasets to verify the robustness of the model in more complex situations and to realize more complicated tasks, such as anomalous aliasing, to provide more powerful robustness to MTSD processing. However, deep learning models inherently have complex computational degree issues, but the parallel architecture of the proposed model provides a technical basis for distributed deployment at the edge. However, the robustness of the model still needs to be further improved in the face of extreme perturbations. Future work will focus on dynamic noise modeling and adaptive threshold optimization to improve its scalability in smart manufacturing systems.