#
Autoencoders for Anomaly Detection in an Industrial Multivariate Time Series Dataset^{ †}

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Theoretical Background

#### 2.1. Long Short-Term Memory Recurrent Neural Networks

#### 2.2. One-Dimensional Convolutional Neural Networks

#### 2.3. Anomaly Detection with Autoencoders

## 3. Implementation on Industrial Dataset

#### 3.1. Description of the Dataset

#### 3.2. Data Preprocessing

#### 3.3. Synthetic Data for Anomaly Detection

**Duration**. Point anomalies seem to be of minor importance, at least in the beginning of testing. Hence, anomalies of finite duration were chosen randomly with a minimum of 10 time-steps.**Magnitude**. Both positive and negative deviations were induced, with addition or subtraction of random numbers with magnitude: [1.6 bar, 2.6 bar] for pressure, [1 dB, 4 dB] for noise and [5, 20]% of the maximum speed value of the curve for speed.**Location**. Location of the anomalies was chosen randomly between the timesteps that the test is performed (operational values > 0).

#### 3.4. Description of the AE Architectures

**ANN-based AE.**The first implementation was an ANN-based AE. It consisted of 10 fully connected (dense) layers, a layer that flattens the matrix and a reshape layer in the decoding operation that transforms the vector back to a matrix. In particular, the architecture is as follows: Input (128-64-32)-flatten-1024-latent space (128)-(1024-6432)-reshape (64-128)-output. For all hidden layers ReLU activation function was used.

**LSTM-based AE.**The LSTM-based AE is a shallower network compared to the previous one. It consists of 4 LSTM layers, a layer that repeats the vector in the corresponding timesteps and a skip connection layer. Hyperbolic tangent and sigmoid activation functions were used in LSTM units for the input and the recurrent state respectively. Skip-connections were employed in the stacked LSTM layers, following the practice of [16,17,18], to boost model’s reconstruction performance. The architecture is shown in Figure 5.

**CNN-based AE.**The CNN-based AE consists of convolutions, transpose convolutions and pooling operation. Convolutions and pooling operation were part of the encoding process while transpose convolutions [19] were used to perform up-sampling during decoding. Convolutional and transpose convolutional layers employed the “same-padding” method, so that the size of the output matched the size of input. ReLU was used as the activation function. The complete architecture is presented in Figure 6.

#### 3.5. Results

## 4. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Chandola, V.; Banerjee, A.; Kumar, V. Anomaly Detection: A Survey. ACM Comput. Surv. (CSUR)
**2009**, 41, 1–58. [Google Scholar] [CrossRef] - Gupta, M.; Gao, J.; Aggarwal, C.C.; Han, J. Outlier Detection for Temporal Data: A Survey. IEEE Trans. Knowl. Data Eng.
**2014**, 26, 2250–2267. [Google Scholar] [CrossRef] - Breunig, M.; Kriegel, H.-P.; Ng, R.; Sander, J. LOF: Identifying Density-Based Local Outliers. ACM SIGMOD Rec.
**2000**, 29, 93–104. [Google Scholar] [CrossRef] - Lei, D.; Zhu, Q.; Chen, J.; Lin, H.; Yang, P. Automatic K-Means Clustering Algorithm for Outlier Detection. In Proceedings of the International Conference on Information Engineering and Applications, Chongqing, China, 26–28 October 2012; Zhu, R., Ma, Y., Eds.; Springer: London, UK, 2012; pp. 363–372. [Google Scholar]
- Kozitsin, V.; Katser, I.; Lakontsev, D. Online Forecasting and Anomaly Detection Based on the ARIMA Model. Appl. Sci.
**2021**, 11, 3194. [Google Scholar] [CrossRef] - Pang, G.; Shen, C.; Cao, L.; Hengel, A.V.D. Deep Learning for Anomaly Detection: A Review. ACM Comput. Surv. (CSUR)
**2021**, 54, 1–38. [Google Scholar] [CrossRef] - Malhotra, P.; Vig, L.; Shroff, G.; Agarwal, P. Long Short Term Memory Networks for Anomaly Detection in Time Series. Proceedings
**2015**, 89, 89–94. [Google Scholar] - Ding, S.; Morozov, A.; Vock, S.; Weyrich, M.; Janschek, K. Model-Based Error Detection for Industrial Automation Systems Using LSTM Networks. In Proceedings of the 7th International Symposium, IMBSA 2020, Lisbon, Portugal, 14–16 September 2020; Zeller, M., Höfig, K., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 212–226. [Google Scholar]
- Carletti, M.; Masiero, C.; Beghi, A.; Susto, G.A. A Deep Learning Approach for Anomaly Detection with Industrial Time Series Data: A Refrigerators Manufacturing Case Study. Procedia Manuf.
**2019**, 38, 233–240. [Google Scholar] [CrossRef] - Malhotra, P.; Ramakrishnan, A.; Anand, G.; Vig, L.; Agarwal, P.; Shroff, G. LSTM-Based Encoder-Decoder for Multi-Sensor Anomaly Detection. arXiv
**2016**, arXiv:1607.00148. [Google Scholar] - Hinton, G.E.; Salakhutdinov, R.R. Reducing the Dimensionality of Data with Neural Networks. Science
**2006**, 313, 504–507. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Provotar, O.I.; Linder, Y.M.; Veres, M.M. Unsupervised Anomaly Detection in Time Series Using LSTM-Based Autoencoders. In Proceedings of the 2019 IEEE International Conference on Advanced Trends in Information Theory (ATIT), Kyiv, Ukraine, 18–20 December 2019; pp. 513–517. [Google Scholar]
- Yu, J.; Zhou, X. One-Dimensional Residual Convolutional Autoencoder Based Feature Learning for Gearbox Fault Diagnosis. IEEE Trans. Ind. Inform.
**2020**, 16, 6347–6358. [Google Scholar] [CrossRef] - Canizo, M.; Triguero, I.; Conde, A.; Onieva, E. Multi-Head CNN–RNN for Multi-Time Series Anomaly Detection: An Industrial Case Study. Neurocomputing
**2019**, 363, 246–260. [Google Scholar] [CrossRef] - Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet Classification with Deep Convolutional Neural Networks. Adv. Neural Inf. Processing Syst.
**2012**, 25, 1097–1105. [Google Scholar] [CrossRef] - He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Rasmus, A.; Berglund, M.; Honkala, M.; Valpola, H.; Raiko, T. Semi-Supervised Learning with Ladder Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015. [Google Scholar]
- Zhang, S.; Qiu, T. Semi-Supervised LSTM Ladder Autoencoder for Chemical Process Fault Diagnosis and Localization. Chem. Eng. Sci.
**2022**, 251, 117467. [Google Scholar] [CrossRef] - Zeiler, M.D.; Krishnan, D.; Taylor, G.W.; Fergus, R. Deconvolutional Networks. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 2528–2535. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv
**2014**, arXiv:1412.6980. [Google Scholar] - Shahin, M.A.; Maier, H.R.; Jaksa, M.B. Data Division for Developing Neural Networks Applied to Geotechnical Engineering. J. Comput. Civ. Eng.
**2004**, 18, 105–114. [Google Scholar] [CrossRef]

**Figure 2.**Indicative diagram from a HPU quality test. Speed, pressure and noise are captured for both translational directions.

**Figure 8.**The reconstruction of the proposed model for a pair of normal and the corresponding with oscillation anomalies sample.

**Figure 9.**The reconstruction of the proposed model for a pair of normal and the corresponding with dips/rises anomalies sample.

Dataset | Population |
---|---|

Training | 12.960 × 201 × 3 |

Testing (w/o anomalies) | 1.440 × 201 × 3 |

Testing (with anomalies) | 1.440 × 201 × 3 |

AE | Training Time (s) | Epochs | Total Parameters | Threshold | Accuracy | Precision | Recall |
---|---|---|---|---|---|---|---|

ANN-based | 81 | 122 | 13,465,155 | 0.0063 | 84.79 | 91.27 | 76.94 |

LSTM-based | 677 | 91 | 332,336 | 0.00048 | 91.38 | 96.05 | 86.31 |

CNN-based | 101 | 99 | 109,611 | 0.00514 | 94.09 | 97.10 | 90.90 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Tziolas, T.; Papageorgiou, K.; Theodosiou, T.; Papageorgiou, E.; Mastos, T.; Papadopoulos, A.
Autoencoders for Anomaly Detection in an Industrial Multivariate Time Series Dataset. *Eng. Proc.* **2022**, *18*, 23.
https://doi.org/10.3390/engproc2022018023

**AMA Style**

Tziolas T, Papageorgiou K, Theodosiou T, Papageorgiou E, Mastos T, Papadopoulos A.
Autoencoders for Anomaly Detection in an Industrial Multivariate Time Series Dataset. *Engineering Proceedings*. 2022; 18(1):23.
https://doi.org/10.3390/engproc2022018023

**Chicago/Turabian Style**

Tziolas, Theodoros, Konstantinos Papageorgiou, Theodosios Theodosiou, Elpiniki Papageorgiou, Theofilos Mastos, and Angelos Papadopoulos.
2022. "Autoencoders for Anomaly Detection in an Industrial Multivariate Time Series Dataset" *Engineering Proceedings* 18, no. 1: 23.
https://doi.org/10.3390/engproc2022018023