# Fault Diagnosis via Neural Ordinary Differential Equations

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Graph Neural Ordinary Differential Equations

#### 2.1. ResNets Inspiration

#### 2.2. Training Using ODE Solvers

#### 2.3. External Inputs

## 3. Model Representation

#### 3.1. State-Space

#### 3.2. Neural ODE Networks

## 4. Fault Diagnosis via Neural ODE Applied to Nonlinear Benchmark System

#### 4.1. System Model

#### 4.2. Data-Set

#### 4.3. Actuator Faults

#### 4.3.1. Training

#### 4.3.2. Results

#### 4.4. Sensor Faults

#### 4.4.1. Training

#### 4.4.2. Results

#### 4.5. Process Faults

#### 4.5.1. Training

#### 4.5.2. Results

## 5. Visual Representation of the Network

## 6. Discussion

## 7. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ODE | Ordinary Differential Equations |

LMI | Linear Matrix Inequalities |

PCA | Principal Component Analysis |

SVM | Support Vector Machine |

ANN | Artifical Neural Network |

ResNet | Residual Networks |

MIMO | Multiple Inputs Multiple Outputs |

FDI | Fault Detection and Idetification |

## References

- Ding, S.X. Model-Based Fault Diagnosis Techniques, 2nd ed.; Springer: London, UK, 2013. [Google Scholar]
- Pérez-Zuñiga, G.; Rivas-Perez, R.; Sotomayor-Moriano, J.; Sánchez-Zurita, V. Fault Detection and Isolation System Based on Structural Analysis of an Industrial Seawater Reverse Osmosis Desalination Plant. Processes
**2020**, 8, 1100. [Google Scholar] [CrossRef] - Krysander, M.; Åslund, J.; Nyberg, M. An efficient algorithm for finding minimal overconstrained subsystems for model-based diagnosis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum.
**2008**, 38, 197–206. [Google Scholar] [CrossRef] - Pérez, C.G.; Travé-Massuyès, L.; Chanthery, E.; Sotomayor, J. Decentralized diagnosis in a spacecraft attitude determination and control system. J. Phys. Conf. Ser.
**2015**, 659, 1–12. [Google Scholar] [CrossRef] - Dai, X.; Gao, Z.; Breikin, T.; Wang, H. Disturbance attenuation in fault detection of gas turbine engines: A discrete robust observer design. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev.
**2009**, 39, 234–239. [Google Scholar] [CrossRef] - Gao, Z.; Cecati, C.; Ding, S.X. A survey of fault diagnosis and fault-tolerant techniques-part II: Fault diagnosis with knowledge-based and hybrid/active approaches. IEEE Trans. Ind. Electron.
**2015**, 62, 3768–3774. [Google Scholar] [CrossRef] [Green Version] - Karimi, H.R.; Zapateiro, M.; Luo, N. A linear matrix inequality approach to robust fault detection filter design of linear systems with mixed time-varying delays and nonlinear perturbations. J. Frankl. Inst.
**2010**, 347, 957–973. [Google Scholar] [CrossRef] [Green Version] - Liu, C.; Jiang, B.; Zhang, K. Sliding mode observer-based actuator fault detection for a class of linear uncertain systems. In Proceedings of the 2014 IEEE Chinese Guidance, Navigation and Control Conference, CGNCC 2014, Yantai, China, 8–10 August 2014; pp. 230–234. [Google Scholar] [CrossRef]
- Guo, F.; Ren, X.; Li, Z.; Han, C. Kalman filter based fault detection of dual motor systems. In Proceedings of the Chinese Control Conference, CCC 2017, Dalian, China, 26–28 July 2017; pp. 7133–7137. [Google Scholar] [CrossRef]
- Nguang, S.K.; Zhang, P.; Ding, S. Parity relation based fault estimation for nonlinear systems: An LMI approach. IFAC Proc. Vol. (IFAC-PapersOnline)
**2006**, 6, 366–371. [Google Scholar] [CrossRef] - Basri, H.M.; Lias, K.; Abidin, W.A.W.Z.; Tay, K.M.; Zen, H. Fault Detection Using Dynamic Parity Space Approach H. In Proceedings of the 2012 IEEE International PEOCO2012, Melaka, Malaysia, 6–7 June 2012. [Google Scholar]
- Pérez, C.G.; Chanthery, E.; Travé-massuyès, L.; Sotomayor, J. Fault-Driven Minimal Structurally Overdetermined Set in a Distributed Context To cite this version: HAL Id: Hal-01392572, In Proceedings of the 27th International Workshop on Principles of Diagnosis: DX-2016, Denver, CO, USA, 4–7 October 2016.
- Pérez-Zuniga, C.; Chantery, E.; Travé-Massuyes, L.; Sotomayor, J.; Artigues, C. Decentralized Diagnosis via Structural Analysis and Integer Programming. IFAC-PapersOnLine
**2018**, 51, 168–175. [Google Scholar] [CrossRef] - Garcia-Alvarez, D. Fault detection using Principal Component Analysis (PCA) in a Wastewater Treatment Plant (WWTP). In Proceedings of the 62-th International Student’s Scientific Conference, San Diego, CA, USA, 17–19 November 2009; p. 6. [Google Scholar]
- Wang, X.; Kruger, U.; Irwin, G.W.; McCullough, G.; McDowell, N. Nonlinear PCA with the local approach for diesel engine fault detection and diagnosis. IEEE Trans. Control Syst. Technol.
**2008**, 16, 122–129. [Google Scholar] [CrossRef] - Wang, X.; Liu, X.; Japkowicz, N.; Matwin, S. Ensemble of multiple kernel SVM classifiers. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer International Publishing: Cham, Switzerland, 2014; Volume 8436, pp. 239–250. [Google Scholar] [CrossRef]
- Mohd Amiruddin, A.A.A.; Zabiri, H.; Taqvi, S.A.A.; Tufa, L.D. Neural network applications in fault diagnosis and detection: An overview of implementations in engineering-related systems. Neural Comput. Appl.
**2020**, 32, 447–472. [Google Scholar] [CrossRef] - Hoang, D.T.; Kang, H.J. A survey on Deep Learning based bearing fault diagnosis. Neurocomputing
**2019**, 335, 327–335. [Google Scholar] [CrossRef] - Mandal, S.; Santhi, B.; Sridhar, S.; Vinolia, K.; Swaminathan, P. Nuclear Power Plant Thermocouple Sensor-Fault Detection and Classification Using Deep Learning and Generalized Likelihood Ratio Test. IEEE Trans. Nuclear Sci.
**2017**, 64, 1526–1534. [Google Scholar] [CrossRef] - Liang, T.; Wu, S.; Duan, W.; Zhang, R. Bearing fault diagnosis based on improved convolutional deep belief network. IOP Conf. Ser. J. Phys.
**2018**. [Google Scholar] [CrossRef] - Jia, F.; Lei, Y.; Lin, J.; Zhou, X.; Lu, N. Deep neural networks: A promising tool for fault characteristic mining and intelligent diagnosis of rotating machinery with massive data. Mech. Syst. Signal Process.
**2016**, 72–73, 303–315. [Google Scholar] [CrossRef] - Wei, Z.; Gaoliang, P.; Chuanhao, L. Bearings Fault Diagnosis Based on Convolutional Neural Networks with 2- D Representation of Vibration Signals as Input. MATEC Web Conf.
**2017**, 13001, 1–5. [Google Scholar] - Zhang, B.; Li, W.; Hao, J.; Li, X.L.; Zhang, M. Adversarial adaptive 1-D convolutional neural networks for bearing fault diagnosis under varying working condition. arXiv
**2018**, arXiv:1805.00778. [Google Scholar] - Guo, L.; Li, N.; Jia, F.; Lei, Y.; Lin, J. A recurrent neural network based health indicator for remaining useful life prediction of bearings. Neurocomputing
**2017**, 240, 98–109. [Google Scholar] [CrossRef] - Abed, W.; Sharma, S.; Sutton, R.; Motwani, A. A Robust Bearing Fault Detection and Diagnosis Technique for Brushless DC Motors Under Non-stationary Operating Conditions. J. Control Autom. Electr. Syst.
**2015**, 26, 241–254. [Google Scholar] [CrossRef] [Green Version] - Zhang, C.; Xu, L.; Li, X.; Wang, H. A Method of Fault Diagnosis for Rotary Equipment Based on Deep Learning. In Proceedings of the 2018 Prognostics and System Health Management Conference, PHM-Chongqing 2018, Chongqing, China, 26–28 October 2018; pp. 958–962. [Google Scholar] [CrossRef]
- Frank, S.; Heaney, M.; Jin, X.; Robertson, J.; Cheung, H.; Elmore, R.; Henze, G.P. Hybrid Model-Based and Data-Driven Fault Detection and Diagnostics for Commercial Buildings. In Proceedings of the ACEEE Summer Study on Energy Efficiency in Buildings, Pacific Grove, CA, USA, 21–26 August 2016; pp. 12.1–12.4. [Google Scholar]
- Jung, D.; Ng, K.Y.; Frisk, E.; Krysander, M. A combined diagnosis system design using model-based and data-driven methods. In Proceedings of the Conference on Control and Fault-Tolerant Systems, SysTol, Barcelona, Spain, 7–9 September 2016; pp. 177–182. [Google Scholar] [CrossRef]
- Jung, D.; Ng, K.Y.; Frisk, E.; Krysander, M. Combining model-based diagnosis and data-driven anomaly classifiers for fault isolation. Control Eng. Pract.
**2018**, 80, 146–156. [Google Scholar] [CrossRef] - Jung, D.; Sundstrom, C. A Combined Data-Driven and Model-Based Residual Selection Algorithm for Fault Detection and Isolation. IEEE Trans. Control Syst. Technol.
**2019**, 27, 616–630. [Google Scholar] [CrossRef] [Green Version] - Jiang, N.; Hu, X.; Li, N. Graphical temporal semi-supervised deep learning–based principal fault localization in wind turbine systems. Proc. Inst. Mech. Eng. Part I J. Syst. Control Eng.
**2020**, 234, 985–999. [Google Scholar] [CrossRef] - Li, D.; Wang, Y.; Wang, J.; Wang, C.; Duan, Y. Recent advances in sensor fault diagnosis: A review. Sens. Actuators A Phys.
**2020**, 309, 111990. [Google Scholar] [CrossRef] - Chen, R.T.Q.; Rubanova, Y.; Bettencourt, J.; Duvenaud, D. Neural Ordinary Differential Equations. arXiv
**2018**, arXiv:1806.07366. [Google Scholar] - He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar] [CrossRef] [Green Version]
- Deng, Z.; Nawhal, M.; Meng, L.; Mori, G. Continuous Graph Flow. arXiv
**2019**, arXiv:908.02436. [Google Scholar] - Greydanus, S.; Dzamba, M.; Yosinski, J. Hamiltonian Neural Networks. arXiv
**2019**, arXiv:1906.01563v3. [Google Scholar] - Tzen, B.; Raginsky, M. Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit. arXiv
**2019**, arXiv:1905.09883v2. [Google Scholar] - Poli, M.; Massaroli, S.; Park, J.; Yamashita, A.; Asama, H.; Park, J. Neural ordinary differential equations. In Proceedings of the Advances in Neural Information Processing Systems, Virtual, 6–12 December 2020; pp. 6571–6583. [Google Scholar]
- Ruthotto, L.; Haber, E. Deep Neural Networks Motivated by Partial Differential Equations. J. Math. Imaging Vis.
**2020**, 62, 352–364. [Google Scholar] [CrossRef] [Green Version] - Johansson, K.H. The quadruple-tank process: A multivariable laboratory process with an adjustable zero. IEEE Trans. Control Syst. Technol.
**2000**, 8, 456–465. [Google Scholar] [CrossRef] [Green Version] - Sotomayor-Moriano, J.; Pérez-Zúñiga, G.; Soto, M. A Virtual Laboratory Environment for Control Design of a Multivariable Process. IFAC PapersOnLine
**2019**, 52, 218–223. [Google Scholar] [CrossRef] - Sánchez-Zurita, V.; Pérez-Zúñiga, G.; Sotomayor-Moriano, J. Reconfigurable Model Predictive Control applied to the Quadruple Tank Process. In Proceedings of the 15th European Workshop on Advanced Control and Diagnosis, Palazzo Grassi, Bologna, 21–22 November 2019. [Google Scholar]
- Jager, G.; Zug, S.; Brade, T.; Dietrich, A.; Steup, C.; Moewes, C.; Cretu, A.M. Assessing neural networks for sensor fault detection. In Proceedings of the CIVEMSA 2014—2014 IEEE Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, Ottawa, ON, Canada, 5–7 May 2014; pp. 70–75. [Google Scholar] [CrossRef]
- Shao, H.; Jiang, H.; Zhao, H.; Wang, F. A novel deep autoencoder feature learning method for rotating machinery fault diagnosis. Mech. Syst. Signal Process.
**2017**, 95, 187–204. [Google Scholar] [CrossRef] - Wen, Q.; Sun, L.; Song, X.; Gao, J.; Wang, X.; Xu, H. Time Series Data Augmentation for Deep Learning: A Survey. arXiv
**2020**, arXiv:2002.12478. [Google Scholar]

**Figure 4.**Actuator faults in an extract of the data-set, shaded areas indicate that a fault has occurred, and signals are normalized.

**Figure 8.**Actuator faults detection, the faults are represented in the shaded areas, states are given in black for reference. Above: faults labeled in the data-set. Below: faults detected by the system.

**Figure 9.**Sensor faults detection, the faults are represented in the shaded areas, states are given in black for reference. Above: faults labeled in the data-set. Below: faults detected by the system.

**Figure 10.**Process faults detection, the faults are represented in the shaded areas, states are given in black for reference. Above: faults labeled in the data-set. Below: faults detected by the system.

Parameter | Units |
---|---|

Bottom area, ${A}_{i}$, for $i=1,2,3,4$ | ${\mathrm{cm}}^{2}$ |

Outlet pipe cross section, ${a}_{i}$, for $i=1,2$ | $\mathrm{cm}$ |

Outlet pipe cross section, ${a}_{i}$, for $i=3,4$ | $\mathrm{cm}$ |

Gravity constant, g | ${\mathrm{cm}}^{2}/\mathrm{s}$ |

Pump constants, ${k}_{i}$, for $i=1,2$ | ${\mathrm{cm}}^{3}/\left(\mathrm{V}\mathrm{s}\right)$ |

Main valve positions, ${\gamma}_{i}$ for $i=1,2$ | |

Heights 1 and 2, ${h}_{1o}$, ${h}_{2o}$ | $\mathrm{cm}$ |

Heights 3 and 4, ${h}_{3o}$, ${h}_{4o}$ | $\mathrm{cm}$ |

Inputs 1 and 2, ${u}_{1o}$, ${u}_{2o}$ | ${\mathrm{cm}}^{3}/\mathrm{s}$ |

Model | Accuracy | Weighted Accuracy |
---|---|---|

Suport Vector Machine | ||

(Polynomial Kernel) | 93.3% | 62.657% |

Artificial Neural Network | ||

(4 Fully connected Layers) | 80.9% | 74.8% |

Convolutional Neural Network | ||

(2 Convolutional Layers + 1 Fully connected Layer) | 77.5% | 55.8% |

Recurrent Neural Network | ||

(4 LSTM Layers + 1 Fully connected Layer) | 80.53% | 73.8% |

Neural ODE & Dense Network | ||

(Neural ODE + 2 Fully connected Layers) | 98.5% | 90% |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Enciso-Salas, L.; Pérez-Zuñiga, G.; Sotomayor-Moriano, J.
Fault Diagnosis via Neural Ordinary Differential Equations. *Appl. Sci.* **2021**, *11*, 3776.
https://doi.org/10.3390/app11093776

**AMA Style**

Enciso-Salas L, Pérez-Zuñiga G, Sotomayor-Moriano J.
Fault Diagnosis via Neural Ordinary Differential Equations. *Applied Sciences*. 2021; 11(9):3776.
https://doi.org/10.3390/app11093776

**Chicago/Turabian Style**

Enciso-Salas, Luis, Gustavo Pérez-Zuñiga, and Javier Sotomayor-Moriano.
2021. "Fault Diagnosis via Neural Ordinary Differential Equations" *Applied Sciences* 11, no. 9: 3776.
https://doi.org/10.3390/app11093776