# Multi-Time Resolution Ensemble LSTMs for Enhanced Feature Extraction in High-Rate Time Series

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Deep Learning Architecture

**x**while sensor measurements are being acquired. The algorithm, shown in Figure 1, consists of (1) a multi-rate sampler, shown in Figure 1 (left), and (2) an ensemble of j LSTM cells arranged in parallel, joined through an attention layer, to conduct the prediction $\widehat{x}$, shown in Figure 1 (right) as making a one-step ahead prediction at step k, ${\widehat{x}}_{k+1}$. Short-sequence LSTMs are used to accelerate computing. A particularity of the algorithm is that the multi-rate sampler is used on part of the acquired sensor measurements $\mathbf{x}=\left[\begin{array}{cccc}{x}_{1}& {x}_{2}& \cdots & {x}_{k}\end{array}\right]$ to represent unique dynamic characteristics. More precisely, it samples a different sequence for each LSTM i ($i=1,2,\cdots ,j$) using a different time delay ${\tau}_{i}$ embedded in a vector of dimension ${d}_{i}$ with the delay vector ${\mathbf{x}}_{k}^{i}$ written

**b**the bias vector associated with the gate in subscript, and ⊙ an element-wise multiplication. Both weights and biases are shared through all time steps. These LSTM cells use internal gating functions (Equations (3)–(8)) to augment memory capabilities. Three gates, consisting of the input gate $\mathbf{i}$, forget gate $\mathbf{f}$, and output gate $\mathbf{o}$, modulate the flow of information inside the cell by assigning a value in the range of $(0,1)$ to write the input to the internal memory $\mathbf{s}$ (${\mathbf{i}}_{m}\odot {\mathbf{g}}_{m}$ in Equation (7)), reset the memory (${\mathbf{f}}_{m}\odot {\mathbf{s}}_{m-1}$ in Equation (7)), or read from it (Equation (8)). Gate values close to zero are less relevant for prediction purposes than those with values close to one.

## 3. Multi-Rate Sampler

#### 3.1. Principal Component Analysis

#### 3.2. Construction of Input Space

#### 3.3. Feature Extractor Training

**X**constructed using associated delay vector characteristics ($\tau $ and d). After, an RNN with LSTM cells (as in Figure 2a), with a hidden state h of size equal to twice its corresponding embedding dimension d, is trained on the original training time series (i.e., pre-decomposition) using a standard sequential back-propagation scheme with the associated loss function

## 4. Validation Methodology

#### 4.1. Experimental Setup

#### 4.2. Performance Metrics

## 5. Implementation and Results

#### 5.1. Input Space Construction

#### 5.2. Time Series Prediction

## 6. Discussion

## 7. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ms | milliseconds |

HRSHM | High-Rate Structural Health Monitoring |

RNN | Recurrent Neural Network |

LSTM | Long Short-Term Memory |

PCA | Principal Component Analysis |

PC | Principal Component |

MHz | Megahertz |

kg${}_{n}$ | Kilo g${}_{n}$ (gravitational acceleration) |

MAE | Mean Absolute Error |

RMSE | Root Mean Square Error |

DTW | dynamic Time Warping |

LOI | Line Of Indifference |

VIO | Variable Input Observer |

GRU | Gated Recurrent Units |

## References

- Hong, J.; Laflamme, S.; Dodson, J. Study of input space for state estimation of high-rate dynamics. Struct. Control Health Monit.
**2018**, 25, e2159. [Google Scholar] [CrossRef] [Green Version] - Hong, J.; Laflamme, S.; Dodson, J.; Joyce, B. Introduction to State Estimation of High-Rate System Dynamics. Sensors
**2018**, 18, 217. [Google Scholar] [CrossRef] [Green Version] - Joyce, B.; Dodson, J.; Laflamme, S.; Hong, J. An experimental test bed for developing high-rate structural health monitoring methods. Shock Vib.
**2018**, 2018, 3827463. [Google Scholar] [CrossRef] - Downey, A.; Hong, J.; Dodson, J.; Carroll, M.; Scheppegrell, J. Millisecond model updating for structures experiencing unmodeled high-rate dynamic events. Mech. Syst. Signal Process.
**2020**, 138, 106551. [Google Scholar] [CrossRef] - Yan, J.; Laflamme, S.; Hong, J.; Dodson, J. Online Parameter Estimation under Non-Persistent Excitations for High-Rate Dynamic Systems. Mech. Syst. Signal Process. In Review.
- Vlachas, P.R.; Byeon, W.; Wan, Z.Y.; Sapsis, T.P.; Koumoutsakos, P. Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. Proc. R. Soc. A Math. Phys. Eng. Sci.
**2018**, 474, 20170844. [Google Scholar] [CrossRef] [Green Version] - Lim, B.; Zohren, S. Time Series Forecasting with Deep Learning: A Survey. arXiv
**2004**, arXiv:2004.13408. [Google Scholar] - Lara-Benítez, P.; Carranza-García, M.; Riquelme, J.C. An Experimental Review on Deep Learning Architectures for Time Series Forecasting. Int. J. Neural Syst.
**2021**, 31, 2130001. [Google Scholar] [CrossRef] - Sen, R.; Yu, H.F.; Dhillon, I. Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting. arXiv
**2019**, arXiv:1905.03806. [Google Scholar] - Arisdakessian, C.; Poirion, O.; Yunits, B.; Zhu, X.; Garmire, L.X. DeepImpute: An accurate, fast, and scalable deep neural network method to impute single-cell RNA-seq data. Genome Biol.
**2019**, 20, 1–4. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Fayyad, J.; Jaradat, M.A.; Gruyer, D.; Najjaran, H. Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review. Sensors
**2020**, 20, 4220. [Google Scholar] [CrossRef] [PubMed] - Kuutti, S.; Bowden, R.; Jin, Y.; Barber, P.; Fallah, S. A Survey of Deep Learning Applications to Autonomous Vehicle Control. IEEE Trans. Intell. Transp. Syst.
**2020**, 22, 1–22. [Google Scholar] [CrossRef] - Rengasamy, D.; Jafari, M.; Rothwell, B.; Chen, X.; Figueredo, G.P. Deep Learning with Dynamically Weighted Loss Function for Sensor-Based Prognostics and Health Management. Sensors
**2020**, 20, 723. [Google Scholar] [CrossRef] [Green Version] - Mondal, T.G.; Jahanshahi, M. Autonomous vision-based damage chronology for spatiotemporal condition assessment of civil infrastructure using unmanned aerial vehicle. Smart Struct. Syst.
**2020**, 25, 733–749. [Google Scholar] - Sajedi, S.O.; Liang, X. Uncertainty-assisted deep vision structural health monitoring. Comput.-Aided Civ. Infrastruct. Eng.
**2020**, 36, 126–142. [Google Scholar] [CrossRef] - Graves, A. Supervised Sequence Labelling. In Studies in Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2012; pp. 5–13. [Google Scholar] [CrossRef]
- Greff, K.; Srivastava, R.K.; Koutnik, J.; Steunebrink, B.R.; Schmidhuber, J. LSTM: A Search Space Odyssey. IEEE Trans. Neural Netw. Learn. Syst.
**2017**, 28, 2222–2232. [Google Scholar] [CrossRef] [Green Version] - Zhu, L.; Laptev, N. Deep and Confident Prediction for Time Series at Uber. In Proceedings of the 2017 IEEE International Conference on Data Mining Workshops (ICDMW), New Orleans, LA, USA, 18–21 November 2017. [Google Scholar] [CrossRef] [Green Version]
- Guen, V.L.; Thome, N. Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models. arXiv
**2019**, arXiv:1909.09020. [Google Scholar] - Cui, Z.; Ke, R.; Pu, Z.; Wang, Y. Stacked Bidirectional and Unidirectional LSTM Recurrent Neural Network for Forecasting Network-wide Traffic State with Missing Values. Transp. Res. Part C Emerg. Technol.
**2020**, 118, 102674. [Google Scholar] [CrossRef] - Vochozka, M.; Vrbka, J.; Suler, P. Bankruptcy or Success? The Effective Prediction of a Company’s Financial Development Using LSTM. Sustainability
**2020**, 12, 7529. [Google Scholar] [CrossRef] - Lai, G.; Chang, W.C.; Yang, Y.; Liu, H. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. arXiv
**2017**, arXiv:1703.07015v3. [Google Scholar] - Liu, Y.; Gong, C.; Yang, L.; Chen, Y. DSTP-RNN: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction. Expert Syst. Appl.
**2020**, 143, 113082. [Google Scholar] [CrossRef] - Gilpin, W. Deep reconstruction of strange attractors from time series. arXiv
**2002**, arXiv:2002.05909v3. [Google Scholar] - Hua, Y.; Zhao, Z.; Li, R.; Chen, X.; Liu, Z.; Zhang, H. Deep Learning with Long Short-Term Memory for Time Series Prediction. IEEE Commun. Mag.
**2019**, 57, 114–119. [Google Scholar] [CrossRef] [Green Version] - Yeo, K.; Melnyk, I. Deep learning algorithm for data-driven simulation of noisy dynamical system. J. Comput. Phys.
**2019**, 376, 1212–1231. [Google Scholar] [CrossRef] [Green Version] - Raissi, M.; Perdikaris, P.; Karniadakis, G. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys.
**2019**, 378, 686–707. [Google Scholar] [CrossRef] - Diligenti, M.; Roychowdhury, S.; Gori, M. Integrating Prior Knowledge into Deep Learning. In Proceedings of the 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), Cancun, Mexico, 18–21 December 2017. [Google Scholar] [CrossRef]
- Karpatne, A.; Watkins, W.; Read, J.; Kumar, V. Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling. arXiv
**2017**, arXiv:1710.11431v2. [Google Scholar] - Cully, A.; Clune, J.; Tarapore, D.; Mouret, J.B. Robots that can adapt like animals. Nature
**2015**, 521, 503–507. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Pfrommer, J.; Zimmerling, C.; Liu, J.; Kärger, L.; Henning, F.; Beyerer, J. Optimisation of manufacturing process parameters using deep neural networks as surrogate models. Procedia CIRP
**2018**, 72, 426–431. [Google Scholar] [CrossRef] - Hong, J.; Laflamme, S.; Cao, L.; Dodson, J.; Joyce, B. Variable input observer for nonstationary high-rate dynamic systems. Neural Comput. Appl.
**2018**, 32, 5015–5026. [Google Scholar] [CrossRef] [Green Version] - Takens, F. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980; Springer: Berlin/Heidelberg, Germany, 1981; pp. 366–381. [Google Scholar]
- Li, F.; Shirahama, K.; Nisar, M.A.; Huang, X.; Grzegorzek, M. Deep Transfer Learning for Time Series Data Based on Sensor Modality Classification. Sensors
**2020**, 20, 4271. [Google Scholar] [CrossRef] - Holger Kantz, T.S. Nonlinear Time Series Analysis; Cambridge University Press: Cambridge, UK, 2006. [Google Scholar]
- Stark, J. Delay Embeddings for Forced Systems. I. Deterministic Forcing. J. Nonlinear Sci.
**1999**, 9, 255–332. [Google Scholar] [CrossRef] - Caballero, V. On an Embedding Theorem. Acta Math. Hung.
**2000**, 88, 269–278. [Google Scholar] [CrossRef] - Stark, J.; Broomhead, D.; Davies, M.; Huke, J. Delay Embeddings for Forced Systems. II. Stochastic Forcing. J. Nonlinear Sci.
**2003**, 13, 519–577. [Google Scholar] [CrossRef] - Belghazi, M.I.; Baratin, A.; Rajeshwar, S.; Ozair, S.; Bengio, Y.; Courville, A.; Hjelm, D. Mutual information neural estimation. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 June 2018; pp. 531–540. [Google Scholar]
- Fraser, A.M.; Swinney, H.L. Independent coordinates for strange attractors from mutual information. Phys. Rev. A
**1986**, 33, 1134–1140. [Google Scholar] [CrossRef] [PubMed] - Pan, S.J.; Yang, Q. A Survey on Transfer Learning. IEEE Trans. Knowl. Data Eng.
**2010**, 22, 1345–1359. [Google Scholar] [CrossRef] - Müller, M. Dynamic Time Warping. In Information Retrieval for Music and Motion; Springer: Berlin/Heidelberg, Germany, 2007; pp. 69–84. [Google Scholar] [CrossRef]
- Wu, J.; Chen, X.Y.; Zhang, H.; Xiong, L.D.; Lei, H.; Deng, S.H. Hyperparameter optimization for machine learning models based on Bayesian optimization. J. Electron. Sci. Technol.
**2019**, 17, 26–40. [Google Scholar] - Perrone, V.; Shen, H.; Seeger, M.; Archambeau, C.; Jenatton, R. Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning. arXiv
**2019**, arXiv:1909.12552v1. [Google Scholar] - Kim, J.Y.; Cho, S.B. Evolutionary Optimization of Hyperparameters in Deep Learning Models. In Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand, 10–13 June 2019. [Google Scholar] [CrossRef]
- Martínez-Álvarez, F.; Asencio-Cortés, G.; Torres, J.F.; Gutiérrez-Avilés, D.; Melgar-García, L.; Pérez-Chacón, R.; Rubio-Escudero, C.; Riquelme, J.C.; Troncoso, A. Coronavirus Optimization Algorithm: A Bioinspired Metaheuristic Based on the COVID-19 Propagation Model. Big Data
**2020**, 8, 308–322. [Google Scholar] [CrossRef] [PubMed] - Chung, J.; Gulcehre, C.; Cho, K.; Bengio, Y. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv
**2014**, arXiv:1412.3555v1. [Google Scholar]

**Figure 4.**Drop tower experimental setup [32].

**Figure 5.**Deceleration time series obtained from five consecutive tests: (

**a**) TS${}_{1}$; and (

**b**) TS${}_{2}$.

**Figure 6.**Comparison of the source domain measurements and its reconstruction using the first five PCs.

**Figure 7.**Selection of input space hyper-parameters: (

**a**) PCA decompositions of the source domain measurements; (

**b**) selection of $\tau $ based on MI analysis for each PC decomposition; and (

**c**) selection of d based on percentage of FNNs for each PC decomposition.

**Figure 9.**Prediction performance for 14 steps ahead, “PCA inputs” versus “GS inputs”: (

**a**) prediction; and (

**b**) scaled typical feature extracted by the LSTMs using $\tau =11$ under both methodes.

**Figure 12.**(

**a**) Mean DTW over all LSTMs and five tests; (

**b**) mean DTW of individual LSTMs over all five tests—“GS inputs”; (

**c**) mean DTW of individual LSTMs over all five tests—“PCA inputs”.

GS Inputs | PCA Inputs | |||
---|---|---|---|---|

LSTM | $\mathit{\tau}$ | d | $\mathit{\tau}$ | d |

1 | 14 | 8 | 25 | 5 |

2 | 11 | 10 | 15 | 6 |

3 | 8 | 12 | 11 | 8 |

4 | 5 | 14 | 7 | 12 |

5 | 4 | 15 | 5 | 15 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Barzegar, V.; Laflamme, S.; Hu, C.; Dodson, J.
Multi-Time Resolution Ensemble LSTMs for Enhanced Feature Extraction in High-Rate Time Series. *Sensors* **2021**, *21*, 1954.
https://doi.org/10.3390/s21061954

**AMA Style**

Barzegar V, Laflamme S, Hu C, Dodson J.
Multi-Time Resolution Ensemble LSTMs for Enhanced Feature Extraction in High-Rate Time Series. *Sensors*. 2021; 21(6):1954.
https://doi.org/10.3390/s21061954

**Chicago/Turabian Style**

Barzegar, Vahid, Simon Laflamme, Chao Hu, and Jacob Dodson.
2021. "Multi-Time Resolution Ensemble LSTMs for Enhanced Feature Extraction in High-Rate Time Series" *Sensors* 21, no. 6: 1954.
https://doi.org/10.3390/s21061954