#
Improving the Predictive Power of Historical Consistent Neural Networks^{ †}

^{1}

^{2}

^{3}

^{4}

^{*}

^{†}

^{‡}

## Abstract

**:**

## 1. Introduction

## 2. Reminder of Recurrent Neural Networks for Dynamical Systems

## 3. Historical Consistent Neural Networks

## 4. Long-Term Memory Improvement Methods

#### 4.1. HCNN with Partial Teacher Forcing

#### 4.2. HCNN with Large Sparse State Transition Matrix

- The matrix–vector computation between A and $tanh\left({s}_{t}\right)$, which includes the addition of randomly generated (and learned) scalar values will likely blow up to infinity (∞).
- The superposition of additional information brought in by the large dimensionality of A could destroy the longer memory information acquired throughout.

#### 4.3. HCNN with LSTM Formulation

## 5. Experimental Setup

#### 5.1. The Rabinovich–Fabrikant System

#### 5.2. The Rossler System

#### 5.3. The Lorenz System

#### 5.4. Traning Strategies

#### 5.5. Evaluation Metrics

## 6. Results and Analysis

#### 6.1. On the Rabinovich-Fabrikant System

#### 6.2. On the Rossler System

#### 6.3. On the Lorenz System

## 7. Ensemble Computations

## 8. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Brunton, S.L.; Kutz, J.N. Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control; Cambridge University Press: Cambridge, UK, 2022. [Google Scholar]
- Schäfer, A.M.; Zimmermann, H.G. Recurrent neural networks are universal approximators. In Proceedings of the 16th International Conference, Athens, Greece, 10–14 September 2006; Springer: Berlin/Heidelberg, Germany, 2006. [Google Scholar]
- Hornik, K.; Stinchcombe, M.; White, H. Multilayer feedforward networks are universal approximators. Neural Netw.
**1989**, 2, 359–366. [Google Scholar] [CrossRef] - Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2017. [Google Scholar]
- Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning representations by back-propagating errors. Nature
**1986**, 323, 533–536. [Google Scholar] [CrossRef] - Haykin, S.; Network, N. A comprehensive foundation. Neural Netw.
**2004**, 2, 41. [Google Scholar] - Zimmermann, H.G.; Tietz, C.; Grothmann, R. Forecasting with recurrent neural networks: 12 tricks. In Neural Networks: Tricks of the Trade; Springer: Berlin/Heidelberg, Germany, 2012; pp. 687–707. [Google Scholar]
- Zimmermann, H.G.; Grothmann, R.; Tietz, C.; Jouanne-Diedrich, H.V. Market modelling, forecasting and risk analysis with historical consistent neural networks. In Operations Research Proceedings 2010; Springer: Berlin/Heidelberg, Germany, 2011; pp. 531–536. [Google Scholar]
- Mvubu, M.; Kabuga, E.; Plitz, C.; Bah, B.; Becker, R.; Zimmermann, H.G. On Error Correction Neural Networks for Economic Forecasting. In Proceedings of the 2020 IEEE 23rd International Conference on Information Fusion (FUSION), Rustenburg, South Africa, 6–9 July 2020. [Google Scholar]
- Zimmermann, H.G.; Neuneier, R.; Grothmann, R. Multi-agent modelling of multiple FX-markets by neural networks. IEEE Trans. Neural Netw.
**2001**, 12, 735–743. [Google Scholar] [CrossRef] [PubMed] - Danny, P.; Allon, J. Multivariate exponential smoothing: Method and practice. Int. J. Forecast.
**1989**, 5, 83–98. [Google Scholar] - Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed] - Serrano-Pérez, J.D.; Fernández-Anaya, G.; Carrillo-Moreno, S.; Yu, W. New results for prediction of chaotic systems using deep recurrent neural networks. Neural Process. Lett.
**2021**, 53, 1579–1596. [Google Scholar] [CrossRef] - Rabinovich, M.I.; Fabrikant, A.L. Stochastic self-modulation of waves in nonequilibrium media. J. Exp. Theor. Phys.
**1979**, 77, 617–629. [Google Scholar] - Rössler, O.E. An equation for continuous chaos. Phys. Lett. A
**1976**, 57, 397–398. [Google Scholar] [CrossRef] - Curry, J.H. A generalized Lorenz system. Commun. Math. Phys.
**1978**, 60, 193–204. [Google Scholar] [CrossRef] - Edward, O.; Sauer, T.; Yorke, J.A. Coping with Chaos. Analysis of Chaotic Data and the Exploitation of Chaotic Systems; Wiley Series in Nonlinear Science; John Wiley and Sons: Hoboken, NJ, USA, 1994. [Google Scholar]

**Figure 7.**The Rabinovich–Fabrikant attractor and the corresponding time series, split into training and test data.

**Figure 8.**The Rossler attractor and the corresponding time series, split into training and test data.

**Figure 9.**The Lorenz attractor and the corresponding time series, split into training and test data.

System | Initial Conditions | Sample Size | Step Size | Truncation Parameter | Training Size | Test Size |
---|---|---|---|---|---|---|

Rossler | $(1,1,1)$ | 10,000 | 0.01 | 5 | 1800 | 200 |

Lorenz | $(0,1,1.05)$ | 12,000 | 0.01 | 8 | 1400 | 100 |

Rab-Fabrikant | $(-1,0,0.5)$ | 20,000 | 0.01 | 10 | 1800 | 200 |

Model | Rabi-Fabrikant $(\times {\mathbf{10}}^{-\mathbf{3}})$ | Rossler $(\times {\mathbf{10}}^{-\mathbf{3}})$ | Lorenz $(\times {\mathbf{10}}^{-\mathbf{3}})$ |

Vanilla HCNN | $0.7$ | $2.88$ | $3.52$ |

Vanilla RNN | $6.84$ | $3.68$ | $18.5$ |

HCNN p-Teacher Forcing | $\mathbf{0}.\mathbf{023}$ | $\mathbf{0}.\mathbf{17}$ | $\mathbf{0}.\mathbf{173}$ |

HCNN Lar-Sparse Tran. Mat | $0.072$ | $0.29$ | $9.01$ |

HCNN with LSTM Form. | $0.6$ | $0.22$ | $8.07$ |

RNN with LSTM Form. | $1.25$ | $1.39$ | $9.56$ |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Rockefeller, R.; Bah, B.; Marivate, V.; Zimmermann, H.-G.
Improving the Predictive Power of Historical Consistent Neural Networks. *Eng. Proc.* **2022**, *18*, 36.
https://doi.org/10.3390/engproc2022018036

**AMA Style**

Rockefeller R, Bah B, Marivate V, Zimmermann H-G.
Improving the Predictive Power of Historical Consistent Neural Networks. *Engineering Proceedings*. 2022; 18(1):36.
https://doi.org/10.3390/engproc2022018036

**Chicago/Turabian Style**

Rockefeller, Rockefeller, Bubacarr Bah, Vukosi Marivate, and Hans-Georg Zimmermann.
2022. "Improving the Predictive Power of Historical Consistent Neural Networks" *Engineering Proceedings* 18, no. 1: 36.
https://doi.org/10.3390/engproc2022018036