A Hybrid Method Using HAVOK Analysis and Machine Learning for Predicting Chaotic Time Series

Abstract

The prediction of chaotic time series systems has remained a challenging problem in recent decades. A hybrid method using Hankel Alternative View Of Koopman (HAVOK) analysis and machine learning (HAVOK-ML) is developed to predict chaotic time series. HAVOK-ML simulates the time series by reconstructing a closed linear model so as to achieve the purpose of prediction. It decomposes chaotic dynamics into intermittently forced linear systems by HAVOK analysis and estimates the external intermittently forcing term using machine learning. The prediction performance evaluations confirm that the proposed method has superior forecasting skills compared with existing prediction methods.

1. Introduction

A chaotic system refers to a deterministic system where there are irregular movements that appear to be random, and its behavior is uncertain, unrepeatable, and unpredictable. The high sensitivity to the initial conditions and the fact that they are inherently unpredictable are the main characteristics of the chaotic systems. Chaotic phenomena are ubiquitous in several scientific fields, such as atmosphere motions [1], population dynamics [2,3,4], epidemiology [5], and economics. It has been a hot topic in such fields and attracted the attention of many people. It is worth noting that the chaotic system is not completely random as its name suggests but has certain structure and patterns. However, due to the lack of understanding of the dynamic mechanism of chaotic systems, the prediction of chaotic time series is still a very important but challenging problem.

With the development of big data and advanced algorithms in machine learning, it has become a new research direction to solve prediction problems of chaotic systems using a data-driven way. Several empirical models to predict chaotic time series based on machine learning are proposed. Many famous artificial neural networks (ANN) models such as Radial Basis Function (RBF) neural network [6], neuro-fuzzy model with Locally Linear Model Tree (LoLiMoT) [7], feedforward neural network [8], multi-layer perceptron (MLP) [9], recurrent neural networks (RNN) [10], finite impulse response (FIR) neural network [11], deep belief nets (DBN) [12], Elman neural network [11,13,14], and wavelet neural network (WNN) [15,16] have been introduced in the literature.

However, the setting of neural network model parameters will greatly affect the performance of these models. Consequently, a substantial amount of work has also been put into the optimization algorithm and parameter settings. Min Gan et al. present a state-dependent autoregressive (SD-AR) model, which uses a set of locally linear radial basis function networks (LIRBFNs) to approximate its functional coefficients [17]. In addition, Pauline Ong and Zarita Zainuddin presented a modified cuckoo search algorithm (MCSA) to initialize WNN models [16]. A hybrid learning algorithm, called HGAGD, which combines genetic algorithm (GA) with gradient descent (GD) is proposed to optimize the parameters of a quantum-inspired neural network (QNN) [18]. In the proposed methodology, the embedding method is used along with ENN to predict the residual time series [13]. A single hidden Markov model (HMM) combined with fuzzy inference systems is introduced for time series predicting [19]. What is more, many hybrid methods are also developed for improving the performance of these prediction models [20,21].

As mentioned above, model structure and parameter tuning are important factors for chaotic time series prediction with machine learning, and a lot of research has focused on it. To simplify the learning model, a hybrid method using Hankel Alternative View Of Koopman (HAVOK) analysis and machine learning (HAVOK-ML) is developed to predict chaotic time series in this research. Hankel Alternative View Of Koopman (HAVOK) analysis was proposed by Brunton [22]. It combines the delay embedding method [23] and the Koopman theory [24] to decompose chaotic dynamics into a linear model with intermittent forcing. HAVOK-ML decomposes chaotic dynamics into intermittently forced linear systems with HAVOK; then, it estimates the forcing term using machine learning. Essentially, the prediction of the chaotic time series using the HAVOK-ML method is conducted as solving linear ordinary differential equations, which can be calculated efficiently. It can take different types of regression methods such as Linear Regression or Random Forest Regression (RFR) [25] into the prediction framework and combines the advantages of HAVOK theory and machine learning. Therefore, it can obtain better prediction results than directly using those machine learning models.

This paper is organized as follows. Section 2 briefly describes the theory of the HAVOK analysis combined with the machine learning method for time series prediction. Section 3 applies the proposed combined method to perform multi-step ahead prediction for some well-known chaotic time series and also compares the obtained prediction performance with that of existing prediction models. Finally, conclusions are given in Section 4.

2. HAVOK-ML Method

Consider a nonlinear system of the form:

$$\frac{d\mathbf{x}\left(t\right)}{dt}=f\left(\mathbf{x}\left(t\right)\right)$$

(1)

where $\mathbf{x}\left(t\right)\in {\mathbb{R}}^n$ is the state of the system at time t and f denotes the dynamics of the system. For a given state $x\left(t_0\right)$ at time $t_0$, $x(t_0+t)$ can be given discretely by:

$$\mathbf{x}(t_0+t)=\mathbf{x}\left(t_0\right)+{\int }_{t_0}^{t_0+t}\mathbf{f}\left(x\left(\tau \right)\right)d\tau$$

(2)

Generally speaking, for an observed chaotic time series x(t), the governing equation f is highly nonlinear and unknown. HAVOK analysis [22] provides linear representations for those unknown nonlinear systems. A Hankel matrix $\mathbf{H}$, for a single measurement $x\left(t\right)$, by taking singular value decomposition (SVD), is given by:

$$\mathbf{H}=\left(\begin{array}{cccc}{x}_{(}{t}_1)& x_{(}{t}_2)& \dots & x_{(}{t}_p)\\ x_{(}{t}_2)& x_{(}{t}_3)& \dots & x_{(}{t}_p+1)\\ ⋮& ⋮& \ddots & ⋮\\ x_{(}{t}_q)& x_{(}{t}_{q+1})& \dots & x_{(}{t}_m)\end{array}\right)=\mathbf{U}\Sigma {\mathbf{V}}^T$$

(3)

where $m=p+q-1$, p and q are two parameters that determine the dimension of $\mathbf{H}$. The columns of $\mathbf{H}$ are defined by:

$$\mathbf{h}\left(i\right)={[x\left(t_i\right),x\left(t_{i+1}\right),\cdots ,x\left(t_{i+q-1}\right)]}^T(i=1,\cdots ,p)$$

(4)

then

$$\mathbf{H}=\left[\mathbf{h}\right(1\left)\mathbf{h}\right(2)\cdots \mathbf{h}(p\left)\right]$$

(5)

Usually, $\mathbf{H}$ can be well approximated by the first r columns of $\mathbf{U}$, $\mathbf{V}$. According to the HAVOK analysis [22], the first $r-1$ variables in V can be built as a linear model with the last variable $v_r$ as a forcing term:

$$\frac{d{\mathbf{v}}_{r-1}\left(t\right)}{dt}=\mathbf{A}{\mathbf{v}}_{r-1}\left(t\right)+\mathbf{B}\left(t\right)$$

(6)

where ${\mathbf{v}}_{r-1}={[v_1,v_2,\cdots ,v_{r-1}]}^T$ is the vector of the first $r-1$ eigen-time-delay coordinates. Note that Equation (6) is not a closed model because $v_r\left(t\right)$ is an external input forcing. In the linear HAVOK model, matrix $\mathbf{A}$ and vector $\mathbf{B}$ may be obtained by the Sparse Identification of Nonlinear Dynamics (SINDy) algorithm [26] or by a straightforward linear regression procedure. $v_r$ is given by the rth column of $\mathbf{V}$.

A machine learning method is used to predict $v_r(t+1)$ by using previous observed values $\left[x\right(t-D\Delta t),x(t-(D-1)\Delta t,\cdots ,x(t-\Delta t)]$, as shown in Figure 1. Suppose $v_r$ evenly varies within interval $[t,t+1]$. The evolution of $v_r\left(t\right)$ can be approximated by:

$$\frac{d{v}_r\left(t\right)}{dt}=(v_r(t+1)-v_r\left(t\right))/\Delta t$$

(7)

Then, the first $r-1$ variables ${\mathbf{v}}_{r-1}(t+1)$ are obtained by solving the linear model Equation (6):

$$\mathbf{H}=\left(\begin{array}{c}{\mathbf{v}}_{r-1}(t+1)\\ v_r(t+1)\\ v_r(t+1)-v_r\left(t\right)\end{array}\right)=exp\left(\begin{array}{ccc}\mathbf{A}dt& \mathbf{B}dt& 0\\ 0& 0& \mathbf{I}\\ 0& 0& 0\end{array}\right)\left(\begin{array}{c}{\mathbf{v}}_{r-1}\left(t\right)\\ v_r\left(t\right)\\ v_r(t+1)-v_r\left(t\right)\end{array}\right)$$

(8)

Assume that the integration starts at time $t_p$ for an input $\mathbf{h}\left(p\right)$. Then, the next step of $\mathbf{h}(p+1)$ can be written as:

$$\mathbf{h}(p+1)=\mathbf{U}\Sigma {\mathbf{v}}_r(t_p+1)$$

(9)

where ${\mathbf{v}}_r=[v_1,v_2,\cdots ,v_{r-1},v_r]$ is a vector containing the first r variables. In order to evaluate the efficiency of HAVOK-ML, RMSE, NMSE, and $R^2$ score defined below are used as a performance index.

$$RMSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2}$$

(10)

$$NMSE=\frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^N{(y_i-{\overline{y}}_i)}^2}$$

(11)

$$error=\frac{{\sum }_{i=1}^N{({\widehat{y}}_i-{\overline{y}}_i)}^2}{{\sum }_{i=1}^N{(y_i-{\overline{y}}_i)}^2}$$

(12)

where $y_i,{\widehat{y}}_i$, and $\overline{y}$ represent the observed data, the predicted data, and the mean of the observed data, respectively. The Root Mean Squared Error (RMSE) and the Normalized Mean Squared Error (NMSE) are used to assess the accuracy of the prediction and to compare the results with those of the literature. The $R^2$ score is used to evaluate the score of the machine learning based prediction for $v_r$.

3. Numerical Experiments

In this section, three different type of time series—Lorenz [1], Mackey–Glass [26], and Sunspot—are applied to verify our proposed HAVOK_ML method. The parameters adopted in the HAVOK analysis of these series are listed in

Table 1. HAVOK analysis parameters for each system.

System	Samples	dt	$\Delta t$	q	Rank (r)	Regressor for ${\mathit{v}}_{\mathit{r}}$
Lorenz	20,000	0.01 s	0.001 s	40	11	RandomForest
Mackey-Glass	50,000	0.1 s	/	5	5	LoLiMoT
Sunspot	2000	1 month	0.02 month	140	7	LoLiMoT

.

3.1. Lorenz Time Series

The Lorenz system [1] is among the most famous chaotic systems, which is described by:

$$\begin{array}{cc}\hfill \frac{dx}{dt}& =\sigma (y-x)\hfill \\ \hfill \frac{dy}{dt}& =x(\rho -z)-y\hfill \\ \hfill \frac{dz}{dt}& =xy-\beta z\hfill \end{array}$$

(13)

The chaotic time series is obtained with parameters $\sigma =10$, $\rho =28$, $\beta =8/3$, and $dt=0.01$ in the second sampling. In this study, only the time series of variable $x\left(t\right)$, shown in Figure 2, is considered.

In this research, HAVOK-ML decomposes chaotic dynamics into intermittently forced linear systems by HAVOK analysis; the settings of HAVOK analysis are given in Table 1, and the sampling time step for each system is consistent with other references listed in

Table 2. Comparison of the models in one-step predicting for Lorenz chaotic series $x\left(t\right)$, with 1000 testing samples. The last row represents the proposed HAVOK-ML combined with the RFR method. The highest prediction accuracies achieved by the models are shown in bold.

Model	RMSE	NMSE	Reference
Deep Belief Network	1.02 × 10${}^{-2}$	/	[12]
Elman–NARX neural networks	1.08 × 10 ${}^{-4}$	1.98 × 10${}^{-10}$	[13]
WNN	/	9.84 × 10${}^{-15}$	[15]
Fuzzy Inference System	3.1 × 10${}^{-3}$	/	[19]
Local Linear Neural Fuzzy	/	9.80 × 10${}^{-10}$	[7]
Local Linear Radial Basis Function Networks	/	4.53 × 10${}^{-12}$	[27]
WNNs with MCSA	8.20 × 10${}^{-3}$	1.22 × 10${}^{-6}$	[17]
HAVOK_ML(RFR)	1.43 × 10${}^{-5}$	3.23 × 10${}^{-12}$

. However, according to the advice in paper [14], the samples of the Lorenz system are interpolated at 0.001 s resolution in the HAVOK analysis.

By using HAVOK analysis for the training data, a linear HAVOK model (Equation (6)) is developed. As shown in Figure 3, matrix $\mathbf{A}$ and vector $\mathbf{B}$ are sparse, and the reconstruction of $v_1\left(t\right)$ and $x\left(t\right)$ is coherent with the actual values for the full range of time. Since the $v_r$ (Figure 4) is not smooth enough, many experiment results demonstrate that the RFR method [25] can predict the $v_r$ best. Hence, an RFR method is adopted to train and estimate the next step $v_r(t+1)$ based on previously observed values $\left[x\right(t-40),x(t-35),\cdots ,x(t-10),x(t-5\left)\right]$. The samples from the 3rd to the 100th seconds are spilled into training set (first 80%) and test set (20%). The $R^2$ score for the RFR method on the test set is 0.87. We can observe that the estimated results are mainly consistent with the actual values, as shown in Figure 4.

In the next experiment, the HAVOK-ML method is used in the N-step recursive prediction of Lorenz time series. In the recursive prediction, the current predicted values are used for next predictions without any correction to the actual values. A comparison between the multi-step predicted values and the original time series, with 1000 testing samples, is shown in Figure 5. It can be seen that in the initial steps of predict (less than 10), the prediction results are coherent with the actual values. The error increases with predict steps, especially at the region near the extreme point of the curve. The RMSE of the prediction function of time is presented in Figure 6. It can be observed that the error quickly increases with the increase of the predicted time, which means that a long-term prediction is basically impossible. At step 10, the obtained RMSE is 9.003× 10${}^{-3}$ (Figure 6), which is significantly better than the result of the literature (0.014) [27].

Table 2 presents the one-step ahead prediction errors (RMSE and NMSE) for the proposed method as well as some results obtained by existing methods, which were extracted from the literature. It can be shown that the RMSE index of the proposed method is optimal, while the NMSE index shows that the proposed method is second only to the functional weight WNN state-dependent AR (FWWNN-AR) model [15].

3.2. Mackey–Glass Time Series

The Mackey–Glass chaotic time series has been introduced as a white blood cell production [26]. It is described by:

$$\frac{dx\left(t\right)}{dt}=\frac{ax(t-\tau )}{1+x^r(t-\tau )}-bx\left(t\right),t>0$$

(14)

where $a=0.2$, $b=0.1$, $r=10$ and $\tau =17$, similar to other published papers presented in

Table 3. Comparison of the models in six-time step ahead predicting Mackey–Glass time series, with 4000 testing samples. The last row shows the proposed HAVOK-ML method with the LLN model as the regressor. The values in bold are the highest prediction accuracies achieved by the models.

Model	RMSE	NMSE	Reference
ARMA with Maximal Overlap Discrete Wavelet Transform	/	5.3373 × 10${}^{-7}$	[16]
Ensembles of Recurrent Neural Network	7.533 × 10${}^{-3}$	8.29 × 10${}^{-4}$	[20]
Quantum-Inspired Neural Network	9.70 × 10${}^{-4}$	/	[28]
Recurrent Neural Network	6.25 × 10${}^{-4}$	/	[18]
Type-1 Fuzzy System	4.8 × 10${}^{-4}$	/	[21]
Fuzzy Inference System	7.1 × 10${}^{-4}$	/	[19]
WNNs with MCSA	5.60 × 10${}^{-5}$	6.25 × 10${}^{-8}$	[17]
HAVOK_ML(RFR)	9.92 × 10${}^{-6}$	1.86 × 10${}^{-9}$

. The Mackey–Glass equation is solved using the delay differential equation method dde23 of MATLAB. A chaotic time series samples set of 25,000 lengths, with time step $dt=0.1$, is generated. The samples from the 300th to 2000th seconds, shown in Figure 7, are chosen as the training set, while the rest is used as the test set.

The HAVOK analysis settings for Mackey–Glass time series are given in Table 1. The rows in the H matrix are for $q=5$, and the rank of the SVD decomposition is $r=5$. More details on the HAVOK analysis and the multi-step ahead prediction for the Mackey–Glass time series are presented in Figure A1, Figure A2, Figure A3 and Figure A4 in Appendix A. By considering the properties of the $v_r$ curve, an LLN model with the LoliMoT optimization method [7] is determined through experiments as a regressor to predict $v_r$. A comparison between the prediction accuracies of the proposed method and other models of the literature are summarized in Table 3. As shown in Table 3, whether RMSE or NMSE, the effect of the proposed method outperforms the existing models.

3.3. Sunspot Time Series

The number of sunspots observed in the solar surface varies within a period of approximately 11 years. The variation of the number of sunspots has a large impact on Earth and on the climate. The monthly smoothed sunspot number time series, observed by the Solar Influences Data Analysis Center (http://sidc.oma.be/index.php (accessed on 7 August 2021)), is used for training and forecasting the trend of the sunspot variation. In order to compare the results with those extracted from the existing published papers of the literature (listed in

Table 4. Comparison of the models in one-time step ahead predicting sunspot time series, with 1000 testing samples. The last row shows the proposed HAVOK analysis with the LLN model as the regressor. The values in bold are the highest prediction accuracies achieved by the models.

Model	RMSE	NMSE	Reference
Elman-NARX Neural Networks	1.19 × 10${}^{-2}$	5.90 × 10${}^{-4}$	[13]
Elman Recurrent Neural Networks	5.58 × 10${}^{-2}$	1.92 × 10${}^{-2}$	[29]
Ensembles of Recurrent Neural Network	1.52 × 10${}^{-2}$	9.64 × 10${}^{-4}$	[20]
Fuzzy Inference System	1.18 × 10${}^{-2}$	5.32 × 10${}^{-4}$	[19]
Functional Weights WNNs State Dependent Autoregressive Model	1.12 × 10${}^{-2}$	5.24 × 10${}^{-4}$	[21]
WNNs with MCSA	1.13 × 10${}^{-2}$	5.30 × 10${}^{-4}$	[17]
HAVOK_ML(RFR)	4.25 × 10${}^{-3}$	7.40 × 10${}^{-5}$

), the series from November 1834 to June 2001 (2000 data points) are chosen and scaled between 0 and 1. The first 1000 samples are selected as the training set (Figure 8), and the remaining 1000 samples are used as the testing set. The settings of HAVOK analysis for sunspot time series are given in Table 1. Similar to the Mackey–Glass time series, an LLN model is used as the regressor to predict $v_r$. The details of the HAVOK decomposition and the multi-step prediction for the sunspot time series are shown in Figure A5, Figure A6, Figure A7 and Figure A8 in Appendix A. A comparison between the prediction errors (RMSE and NMSE) obtained with the 1000 samples of the testing set and with other models are presented in Table 4. It can be seen that the proposed HAVOK-ML method outperforms the existing methods in predicting the sunspot chaotic time series.

4. Discussion and Conclusions

In this paper, a HAVOK-ML method combining the HAVOK analysis with machine learning to predict chaotic time series is proposed. Based on the HAVOK analysis, the observed chaotic dynamic system could be reconstructed as a linear model with an external intermittent forcing. A machine learning method was applied to predict the external forcing term by using previously observed values. Finally, the combination of the HAVOK analysis with machine learning produces a closed model for prediction. It is worth noting that the machine learning method used in HAVOK-ML will vary depending on the property of the external forcing term. The developed method has been validated for multi-step ahead prediction of several classic chaotic time series (the Lorenz time series, the Mackey–Glass time series, and the Sunspot time series). The experimental results show that our method can produce accurate forecasts even with simple machine learning algorithms. The prediction performance of the proposed method has been compared with other forecasting models of the literature. The comparison shows that the proposed method outperforms the existing ones in terms of superior forecasting ability. Although HAVOK-ML can be combined with different machine learning methods, it does not give suggestions on how to choose machine learning methods for different time series forecasting problems. This is worth studying in the future.