Nonstationary Time Series Prediction Based on Deep Echo State Network Tuned by Bayesian Optimization

Abstract

The predictions from time series data can help us sense development trends and make scientific decisions in advance. The commonly used forecasting methods with backpropagation consume a lot of computational resources. The deep echo state network (DeepESN) is an advanced prediction method with a deep neural network structure and training algorithm without backpropagation. In this paper, a Bayesian optimization algorithm (BOA) is proposed to optimize DeepESN to address the problem of increasing parameter scale. Firstly, the DeepESN was studied and constructed as the basic prediction model for the time series data. Secondly, the BOA was reconstructed, based on the DeepESN, for optimal parameter searching. The algorithm is proposed within the framework of the DeepESN. Thirdly, an experiment was conducted to verify the DeepESN with a BOA within three datasets: simulation data generated from computer programs, a real humidity dataset collected from Beijing, and a power load dataset obtained from America. Compared with the models of BP (backpropagation), LSTM (long short-term memory), GRU (gated recurrent unit), and ESN (echo state network), DeepESN obtained optimal results, which were 0.0719, 18.6707, and 764.5281 using RMSE evaluation. While getting better accuracy, the BOA optimization time was only 323.4 s, 563.2 s, and 9854 s for the three datasets. It is more efficient than grid search and grey wolf optimizer.

1. Introduction

Data and information have been significant resources for management and control within many domains. Time series data become vital because they can represent the historical and future trends of the systems and elements involved in the economy [1,2,3,4], agriculture [5,6], and air monitoring [7,8,9,10]. Advanced time series prediction can help us make rational decisions, avoid risks, and reduce losses—contributing significantly to social management, economic manufacturing, and human life. In the real world, time series data show complex characteristics because they are affected by many factors in different systems. The nonstationary and nonlinear characteristics have been a typical issue of time series analysis, making it difficult to predict using subjective and empiric knowledge [11].

The field of time series prediction focuses on various types of methods, including statistical methods [12,13] and machine learning methods [14,15,16,17], which can be classified into shallow and deep models. Statistical methods, such as the autoregressive (AR), moving average (MA), autoregressive moving average (ARMA), and autoregressive integrated moving average (ARIMA) models, find data patterns through mathematical inference. The statistical methods run well based on the exploration of the data patterns. The typical processing methods transform nonstationary data into stationary data. However, during the process of processing, the problem of data feature loss will make the data incomplete, which results in the data lacking original meaning and interpretability; this means that there will be corresponding errors, thus resulting in prediction uncertainty. These classical methods have significant errors and low prediction performance for unstable and nonstationary data, which may fail during nonstationary time series prediction. Deep learning was developed, which has more robust data feature extraction abilities and improved computer operation capabilities. The deep network structure can build the model with more vital learning ability. By finely adjusting the learning rate, the region with more substantial fluctuations in nonstationary time series can obtain better learning and more accurate model parameters. Therefore, deep networks have received much attention in research and application within time series prediction. Parts of machine learning methods apply to the regression of the time series modeling, including the support vector machine, the recurrent neural network (RNN), long short-term memory (LSTM), and the gated recurrent unit (GRU). They can construct the mapping relation of the time series trend. Various machine learning methods perform differently due to the time series type, method structure, and computing resource.

The decomposition method is a data processing method that can reduce data complexity by decomposing a nonstationary time series into multiple series of lower complexity. In recent years, many researchers have widely used decomposition methods for time series prediction [18,19,20]. The decomposition method will generate random terms in decomposition, and the degree of nonstationarity of the random terms remains high and difficult to predict. In addition, the decomposition method creates a separate model for each component, making the overall model architecturally complex. Therefore, networks with high prediction accuracy and simple model structures have been a hot topic for researchers.

Performance improvement in machine learning usually relies on increased network structure and computing resources. It is expected to be able to obtain high prediction precision with relatively simple networks. The echo state network (ESN) [21] is a recurrent neural network that does not need to update parameters through a backpropagation algorithm. The ESN completes system modeling through the matrix operation of internal weights, which does not require parameter tuning in an iterative process, and thus, consumes fewer resources. The DeepESN [22] deepens the reservoir structure of the ESN with more prosperous feature extraction capabilities, further improving the prediction performance. The DeepESN is suitable for various scenarios and widely used in multiple time series prediction tasks. However, the DeepESN model has a complex structure and numerous parameters, which weakens the original advantage of the ESN in terms of network structure and learning mechanisms. Therefore, the DeepESN should be optimized in view of redundant parameters and appropriate structures.

In this paper, the DeepESN is studied to predict nonstationary time series. Hyperparameter tuning in the machine learning model is generally considered a black-box optimization problem. During the tuning process, only the input and output of the model are seen. For machine learning, the choice of different hyperparameters has a crucial influence on the result. It is essential to determine the optimal values of hyperparameters. The heuristic optimization algorithm is a comprehensive way to adjust parameters. Some heuristic optimization algorithms have been proposed and applied to solve many parameter optimization problems [23,24,25,26,27]. However, most optimization algorithms require enough initial sample points, thus increasing the optimization cost, and the optimization efficiency is not exceptionally high. We noticed that the Bayesian optimization algorithm could fully use the previous evaluation information when selecting the next set of hyperparameters. This can reduce the number of attempts required to determine hyperparameters and improve the model’s estimation and generalization abilities. The optimization efficiency is higher than the heuristic optimization algorithms and grid search. Therefore, Bayesian optimization was chosen as the optimization method—an efficient optimization algorithm suitable for DeepESN. In the optimization method, the parameters of DeepESN are determined adaptively, which can reduce the parameter search time and maintain model precision.

The prediction model and optimization methods were verified using simulated and real time series data, which are the data from mixed signal oscilloscopes (MSO), humidity data collected from Beijing, and electric load data from American Electric Load Company. The datasets cover the simulation data based on typical mathematic signals, natural phenomena, and human social behavior. Our proposed model achieved the best prediction accuracy on several metrics when several models were compared. This paper is organized as follows. Section 2 introduces the related works on time series prediction. Section 3 presents the DeepESN and BOA. Section 4 displays the experiment and results. The method is discussed and concluded, finally, in Section 5.

2. Related Works

Nonstationary time series have high complexity. In recent years, time series forecasting methods have mainly included statistical and neural network methods. In addition, the echo state network and its deformed structure have increasingly been widely used in time series research. In this section, related research is introduced from three aspects. Then, using the method proposed in this article, the existing problems are analyzed.

2.1. Sample Entropy Method and Autocorrelation Functions

Sample entropy [28] is a method used to measure the fluctuation and complexity of time series. It measures the complexity of the time series by calculating the probability of generating new patterns in the signal. The greater the probability of a new model, the higher the complexity of the time series.

For a set of time series $\{u(i),i=1,\cdots ,N\}$, construct a $m$ dimensional vector:

$$Xm(i)=\{u(i),u(i+1),\cdots ,u(i+m-1)\},1⩽i⩽N-m+1$$

(1)

Define the maximum distance function $d [X_m(p),X_m(q)]$ of the vectors $X_m(p)$ and $X_m(q)$, which is recorded as the number that meets the condition $d [X_m(p),X_m(q)]\le r$. Define $n_{xi}^m$ as the number of $X_m(q)$ that have a distance between $X_m(p)$ and a vector $X_m(q)$ within $r$.

The sample entropy is:

$$SampEn(m,r,N)=-\ln \frac{{{\displaystyle \sum }}_{i=1}^{N-m-1}{n}_{xi}^{m+1}}{{{\displaystyle \sum }}_{i=1}^{N-m-1}{n}_{xi}^m}$$

(2)

The fluctuation and complexity of the time series can be measured using the method of sample entropy. The sample entropy measures the distance between two sub-vectors in a reconstructed time series. If the distance between any two sub-vectors is relatively close, the fluctuation and complexity of the time series are relatively weak. On the contrary, if the short distances between two vectors are relatively small, the time series has strong fluctuation and complexity.

The sample entropy can be set as a time series fluctuation and complexity measurement index. A threshold value can be selected, and a model with high prediction ability should be built when the entropy of a dataset is over the threshold value. The prediction models can be deep networks, as introduced in the following subsection.

The autocorrelation function refers to the correlation between neighboring variables of the time series. Assuming that the observed value of the series is $x_t,x_{t-1},\dots ,x_{t-k}$, after the order of k, the correlation degree is defined as:

$${\gamma }_k=\frac{E\left(x_t,x_{t+k}\right)}{E\left(x_t^2\right)}=\frac{{\displaystyle {\displaystyle \sum }_{t=1}^{n-k}}(x_t-\overline{x})(x_{t+k}-\overline{x})}{{\displaystyle {\displaystyle \sum }_{t=1}^{n-k}}{(x_t-\overline{x})}^2}$$

(3)

where $\overline{x}$ represents the sample mean and $E$ represents the expectation function. The value of ${\gamma }_k$ is between −1 and 1; the closer to 1, the higher the degree of autocorrelation.

The autocorrelation function is either trailing or truncated for a stationary time series. The trailing refers to the function slowly approaching 0 at a negative exponential rate as the lag value $k$ becomes larger. The truncation refers to the autocorrelation function becoming 0 when the lag value $k$ is a specific value, like a truncated tail. The autocorrelation function does not have trailing and truncated tails for nonstationary time series.

Figure 1 shows the graph of the three data autocorrelation functions that were used. Its autocorrelation function has no tail or truncation, so it can be seen that they are non-stationary time series.

2.2. Statistical Models and Classical Neural Networks

Statistical methods are the earliest time series modeling methods. The ARMA [29,30] combines AR and MA for the stationary time series. The AR model treats subsequent data as a linear combination of previous data, and the MA model introduces a sliding window that can dynamically respond to time series characteristics. ARMA combines the advantages of the AR and MA models, which use statistical methods to model historical data, are simple to train, and have a good effect on dealing with complex time series. The ARIMA [31,32] model adds a different process, based on ARMA, which improves the nonlinear modeling ability. Amini et al. [33] built an ARIMA model to predict the electric vehicle charging demand for stochastic power system operation. The GARCH model can accurately simulate the fluctuation change of time series variables, and Caiado et al. [34] constructed it on finance time series. These methods are computationally small but very limited in their applicability to tasks and scenarios.

The machine learning method can extract data features more effectively using the computer’s computing power, thus capturing features more effectively when establishing the model. The classical time series neural network prediction fits the data and builds the model using error backpropagation. BP is the most basic neural network model, which reduces the error by continuously adjusting the model parameters. With the development of theory, the network structure with a deeper structure has been developed and studied. The deep network can mine the dependence on historical data and further extract the change rule of data. The RNN [35] model considers the influence of historical information on the current moment’s data. The LSTM [36] is one of the most widely used networks, and it further optimizes problems, such as gradient disappearance and explosion, in RNN. The GRU [37,38,39] reduces the number of gate units in the network technology of the LSTM and reduces computing consumption while achieving similar precision. Machine learning methods have been used for many tasks. Xue et al. [40] used the ELM to predict finance time series. Simran et al. [41] used the butterfly optimization algorithm (BOA) to optimize the LSTM network, and they utilized the LSTM network model to predict electricity consumption. Limei et al. [42] applied dynamic sliding windows to LSTM networks for water flow prediction. Xu et al. [43] built a dual-scale deep belief network, to predict the demand volume of urban water, that outperforms the single statistical model.

A single model is incapable of handling high-complexity nonstationary data. Thus, the hybrid modeling method of multiple models has been developed. Qiao et al. [44] established a hybrid model based on wavelet transform and an improved deep learning algorithm to predict PM2.5. Bao et al. [45] constructed stacked autoencoders and an LSTM to predict financial time series. Hanhong et al. [46] proposed a hybrid neural network model based on TCN and GRU to predict the short-term electricity load. The integrated model takes advantage of the advantages of multiple models to build models separately, which improves the predictive ability [47].

2.3. Echo State Network

Statistical methods are relatively simple to model but have insufficient predictive power. The machine learning method requires many calculations and takes a long time. The integrated learning method makes the model structure more complicated, which is not conducive to updating the model. The ESN is considered the most effective method for training recurrent neural networks, which have the advantages of a simple training process and short time consumption [48].

Figure 2 shows the structure diagram of the ESN. We define the $u(n)={[u_1(n) \cdots u_k(n)]}^T$ as the input sample at time $n$. $y(n)$ is the output corresponding to $u(n)$. The input matrix $W_{in}$ and the reservoir layer weight matrix $\hat{W}$ are uniformly distributed between −1 and 1, which remain fixed. To make the echo state network have echo properties, the spectral radius of the reservoir layer should be set between 0 and 1. During the training process, with the sample’s input, the reservoir layer’s state is updated using the following formula.

$$x(n+1)=f\left(W_{in}u(n+1)+\hat{W}x(n)\right)$$

(4)

where $W_{in}$ denotes the input connection matrix, $\hat{W}$ denotes the reservoir layer connection matrix, and $f$ denotes the activation function inside the reservoir layer, which is usually taken strictly as a hyperbolic tangent function. According to the above state of the reservoir layer, the output of the ESN can be calculated using the following equation.

$$y(n+1)=f_{\mathit{out}}\left(W_{\mathit{out}}x(n+1)\right)$$

(5)

where $W_{out}$ denotes the output connection matrix and $f_{out}$ denotes the output excitation function. During training, the reservoir layer states are collected into a state matrix X. The final network output weight $W_{out}$ can be calculated using the following equation.

$$W_{out}={\left({\mathit{X}}^T\mathit{X}\right)}^{-1}{\mathit{X}}^T\mathit{Y}$$

(6)

where X denotes the matrix form of the input reservoir state and Y denotes the output matrix form. The superscript $T$ represents the matrix transpose and $-1$ represents the inverse of the matrix. The optimal solution of the output matrix can be found using least squares or MSE alone. Since the output function is a standard linear function, Equation (6) can be derived from the loss function $L=\left|W_{out}X-Y\right|$, where $L$ denotes the loss matrix.

Researchers have implemented some optimizations and improvements to the structure of the ESN, such as the quantum echo state network [49] and the robust echo state network [50], and the DeepESN [22] is a neural network composed of multiple reservoirs. It has excellent predictive capabilities and has been applied to various tasks. It has been a research hotspot in recent years, and we will study it in this paper. The DeepESN has many network parameters, so it is difficult to determine a suitable network structure. Therefore, we propose the BOA to optimize the DeepESN, which can adaptively determine the network model structure within a limited amount of time, thus saving network modeling time and making the network more suitable for various tasks [51,52,53,54,55,56,57].

Significantly different from previous studies, we propose a DeepESN optimized by the BOA for predicting complex time series. Our innovative contributions are highlighted as follows:

(1)

The BOA is used to optimize the DeepESN so that the network parameters are easily set and the final model is easier to determine.

(2)

Bayesian optimization is applied to nonstationary time series prediction, and excellent prediction results are obtained.

3. Bayesian Optimization of Deep Echo State Network

3.1. Deep Echo State Network

The DeepESN is a novel deep network structure that replaces a single reservoir layer with an ESN with multiple reservoir layers. It consists of an input layer, a multiple-reservoir layer, and an output layer. The input layer feeds data into the entire network. The reservoir layers process and compute the data to obtain the internal state. The output layer fits the internal states to the actual values to obtain the model parameters. Compared with the backpropagation algorithm, the DeepESN only requires matrix operations and does not require many backpropagation iterations. Further, the DeepESN has a more robust feature extraction ability and better prediction accuracy than a simple ESN.

Figure 3 shows the structure of the DeepESN. We define $u(n)={[u_1(n) \cdots u_k(n)]}^T$ as the input sample at time $n$. After each sample is inputted into the DeepESN network, the state of the reservoir layers in the DeepESN will be updated. For the state of the $l$ reservoir layer, it is updated using the following formula:

$$x^{(l)}(n)=\left(1-a^{(l)}\right)x^{(l)}(n-1)+a^{(l)}\tanh \left(W_{in}^{(l)}{i}^{(l)}(n)+{\theta }^{(l)}+{\hat{W}}^{(l)}{x}^{(l)}(n-1)\right)$$

(7)

$$i^{(l)}(n)=\{\begin{array}{l}u(n),l=1\\ x^{(l-1)}(n),l>1\end{array}$$

(8)

where $x^{(l)}(n)$ represents the state obtained by the reserved layer $l$ when the first sample is inputted. $a^{(l)}$ represents the leaking rate of the layer $l$, $\tanh$ is the hyperbolic tangent activation function, $W_{in}^{(l)}$ represents the input matrix of the input layer, ${\hat{W}}^{(l)}$ represents the internal matrix of the reservoir layer, and $W_{in}^{(l)}$ and ${\hat{W}}^{(l)}$ are the fixed parameters. ${\mathit{i}}^{(l)}(n)$ represents the input of the reservoir layer. When $l$ is 1, the input of the reservoir layer is the original sample; when $l$ is greater than 1, the input layer of the reservoir layer is the state of the reservoir layer.

Finally, the state of each reservoir layer is collected to get the state of the entire network.

$$x(n)={[x^{(1)}(n)x^{(2)}(n)\cdots x^{(l)}(n)]}^T$$

(9)

The output weight and the state collected by the network will be multiplied by the matrix to get the final network output value.

$$y\left(n\right)=g\left(W_{out}x\left(n\right)\right)$$

(10)

The output weights $W_{out}$ are the model parameters that need to be calculated by the network. The $W_{out}$ is the parameter that only needs to be trained. Sort all $x(n)$ and $y(n)$ by row to get $X$ and $Y$, the final model parameters. $W_{out}$ can be calculated using the following formula:

$$W_{out}=Y{X}^T{\left(X{X}^T+\alpha E\right)}^{-1}$$

(11)

where $\alpha$ is the regularization coefficient, $E$ is the unit matrix, the superscript $T$ represents the matrix transpose, and $-1$ represents the inverse of the matrix. The derivation principle of Equation (11) is similar to that of Equation (6) and can be derived from the loss function $L=\left|W_{out}X-Y\right|+\alpha W_{out}$, where $L$ denotes the loss matrix.

3.2. Bayesian Optimization Algorithm

The BOA is an effective hyperparameter optimization algorithm. It can find the best model parameters faster than a grid and random search. Each parameter is in a specific interval, and all parameters constitute a hyperparameter space. For different hyperparameter combinations, the prediction performance of the model is different. The model’s accuracy gets higher and higher through the optimization process, and the predicted result is closer to the actual value. The objective optimization function can be expressed as:

$$f(x)=\sqrt{\frac{{{\displaystyle {\sum }_{i=1}^{num}(y(x_i)-{\hat{y}}_i)}}^2}{num}}$$

(12)

where ${\hat{y}}_i$ is the true value and $y(x_i)$ is the predicted value. $num$ is the length of the input time series. The optimal parameters can be obtained through the BOA, which makes the optimized target reach the minimum value.

The objective function of the BOA can be simplified as:

$$x^{\ast }=\underset{x\in X}{argmin}f(x)$$

(13)

where $X$ stands for parameter combination, $x^{\ast }$ represents the best parameters obtained, and $x$ is a group of hyperparameter combinations.

The BOA finds new evaluation points by maximizing the acquisition function to weigh the distribution of evaluation points and the improvement of prediction performance; it then re-imports them as inputs in the model to obtain new outputs so that it can continuously update and find model parameters. We chose the Gaussian function as the distribution hypothesis of the prior function and then used the acquisition function to select the next point in the posterior process for evaluation. The Gaussian process (GP) is an extension of the multidimensional Gaussian distribution, which can be defined using mean and covariance. The covariance function of a GP is its kernel function $k(x,x^{\prime })$. The kernel function measures the role of the distance between any two points, $x$ and $x^{\prime }$.

$$f(x)\sim GP(\mu (x),k(x,x^{\prime }))$$

(14)

$$\mu (x)=E[f(x)]$$

(15)

$$k(x,x^{\prime })=E[(f(x)-\mu (x))(f(x^{\prime })-\mu (x^{\prime }))]$$

(16)

Usually, the mean function is set to zero; then, the above Gaussian process can be expressed as:

$$K_n=\left(\begin{array}{ccc}k(x_1,x_1)& \dots & k(x_1,x_n)\\ ⋮& \ddots & ⋮\\ k(x_n,x_1)& \cdots & k(x_n,x_n)\end{array}\right)$$

(17)

When a new set of evaluation samples are added to the set of all evaluation points, the covariance matrix formula is updated to:

$$\{\begin{array}{l}{K}_{n+1}=\left(\begin{array}{cc}{K}_n& k_n^T\\ k_n& k(x_{n+1},x_{n+1})\end{array}\right)\\ k_n=\left[k(x_{n+1},x_1),k(x_{n+1},x_2),\cdots ,k(x_{n+1},x_n)\right]\end{array}$$

(18)

Using the updated covariance matrix, we can get the posterior probability:

$$P(f_{n+1}|D_{n+1},x_{n+1})\sim N({\mu }_{n+1}(x),{\sigma }_{n+1}^2(x))$$

(19)

where $D$ is the observation data, ${\mu }_{i+1}(x)$ is the mean value of $f(x)$ at step $i+1$, and ${\sigma }_{i+1}^2(x)$ is the variance of $f(x)$ at step $i+1$.

The value of the function can be sampled from the joint posterior distribution by evaluating the mean value and covariance matrix. The sampling function can determine the next point to be evaluated to find the optimal parameter value faster and reduce resource consumption. We choose the UCB function as the sampling function, and the expression is as follows:

$$x_{i+1}=\arg \max H(x|D_i)=\arg \max \mu (x)+{\zeta }_{i+1}^{\frac{1}{2}}{\sigma }_i(x)$$

(20)

where ${\zeta }_{i+1}^{\frac{1}{2}}$ is a constant, $H(x|D_i)$ is the UCB acquisition function, and $x_{i+1}$ is the selected hyperparameter of step $i+1$.

In this paper, we optimized five important parameters of the DeepESN: layers, number of neurons, leaking rate, spectral radius rate, and input scaling. Among them, the spectral radius is the maximum value of all of the eigenvalues of the reservoir weights, and it is the main factor affecting the memory capacity of the reserve. It is necessary to make the spectral radius less than 1 to ensure the convergence of the network. Input scaling is the pre-processing data method used to limit the range of input data. The entire algorithm flow is shown in Algorithm 1.

Algorithm 1: DeepESN optimized by the BOA.

Input: Select the parameters to be optimized in the DeepESN model and determine the optimization space $X$. Set the number of iterations of the BOA as $n$.

Output: returns the optimal hyperparameter group $x^{\ast }$.

For i = 1:$n$

Select a set of optimized parameters $x$ from $X$.

Train the DeepESN model with parameters $x$.

Evaluate model prediction performance using Equation (11).

Update the covariance matrix and get the posterior probability.

Based on the current results, obtain the next set of model parameters.

Evaluate the model using Equation (12) and obtain the optimal model parameters $x^{\ast }$.

4. Experiment and Results

4.1. Experimental Setup

The data used in the experiments included two types of data, which were simulation data and actual data. The simulation data were obtained through computer simulation, and the real data were collected from the actual generation. The MSO dataset, obtained through computer simulation, was used as the simulation data. The actual data were the collected humidity data from the meteorological field and the electric load data from the electric power industry. The sample entropy was calculated for the three datasets according to the method in Section 2.1, and the results are shown in

Table 1. The sample entropy of each dataset.

Datasets	Data Interval 1 (0–500)	Data Interval 2 (500–1000)	Data Interval 3 (1000–1500)	Data Interval 4 (1500–2000)
MSO	1.5874	1.5361	1.5141	1.6081
Humidity	0.5050	0.7246	0.8705	0.8114
Power load	0.6351	0.8671	0.9063	0.8376

.

For each dataset, four value intervals were selected as 0–500, 500–1000, 1000–1500, 1500–2000. The sample entropy was calculated for these four intervals, and Table 1 shows the specific results. It can be seen that the sample entropy was large, and the magnitude of the sample entropy varied widely between segments, implying a high level of fluctuation and complexity within the dataset. This shows that the models of high prediction ability should be applied to the time series.

The experimental platform of this paper was based on 64-bit Windows. The RAM was 8 GB, and the core was i7-8565u at 1.8 GHz. The deep learning framework used the Tensorflow interface of Keras, the optimizer was Adam, and the code was executed in python.

In the experiments in this paper, several typical models were chosen to compare with the Bayesian-optimized DeepESN to demonstrate the superior predictive power of the model. The typical models of the backpropagation neural network and the primary echo state network were chosen as the comparison models. Among them, the BP model used for comparison is the most classic neural network model and the most basic method used for non-stationary time series forecasting. The LSTM is a deep learning model method that has achieved good results in many time series forecasting tasks. The GRU is the optimized structure of the LSTM and is also used as a comparison model. These backpropagation models were used for comparison with the deep echo state networks to illustrate the learning ability of non-backpropagation networks and prove that deep echo state networks have good accuracy while consuming fewer resources. The ESN was used as the basic model, and the DeepESN was used for comparison in regard to prediction performance, thus confirming deep networks’ superiority in prediction accuracy. Therefore, we chose the above model as the comparison model.

To verify the efficiency of the BOA in parameter search, we compared the Bayesian optimization with the grid search method and the grey wolf optimizer (GWO) [26]. Too many grid search cases for five parameters would consume significant computational time. Therefore, we selected four values at equal intervals instead of the original parameter interval for each parameter. Thus, with four cases per parameter, the total number of parameter selection cases, by grid search for five parameters, was 1024. The GWO used 10 initial sample points and iterated 50 times.

In the field of time series forecasting, there are many indicators used to evaluate the forecasting effect. In this paper, we chose RMSE, MAE, R², and CC as the evaluation indexes. RMSE and MAE can most directly measure the forecasting effect. The smaller their values, the more accurate the prediction. R² and CC measure the linear relationship between predicted and true values. The closer the value is to 1, the stronger the linear relationship between the predicted and true values. In addition to the calculated value of the CC, it is also necessary to check its significance level, represented by the P-value. Generally, when the P-value is less than 0.05, it can indicate a significant correlation between the two data groups, and chance factors do not cause the CC value. The P-values of the three data groups were 0.000492, 3.212 × 10⁻¹³, and 6.338 × 10⁻¹¹, which meant that the datasets were significantly correlated and their evaluation indicators for CC were reliable.

$$RMSE=\sqrt{\frac{1}{T}{{\displaystyle \sum _{i=1}^T(y_r-y_p)}}^2}$$

(21)

$$MAE=\frac{{\displaystyle \sum _{i=1}^T\left|y_r-y_p\right|}}{T}$$

(22)

$$R^2=1-\frac{{\displaystyle \sum _{i=1}^T{(y_p-y_r)}^2}}{{\displaystyle \sum _{i=1}^T{(y_p-y_{rv})}^2}}$$

(23)

$$CC=\frac{{\displaystyle \sum _{i=1}^T\left[y_r(i)-y_{rv}(i)\right]}\left[y_p(i)-y_{pv}(i)\right]}{\sqrt{{{\displaystyle \sum _{i=1}^T\left[y_r(i)-y_{rv}(i)\right]}}^2}\sqrt{{{\displaystyle \sum _{i=1}^T\left[y_p(i)-y_{pv}(i)\right]}}^2}}$$

(24)

where $y_r$ represents the real value, $y_p$ represents the predicted value, $T$ represents the number of data, $y_{rv}$ represents the average value of the real value, and $y_{pv}$ represents the average value of the predicted value.

4.2. Datasets

MSO simulation data is a typical nonstationary time series. It consists of several sinusoidal components of different frequencies superimposed on each other. The following formula can represent it:

$$\mathrm{y}(n)={\displaystyle \sum _{i=1}^s\sin ({\alpha }_{i}n)}$$

(25)

where $s$ represents the number of sinusoidal frequencies, ${\alpha }_i$ represents the frequency of the ith component, and $\mathrm{y}(n)$ represents the n-th sample. In our experiment, we selected MSO data composed of 8 frequencies. The frequency components were: ${\alpha }_1=0.2$, ${\alpha }_2=0.311$, ${\alpha }_3=0.42$, ${\alpha }_4=0.51$, ${\alpha }_5=0.63$, ${\alpha }_6=0.74$, ${\alpha }_7=0.85$, ${\alpha }_8=0.97$. The MSO data are shown in Figure 4. The first 80% of the data was used as the training set, and the last 20% was used as the test set.

The data on humidity are recorded every hour in Beijing, China. The data of 7680 h across 320 days were set as the training data, and data of 1920 h across the next 80 days were the test data. In this experiment, a 24-step was set as the forward prediction length. The original humidity data are shown in Figure 5.

Power load data has strong non-stationarity due to the influence of season and temperature. We collected electricity data from the American Electric Load Company from 1 January 2017 to 1 January 2020. The data were collected every hour, for a total of 26,280 h. In our research, we selected the first 500 days of data, a total of 12,000 h, as the dataset. The first 80% of the data was used as the training set, and the last 20% was used as the test set. Figure 6 shows the original power load data.

The datasets are also presented with the statistical indicators in

Table 2. Statistical indicators for the three datasets.

Datasets	Minimum	Maximum	Mean	Standard Deviation	Variance
MSO	−7.4441	7.3918	−0.0261	2.0009	4.0034
Humidity	6	100	53	26.267	689.957
Power load	7196	22,759	14,784.716	2388.0759	5,702,906

. The statistics include the minimum, maximum, mean, standard deviation, and variance. Figure 4, Figure 5 and Figure 6 and Table 2 show the datasets have complex trends in different scales. When performing the Bayesian optimization of the DeepESN, the parameters to be optimized differed for each dataset, depending on the data type and size. For the above three datasets, we set the parameter ranges as

Table 3. The parameter ranges of the three datasets.

Hyperparameter	Type	MSO Data Parameter Range	Humidity Data Parameter Range	Power Load Data Parameter Range
layers	integer	1–30	1–40	1–30
Number of neurons	integer	100–4000	100–4000	100–6000
Leaking rate	uniform	0.1–1.0	0.1–1.0	0.1–1.0
Spectral radius rate	uniform	0.1–1.0	0.1–1.0	0.1–1.0
Input scaling	uniform	0.1–1.0	0.1–1.0	0.1–1.0

. For the comparison networks, the relevant parameters are shown in

Table 4. Parameter setting for comparison models.

Model	Layers	Number of Neurons	Reservoir Size	Spectral Radius Rate	Leaking Rate
BP	3	24	NA	NA	NA
LSTM	1	32	NA	NA	NA
GRU	1	32	NA	NA	NA
ESN	NA	NA	600	0.9	0.2

. In particular, the comparison networks were optimized using the grid searching method. This guaranteed that the prediction results of the BP, LSTM, GRU, and ESN had reached the best level of the experimental data.

4.3. Results

4.3.1. Results for the MSO Data

Figure 7 shows the prediction results of each model for the MSO simulation data, which are represented by curves of different colors. It can be seen from the figure that the Bayesian-optimized DeepESN, represented by the purple curve, is closest to the true value among all the models.

To intuitively display the predicted results of each model, the above evaluation indexes were used to calculate the experimental results. The specific results are shown in

Table 5. Evaluation indicators of the MSO dataset.

Model	RMSE	MAE	CC	R2
BP	1.7607	1.3953	0.5045	0.2221
LSTM	1.2252	0.9274	0.7982	0.6234
GRU	1.4290	1.0535	0.7053	0.4876
ESN	0.1024	0.0813	0.9987	0.9974
DeepESN	0.0719	0.0573	0.9994	0.9987

. The bold in Table 5 represents the model proposed in this paper and the best results in the evaluation indexes.

It can be seen that the DeepESN achieved optimal values for RMSE, MAE, CC, and R². Compared with the neural network algorithm, the prediction ability of the DeepESN has a significant advantage. Meanwhile, the prediction performance is improved based on the ESN. This experiment verifies the prediction ability of the DeepESN on simulated datasets.

Figure 8 shows the histogram of the prediction errors of different models on the MSO dataset. The minimum value of the errors was −4.7455, and the maximum was 4.9751. The errors were divided uniformly into 10 intervals. The prediction errors of the DeepESN, indicated in purple, were concentrated in the interval closest to [−0.85726, 0.1148]. Compared to other models, the overall errors of the DeepESN were the smallest, and the best prediction results were obtained.

4.3.2. Result of Humidity Data

Figure 9 shows the prediction results of each model on the actual humidity data. It can be seen from the figure that the Bayesian optimization DeepESN, represented by the purple curve, was the closest to the true value among all the models.

The results of the evaluation metrics calculations are shown in

Table 6. Evaluation indicators of the humidity dataset.

Model	RMSE	MAE	CC	R2
BP	21.5003	15.6253	0.6138	0.2787
LSTM	20.7405	15.2197	0.6249	0.3289
GRU	19.0048	14.0715	0.6897	0.4365
ESN	22.1146	17.3104	0.6069	0.2369
DeepESN	18.6707	14.7007	0.6786	0.4561

. The bold in Table 6 represents the model of this paper and the best results. The results show that the DeepESN model had the smallest RMSE, the smallest MAE, the largest R², and the largest CC. The DeepESN model had fewer errors and more accurate predicted values than the other models.

The histogram of prediction errors is shown in Figure 10. For the different models, the minimum value of the errors was −4.8994, and the maximum was 4.9751. The error distribution shows that the DeepESN, indicated in purple, was concentrated at [−0.9496, 0.03789]. The errors of the other models were located in intervals of big values.

4.3.3. Result of Power Load Data

Figure 11 shows the prediction results of each model on the actual power load data. Compared with the other curves, the DeepESN, represented by the purple curve, was the closest to the true value.

The results of the evaluation metrics calculation are shown in

Table 7. Evaluation indicators of the power load dataset.

Model	RMSE	MAE	CC	R2
BP	801.637	626.660	0.8922	0.7926
LSTM	840.739	623.628	0.8926	0.7719
GRU	791.691	583.079	0.9115	0.7977
ESN	823.565	586.355	0.8945	0.7811
DeepESN	764.5281	584.296	0.9011	0.8114

. The bold in Table 7 represents the model selected in this paper and the optimal eval-ua-tion index, respectively. As with the prediction results of the humidity dataset, the DeepESN model had the smallest RMSE and MAE and the largest R² and CC for the power load dataset, which confirms the prediction ability of the DeepESN for the actual dataset.

Through the above three groups of experiments and comparisons of the prediction curves and several evaluation indexes, it can be concluded that the Bayesian-optimized DeepESN has good prediction performance. Compared with other models, the DeepESN achieved better prediction results on both the simulated and real datasets, which confirms the advantages of the DeepESN in nonstationary time series prediction.

Figure 12 shows the histogram of prediction errors for the power load dataset. The data of power loads is complicated to predict. The minimum error value was −2968.06, and the maximum was 2993.299. The errors were divided into 15 intervals. The prediction errors of the DeepESN, indicated in purple, were concentrated at [−186.0927, 211.3312]. The Gaussian distribution of the errors showed that the DeepESN obtained the best prediction results.

4.3.4. Result of Bayesian Optimization Algorithm

Grid search and GWO were performed on the same dataset, and the optimization results were compared with the BOA method.

Table 8. Prediction results of different optimization methods.

Datasets	Optimization Method	RMSE	MAE	CC	R2	Time (s)
MSO	BOA	0.0719	0.0573	0.9994	0.9987	323.4
MSO	Grid search	0.0689	0.0544	0.9996	0.9989	3587.2
MSO	GWO	0.1204	0.0971	0.9881	0.9855	672.9
Humidity	BOA	18.6707	14.7007	0.6786	0.4561	563.2
Humidity	Grid search	18.5819	14.6893	0.6629	0.4344	6023.3
Humidity	GWO	20.6907	17.3544	0.5803	0.3321	964.1
Power load	BOA	764.5281	584.296	0.9011	0.8114	985.4
Power load	Grid search	749.8249	573.354	0.9259	0.8319	9987.8
Power load	GWO	796.5818	613.141	0.8934	0.7952	1367.2

compares the experimental results of the three optimization methods for the three datasets.

Table 8 shows the metrics of the three optimization methods on the three datasets. By comparison, it was found that the BOA obtained the best results across four prediction accuracy metrics while consuming the least amount of time. Compared with grid search and GWO, the BOA can obtain better prediction results in the shortest time

Table 9. Parameters on each dataset obtained through Bayesian optimization.

Datasets	Optimization Method	Layers	Number of Neurons	Leaking Rate	Spectral Radius Rate	Input Scaling
MSO	BOA	12	2500	0.20	0.79	0.10
MSO	Grid search	5	1700	0.39	0.44	0.26
MSO	GWO	18	3998	0.37	0.71	0.10
Humidity	BOA	5	3100	0.89	0.82	0.03
Humidity	Grid search	3	1600	0.70	0.37	0.41
Humidity	GWO	6	167	0.12	0.10	0.11
Power Load	BOA	9	5900	0.99	0.47	0.29
Power Load	Grid search	8	3400	0.29	0.36	0.28
Power Load	GWO	9	5471	0.11	0.10	0.21

shows the parameters finally selected by the DeepESN for the three experiments by the three optimization methods.

For a more visual comparison of the results, we plotted the comparison of the evaluation results of different optimization approaches for each dataset using bar charts. Evaluation indicators included RMSE, MAE, R², CC, and consumption time. Figure 13, Figure 14 and Figure 15 show each dataset’s evaluation indicators for different optimization methods. In the figures, subplot (a) shows the comparison of RMSE and MSE, subplot (b) shows the CC and R², and subplot (c) shows the comparison of running time.

As shown in Figure 13, the simulation data MSO, BOA, and grid search achieved extremely close results in the evaluation indicators, indicating that the two optimization methods can achieve close results, while GWO had poor prediction results. Regarding time consumption, The BOA method consumed less time, followed by GWO and grid search, which indicates that the BOA method is efficient in parameter search.

Figure 14 shows that the BOA and grid search optimization methods had the highest prediction accuracy for the humidity data. Moreover, the time consumed by the BOA was relatively lower among the three optimization algorithms.

As shown in Figure 15, similar to the results in the MSO and humidity datasets, the BOA can achieve similar prediction accuracy to a grid search in power load experiments, and it has the shortest optimization time compared with grid search and GWO.

In three experiments comparing the results of three experiments by three different optimization methods, the results of the BOA were close to the grid search for multiple indicators of RMSE, MAE, CC, and R². However, the results predicted using the GWO method were relatively poor. Meanwhile, in these three cases, the BOA method consumed the least time of all three cases, which means that fewer computational resources were used, showing the efficiency of the BOA in the parameter search task. It is confirmed that the Bayesian optimization method is an effective way to optimize the DeepESN.

5. Discussion and Conclusions

This paper used the Bayesian-optimized DeepESN to model and predict simulated and real data. The prediction results showed that, compared to ARIMA, BP, LSTM, GRU, and ESN networks, the Bayesian-optimized DeepESN was closer to the curve the actual values represented. The evaluation metrics showed that the Bayesian-optimized DeepESN achieved the best prediction results across several evaluation metrics compared to other network types.

Experiments were conducted to compare the BOA, grid search methods, and GWO to optimize the DeepESN in three ways. The BOA can use previous validation results to reasonably collect the parameters for the next time. With almost the same prediction accuracy, the BOA significantly reduces the parameter search time, decreases time complexity, and proves that the BOA has a very efficient effect in performing the parameter optimization of the model. Therefore, the Bayesian-optimized DeepESN can significantly reduce the model tuning time, improve the prediction accuracy and reduce resource consumption.

Compared with echo state networks, the DeepESN has richer dynamic characteristics, stronger short-term memory storage capacity, and more accurate temporal prediction ability though. However, the deep echo state network still has inherent characteristics and limitations, such as covariance, discomfort, and poor numerical stability. The neural network needs to continuously adjust the learning rate to finalize the model parameters in the modeling process. The training should be conducted based on adequate data. For small sample datasets, neural networks are prone to overfitting, reduced generalization ability, and ineffective tuning and learning of internal parameters. The BOA helps to determine the parameters faster with the smaller dataset. It reduces the requirement of data volume to a certain extent. However, how to adjust the method in the small dataset should be studied continuously.

Based on the research in this paper, we will continue to explore the following directions, such as scouring the effect of the latest heuristic optimization algorithms combined with the DeepESN, the results enhancement by data enhancement methods, and how to optimize the structure of the DeepESN. The DeepESN with the BOA provides a feasible solution as the basic time series prediction model. It can be applied to exploring integrated methods, such as the prediction based on data decomposition and the integration of various models. The performance can be improved by utilizing the prediction ability of the DeepESN with the BOA and the data-featured extraction ability of other methods.