Time Series Determinism Recognition by LSTM Model

Abstract

The problem of time series determinism measurement is investigated. It is shown that a deep learning model can be used as a determinism measure of a time series. Three distinct time series classes were utilised to verify the feasibility of differentiating deterministic time series: deterministic, deterministic with noise, and stochastic. The LSTM model was constructed for each time series, and its features were thoroughly investigated. The findings of this study demonstrate a strong correlation between the root mean square error (RMSE) of the trained models and the determinism of a time series.

1. Introduction

The scientific approach to studying nature is predicated on identifying laws, rules, and relationships that describe observed phenomena. Within time series analysis, the forecast is one of the frequently raised issues. Therefore, the existence of a function that governs the evolution of the system is expected, and within this work, it is considered the determinism of the system. It is acknowledged that the collected data (or measurement results) are inherently imperfect. In addition to the deterministic component, these data generally contain a stochastic element (i.e., a noise) related to the nature of the phenomenon under investigation, the quality of the experiment, and the data collection process. Consequently, numerous methodologies have been developed to determine the stochasticity of the data. The entropy-based methods are the most frequently used (and the best known). Information entropy, in particular, encompasses the Shannon entropy, Rényi entropy, Tsallis entropy, Theil index, and q-Theil index, among the most frequently employed methods [1,2,3,4,5]. Other approaches to the time series randomness are the strictly statistical methods, i.e., randomness statistical tests [6,7]. The predictability problem is associated with the signal-to-noise ratio and the analysis of chaotic systems, for example, the use of Lyapunov exponents [8]. Correlations can be local or long-range, and in addition to correlograms, Hurst exponents can be estimated to measure long-range correlations [9,10,11]. A commonality among the aforementioned methods is that they are designed to investigate specific features and do not provide a comprehensive answer to the determinism of the observed data. A more general approach is, therefore, required. This issue can be further articulated through a time series model. This issue is of fundamental importance, superseding the estimation of any of the aforementioned parameters. All possible characteristics can be estimated if the time series model is known. Despite advances in the time series models [12,13,14], no general methods can be applied. Each of the time series models has to be fitted individually, so failure of fitting does not mean that there is no other model that can explain the data. Therefore, the approach based on the time series model cannot be used as a determinism measure.

However, deep learning methods have recently allowed stating a more general method of the determinism estimation of a time series. Deep learning methods are, in fact, algorithms that are deterministic on one side and capable of being trained on the data provided. The quality of the trained model depends on several factors. However, one key feature is the existence of some deterministic patterns among the data discovered during the training stage. Of course, the model is not always satisfactory. Besides the possible mistakes in the model design, the main reason is the randomness of the analysed data. This feature allows designing a more general tool than the existing time series models for the determinism measure method. The main idea of the proposed methods is to use the quality of the trained model as a measure of the determinism of the time series. The capacity of AI models to reconstruct a wide range of time series suggests the potential for verifying the determinism of the analysed data. In this study, a novel method for assessing determinism is proposed. This method is based on the quality of the AI model trained on the time series data. It is hypothesised that highly deterministic time series will be reconstructed with high accuracy. In contrast, models trained on stochastic time series will demonstrate poor concordance with the time series. Due to the inherent characteristics of AI modelling, it is not feasible to provide a mathematical proof of the quality of the methods. The scheme of the proposed approach is presented in Figure 1. The discussion of the possible AI models and reasoning for the model choice are provided in Section 2. This paper verifies the possibility of establishing such a measure of determinism of the time series. The hypothesis is verified on the chosen examples and is performed in three classes of models: (i) deterministic time series models with a stochastic component; (ii) deterministic systems—however, in this case, the chaotic models will be used; (iii) real, stochastic models.

The autoregressive time series AR(p) will be used as the first class (i) examples. The primary advantage of AR(p) models is that, in addition to their purely deterministic component, they possess a stochastic component, thereby regulating the noise level. The Lorentz Attractor will represent the second class (ii). This is a deterministic system, yet it exhibits chaotic solutions. The system’s unpredictability, stemming from its sensitivity to initial conditions, challenges the conventional notion of determinism. Finally, the financial time series represent stochastic time series (iii) despite significant effort of scholars resulting in the observation of deterministic signs such as trends, seasonality, log-periodicity, long correlations, multifractality, etc. There is no clear and broadly accepted deterministic model of financial time series. Even the trends can be explained as random walks with a nonzero mean. Of course, there are obvious external factors that influence some of the financial time series (e.g., day and night periodicity in electricity demand or other feature, see, e.g., [15]). However, at present, reliable forecasting is difficult, particularly if the only information is the past data without external information. Therefore, the financial time series were chosen as an example of stochastic time series.

2. AI Time Series Models

Several classes of AI models can be used to analyse the determinism of data. Of particular note are natural language processing (NLP) models, which have recently garnered significant popularity. These models are founded upon the principle of encoder models. The primary advantage of these models lies in their capacity to capture the essential features of the input data. The encoders’ primary function is converting the input data into a numerical representation, which can subsequently be utilised to reconstruct or generate analogous data. The application of these encoder models is extensive, encompassing a wide range of disciplines including natural language processing, pattern recognition, and graph recognition [16,17]. However, this study focuses on the analysis of time series data.

The second category comprises recurrent neural networks (RNNs) [18,19,20], which have been defined for processing sequential data and are frequently employed for time series forecasting, including applications such as weather and energy production (see [21,22]). Recurrent neural networks (RNNs) can retain previous inputs, thereby enabling the capture of correlations among data points.

The long short-term memory (LSTM) model, as proposed in [23], can be regarded as a refinement of the RNN model, endowed with the capacity to retain information over extended periods. It is acknowledged that particular forecasting time series may encompass spatial challenges or exhibit patterns; for instance, energy forecasting may be contingent on a specific region. In such cases, the convolutional neural network (CNN) is employed (see [24,25]). In the present study, the LSTM model will be used to assess the determinism level of the input data.

A comprehensive overview of recurrent neural networks (RNNs) and long short-term memory (LSTM) can be found in [26,27,28]. The key elements of the RNN model are layers and the gates which control the flow of information between layers. In contrast to the RNN, the LSTM possesses an internal memory and multiplicative gates. The LSTM architecture employs three distinct types of gates: the input gate, which controls the information added to the memory cell; the forget gate, which determines the information to be removed from the memory cell; and the output gate, which controls the output from the memory cell.

The LSTM detailed description can be found in the literature, e.g., [26,27,28], so only the key elements are presented here. Let the time series be denoted as $X=(X_1,X_2,\dots ,X_t)$. Typically, $X_i$ is considered an N-dim variable, thus facilitating multivariate time series analysis. However, it should be noted that the present research will be constrained to one-dimensional variables. The output is the corresponding predicted value $Y=(Y_{t+1},Y_{t+2},\dots ,Y_{t+L})$, where L is the horizon of the prediction. In addition to input and output states, the system encompasses data from hidden layers, denoted by $h_t$. The matrices $W_c$ and $b_c$ denote learned weights and biases, while $c_t$ and ${\stackrel{\sim }{c}}_t$ represent the cell state and the candidate layer, respectively. The LSTM layer processes information at each step, calculating a cell state and a hidden state, utilising the following equations

$${\stackrel{\sim }{c}}_t=tanh(W_c[x_t,h_{t-1}]+b_c)$$

(1)

$$c_t=f_t\circ c_{t-1}+i_t\circ {\stackrel{\sim }{c}}_t$$

(2)

$$h_t=o_t\circ tanh\left(c_t\right),$$

(3)

where ∘ denotes pointwise multiplication, $f_t$ and $i_t$ are the forget and input gates, respectively, as shown in Equations (4) and (6). At Equation (1) the standard tanh function was used as the activation function. However, alternative activation functions are used in various applications, see, for example, refs. [29,30]. Naturally, a proper choice of activation function can increase the quality of the fitted model, and we believe that optimisation methods will be developed. However, the main goal is to demonstrate the ability of determinism assessment, so the approach of the universal model (with a chosen activation function) is used. However, if the optimisation method for the choice of activation function is developed, this element can also extend the model.

$$f_t=\sigma (W_f[x_t,h_{t-1}]+b_f)$$

(4)

$$i_t=\sigma (W_i[x_t,h_{t-1}]+b_i)$$

(5)

$$o_t=\sigma (W_o[x_t,h_{t-1}]+b_o)$$

(6)

Here, $W_f$, $W_i$, and $W_o$ are the learned matrices for the forget, input, and output gates. $\sigma$ is the sigmoid activation function.

The quality of the model fit to the data is usually measured by three characteristics: mean square error (MSE) Equation (7), root mean square error (RMSE) Equation (8), mean absolute error (MAE) Equation (9), and the coefficient of determination $R^2$ Equation (10).

$$MSE={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2$$

(7)

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2}$$

(8)

$$MAE={\displaystyle \frac{1}{N}}\sum _{i=1}^N|y_i-{\widehat{y}}_i|$$

(9)

$$R^2=1-{\displaystyle \frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^N{(y_i-\overline{y})}^2}}$$

(10)

where $\overline{y}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{y}_i$, and ${\widehat{y}}_i$ is the value estimated from the model.

The measure Equations (7)–(9) provide comparable information. The significant difference is in the magnitude, particularly between MSE and RMSE. RMSE is defined as the square root of MSE. A further distinction is that the RMSE and MAE are expressed in units consistent with the data. This fact may be significant for analysing experimental (or physical observation) data. Furthermore, the quality of the fitted models may be contingent upon the evaluation matrix. In the present study, the RMSE will be utilised. The $R^2$ measure represents the proportion of variance that has been explained by the independent variables in the model. The $R^2$ measure takes a value from 1 to arbitrary small values, where 1 indicates perfect agreement between the model and the data.

The estimation of model parameters is a subtle problem. Indeed, several methods have been developed for the optimisation of model parameters. These include Adagrad, Adadelta, RMSprop, ADAM, AdaMax, and others [31,32,33]. The optimisation method features can be found in review works, e.g., [32]. However, no rigorous method exists for choosing the “best” algorithm. The efficacy of these methods is contingent upon the specific nature of the problem under investigation. In the context of the present paper, the objective is to ascertain the viability of the AI model as a method for time series determinism estimation. In this regard, the standardisation of the model assumes greater significance than the pursuit of enhanced performance. This study employed the Adagrad, RMSprop, and ADAM optimisation methods. However, further investigation revealed that the ADAM method yielded the lowest root mean square error (RMSE) estimates in the analysed dataset. Nonetheless, this aspect is not considered fundamental and can be adapted if it enhances analysis quality. Undoubtedly, implementing the optimisation algorithm helps streamline the model estimation process and enhance the quality of the generated models.

The subsequent pivotal facet of the AI model design is selecting the regularisation method. It is imperative to recognise that during its training phase, the model is designed to recognise deterministic patterns and incorporate stochastic elements, thereby ensuring the incorporation of these elements into the generated data. This phenomenon is referred to as “overfitting” in the context of a neural network. A proper regularisation usually solves the problem [34,35,36]. However, the objective of this study is not to propose the most optimal model for a specific problem but rather to demonstrate the potential of an AI model as a metric for assessing the determinism level of a time series. Consequently, the dropout technique was employed as a regularisation method. The Optuna framework [37] optimised the dropout probability and the number of neurons.

3. Time Series

The stated hypothesis was verified using a set of examples: AR(2) models, a Lorentz Attractor, and financial time series.

3.1. AR Model

Autoregression (AR) models are one of the very classical time series models [38,39]. The main feature of the AR(p) model is that the state $a_t$ depends on the p previous states, as shown in Equation (11).

$$x_t=a_0+\sum _{i=1}^p{a}_i{x}_{t-i}+{ϵ}_t$$

(11)

$a_i$—model parameters, ${ϵ}_t$—noise, which expectation value equals zero $\langle {ϵ}_t\rangle =0$, and variance $Var\left({ϵ}_t\right)=\sigma$ (constant value). The most important features of AR(p) models are the state at moment t, which depends on p previous states (so this is the deterministic part, because if the ${ϵ}_t=0$ is taken, then the process become purely deterministic), and there is a stochastic part ${ϵ}_t$.

The present study uses the model with $p=2$, so AR(2). The choice of the memory length in AR model was made to balance two opposite factors—the determinism and the noise. Extending the length of the time series would simplify the task for the AI model since it is sensitive to the “patterns” presence; on the other hand, the $p=1$ is relatively trivial. Therefore, the AR(2) model was chosen as the first reasonable model to verify the ability of assessing the determinism of the time series. Naturally, the model can be adopted to any AR(p). On the other hand, the AR(2) model might be problematic for AI model training since it depends only on two previous states, and the noise might cover the determinism of the time series.

3.2. Lorentz Attractor

The deterministic chaotic models [40] are the systems, where despite purely deterministic evolution, the prediction of the state is impossible due to the sensitivity to the initial conditions. One such system is the Lorentz Attractor defined by Equation (12). Lorentz presented the system in [41]. More detailed analysis of the Lorentz Attractor can be found, e.g., [42,43]

$$\left\{\begin{array}{c}{\displaystyle \frac{dx}{dt}}=\sigma (y-x)\hfill \\ {\displaystyle \frac{dy}{dt}}=x(r-z)-y\hfill \\ {\displaystyle \frac{dz}{dt}}=xy-bz\hfill \end{array}\right.$$

(12)

For parameter $\sigma =10, r=28, b={\displaystyle \frac{8}{3}}$, the system shows chaotic behaviour. The example of the system evolution is presented in Figure 2. In the present study, the determinism problem of the time series is investigated for the one-dimensional system; therefore, the Lorenz Attractor system, although 3D, is analysed as one-dimensional, in which the evolution of one of the coordinates is modelled by LSTM and the quality of the model is verified.

3.3. Financial Time Series

The last type of data verified here is the financial time series. This is a very special type of data. In this field, despite massive effort and numerous models, e.g., [44,45,46,47], financial markets are still unpredictable. Of course, scholars cannot consider the lack of a reliable method for stock value forecasting a failure because, through theory development, we understand the stock market issue much better. However, we cannot claim that the system is deterministic. The stochastic nature of stock markets is a natural consequence of being a complex system in which human decisions play a key role.

This study used two time series: the daily close exchange rate of USD/EUR (FOREX market) and Microsoft’s daily close price (MSFT) of the New York stock exchange. The most important features that determined the choice of the examples were the liquidity of those markets and their importance to the global market. The data were downloaded from the Yahoo Finance web page, covering 2022/2023. The evolution of EUR/USD is presented in Figure 3, while the evolution of MSFT daily close price can be seen in Figure 4.

The descriptive statistics of the analysed data are presented in

Table 1. The basic statistical characteristic of the financial data.

	Mean Value	Max	Min	Var	Median
USD/EUR	1.0677	1.1457	0.9596	0.0016	1.0730
MSFT	291.39 USD	382.70 USD	214.25 USD	1676.01 USD2	287.18 USD

. However, we do not discuss the economic background and evolution of the considered time series, as it is beyond the scope of the present study.

4. LSTM Implementation (Results)

The LSTM model requires splitting the available data into three sets: training, validation, and testing. The proportion can be arbitrary, depending on the availability of the data. In the case of artificial data, the splitting is less important because the data can be easily generated. However, in the case of real data, there are significant limits due to the size of the available data. To assess the quality of trained models, the following data split pattern was used: 60% were used for training, 20% for validating the model during training, and 20% for testing the model. For all the considered time series, the same splitting pattern is used.

4.1. AR(2) Model

The AR(2) model allows for generating an arbitrary amount of data. However, generating extremely long time series would significantly lower the model convergence’s performance, since the time series memory is restricted to two consecutive time steps. Therefore, the time series of 5000 data points was used, the model’s training was performed on 3000 data points, and the test was conducted on 1000 data points. The main task in this part was verifying the quality of the model trained concerning the noise level in the time series. The following AR(2) models were verified: the memory coefficient $a_1=a_2\in \{0.15,0.25,0.35,0.45\}$ and the noise standard deviation $\sigma \in \{0.2,0.5,0.8\}$. The assumed values of the AR(2) parameters gave 12 different time series which represented a low dependence of past values $a_1=a_2=0.15$ and low noise $\sigma =0.2$ up to a high dependence of the past values $a_1=a_2=0.45$ and high value standard deviation $\sigma =0.8$. It is important to notice that, due to the definition Equation (11), the past noise impulses are multiplied by the $a_1$ and $a_2$ coefficients.

The applied optimisation process of the single-layer LSTM model allows setting the models trained on AR(2) time series for different parameters. The model’s details and quality are measured as RMSE, which are presented in

Table 2. Parameter of the optimised single-layer LSTM models trained for the AR(2) time series generated for different memory coefficients $a_1,a_2$ and the noise level $\sigma$. The quality of the trained single-layer LSTM model is presented in the last column.

$\mathit{\sigma }$	${\mathit{a}}_1={\mathit{a}}_2$	Neuron Number	Dropout Rate	RMSE	MAE	${\mathit{R}}^2$
0.2	0.15	324	0.003	0.206	0.168	0.059
	0.25	63	0.054	0.206	0.168	0.179
	0.35	196	0.072	0.207	0.168	0.391
	0.45	103	0.134	0.206	0.168	0.766
0.5	0.15	229	0.035	0.482	0.389	−0.004
	0.25	61	0.441	0.478	0.385	0.086
	0.35	22	0.242	0.478	0.385	0.249
	0.45	224	0.228	0.477	0.384	0.632
0.8	0.15	228	0.450	0.791	0.622	0.037
	0.25	153	0.238	0.791	0.623	0.148
	0.35	27	0.050	0.794	0.623	0.342
	0.45	161	0.104	0.790	0.620	0.691

. For clarity, the result are also presented as Figure 5.

The most important observation of RMSE of the trained models in Table 2 is that the quality of the models decreases with the noise level (defined by $\sigma$). It is noticeable that the AR(q) time series combines the deterministic part with the stochastic noise. Here, the value of the memory coefficient does not influence the quality of the trained models. In the case of low noise $\sigma =0.2$, RMSE is ≈0.21. A much higher RMSE is observed for high noise $\sigma =0.8$ for which RMSE $\approx 0.8$. Naturally, the noise disturb the possibility to recognise the determinism of the time series as it is an internal part of the time series—the consecutive time series element is calculated not only on the deterministic part but also takes into account the noise of the previous point (compared to Equation (11)). Besides the clear observation of the standard deviation influence, one can also observe that the $R^2$ coefficient increases as the value of the memory parameters increases. So, the quality of the trained model depends on the determinism and the randomness of the investigated time series.

4.2. Lorentz Attractor

Despite its chaotic nature, the Lorentz Attractor is a fully deterministic system, which renders it challenging to forecast. This is particularly the case due to its sensitivity to the initial conditions. However, as a theoretical problem, it is possible to generate sufficiently long time series to train and verify the quality of the model.

The model generates a trajectory in ${\Re }^3$ space, so it was expressed as X, Y, Z time series and used as three features for the model. For the analysis, ${10}^4$ iteration steps were generated, using a 20-point lookback length. Six thousand four hundred data points were used for training with the overlapping samples, and the model was verified on 2000 data points. Two types of LSTM networks were generated: one-layer LSTM and two-layer LSTM. And the model quality was verified in the X and Z coordinate time series. The details of optimised models and the quality of prediction are presented in

Table 3. The results of Lorentz Attractor models. The quality of the trained LSTM models is presented in the last column.

Type of Network, Coordinate	Neuron Number	Dropout Rate	RMSE	MAE	${\mathit{R}}^2$
one layer, X	23	0.003	0.021	0.015	1
two layers, X	151; 93	0.002	0.044	0.029	1
one layer, Z	156	0.018	0.324	0.309	0.999

and as Figure 6. The two-layer model was used because the training of the one-layer model required significant computational time, and we decided to verify the results on the two-layer model.

All the fitted models have low RMSE values, even below the RMSE for AR(2) models. So, despite the chaotic and nonlinear system, the quality of the fitted model is better than that of the low-noise AR(2). Particularly interesting are the values of $R^2$ measures, which are practically equal one, that indicate perfect agreement between the model and the analysed data.

4.3. Financial Time Series

The main difficulty in model estimation for financial time series is the short time series. For the exchange rate of EUR/USD, the time series consists of 520 data points, while the MSFT time series has 501 data points. These values are significantly smaller than the length of the generated time series AR and Lorentz Attractor. As a consequence of the limited size of the data, the model was trained on 312 data points.

The details and quality measure of the fitted model are presented in

Table 4. The quality of one-layer LSTM models trained on the financial time series EUR/USD and MSFT. The quality of the trained one-layer LSTM model is presented in the last column.

Time Series	Neuron Number	Dropout Rate	RMSE	MAE	${\mathit{R}}^2$
EUR/USD	186	0.322	0.008	0.005	0.860
MSFT	138	0.166	4.720	3.754	0.958

and in Figure 7. The quality of the LSTM model trained on the exchange rate EUR/USD is surprisingly low. However, it should be compared with the value within the time series. The range of the values observed is in the interval (0.9596, 1.1457). The difference between two consecutive data points is in the interval $(1.3·{10}^{-5},0.0192)$, and the averaged increment is 0.0045. Therefore, RMSE = 0.008 means that the relative error of prediction is approximately 150%, which is high. Analogously, the observed values for MSFT stocks are in the interval (214.25, 382.70), and the increments are in (0.01, 19.95), with an average value of 4.16. Comparing RMSE, it gives the relative error of prediction approximately 90%, so, once again, very high. It should be noticed that the trained model are characterised as relatively high values of the $R^2$ measures 0.860 and 0.958. One important point is that the considered time series are short (particularly comparing with the model time series), so it influences the model quality; on the other hand, the $R^2$ measure focuses on the variance. Indeed, the intervals for the analysis were chosen beyond the extreme event on the stock market, aside from bot time series which belongs to the highly liquid reflected in high $R^2$ as a result.

5. Conclusions and Future Work

5.1. Key Contributions

• This study introduces a novel application of deep learning models—specifically LSTM networks—as a metric for assessing determinism in time series data.
• Unlike traditional uses of AI for prediction or classification, this research inverts the paradigm by using the model’s performance as an indicator of the underlying structure in the data.
• Theoretical insights are provided regarding the deterministic nature of AI models, emphasizing their reliance on pattern recognition and the importance of avoiding overfitting.
• The approach was empirically validated on three types of time series:

  –

  Deterministic with noise: AR(2) model—performance degraded with noise.

  –

  Deterministic: Lorenz attractor—accurately reconstructed.

  –

  Stochastic: Financial time series—poor model performance, supporting the hypothesis.

• The quality of the LSTM model, measured via RMSE, MAE, and $R^2$, correlates with the degree of determinism in the time series.

5.2. Shortcomings

• The process of constructing AI models is not yet standardised, which may affect reproducibility and scalability.
• Designing optimal architectures remains a complex task, even though an optimization algorithm was employed.
• This study primarily used RMSE as the determinism indicator; however, this may not capture all aspects of model quality or data structure.

5.3. Future Prospects

• Develop a standardised pipeline for generating and validating AI models for determinism analysis.
• Develop a standardised metric for determinism analysis.