# Evaluating Temporal Correlations in Time Series Using Permutation Entropy, Ordinal Probabilities and Machine Learning

^{1}

^{2}

^{3}

^{4}

^{5}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methodology

- 1.
- Calculate the ordinal probabilities (OPs) of a large set of FN time series generated with different values of $\alpha $ and use them as features to train a ML algorithm to return the (known) value of ${\alpha}_{\mathrm{e}}$;
- 2.
- Calculate the OPs of the time series of interest, $x\left(t\right)$, and use them as features to the trained ML algorithm, which returns a value ${\alpha}_{e}$ (see Section 2.1);
- 3.
- Generate a FN time series with $\alpha ={\alpha}_{e}$ and calculate its permutation entropy (PE), ${\overline{S}}_{\mathrm{FN}}$ (see Section 2.2);
- 4.
- Calculate the relative difference, $\mathrm{\Omega}$, between the PE of $x\left(t\right)$, $\overline{S}$ and ${\overline{S}}_{\mathrm{FN}}$:$$\mathrm{\Omega}=\frac{|{\overline{S}}_{\mathrm{FN}}-\overline{S}|}{{\overline{S}}_{\mathrm{FN}}};$$
- 5.
- Use the value of ${\alpha}_{e}$ to quantify the strength of the temporal correlations in the time series of interest and use the value of $\mathrm{\Omega}$ to identify underlying determinism: if $\mathrm{\Omega}\approx 0$, $x\left(t\right)$ is mainly stochastic, otherwise there is some determinism.

#### 2.1. Machine Learning Algorithm

#### 2.2. Ordinal Analysis and Permutation Entropy

#### 2.3. Quantifiers of Chaos and Complexity

## 3. Datasets

#### 3.1. Flicker Noise

#### 3.2. Uniform Noise

#### 3.3. Random Walk

#### 3.4. Periodic or Chaotic Signals Contaminated by Noise

#### 3.5. Logistic Map

#### 3.6. $\beta x$ Map

#### 3.7. Schuster Map

#### 3.8. Lorenz System

#### 3.9. Rossler System

#### 3.10. Three Waves System

#### 3.11. Hindmarsh–Rose Model

## 4. Results

#### 4.1. Comparison with Standard Quantifiers

#### 4.2. Influence of the Length of the Time Series

#### 4.3. Analysis of Periodic Signals Contaminated by Noise

#### 4.4. Analysis of Two Stochastic Processes

## 5. Discussions and Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Huang, N.E.; Shen, Z.; Long, S.R.; Wu, M.L.C.; Shih, H.H.; Zheng, Q.N.; Yen, N.C.; Tung, C.C.; Liu, H.H. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proc. R. Soc. A
**1998**, 454, 903–995. [Google Scholar] [CrossRef] - Mallat, S. A Wavelet Tour of Signal Processing; Academic Press: Cambridge, MA, USA, 1998. [Google Scholar]
- Kantz, H.; Schreiber, T. Nonlinear Time Series Analysis; Cambridge University Press: Cambridge, UK, 2005. [Google Scholar]
- Brunton, S.L.; Kutz, J.N. Data-driven Science and Engineering: Machine Learning, Dynamical Systems, and Control; Cambridge University Press: Cambridge, UK, 2019. [Google Scholar]
- Fulcher, B.D.; Jones, N.S. A computational framework for automated time-series phenotyping using massive feature extraction. Cell Syst.
**2017**, 5, 527–531. [Google Scholar] [CrossRef] - Zou, Y.; Donner, R.V.; Marwan, N.; Donges, J.F.; Kurths, J. Complex network approaches to nonlinear time series analysis. Phys. Rep.
**2019**, 787, 1–97. [Google Scholar] [CrossRef] - Carlsson, G. Topological methods for data modelling. Nat. Rev. Phys.
**2020**, 2, 697–708. [Google Scholar] [CrossRef] - Lancaster, G.; Iatsenko, D.; Pidde, A.; Ticcinelli, V.; Stefanovska, A. Surrogate data for hypothesis testing of physical systems. Phys. Rep.
**2018**, 748, 1–60. [Google Scholar] [CrossRef] - Boaretto, B.R.R.; Budzinski, R.C.; Rossi, K.L.; Prado, T.L.; Lopes, S.R.; Masoller, C. Discriminating chaotic and stochastic time series using permutation entropy and artificial neural networks. Sci. Rep.
**2021**, 11, 15789. [Google Scholar] [CrossRef] - Beran, J.; Feng, Y.; Ghosh, S.; Kulik, R. Long-Memory Processes; Springer: Berlin/Heidelberg, Germany, 2016. [Google Scholar]
- Bandt, C.; Pompe, B. Permutation entropy: A natural complexity measure for time series. Phys. Rev. Lett.
**2002**, 88, 174102. [Google Scholar] [CrossRef] - Rosso, O.A. Permutation Entropy & Its Interdisciplinary Applications. Available online: https://www.mdpi.com/journal/entropy/special_issues/Permutation_Entropy (accessed on 4 August 2021).
- Tenenbaum, J.B.; De Silva, V.; Langford, J.C. A global geometric framework for nonlinear dimensionality reduction. Science
**2000**, 290, 2319–2323. [Google Scholar] [CrossRef] [PubMed] - Maaten, L.V.D.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res.
**2008**, 9, 2579–2605. [Google Scholar] - Van Der Maaten, L.; Postma, E.; Van den Herik, J. Dimensionality reduction: A comparative review. J. Mach. Learn. Res.
**2009**, 10, 66–71. [Google Scholar] - Rosso, O.A.; Larrondo, H.A.; Martin, M.T.; Plastino, A.; Fuentes, M.A. Distinguishing noise from chaos. Phys. Rev. Lett.
**2007**, 99, 154102. [Google Scholar] [CrossRef] [Green Version] - Olivares, F.; Plastino, A.; Rosso, O. Contrasting chaos with noise via local versus global information quantifiers. Phys. Lett. A
**2012**, 376, 1577–1583. [Google Scholar] [CrossRef] - Ravetti, M.G.; Carpi, L.C.; Gonçalves, B.A.; Frery, A.C.; Rosso, O.A. Distinguishing noise from chaos: Objective versus subjective criteria using horizontal visibility graph. PLoS ONE
**2014**, 9, e108004. [Google Scholar] [CrossRef] - Spichak, D.; Kupetsky, A.; Aragoneses, A. Characterizing complexity of non-invertible chaotic maps in the Shannon-Fisher information plane with ordinal patterns. Chaos Solitons Fractals
**2021**, 142, 110492. [Google Scholar] [CrossRef] - Daw, C.S.; Finney, C.E.A.; Tracy, E.R. A review of symbolic analysis of experimental data. Rev. Sci. Inst.
**2003**, 74, 913–930. [Google Scholar] [CrossRef] - Corso, G.; Prado, T.d.L.; Lima, G.Z.S.; Kurths, J.; Lopes, S.R. Quantifying entropy using recurrence matrix microstates. Chaos Interdiscip. Nonlinear Sci.
**2018**, 28, 083108. [Google Scholar] [CrossRef] [PubMed] - Tsallis, C. Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys.
**1988**, 52, 479–487. [Google Scholar] [CrossRef] - Richman, J.S.; Moorman, J.R. Physiological time-series analysis using approximate entropy and sample entropy. Am. J. Physiol.
**2000**, 278, H2039–H2049. [Google Scholar] [CrossRef] [Green Version] - Prado, T.L.; Corso, G.; dos Santos Lima, G.Z.; Budzinski, R.C.; Boaretto, B.R.R.; Ferrari, F.A.S.; Macau, E.E.N.; Lopes, S.R. Maximum entropy principle in recurrence plot analysis on stochastic and chaotic systems. Chaos Interdiscip. Nonlinear Sci.
**2020**, 30, 043123. [Google Scholar] [CrossRef] [PubMed] - Martin, M.T.; Plastino, A.; Rosso, O.A. Generalized statistical complexity measures: Geometrical and analytical properties. Physica A
**2006**, 369, 439–462. [Google Scholar] [CrossRef] - Parlitz, U.; Berg, S.; Luther, S.; Schirdewan, A.; Kurths, J.; Wessel, N. Classifying cardiac biosignals using ordinal pattern statistics and symbolic dynamics. Comput. Biol. Med.
**2012**, 42, 319–327. [Google Scholar] [CrossRef] - Ott, E. Chaos in Dynamical Systems; Cambridge University Press: Cambridge, UK, 2002. [Google Scholar]
- Mitschke, F.; Damming, M. Chaos vs. noise in experimental data. Int. J. Bifurc. Chaos
**1993**, 3, 693–702. [Google Scholar] [CrossRef] - Rosso, O.A.; Masoller, C. Detecting and quantifying stochastic and coherence resonances via information-theory complexity measurements. Phys. Rev. E
**2009**, 79, 040106. [Google Scholar] [CrossRef] [Green Version] - Library to Generate a Flicker Noise. Available online: https://github.com/felixpatzelt/colorednoise (accessed on 4 August 2021).
- Timmer, J.; Koenig, M. On generating power law noise. Astron. Astrophys.
**1995**, 300, 707. [Google Scholar] - Schuster, H.G.; Just, W. Deterministic Chaos: An Introduction; John Wiley & Sons: Hoboken, NJ, USA, 2006. [Google Scholar]
- Lorenz, E.N. Deterministic nonperiodic flow. J. Atmos. Sci.
**1963**, 20, 130–141. [Google Scholar] [CrossRef] [Green Version] - Rössler, O.E. An equation for continuous chaos. Phys. Lett. A
**1976**, 57, 397–398. [Google Scholar] [CrossRef] - Lopes, S.; Chian, A.L. Controlling chaos in nonlinear three-wave coupling. Phys. Rev. E
**1996**, 54, 170. [Google Scholar] [CrossRef] - Hindmarsh, J.L.; Rose, R.M. A model of neuronal bursting using three coupled first order differential equations. Proc. R. Soc. Lond. B Biol. Sci.
**1984**, 221, 87–102. [Google Scholar] - Little, D.J.; Kane, D.M. Permutation entropy of finite-length white-noise time series. Phys. Rev. E
**2016**, 94, 022118. [Google Scholar] [CrossRef] - De Micco, L.; Fernández, J.G.; Larrondo, H.A.; Plastino, A.; Rosso, O.A. Sampling period, statistical complexity, and chaotic attractors. Phys. A Stat. Mech. Its Appl.
**2012**, 391, 2564–2575. [Google Scholar] [CrossRef] - Zunino, L.; Soriano, M.C.; Fischer, I.; Rosso, O.A.; Mirasso, C.R. Permutation-information-theory approach to unveil delay dynamics from time-series analysis. Phys. Rev. E
**2010**, 82, 046212. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Aragoneses, A.; Carpi, L.; Tarasov, N.; Churkin, D.V.; Torrent, M.C.; Masoller, C.; Turitsyn, S.K. Unveiling Temporal Correlations Characteristic of a Phase Transition in the Output Intensity of a Fiber Laser. Phys. Rev. Lett.
**2016**, 116, 033902. [Google Scholar] [CrossRef] [Green Version] - Lopes, S.R.; Prado, T.d.L.; Corso, G.; Lima, G.Z.d.S.; Kurths, J. Parameter-free quantification of stochastic and chaotic signals. Chaos Solitons Fractals
**2020**, 133, 109616. [Google Scholar] [CrossRef] [Green Version] - Repository with the ANN. Available online: https://github.com/brunorrboaretto/chaos_detection_ANN/ (accessed on 4 August 2021).
- Uchida, A.; Amano, K.; Inoue, M.; Hirano, K.; Naito, S.; Someya, H.; Oowada, I.; Kurashige, T.; Shiki, M.; Yoshimori, S. Fast physical random bit generation with chaotic semiconductor lasers. Nat. Photonics
**2008**, 2, 728–732. [Google Scholar] [CrossRef] - Sakuraba, R.; Iwakawa, K.; Kanno, K.; Uchida, A. Tb/s physical random bit generation with bandwidth-enhanced chaos in three-cascaded semiconductor lasers. Opt. Express
**2015**, 23, 1470–1490. [Google Scholar] [CrossRef] - Zhang, L.; Pan, B.; Chen, G.; Guo, L.; Lu, D.; Zhao, L.; Wang, W. 640-Gbit/s fast physical random number generation using a broadband chaotic semiconductor laser. Sci. Rep.
**2017**, 7, 45900. [Google Scholar] [CrossRef] - Oliver, N.; Soriano, M.C.; Sukow, D.W.; Fischer, I. Dynamics of a semiconductor laser with polarization-rotated feedback and its utilization for random bit generation. Opt. Lett.
**2011**, 36, 4632–4634. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zhang, J.; Feng, C.; Zhang, M.; Liu, Y.; Zhang, Y. Suppression of Time Delay Signature Based on Brillouin Backscattering of Chaotic Laser. IEEE Photon J.
**2017**, 9, 1502408. [Google Scholar] [CrossRef] - Li, S.S.; Chan, S.C. Chaotic Time-Delay Signature Suppression in a Semiconductor Laser With Frequency-Detuned Grating Feedback. IEEE J. Sel. Top. Quantum Electron.
**2015**, 21, 541–552. [Google Scholar] - Correa-Mena, A.G.; Lee, M.W.; Zaldívar-Huerta, I.E.; Hong, Y.; Boudrioua, A. Investigation of the Dynamical Behavior of a High-Power Laser Diode Subject to Stimulated Brillouin Scattering Optical Feedback. IEEE J. Quantum Electron.
**2020**, 56, 1–6. [Google Scholar] [CrossRef] - Bouchez, G.; Malica, T.; Wolfersberger, D.; Sciamanna, M. Optimized properties of chaos from a laser diode. Phys. Rev. E
**2021**, 103, 042207. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**(

**a**,

**b**) Uniformly distributed white noise and its probability distribution function (PDF). (

**c**,

**d**) Time series generated by iteration of the $\beta x$ map with $\beta =2$ and its PDF. We observe that the deterministic map depicts a very similar PDF of the white noise. (

**e**) Power spectral density (PSD) of flicker noise (FN) $\alpha =0$ (black), $\alpha =1$ (magenta) and $\alpha =2$ (red). By definition, the PSD of the FN decays as $1/{f}^{\alpha}$. (

**f**,

**g**) Time evolution of two continuous chaotic systems: (

**f**) $\left|C1\right|$ of the three-waves system and (

**g**) x of the Hindmarsh–Rose model.

**Figure 2.**Characterization of stochastic and chaotic time series in the plane (normalized permutation entropy $\overline{S}$; correlation coefficient $\alpha $ returned by the ML algorithm). Panel (

**a**) presents results for discrete systems. The black line represents a set of 10,000 FN signals and $\overline{S}$ decreases as the temporal correlation increases. The stochastic cases (uniform noise and random walk) are very close to the FN signals. Otherwise, the chaotic signal ($\beta x$, logistic and Schuster maps signals) depict a lower $\overline{S}$ compared to the FN signals. Panel (

**b**) presents results for continuous systems. Here, we analyze the sequence of maxima of each variable and, again, we observe that the vertical distance to the FN curve reveals that the time series are not fully stochastic. All time series posses $N={2}^{14}$ points and the error bars represent the standard deviation over 1000 time series generated with different initial conditions or noise seeds. Small error bars are not shown.

**Figure 3.**Analysis of deterministic signals generated by chaotic maps. First column depicts the normalized permutation entropy $\overline{S}$, the complexity C (left vertical scale) and the Lyapunov exponent $\lambda $ (right scale). Second column depicts the new quantifiers ${\alpha}_{\mathrm{e}}$ (left scale) and $\mathrm{\Omega}$ (right scale). The third column presents $\overline{S}$ vs. $\mathrm{\Omega}$; the color codes represent the control parameter of each map. Panels (

**a**–

**c**) illustrate the case of $\beta x$ map as a function of $\beta $; (

**d**–

**f**) show the Logistic map as a function of r; and (

**g**–

**i**) depict the Schuster map as a function of z.

**Figure 4.**The role of time series length N is studied using $\overline{S}$ (

**a**), the output of the ANN ${\alpha}_{\mathrm{e}}$ (

**b**) and $\mathrm{\Omega}$ (

**c**) for both chaotic and stochastic signals. The results are stable for $N\ge {10}^{4}$, indicating the robustness of our methodology. Here, we analyze $\beta x$ time series and also uniform noise and random walk signals. Despite having similar values of $\overline{S}$, our method is able to distinguish between the chaotic and stochastic cases even for $N\le D!=720$ (number of ordinal patterns) for $\beta =2$ and $\beta =3$. However, for $\beta =10$ the chaoticity is too high ($\lambda =ln10$) therefore existing a great local divergence of the trajectory. In this case, the deterministic nature cannot be detected since the ordinal probability distribution is as uniform as a distribution of a white noise signal with $\overline{S}\approx 1$.

**Figure 5.**Robustness of the method to the application of additive noise in the $\beta x$ map. The upper row (

**a**–

**c**) depicts the ${\alpha}_{\mathrm{e}}$ values in color-code as a function of the percentage of noise $\eta $ and the series’ size N for $\beta =2,3,10$, respectively. The lower row (

**d**–

**f**) shows $\mathrm{\Omega}$ for the same simulations.

**Figure 6.**Analysis of periodic signals contaminated with white noise using the temporal correlation ${\alpha}_{\mathrm{e}}$ (

**a**) and the quantifier $\mathrm{\Omega}$ (

**b**). The results show that, when the noise strength, $\eta $, increases, the deterministic nature of the periodic signal gradually vanishes and, for large enough $\eta $, only stochastic dynamics is identified. However, the frequency $\omega =2\pi /\tau $ of the signal is important, because low frequency signals are identified as stochastic at lower noise levels. Moreover, even when the signal is characterized as stochastic, a nonzero temporal correlation can be estimated.

**Figure 7.**The effects of noise on a periodic signal do not depend on the noise’s specific realization. The error bars indicate the dispersion over 1000 analyses with different realization of the added noise. Here, the upper row represents three examples of ${\alpha}_{\mathrm{e}}$ (

**a**) and $\mathrm{\Omega}$ (

**b**) as a function of $\eta $ for a fixed value of $\omega $. The lower row (

**c**,

**d**) shows three examples as a function of $\omega $ for fixed values of $\eta $ following the same representation. The dispersion in all cases is sufficiently low that the trends discussed in Figure 6 remain.

**Figure 8.**Analysis of FN time series contaminated with uniform noise. (

**a**) The addition of uniform noise decreases the ${\alpha}_{\mathrm{e}}$ predicted by the ML algorithm, leading from the value $\alpha $ of the FN time series to value to ${\alpha}_{\mathrm{e}}=0$ for uniform noise. $\eta $ also decreases the value of $\mathrm{\Omega}$ (

**b**), revealing the increase in the degree of stochasticity of the time series. The addition of uniform noise is, thus, similar in FN and in deterministic time series.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Boaretto, B.R.R.; Budzinski, R.C.; Rossi, K.L.; Prado, T.L.; Lopes, S.R.; Masoller, C.
Evaluating Temporal Correlations in Time Series Using Permutation Entropy, Ordinal Probabilities and Machine Learning. *Entropy* **2021**, *23*, 1025.
https://doi.org/10.3390/e23081025

**AMA Style**

Boaretto BRR, Budzinski RC, Rossi KL, Prado TL, Lopes SR, Masoller C.
Evaluating Temporal Correlations in Time Series Using Permutation Entropy, Ordinal Probabilities and Machine Learning. *Entropy*. 2021; 23(8):1025.
https://doi.org/10.3390/e23081025

**Chicago/Turabian Style**

Boaretto, Bruno R. R., Roberto C. Budzinski, Kalel L. Rossi, Thiago L. Prado, Sergio R. Lopes, and Cristina Masoller.
2021. "Evaluating Temporal Correlations in Time Series Using Permutation Entropy, Ordinal Probabilities and Machine Learning" *Entropy* 23, no. 8: 1025.
https://doi.org/10.3390/e23081025