A Super Fast Algorithm for Estimating Sample Entropy

Abstract

Sample entropy, an approximation of the Kolmogorov entropy, was proposed to characterize complexity of a time series, which is essentially defined as $-log(B/A)$, where B denotes the number of matched template pairs with length m and A denotes the number of matched template pairs with $m+1$, for a predetermined positive integer m. It has been widely used to analyze physiological signals. As computing sample entropy is time consuming, the box-assisted, bucket-assisted, x-sort, assisted sliding box, and kd-tree-based algorithms were proposed to accelerate its computation. These algorithms require $O\left(N^2\right)$ or $O\left(N^{2-\frac{1}{m+1}}\right)$ computational complexity, where N is the length of the time series analyzed. When N is big, the computational costs of these algorithms are large. We propose a super fast algorithm to estimate sample entropy based on Monte Carlo, with computational costs independent of N (the length of the time series) and the estimation converging to the exact sample entropy as the number of repeating experiments becomes large. The convergence rate of the algorithm is also established. Numerical experiments are performed for electrocardiogram time series, electroencephalogram time series, cardiac inter-beat time series, mechanical vibration signals (MVS), meteorological data (MD), and $1/f$ noise. Numerical results show that the proposed algorithm can gain 100–1000 times speedup compared to the kd-tree and assisted sliding box algorithms while providing satisfactory approximate accuracy.

1. Introduction

Kolmogorov entropy is a well-suited measure for the complexity of dynamical systems containing noises. Approximate entropy (AppEn), proposed by Pincus [1], is an approximation of the Kolmogorov entropy. To overcome the biasedness of AppEn caused by self-matching, Richman proposed sample entropy (SampEn) [2] in 2000. SampEn is essentially defined as $-log(B/A)$, where B denotes the number of matched template pairs with length m and A denotes the number of matched template pairs with $m+1$. SampEn has prevailed in many areas, such as cyber-physical systems, mechanical systems, health monitoring, disease diagnosis, and control. Based on AppEn and SampEn, multiscale entropy [3] and hierarchical entropy [4] were developed for measuring the complexity of physiological time series in multiple time scales. Since low-frequency filters are involved, multiscale entropy can weaken the influence of meaningless structures such as noise on complexity measurement. By adding the sample entropy of the high-frequency component of the time series, the hierarchical entropy provides more comprehensive and accurate information and improves the ability to distinguish different time series. Multiscale entropy, hierarchical entropy, and their variants have been applied to various fields such as fault identification [5,6] and feature extraction [7], beyond physiological time series analysis.

Computing SampEn requires counting the number of similar templates of time series. In other words, it requires counting the number of matched template pairs for a given time series. Clearly, direct computing of SampEn requires computational complexity of $O\left(N^2\right)$, where N is the length of the time series analyzed. To accelerate the computation of SampEn, kd-tree based algorithms for sample entropy were proposed, which reduce the time complexity to $O\left(N^{2-\frac{1}{m+1}}\right)$, where m is the template (also called pattern) length [8,9]. In addition, box-assisted [10,11], bucket-assisted [12], lightweight [13], and assisted sliding box (SBOX) [14] algorithms were developed. However, the complexity of all these algorithms is $O\left(N^2\right)$. Recently, an algorithm proposed in [15] for computing approximate values of AppEn and SampEn, without theoretical error analysis, still requires $O\left(N^2\right)$ computational costs in the worst scenario, even though it requires only $O\left(N\right)$ number of operations in certain best cases. Developing fast algorithms for estimating SampEn is still of great interest.

The goal of this study is to develop a Monte-Carlo-based algorithm for calculating SampEn. The most costly step in computing SampEn is to compute the matched template ratio $B/A$ of length m over length $m+1$. Noting that $\frac{A}{N(N-1)}$ (resp. $\frac{B}{N(N-1)}$) is the probability that templates of length m (resp. $m+1$) are matched, the ratio $B/A$ can be regarded as a conditional probability. From this viewpoint, we can approximate this conditional probability of the original data set by that of a data set randomly down-sampled from the original one. Specifically, we randomly select $N_0$ templates of lengths m and $N_0$ templates of $m+1$ from the original time series. We then count the number $\tilde{A}$ (resp. $\tilde{B}$) of matched pairs among the selected templates of lengths m (resp. $m+1$). We repeat this process $N_1$ times, and compute the mean ${\overline{A}}_{N_1}$ (resp. ${\overline{B}}_{N_1}$) of $\tilde{A}$ (resp. $\tilde{B}$). Then, we use $-log({\overline{B}}_{N_1}/{\overline{A}}_{N_1})$ to approximate $-log(B/A)$ for the time series to measure its complexity. We establish the computational complexity and convergence rate of the proposed algorithm. We then study the performance of the proposed algorithm, by comparing it with the kd-tree-based algorithm and the SBOX method on the electrocardiogram (ECG) time series, electroencephalogram time series (EEG), cardiac inter-beat (RR) time series, mechanical vibration signals (MVS), meteorological data (MD), and $1/f$ noise. Numerical results show that the proposed algorithm can gain more than 100 times speedup compared to the SBOX algorithm (the most recent algorithm in the literature to the best of our knowledge) for a time series of length $2^{16}-2^{18}$, and more than 1000 times speedup for a time series of length $2^{19}-2^{20}$. Compared to the kd-tree algorithm, the proposed algorithm can again achieve up to 1000 times speedup for a time series of length $2^{20}$.

This article is organized in five sections. The proposed Monte-Carlo-based algorithm for estimating sample entropy is described in Section 2. Section 3 includes the main results of the analysis of approximate accuracy of the proposed algorithm, and the proofs are given in the Appendix A. Numerical results are presented in Section 4, and conclusion remarks are made in Section 5.

2. Sample Entropy via Monte Carlo Sampling

In this section, we describe a Monte-Carlo-based algorithm for estimating the sample entropy of a time series.

We first recall the definition of sample entropy. For all $k\in \mathbb{N}$, let ${\mathbb{Z}}_k:=\{1,2,\dots ,k\}$. The distance of two real vectors $\mathbf{a}:=[a_l:l\in {\mathbb{Z}}_k]$ and $\mathbf{b}:=[b_l:l\in {\mathbb{Z}}_k]$ of length k is defined by

$$\rho (\mathbf{a},\mathbf{b}):=max\left\{\right|a_l-b_l|:l\in {\mathbb{Z}}_k\}.$$

We let $\mathbf{u}:=(u_i\in \mathbb{R}:i\in {\mathbb{Z}}_n)$ be a time series of length $n\in \mathbb{N}$. For $m\in \mathbb{N}$, we let $N:=n-m-1$. We define a set X of N vectors by $X:=\left\{{\mathbf{x}}_i:i\in {\mathbb{Z}}_N\right\}$, where ${\mathbf{x}}_i:=\left[u_{i+l-1}:l\in {\mathbb{Z}}_m\right]$ is called a template of length m for the time series $\mathbf{u}$. We also define a set Y of N vectors by $Y:=\left\{{\mathbf{y}}_i:i\in {\mathbb{Z}}_N\right\}$, where ${\mathbf{y}}_i:=\left[u_{i+l-1}:l\in {\mathbb{Z}}_{m+1}\right]$ is called a template of length $m+1$ for $\mathbf{u}$. To avoid confusion, we call the elements in X and Y the templates for the time series $\mathbf{u}$. We denote by $\#E$ the cardinality of a set E. We use $A_i$, $i\in {\mathbb{Z}}_N$, to denote the cardinality of the set consisting of templates $\mathbf{x}\in X \backslash \left\{{\mathbf{x}}_i\right\}$ satisfying $\rho ({\mathbf{x}}_i,\mathbf{x})\le r$, that is,

$$A_i:=\#\left\{\mathbf{x}\in X \backslash \left\{{\mathbf{x}}_i\right\}:\rho ({\mathbf{x}}_i,\mathbf{x})\le r\right\}.$$

Likewise, for $i\in {\mathbb{Z}}_N$, we let

$$B_i:=\#\left\{\mathbf{y}\in Y \backslash \left\{{\mathbf{y}}_i\right\}:\rho ({\mathbf{y}}_i,\mathbf{y})\le r\right\}.$$

Letting

$$B:=\frac{1}{2}\sum _{i\in {\mathbb{Z}}_N}{B}_i\mathrm{and}A:=\frac{1}{2}\sum _{i\in {\mathbb{Z}}_N}{A}_i,$$

we define the sample entropy of time series $\mathbf{u}$ by

$$\begin{array}{c}\hfill \mathrm{SampEn}(\mathbf{u},m,r):=\left\{\begin{array}{c}-log\left(\frac{B}{A}\right),\mathrm{if}A>0,B>0,\hfill \\ -log\left(\frac{2}{N(N-1)}\right),\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

The definition of sample entropy yields the direct algorithm, which explicitly utilizes two nested loops, where the inner one computes $A_i$ and $B_i$, and the outer one computes A and B. Algorithm 1 will be called repeatedly in the Monte-Carlo-based algorithm to be described later.

Algorithm 1 Direct method for range counting

Require:  Sequence $\mathbf{u}:=(u_i:i\in {\mathbb{Z}}_{N+m})$, subset $\mathbf{s}\subset {\mathbb{Z}}_N$, template length m and threshold r. 1: procedure  DirectRangeCounting ($\mathbf{u},\mathbf{s},m,r$) 2:     Set $count=0$, 3:     Set $L=\#\mathbf{s}$, 4:     for $i=1$ to L do 5:         Set $\mathbf{a}=[u_{s_i+l-1}:l\in {\mathbb{Z}}_m]$, 6:         for $j=i+1$ to L do 7:            Set $\mathbf{b}=[u_{s_j+l-1}:l\in {\mathbb{Z}}_m]$, 8:            if $\rho (\mathbf{a}-\mathbf{b})\le r$ then 9:                $count=count+1$, 10:     return $count$

The definition of sample entropy shows that sample entropy measures the predictability of data. Precisely, in the definition of sample entropy, $B/A$ measures a conditional probability that when the distance of two templates $\mathbf{a}$ and $\mathbf{b}$ is less than or equal to r, the distance of their corresponding $(m+1)$-th component is also less than or equal to r. From this perspective, we can approximate this conditional probability of the original data set by computing it on a data set randomly down-sampled from the original one. To describe this method precisely, we define the notations as follows.

We choose a positive integer $N_0$, randomly draw $N_0$ numbers from ${\mathbb{Z}}_N$ without replacement, and form an $N_0$-dimensional vector. All of such vectors form a subset $\mathrm{\Omega }$ of the product space

$${\mathbb{Z}}_N^{N_0}:={\mathbb{Z}}_N\otimes {\mathbb{Z}}_N\otimes \cdots \otimes {\mathbb{Z}}_N\left(N_0\text{-folds}\right),$$

that is,

$$\mathrm{\Omega }:=\{\mathbf{s}:=[s_1,\cdots ,s_{N_0}]\in {\mathbb{Z}}_N^{N_0}:s_i\ne s_j\mathrm{for}\mathrm{all}i\ne j\}.$$

Suppose that $\mathcal{F}$ is the power set of $\mathrm{\Omega }$ (the set of all subsets of $\mathrm{\Omega }$, including the empty set and $\mathrm{\Omega }$ itself). We let P be the uniform probability measure satisfying $P\left(\mathbf{s}\right)=1/(\#\mathrm{\Omega })$ for all $\mathbf{s}\in \mathrm{\Omega }$ and define the probability space $\{\mathrm{\Omega },\mathcal{F},P\}$. The definition of $\mathrm{\Omega }$ implies $\#\mathrm{\Omega }=\frac{N!}{(N-N_0)!}$, and thus the probability measure satisfies $P\left(\mathbf{s}\right)=\frac{(N-N_0)!}{N!}$ for all $\mathbf{s}\in \mathrm{\Omega }$. The definition of $\mathcal{F}$ means all events that may occur in the sample space $\mathrm{\Omega }$ are considered in the probability space $\{\mathrm{\Omega },\mathcal{F},P\}$. We randomly select $N_0$ templates of length m and $N_0$ templates of length $m+1$ from the original time series. We then count the number $\tilde{A}$ (resp. $\tilde{B}$) of matched pairs among the selected templates of lengths m (resp. $m+1$). That is,

$$\tilde{A}\left(\mathbf{s}\right):=\frac{1}{2}\#\left\{(i,j):i,j\in {\mathbb{Z}}_{N_0}\mathrm{with}i\ne j,\mathrm{and}\rho ({\mathbf{x}}_{s_i},{\mathbf{x}}_{s_j})\le r\right\},\mathbf{s}\in \mathrm{\Omega },$$

and

$$\tilde{B}\left(\mathbf{s}\right):=\frac{1}{2}\#\left\{(i,j):i,j\in {\mathbb{Z}}_{N_0}\mathrm{with}i\ne j,\mathrm{and}\rho ({\mathbf{y}}_{s_i},{\mathbf{y}}_{s_j})\le r\right\},\mathbf{s}\in \mathrm{\Omega }.$$

We repeat this process $N_1$ times.

Note that $\tilde{A}$ and $\tilde{B}$ are random variables on the probability space $\{\mathrm{\Omega },\mathcal{F},P\}$. Let ${\overline{A}}_{N_1}$ and ${\overline{B}}_{N_1}$ be the averages of random variables $\tilde{A}$ and $\tilde{B}$, respectively, over the $N_1$ repeated processes. That is,

$${\overline{A}}_{N_1}:=\frac{1}{N_1}\sum _{k=1}^{N_1}\tilde{A}\left({\mathbf{s}}_k\right),\mathrm{and}{\overline{B}}_{N_1}:=\frac{1}{N_1}\sum _{k=1}^{N_1}\tilde{B}\left({\mathbf{s}}_k\right),$$

where $\{{\mathbf{s}}_k:k\in {\mathbb{Z}}_{N_1}\}$ is a subset of $\mathrm{\Omega }$. With ${\overline{A}}_{N_1}$ and ${\overline{B}}_{N_1}$, we can estimate the sample entropy $-log(B/A)$ by computing $-log({\overline{B}}_{N_1}/{\overline{A}}_{N_1})$. We summarize the procedure for computing $-log({\overline{B}}_{N_1}/{\overline{A}}_{N_1})$ in Algorithm 2 and call it the Monte-Carlo-based algorithm for evaluating sample entropy (MCSampEn). In MCSampEn, ${\mathbf{s}}_k$, $k\in {\mathbb{Z}}_{N_0}$, are selected by the Hidden Shuffle algorithm proposed in [16].

Algorithm 2 Monte-Carlo-based algorithm for evaluating sample entropy

Require: Sequence $\mathbf{u}=(u_i:i\in {\mathbb{Z}}_{N+m})$, template length m, tolerance $r\in \mathbb{R}$, sample size $N_0$ and number of experiments $N_1$, probability space $\{\mathrm{\Omega },\mathcal{F},P\}$ 1: procedure MCSampEn ($\mathbf{u},m,r,N_0,N_1$) 2:     Set ${\overline{A}}_{N_1}=0$ and ${\overline{B}}_{N_1}=0$, 3:     for $k=1$ to $N_1$ do 4:         Select ${\mathbf{s}}_k\in \mathrm{\Omega }$, randomly, with uniform distribution, 5:         Compute $\tilde{A}\left({\mathbf{s}}_k\right)$ by calling DirectRangeCounting ($\mathbf{u},{\mathbf{s}}^{\left(k\right)},m,r$), 6:         Compute $\tilde{B}\left({\mathbf{s}}_k\right)$ by calling DirectRangeCounting ($\mathbf{u},{\mathbf{s}}^{\left(k\right)},m+1,r$), 7:         ${\overline{A}}_{N_1}={\overline{A}}_{N_1}+\frac{1}{N_1}\tilde{A}\left({\mathbf{s}}^{\left(k\right)}\right)$, 8:         ${\overline{B}}_{N_1}={\overline{B}}_{N_1}+\frac{1}{N_1}\tilde{B}\left({\mathbf{s}}^{\left(k\right)}\right)$, 9:      $entropy=-log\frac{{\overline{B}}_{N_1}}{{\overline{A}}_{N_1}}$, 10:     return $entropy$

We next estimate the computational complexity of MCSampEn measured by the number of arithmetic operations. To this end, we recall Theorem 3.5 of [16] which gives the number of arithmetic operations used in the Hidden Shuffle algorithm.

Theorem 1.

The Hidden Shuffle algorithm generates a random sample of size $N_0$ sequentially from a population of size N with $O\left(N_0\right)$ arithmetic operations in total.

Theorem 2.

The total number of arithmetic operations needed in Algorithm 2 is $O\left(N_1(N_0^2+N_0)\right)$.

Proof. 

For each $k\in {\mathbb{Z}}_{N_1}$, according to Theorem 1, the number of arithmetic operations needed for selecting ${\mathbf{s}}^{\left(k\right)}$ on line 4 of Algorithm 2 is $O\left(N_0\right)$. Moreover, from Algorithm 1 we can see that for each $k\in {\mathbb{Z}}_{N_1}$, the number of arithmetic operations needed for computing $\tilde{A}\left({\mathbf{s}}_k\right)$ and $\tilde{B}\left({\mathbf{s}}_k\right)$ on lines 5 and 6 is $O\left(N_0^2\right)$. Thus, by counting the number of arithmetic operations needed for lines 7, 8, and 9 of Algorithm 2, we obtain the desired result. □

Theorem 2 indicates that the computational complexity of MCSampEn is controlled by setting appropriate sampling parameters $N_0$ and $N_1$. When $N_0$ and $N_1$ are fixed, the computational complexity of MCSampEn is independent of the length N of time series $\mathbf{u}$. Meanwhile, we can also select $N_0$ and $N_1$ depending on N to balance the error and computational complexity of MCSampEn. For example, we can set $N_0:=max\{1024,\lfloor \sqrt{N}\rfloor \}$ and $N_1:=min\left\{5+{log}_{2}N,\lfloor N/N_0\rfloor \right\}$, where $\lfloor a\rfloor$ denotes the greatest integer no bigger than $a\in \mathbb{R}$. In this case, the computational complexity is $O\left(N{log}_{2}N\right)$.

Noting that MCSampEn provides an approximation of the sample entropy, and not the exact value, convergence of MCSampEn is an important issue. We will discuss this in Section 3.

3. Error Analysis

In this section, we analyze the error of MCSampEn. Specifically, we will establish an approximation rate of MCSampEn in the sense of almost sure convergence.

A sequence of $\{V_k:k\in \mathbb{N}\}$ of random variables in probability space $\{\mathrm{\Omega },\mathcal{F},P\}$ is said to converge almost surely to $V\in \{\mathrm{\Omega },\mathcal{F},P\}$, denoted by

$$V_k\stackrel{a.s.}{\to }V,$$

if there exists a set $\mathcal{N}\in \mathcal{F}$ with $P\left(\mathcal{N}\right)=0$ such that for all $\omega \in \mathrm{\Omega }\backslash \mathcal{N}$,

$$\underset{k\to \infty }{lim}{V}_k\left(\omega \right)=V\left(\omega \right).$$

It is known (see [17]) that $\{V_k:k\in \mathbb{N}\}$ converges almost surely to $V\in \{\mathrm{\Omega },\mathcal{F},P\}$ if and only if

$$\underset{k\to +\infty }{lim}P\left(\left\{\underset{i\ge k}{sup}\left|V_i-V\right|>ϵ\right\}\right)=0,\mathrm{for} \mathrm{all}ϵ>0.$$

Furthermore, we can describe the convergence rate of $\{V_i:i\in \mathbb{N}\}$ by the declining rate of the sequence $\left\{P\left(\left\{{sup}_{i\ge k}\left|V_i-V\right|>ϵ\right\}\right):k\in \mathbb{N}\right\}$ for all $ϵ>0$. If for $\alpha >0$,

$$P\left(\left\{\underset{i\ge k}{sup}\left|V_i-V\right|>ϵ\right\}\right)=O\left(k^{-\alpha }\right),\mathrm{for} \mathrm{all}ϵ>0,$$

we say $\{V_i:i\in \mathbb{N}\}$ converges to V almost surely with rate $\alpha$.

To establish the approximation error of MCSampEn, we first derive two theoretical results for the expectations and variations of $\frac{\tilde{A}}{N_0(N_0-1)}$ and $\frac{\tilde{B}}{N_0(N_0-1)}$. Then, by combining these results with the results of the almost sure convergence rate in [18] and the local smoothness of logarithm functions, we obtain the approximation rate of $\{-log({\overline{B}}_{N_1}/{\overline{A}}_{N_1}):N_1\in \mathbb{N}\}$ in the sense of almost sure convergence, which is the main theoretical result of this paper. We state these results below and postpone their proofs to the Appendix A.

The expectations of $\frac{\tilde{A}}{N_0(N_0-1)}$ and $\frac{\tilde{B}}{N_0(N_0-1)}$ are given in the following theorem.

Theorem 3.

It holds that for all $N_0\in {\mathbb{Z}}_N$ with $N_0>1$,

$$E\left[\frac{\tilde{A}}{N_0(N_0-1)}\right]=\frac{A}{N(N-1)},$$

(1)

and

$$E\left[\frac{\tilde{B}}{N_0(N_0-1)}\right]=\frac{B}{N(N-1)}.$$

(2)

The next theorem presents the variations of $\frac{\tilde{A}}{N_0(N_0-1)}$ and $\frac{\tilde{B}}{N_0(N_0-1)}$.

Theorem 4.

It holds that for all $N_0\in {\mathbb{Z}}_N$ with $N_0>1$,

$$Var\left[\frac{\tilde{A}}{N_0(N_0-1)}\right]=\frac{C_{N_0}}{N_0},$$

(3)

and

$$Var\left[\frac{\tilde{B}}{N_0(N_0-1)}\right]=\frac{C_{N_0}}{N_0},$$

(4)

where

$$\begin{array}{ccc}\hfill C_{N_0}& :=& \frac{B}{(N_0-1)N(N-1)}+\frac{N_0-2}{(N_0-1)N(N-1)(N-2)}\left(\sum _{l=1}^N{B}_l^2-2B\right)\hfill \\ & & +\frac{(N_0-2)(N_0-3)}{(N_0-1)N(N-1)(N-2)(N-3)}\left(B^2-\sum _{l=1}^N{B}_l^2+B\right)-\frac{N_0{B}^2}{N^2{(N-1)}^2}.\hfill \end{array}$$

(5)

Moreover, there is $0<C_{N_0}<1+\frac{1}{2(N_0-1)}$.

Based on Theorems 3 and 4, we can obtain $log\frac{{\overline{B}}_{N_1}}{{\overline{A}}_{N_1}}\stackrel{a.s.}{\to }log\frac{B}{A}$ by the Kolmogorov strong law of large numbers and the continuous mapping theorem. However, in practice it is desirable to quantify the approximation rate in the sense of almost sure convergence, so that we can estimate the error between $log\frac{{\overline{B}}_{N_1}}{{\overline{A}}_{N_1}}$ and $log\frac{B}{A}$. To this end, we define ${\tau }_A:=\mathrm{E}\left[\left|\frac{\tilde{A}}{N_0(N_0-1)}-\frac{A}{N(N-1)}\right|\right]$, and ${\tau }_B:=\mathrm{E}\left[\left|\frac{\tilde{B}}{N_0(N_0-1)}-\frac{B}{N(N-1)}\right|\right]$. Let ${\gamma }_A:=\frac{A}{2N(N-1)e}$ and ${\gamma }_B:=\frac{B}{2N(N-1)e}$. For all $\beta >1$ and $0<ϵ<1$, we also let

$$n_{ϵ,\beta }:=max\left\{6{ϵ}^{-1},exp\left({\left(9{\beta }^{-1}{ϵ}^{-1}\right)}^{{\beta }^{-1}/\left(1-{\beta }^{-1}\right)}\right)\right\}.$$

(6)

With the notation defined above, we present below the main theoretical result of this paper, which gives the rate of $\{-log\frac{{\overline{B}}_k}{{\overline{A}}_k}:k\in \mathbb{N}\}$ approximating $-log\frac{B}{A}$ in the sense of almost sure convergence.

Theorem 5.

Let $\beta >1$ and $N_0\in {\mathbb{Z}}_N$ with $N_0>3$. If $A,B>0$, then there exist constants $D_{\beta }$ and ${\tilde{D}}_{\beta }$ (depending only on β) such that for all $0<ϵ<1$ and $N_1>n_{ϵ,\beta }$, such that

$$\begin{array}{ccc}\hfill & & P\left(\underset{k>N_1}{sup}\left|log\frac{{\overline{B}}_k}{{\overline{A}}_k}-log\frac{B}{A}\right|>max\{{\tau }_A,{\tau }_B\}ϵ\right)\hfill \\ & \le & \frac{72C_{N_0}}{{ϵ}^2{N}_0{N}_1}\left(D_{\beta }+{\tilde{D}}_{\beta }{(log{N}_1)}^{\beta -1}\right)\left(\frac{1}{{\tau }_A^2{\gamma }_A^2}+\frac{1}{{\tau }_B^2{\gamma }_B^2}\right).\hfill \end{array}$$

(7)

The proof for Theorems 3–5 are included in the Appendix A. Note that Theorem 5 indicates that $-log\frac{{\overline{B}}_k}{{\overline{A}}_k}$ approximates $-log\frac{B}{A}$ in the sense of almost sure convergence of order 1.

4. Experiments

We present numerical experiments to show the accuracy and computational complexity of the proposed algorithm MCSampEn.

As sample entropy has been prevalently used in a large number of areas, we consider several series with a variety of statistical features, including the electrocardiogram (ECG) series, RR interval series, electroencephalogram (EEG) series, mechanical vibration signals (MVS), meteorological data (MD), and $1/f$ noise. The ECG and EEG data can be downloaded from PhysioNet, a website offering access to recorded physiologic signals (PhysioBank) and related open-source toolkits (PhysioToolkit) [19]. The MVS data can be found in [20] and the website of the Case Western Reserve University Bearing Data Center [21]. The MD data can be downloaded from the website of the Royal Netherlands Meteorological Institute [22]. The databases used in this paper include:

• Long-Term AF Database (ltafdb) [23]. This database includes 84 long-term ECG recordings of subjects with paroxysmal or sustained atrial fibrillation (AF). Each record contains two simultaneously recorded ECG signals digitized at 128 Hz with 12-bit resolution over a 20 mV range; record durations vary but are typically 24 to 25 h.
• Long-Term ST Database (ltstdb) [24]. This database contains 86 lengthy ECG recordings of 80 human subjects, chosen to exhibit a variety of events of ST segment changes, including ischemic ST episodes, axis-related non-ischemic ST episodes, episodes of slow ST level drift, and episodes containing mixtures of these phenomena.
• MIT-BIH Long-Term ECG Database (ltecg) [19]. This database contains 7 long-term ECG recordings (14 to 22 h each), with manually reviewed beat annotations.
• BIDMC Congestive Heart Failure Database (chfdb) [25]. This database includes long-term ECG recordings from 15 subjects (11 men, aged 22 to 71, and 4 women, aged 54 to 63) with severe congestive heart failure (NYHA class 3–4).
• MGH/MF Waveform Database (mghdb) [26]. The Massachusetts General Hospital/ Marquette Foundation (MGH/MF) Waveform Database is a comprehensive collection of electronic recordings of hemodynamic and electrocardiographic waveforms of stable and unstable patients in critical care units, operating rooms, and cardiac catheterization laboratories. Note that only the ECG records were considered in our experiments.
• RR Interval Time Series (RR). The RR interval time series are derived from healthy subjects (RR/Health), and subjects with heart failure (RR/CHF) and atrial fibrillation (RR/AF).
• CHB-MIT Scalp EEG Database (chbmit) [27]. This database contains (EEG) records of pediatric subjects with intractable seizures. The records are collected from 22 subjects, monitored for up to several days.
• Gearbox Database (gearbox) [20]. The gearbox dataset was introduced in [20] and was published on https://github.com/cathysiyu/Mechanical-datasets (accessed on 27 March 2022).
• Rolling Bearing Database (RB) [21]. This database as a standard reference for the rolling bearing fault diagnosis is provided by the Case Western Reserve University Bearing Data Center [21].
• Meteorological Database (MD) [22]. The meteorological database used in this section records the hourly weather data in the past 70 years in the Netherlands.

As each database consists of multiple records from different subjects, we select one record randomly from each database. Specifically, we choose record “00” from ltafdb, “s20011” from ltstdb, “14046” from ltdb, “chf01” from chfdb, “mgh001” from mghdb, “chb07_01” from chbmit, “Miss_30_2” from gearbox, “XE110_DE_Time” from RB, and “380_t” from MD. Moreover, $1/f$ noise signal, an artificial signal, is studied to increase diversity. The time series considered in this section are illustrated in Figure 1, where all samples are normalized to have a standard deviation of 1, since the parameter threshold r is proportional to the standard deviation of the records, and thus the whole range of the records is negligible.

Figure 1. Samples of the dataset records.

4.1. Approximation Accuracy

In the experiments presented in this subsection, we examine the approximation accuracy of the MCSampEn algorithm. Specifically, we set $r:=0.15$ and $m:=4,5$. We vary the sampling size $N_0$ and the number $N_1$ of computations to study the approximation accuracy of the proposed algorithm. In this experiment, records with lengths exceeding ${10}^6$ are truncated to have length ${10}^6$; otherwise, the entire records are used. Since in the MCSampEn algorithm, ${\mathbf{s}}_k\in \mathrm{\Omega }$ are selected randomly, the outcome of the algorithm depends on the selected value of ${\mathbf{s}}_k$. To overcome the effect of the randomness, for every specified pair of $(N_0,N_1)$, we run the algorithm 50 times and calculate the mean errors (MeanErr) and the root mean squared errors (RMeanSqErr) of the 50 outcomes.

In our first experiment, we consider series “mghdb/mgh001”, select parameters $N_0\in \{200i:i\in {\mathbb{Z}}_{20}^{+}\}$, $N_1\in \{10i:i\in {\mathbb{Z}}_{25}^{+}\}$, and show in Figure 2 the mean errors and the root mean squared errors of the MCSampEn outputs as surfaces in the $N_0$-$N_1$ coordinate system. Images (a) and (c) of Figure 2 show the values of $MeanErr$ and images (b), (d), and (f) of Figure 2 show the values of $RMeanSqErr$. Figure 2 clearly demonstrates that both the mean errors and the root mean squared errors of the MCSampEn outputs converge to 0 as $N_0$ or $N_1$ increases to infinity. This is consistent with our theoretical analysis in the previous section.

Figure 2. The values of $MeanErr$ and $RMeanSqErr$ for time series “mghdb/mgh001” with respect to the sample size $N_0$ and the number of computations $N_1$, where parameters $r=0.15$ and $m=4,5$. (a) $MeanErr$ with $m=4$. (b) $RMeanSqErr$ with $m=4$. (c) $MeanErr$ with $m=5$. (d) $RMeanSqErr$ with $m=5$.

In the second experiment, we consider all series illustrated in Figure 1 and show numerical results in Figure 3 and Figure 4. Images (a), (c), and (e) of Figure 3 show the values of $MeanErr$, and images (b), (d), and (f) of Figure 3 show the values of $RMeanSqErr$, with $N_0\in \{200i:i\in {\mathbb{Z}}_{20}^{+}\}$ and fixed $N_1=250$. Images (a), (c), and (e) of Figure 4 show the values of $MeanErr$, and images (b), (d), and (f) of Figure 4 show the values of $RMeanSqErr$, with $N_0=4000$ and $N_1\in \{10i:i\in {\mathbb{Z}}_{25}^{+}\}$. Figure 3 indicates that the outputs of the MCSampEn algorithm converge as $N_0$ increases. We can also see from Figure 3 that when $N_0\ge 1500$, $N_1=150$, and $m=4$, both MeanErr and RMeanSqErr are less than $1\times {10}^{-2}$ for all tested time series. In other words, the MCSampEn algorithm can effectively estimate sample entropy when $N_0\ge 1500$, $N_1=150$, and $m=4$. From Figure 4, we can also observe that the outputs of the MCSampEn algorithm converge as $N_1$ increases. This is consistent with the theoretical results established in Section 3.

Figure 3. The values of $MeanErr$ and $RMeanSqErr$ with respect to $N_0\in \{200i:i\in {\mathbb{Z}}_{20}^{+}\}$ and $N_1=150$, where parameters $r=0.15$ and $m=4,5$. (a) $MeanErr$ with $m=4$. (b) $RMeanSqErr$ with $m=4$. (c) $MeanErr$ with $m=5$. (d) $RMeanSqErr$ with $m=5$.

Figure 4. The values of $MeanErr$ and $RMeanSqErr$ with respect to $N_0=2\times {10}^3$ and $N_1\in \{10i:i\in {\mathbb{Z}}_{25}^{+}\}$, where parameters $r=0.15$ and $m=4,5$. (a) $MeanErr$ with $m=4$. (b) $RMeanSqErr$ with $m=4$. (c) $MeanErr$ with $m=5$. (d) $RMeanSqErr$ with $m=5$.

We next explain how the randomness of a time series effects the accuracy of the MCSampEn algorithm by applying the algorithm to the stochastic process $MIX\left(p\right)$, which has been widely applied to studies of sample entropy [1,2,28]. The $MIX\left(p\right)$ is defined as follows. Let $x_j:={\alpha }^{-1/2}sin(12\pi j/12)$ for all $j\in {\mathbb{Z}}_N$ where

$$\alpha :=\left(\sum _{j=1}^{12}{sin}^2(2\pi j/12)\right)/12.$$

Let $\{y_j:j\in {\mathbb{Z}}_N\}$ be a family of independent identically distributed (i.i.d) real random variables with uniform probability density on the interval $[-\sqrt{3},\sqrt{3}]$. Note that $\{x_j:j\in {\mathbb{Z}}_N\}$ and $\{y_j:j\in {\mathbb{Z}}_N\}$ are sequences with contrary properties: the former is a completely regular sine sequence, and the latter is completely random. Let $p\in [0,1]$, and $\{z_j:j\in {\mathbb{Z}}_N\}$ be a family of i.i.d random variables satisfying $z_j=1$ with probability p and $z_j=0$ with probability $1-p$. Then, the $MIX\left(p\right)$ process is defined as $\{m_j:=(1-z_j)x_j+z_j{y}_j:j\in {\mathbb{Z}}_N\}$. It’s not hard to find that the parameter p controls the ratio of sine sequence and random noise in the $MIX\left(p\right)$ process and the increase in p makes the $MIX\left(p\right)$ process more random. When $p=0$, the $MIX\left(p\right)$ process is a deterministic sine sequence. Meanwhile, when $p=1$, the $MIX\left(p\right)$ process turns out completely unpredictable uniform noise. This feature makes it an ideal series to study how randomness affects the accuracy of the MCSampEn algorithm.

Here, we apply MCSampEn to $MIX\left(p\right)$, $p\in \{0.5+0.5i:i\in {\mathbb{Z}}_{19}\}$ and show the results of $RMeanSqErr$ versus p in Figure 5. From Figure 5, we can observe that the values of $RMeanSqErr$ increase linearly with a very small growth rate when $p\le 0.5$. When $p>0.5$, the values of $RMeanSqErr$ are significantly faster than that of $p\le 0.5$. Therefore, we believe that when the randomness of a time series is weak, the error of the MCSampEn algorithm is small; as the randomness of the time series increases, the error of the MCSampEn grows.

Figure 5. The values of $RMeanSqErr$ with respect to p, where parameters $r=0.15$, $m=4,5$, $N=2^{20}$, $N_0=2000$, and $N_1=150$.

4.2. Time Complexity

In the experiments presented in this subsection, we compare the computing time of the MCSampEn algorithm with that of the kd-tree algorithm [8] and SBOX algorithm [14], under the condition that the value of sample entropy computed by the MCSampEn algorithm is very close to the ground truth value. The computational time experiments are performed on a desktop computer running Windows 11, with an Intel(R) Core(TM) i5-9500 CPU, and 32GB RAM. The implementations of the kd-tree-based algorithm and the MCSampEn algorithm are available on the website https://github.com/phreer/fast_sampen_impl.git (accessed on 30 March 2022). As for the SBOX method, we utilize the implementation given by the original author, published on website https://sites.google.com/view/yhw-personal-homepage (accessed on 25 October 2021). To demonstrate the validity of the MCSampEn algorithm, we also show both the sample entropy estimated by MCSampEn and the corresponding ground truth.

As we have discussed above, the time complexity of the MCSampEn algorithm depends on the parameters $N_0$ and $N_1$. In this subsection, we discuss two strategies for choosing $N_0$ and $N_1$:

S1 

Choose $N_0$ and $N_1$ to be independent of N, for example $N_0=2\times {10}^3$ and $N_1=150$.

S2 

Choose $N_0=max\{1024,\lfloor \sqrt{N}\rfloor \}$ and $N_1=min\left\{5+{log}_{2}N,\lfloor N/N_0\rfloor \right\}$, depending on N.

An intuitive explanation of the second strategy is shown below. We would like to choose $N_0$ and $N_1$ such that the overall time complexity of executing the algorithm is $O(NlogN)$. For this purpose, we expect $N_0$ to grow like $\sqrt{N}$ and $N_1$ to grow logarithmically in N. However, when N is not large enough, lack of sampling templates can seriously impair the accuracy of the algorithm. To overcome this problem, we set a lower bound of $N_0$ to 1024, which is a good trade-off between accuracy and time complexity. The experimental results in this subsection show that this strategy can produce satisfactory output even when N is small.

The results on different signals “ltafdb/00”, “$1/f$ noise”, “chbmit/chb07_01”, and “ltecg/14046” are shown in Figure 6, where the first strategy is adopted by setting $N_0=2\times {10}^3$ and $N_1=150$, and the results for $m=4$ are marked by red color, and the results for $m=5$ are marked by blue. In the left column of Figure 6, the values of computation time consumed by the kd-tree, SBOX, and MCSampEn algorithms are plotted, respectively, with the dashed lines marked “x”, the dash-dot lines marked “+”, and the solid lines marked “o”. From the results shown in the left column of Figure 6, we can find that MCSampEn is faster than the SBOX algorithm when N is greater than $2^{15}$. We also can see when the time series “chbmit/chb07_01” and “ltecg/14046” have length N of $2^{20}$, MCSampEn is nearly 1000 times faster than the SBOX algorithm. Compared to the kd-tree algorithm, the MCSampEn algorithm can still achieve up to hundreds of times acceleration when $N=2^{20}$. In addition, the time complexity of MCSampEn algorithm is close to a constant relative to m, and is much smaller than the kd-tree and SBOX algorithms when N is large enough. Meanwhile, the computational time (shown in the left column of Figure 6) required is hardly affected by the times series length N.

Figure 6. The left column shows the results of computational time versus data length N on different signals. In the right column, the values of $RMeanSqErr$ are presented by error bars “I”, where the larger the value of $RMeanSqErr$, the longer the error bar “I”. In this figure, we set $m=4,5$, $N_0=2\times {10}^3$, and $N_1=150$. (a) Time for “ltafdb/00”. (b) Sample entropy “ltafdb/00”. (c) Time for $1/f$ noise. (d) Sample entropy for $1/f$ noise. (e) Time for “chbmit/chb07_01”. (f) Sample entropy for “chbmit/chb07_01”. (g) Time “ltecg/14046”. (h) Sample entropy for “ltecg/14046”.

The right column of Figure 6 shows the average of 50 outputs of the MCSampEn algorithm for different time series under the settings of $N_0=2\times {10}^3$ and $N_1=150$, where the red solid lines plot the average for the cases of $m=4$, and the blue solid lines plot the average for the cases of $m=5$. In the right column of Figure 6, the values of ground truth for the cases of $m=4$ and $m=5$ are plotted by the red and blue dashed lines, respectively. Meanwhile, in the right column of Figure 6, we use error bars “I” to represent the values of $RMeanSqErr$, where the larger the value of $RMeanSqErr$, the longer the error bar “I”. From the length of error bar “I”, we can see that the values of $RMeanSqErr$ are small compared to the ground truth. Especially on the time series “ltafdb/00”, “chbmit/chb_0701”, and “ltecg/14046”, the values of $RMeanSqErr$ are negligible compared to the values of ground truth. These results imply that when $N_0=2\times {10}^3$ and $N_1=150$, the sample entropy estimated by the MCSampEn algorithm can effectively approximate the ground truth value.

The results of the second strategy are shown in Figure 7, where $N_0=max\{1024,\lfloor \sqrt{N}\rfloor \}$ and $N_1=min\left\{5+{log}_{2}N,\lfloor N/N_0\rfloor \right\}$. The results for $m=4$ are marked by red color, and the results for $m=5$ are marked by blue color. The left column of Figure 6 shows the values of computation time consumed by the kd-tree, SBOX, and MCSampEn algorithms, which are presented by the dashed lines marked “x”, the dash-dot lines marked “+”, and the solid lines marked “o”, respectively. From the left column of Figure 7, we also can see that with the second strategy, the computational time of MCSampEn algorithm is much less than that of the kd-tree and SBOX algorithms, since the computational complexity of Algorithm 2 is $O(NlogN)$. Furthermore, we observe that MCSampEn achieves a speedup of more than 100 compared to the SBOX algorithm when N goes from $2^{16}$ to $2^{18}$, and it is over 1000 times faster when $N=2^{20}$. Compared to the kd-tree algorithm, the MCSampEn algorithm can still obtain up to 1000 times acceleration when $N=2^{20}$.

Figure 7. The left column shows the results of computational time versus data length N on different signals. The right column shows the values of $RMeanSqErr$ by error bar, where the larger the value of $RMeanSqErr$, the longer the error bar “I”. In this figure, we set $m=4,5$, $N_0=max\{1024,\lfloor \sqrt{N}\rfloor \}$, and $N_1=max\{1,\lfloor N/N_0\rfloor \}$. (a) Time for “ltafdb/00”. (b) Sample entropy “ltafdb/00”. (c) Time for $1/f$ noise. (d) Sample entropy for $1/f$ noise. (e) Time for “chbmit/chb07_01”. (f) Sample entropy for “chbmit/chb07_01”. (g) Time “ltecg/14046”. (h) Sample entropy for “ltecg/14046”.

In the right column of Figure 7, we plot the average of 50 outputs of the MCSampEn algorithm for different time series by the red and blue solid lines for $m=4$ and $m=5$, respectively. At the same time, the values of ground truth for the cases of $m=4$ and $m=5$ are plotted by the red and blue dashed lines, respectively. As in Figure 6, we use the error bar “I” to represent the values of $RMeanSqErr$. Comparing the error bar “I” in Figure 6, we can see that the values of the $RMeanSqErr$ in this experiment are larger than that shown in Figure 6. However, the value of $RMeanSqErr$ is still small in terms of the values of ground truth. Moreover, we can observe that the length of the error bars decreases as N increases. This means that we can obtain a better approximation of sample entropy as the time series length increases.

To reveal the effect of randomness on the speedup, we compare the time taken by the kd-tree and MCSampEn algorithms to compute the sample entropy of the time series $MIX\left(p\right)$, $p\in \{0.5+0.5i:i\in {\mathbb{Z}}_{19}\}$. The experimental results are shown in Figure 8, where the results for $m=4$ are marked by red color, and the results for $m=5$ are marked by blue. The values of computation time consumed by the kd-tree and MCSampEn algorithms are plotted, respectively, with the dashed lines marked “x” and the solid lines marked “o”. In this experiment, we set $N=2^{20}$ and $r=0.15$. We also let $N_0=1000+3000p$ and $N_1=80+70p$ to ensure that the relative error $RMeanSqErr/SampEn$ is no greater than $0.02$. From Figure 8, we can see that when the value of p is less than $0.2$, compared with the kd-tree algorithm, the MCSampEn algorithm can achieve 300 to 1000 times speedup. When the value of p is greater than $0.8$, our algorithm can still obtain a 10x speedup relative to the kd-tree algorithm.

Figure 8. The results of computational time with respect to p, where parameters $r=0.15$, $m=4,5$, $N=2^{20}$, $N_0$, and $N_1$ are selected such that relative error $RMeanSqErr/SampEn\le 0.02$.

From the experiments in this subsection, we can observe that the MCSampEn algorithm can achieve a high speedup when it is applied to different types of signals. In fact, compared with kd-tree algorithm, the MCSampEn algorithm can achieve high accuracy and more than 300 times acceleration when the time series has less randomness. When the randomness of the time series is high, our algorithm can still obtain a speedup of nearly 10 times.

4.3. Memory Usage

In order to show the performance of the MCSampEn algorithm more comprehensively, we also compare the memory usage of the kd-tree and MCSampEn algorithms. The memory usage on signal “ltstdb/s20011” is shown in Figure 9, where the memory usage for $m=4$ and $m=5$ is shown in Figure 9a,b, respectively. In this figure, the memory usage of the kd-tree algorithm is plotted by the blue dash-dot lines marked “x”. The memory usage of the MCSampEn algorithm with the first and second strategies is plotted by the green dashed lines marked “+” and the red dotted lines marked “o”, respectively. In Figure 9, the first strategy is adopted by setting $N_0=2048$ and $N_1=150$, and the second strategy is adopted by $N_0=max\{1024,\lfloor \sqrt{N}\rfloor \}$ and $N_1=min\left\{5+{log}_{2}N,\lfloor N/N_0\rfloor \right\}$. We also present the memory usage for storing the data by the black solid lines marked “□”.

Figure 9. The results of memory usage versus data length N with $m=4,5$. (a) Memory usage for $m=4$. (b) Memory usage for $m=5$.

From the results shown in Figure 9, it can be seen that when the size of the data is $2^{20}$, the memory required by the kd-tree algorithm is almost 36 times that of the memory required by the MCSampEn algorithm. This is because the kd-tree algorithm requires a large memory space to save the kd-tree. Meanwhile, the experimental results in Figure 9 also show that the amount of memory required by the MCSampEn algorithm is only about 15 MB more than the amount of memory required to store the data when the length of data is between $2^{14}$ and $2^{24}$. This is because the MCSampEn algorithm requires additional memory for storing $N_0$ templates and to execute the subroutines that generate random numbers.

Because the MCSampEn algorithm is based on Monte Carlo sampling and the law of large numbers, it is an easily parallelizable algorithm. Therefore, combined with distributed storage techniques, the idea of the MCSampEn algorithm can be used to compute sample entropy for large-scale data (for example, where the size of data is larger than 1 TB). Parallel algorithms for computing sample entropy of large-scale data will be our future work.

5. Conclusions

In this paper, we propose a Monte-Carlo-based algorithm called MCSampEn to estimate sample entropy and prove that the outputs of MCSampEn can approximate sample entropy in the sense of almost sure convergence of order 1. We provide two strategies to select the sampling parameters $N_0$ and $N_1$, which appear in MCSampEn. The experiment results show that we can flexibly select the parameters $N_0$ and $N_1$ to balance the computational complexity and error. From the experimental results, we can observe that the computational time consumed by the proposed algorithm is significantly shorter than the kd-tree and SBOX algorithms, with negligible loss of accuracy. Meanwhile, the computational complexity of our MCSampEn method is hardly affected by the time series length N. We also study how the randomness of the time series affects the accuracy and computation time of the MCSampEn algorithm by applying the algorithm to the stochastic process $MIX\left(p\right)$. The results indicate that the proposed algorithm performs well for time series with less randomness.