Entropy Measures for Stochastic Processes with Applications in Functional Anomaly Detection

Abstract

We propose a definition of entropy for stochastic processes. We provide a reproducing kernel Hilbert space model to estimate entropy from a random sample of realizations of a stochastic process, namely functional data, and introduce two approaches to estimate minimum entropy sets. These sets are relevant to detect anomalous or outlier functional data. A numerical experiment illustrates the performance of the proposed method; in addition, we conduct an analysis of mortality rate curves as an interesting application in a real-data context to explore functional anomaly detection.

1. Introduction

The family of $\alpha$-entropies, originally proposed by Rényi [1], plays an important role in information theory and statistics. Consider a random variable Z distributed according to a measure F that admits a probability density function f. Then, for $\alpha \ge 0$ and $\alpha \ne 1$, the $\alpha$-entropy of Z is computed as follows:

$$H_{\alpha }\left(Z\right)=\frac{1}{1-\alpha }log\left(V_{\alpha }\left(Z\right)\right)$$

(1)

where $V_{\alpha }\left(Z\right)={\mathbb{E}}_F\left\{f^{\alpha -1}\right\}$, and ${\mathbb{E}}_F$ stands for the expected value with respect to the F measure. Several renowned entropy measures in the statistical literature are particular cases in the family of $\alpha$-entropies. For instance, when $\alpha =0$, we obtain the Hartley entropy; when $\alpha \to 1$, then $H_{\alpha }$ converges to the Shannon entropy; and when $\alpha \to \infty$, then $H_{\alpha }$ converges to the Min-entropy measure. The contribution of this paper is two-fold. Firstly, we propose a natural definition of entropy for stochastic processes that extends the previous one and a suitable sample estimator for the observation of partial realizations of the process, the typical framework when dealing with functional data. We also show that Minimal Entropy Sets (MES), as formally defined in Section 3, are useful to solve anomaly detection problems, a common task in almost all data analysis contexts.

The paper is structured as follows: In Section 2, we introduce a definition of entropy for a stochastic process and suitable sample estimators for this measure. In Section 3, we show how to estimate minimum-entropy sets of a stochastic process in order to discover atypical functional data in a sample. Section 4 illustrates the theory with simulations and examples, and Section 5 concludes the work.

2. Entropy of a Stochastic Process

In this section, we extend the definition of entropy to a stochastic process. For the sequel, let $(\Omega ,\mathcal{F},P)$ be a probability space, where $\mathcal{F}$ is the $\sigma$-algebra in $\Omega$ and P a $\sigma$-finite measure. We consider random elements (functions) $X(\omega ,t):\Omega \times T\to \mathbb{R}$ in a metric space $(T,\tau )$. As usual in the case of functional data, the realizations of the random elements $X(\omega ,·)$ are assumed in $C\left(T\right)$, the space of real continuous functions in a compact domain $T\subset \mathbb{R}$ endowed with the uniform metric.

The first step is to consider a suitable representation for the stochastic process. We make use of the well-known Karhunen–Loève expansion [2] (p. 25, Theorem 1.5). Let $X(\omega ,t)$ be a centered (zero-mean) stochastic process with continuous covariance function $K_X(s,t)=\mathbb{E}\left(X(\omega ,s)X(\omega ,t)\right)$, then there exists a basis ${\left\{e_i\right\}}_{i\ge 1}$ of $C\left(T\right)$ such that for all $t\in T$:

$$X(\omega ,t)=\sum _{i=1}^{\infty }{\xi }_i\left(\omega \right)e_i\left(t\right),$$

(2)

where the sequence of random coefficients ${\xi }_i\left(\omega \right)={\int }_{T}X(\omega ,t)e_i\left(t\right)dt$ comprises zero mean random variables with (co)variance $\mathbb{E}\left({\xi }_i{\xi }_j\right)={\delta }_{ij}{\lambda }_j$, being ${\delta }_{ij}$ the Kronecker delta and ${\left\{\lambda \right\}}_{j\ge 1}$ the sequence of eigenvalues associated with the eigenfunctions of $K_X(s,t)$.

The equality in Equation (2) must be understood in the mean square sense, that is:

$$\underset{d\to \infty }{lim}\mathbb{E}\{{(X(\omega ,t)-\sum _{i=1}^d{\xi }_i\left(\omega \right)e_i\left(t\right))}^2\}=0,$$

(3)

uniformly in T. Therefore, we can always consider a $\epsilon$-near representation $X_d(\omega ,t)={\sum }_{i=1}^d{\xi }_i\left(\omega \right)e_i\left(t\right)$ such that for all $\epsilon$ arbitrarily small, there exists an integer D such that for $d\ge D$, then $\tau (X,X_d)={sup}_{t\in T}|X(\omega ,t)-X_d(\omega ,t)|\le \epsilon$. From this result, it is possible to establish a suitable way to approximate the entropy of a random element $X(\omega ,t)$ according to the distribution of the “representation coefficients” ${\left\{{\xi }_i\left(\omega \right)\right\}}_i^d$ obtained from $X_d(\omega ,t)$.

Definition 1 (d-truncated entropy for stochastic processes).

Let X be a centered stochastic process with a continuous covariance function. Consider the truncation $X_d(\omega ,t)={\sum }_{i=1}^d{\xi }_i\left(\omega \right)e_i\left(t\right)$ and the random vector $Z=({\xi }_1,\dots ,{\xi }_d)$; then, the d-truncated entropy of X is defined as $H_{\alpha }(X,d)=H_{\alpha }\left(Z\right)$.

The “approximation error” when computing the entropy of the stochastic process X with Definition 1 decreases monotonically with the number of terms retained in the Karhunen–Loève expansion, at a rate that depends on the decay of the spectrum of the covariance function $K_X(s,t)$. In general, the more autocorrelated the process is, the more quickly the eigenvalues of $K_X(s,t)$ converge to zero. In practical functional data applications (see for instance the mortality-rate curves in Section 4), the autocorrelation is usually strong, and the truncation parameter d will be small when approximating the entropy of the process. The next example illustrates the definition.

Example 1.

[Gaussian process] When X is a Gaussian Process (GP), the coefficients in the Karhunen–Loève expansion have the further property that they are independent and zero-mean normally distributed random variables. Therefore, the Shannon entropy ($\alpha =1$) of X can be approximated with the truncated version of the GP as follows:

$$H_1(X,d)=\frac{1}{2}log{\left(2\pi e\right)}^{d}det(\Sigma ),$$

where Σ is the diagonal covariance matrix with elements ${[\Sigma ]}_{i,j}=\mathbb{E}\left({\xi }_i{\xi }_j\right)$ for $i,j=1,\dots ,d$.

In practice, we can only observe some realizations of the stochastic process X, and these observations are sparsely registered. Therefore, to estimate the entropy of $X(\omega ,t)$ from a random sample of discrete realizations of a stochastic process, a first task is the representation of these paths by means of continuous functions. To this end, we consider a reproducing kernel Hilbert space $\mathcal{H}$ of functions, associated with a positive definite and symmetric kernel function $K:T\times T\to \mathbb{R}$.

Estimating Entropy in a Reproducing Kernel Hilbert Space

Most functional data analysis approaches for representing raw data suggest proceeding as follows: (i) choose an orthogonal basis of functions $\Phi =\{{\varphi }_1,\dots ,{\varphi }_N\}$, where each ${\varphi }_i$ belongs to a general function space $\mathcal{H}$; and (ii) represent each functional datum by means of a linear combination in the $Span(\Phi )$ [3,4]. Our choice is to consider $\mathcal{H}$ as a Reproducing Kernel Hilbert Space (RKHS) of functions [5]. In this case, the elements in the spanning set $\Phi$ are the eigenfunctions associated with the positive-definite and symmetric kernel function $K:T\times T\to \mathbb{R}$ that span $\mathcal{H}$ [5] (Moore-Aronszajn Theorem p. 19).

In our setting, the functional representation problem can be framed as follows: We have available m discrete observations, that is a realization path $x\left(t_1\right),\dots ,x\left(t_m\right)$ of the stochastic element $X(\omega ,t)$. We also assume that the discrete path ${\{x\left(t_i\right),t_i\}}_{i=1}^m$, as usual when dealing with real data, contains zero mean $iid$ error measurements. Then, the functional data estimator, denoted onwards as $\tilde{x}\left(t\right)$, is obtained solving the following regularization problem:

$$\tilde{x}\left(t\right):=arg\underset{g\in \mathcal{H}}{min}\sum _{i=1}^{m}V{(x\left(t_i\right),g\left(t_i\right))}^2+\gamma \Omega \left(g\right),$$

(4)

where V is a strictly convex functional with respect to the second argument, $\gamma >0$ is a regularization parameter, frequently chosen by cross-validation, and $\Omega \left(g\right)$ is a regularization term. By the representer theorem [6,7] (Theorem 5.2, p. 91, Proposition 8, p. 51), the solution of the problem stated in Equation (4) exists, is unique and admits a representation of the form:

$$\tilde{x}\left(t\right)=\sum _{i=1}^m{a}_{i}K(t,t_i).$$

(5)

In the particular case of a squared loss function $V(w,z)={(w-z)}^2$ and considering $\Omega \left(g\right)={\int }_T{g}^2\left(t\right)dt$, the coefficients of the linear combination in Equation (5) are obtained solving the following system:

$$(\gamma m\mathbf{I}+\mathbf{K})\mathbf{a}=\mathbf{y},$$

(6)

where $\mathbf{a}={(a_1,\dots ,a_m)}^T$, $\mathbf{y}={(x\left(t_1\right),\dots ,x\left(t_m\right))}^T$, $\mathbf{I}$ is the identity matrix of order m and $\mathbf{K}$ is the Gram matrix with the kernel evaluations, ${\left[\mathbf{K}\right]}_{k,l}=K(t_k,t_l)$, for $k=1,\dots ,m$ and $l=1,\dots ,m$. To relate the Karhunen–Loève expansion in Equation (2) to the RKHS representation, we make use of Mercer’s theorem [2] (Lemma 1.3, p. 24), then $K_X(s,t)={\sum }_{j=1}^{\infty }{\lambda }_j{\varphi }_j\left(s\right){\varphi }_j\left(t\right)$, where ${\lambda }_j$ is the eigenvalue associated with the orthonormal eigenfunction ${\varphi }_j$ for $j\ge 1$, and invoking the reproducing property, then:

$$\begin{array}{cc}\hfill X(\omega ,t)& =⟨X(\omega ,s),K_X(s,t)⟩\hfill \\ & =\sum _{j=1}^{\infty }{\lambda }_j{\varphi }_j\left(t\right){\int }_{T}X(\omega ,s){\varphi }_j\left(s\right)ds.\hfill \end{array}$$

(7)

Therefore, following Equation (2), ${\xi }_j\left(\omega \right):=\sqrt{{\lambda }_j}{\int }_{T}X(\omega ,s){\varphi }_j\left(s\right)ds$ and $e_j\left(t\right)=\sqrt{{\lambda }_j}{\varphi }_j\left(t\right)$; and the connection is clearly established. When working with discrete realizations of a stochastic process, we must solve two sequential tasks. First, we need to represent raw data as functional data and later find a truncated representation of the function. To this end, when combining Equation (5) with Mercer’s theorem and the reproducing property, we obtain:

$${\tilde{x}}_d\left(t\right)=\sum _{j=1}^d\sqrt{{\lambda }_j}{\varphi }_j\left(t\right)\sqrt{{\lambda }_j}\left(\sum _{i=1}^m{a}_i{\varphi }_j\left(t_i\right)\right),$$

and now, $z_j:=\sqrt{{\lambda }_j}{\sum }_{i=1}^m{a}_i{\varphi }_j\left(t_i\right)$ is the realization of the random variable ${\xi }_j$ for $j=1,\dots ,d$; see [8] for further details. For some kernel functions, for instance the Gaussian kernel, the associated sequence of eigen-pairs $({\lambda }_j,{\varphi }_j)$ for $j\ge 1$ is known [9] (pp. 10), and we can obtain an explicit value for all $z_j$. If not, let $({\lambda }_j,{\mathbf{v}}_j)$ be the j-eigenpair associated with the kernel matrix $\mathbf{K}\in {\mathbb{R}}^{m\times m}$, then $z_j=\sqrt{{\lambda }_j}{\sum }_{i=1}^m{a}_i{v}_{i,j}$ for $j=1,\dots ,d$.

In practice, given a sample of n discrete paths (realizations) of the stochastic process X, say $\{x_l\left(t_1\right),\dots ,x_l\left(t_m\right)\}$ for $l=1,\dots ,n$, a suitable input to estimate entropy in Definition 1 is to consider the set of multivariate vectors ${\mathbf{z}}_l=(z_{l,1},\dots ,z_{l,d})$ for $l=1,\dots ,n$, as formally proposed in the next definition.

Definition 2 (K-entropy estimation of a stochastic process).

Let $\{x_1\left(t_i\right),\dots ,x_n\left(t_i\right)\}$ for $i=1,\dots ,m$ be a discrete random sample of X, and let ${\left\{({\lambda }_j,{\mathbf{v}}_j)\right\}}_{j=1}^d$ be the eigen-pairs of the kernel matrix $\mathbf{K}\in {\mathbb{R}}^{m\times m}$, where $d=rank\left(\mathbf{K}\right)$. Consider the corresponding finite dimensional representation $S_n:=\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_n\}$, where ${\mathbf{z}}_l=(z_{l,1},\dots ,z_{l,d})\in {\mathbb{R}}^d$ for $l=1,\dots ,n$ and $z_{l,j}=\sqrt{{\lambda }_{l,j}}{\sum }_{i=1}^m{a}_{l,i}{v}_{i,j}$ for $j=1,\dots ,d$. Then, the estimated kernel entropy of X is defined as ${\widehat{H}}_{\alpha }(X,K)={\widehat{H}}_{\alpha }\left(Z\right)$.

In Definition 2, ${\widehat{H}}_{\alpha }\left(Z\right)$ denotes the estimated entropy using the (finite dimensional) representation coefficients $S_n=\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_n\}$. In Section 3, we formally introduce two approaches to estimate entropy departing from $S_n$. The next example illustrates the estimation procedure in the context of GPs in Example 1.

Illustration with Example 1:

Consider 100 realizations of a GP as follows: 50 curves from $X\left(t\right)={\sum }_{i=1}^3{\xi }_i{e}_i\left(t\right)$ and another 50 curves from $Y\left(t\right)={\sum }_{i=1}^3{\zeta }_i{e}_i\left(t\right)$; where $e_i\left(t\right)$ is a Fourier basis in $T=[0,1]$, ${\xi }_i\sim N(\mu =0,{\sigma }^2=0.5)$, and ${\zeta }_i\sim N(\mu =0,{\sigma }^2=2)$ are independent normally distributed random variables (r.v.) for $i=1,2,3$.

In Figure 1 (left), we illustrate the realizations of the stochastic processes, in black (“—”) the sample paths of $X\left(t\right)$ and in red (“—”) the paths corresponding to $Y\left(t\right)$. In Figure 1 (right), we show the distribution of the linear combination coefficients ${\{{(z_1,z_2,z_3)}_l,{(w_1,w_2,w_3)}_l\}}_{l=1}^{50}$ corresponding to these paths. Following Example 1, we estimate the covariance functions ${\widehat{\Sigma }}_X$ and ${\widehat{\Sigma }}_Y$ using the respective coefficients and plug this covariance matrix into the Shannon entropy expression to obtain the estimated entropies ${\widehat{H}}_1\left(X\right)=1.402$ and ${\widehat{H}}_2\left(Y\right)=99.552$, similar to the true entropies $H_1\left(X\right)=1.428$ and $H_2\left(Y\right)=91.420$, respectively. We formally propose the estimation procedure in Algorithm 1.

Figure 1. Gaussian processes realizations on the left and coefficients for entropy estimation on the right. The sizes of the balls on the right are proportional to the determinants of ${\widehat{\Sigma }}_X$ (in black) and ${\widehat{\Sigma }}_Y$ (in red).

Algorithm 1: Estimation of $H_{\alpha }(X,K)$ from a sample of random paths.

The choice of kernel parameters in Algorithm 1 is made by cross-validation. This ensures that the curve fitting method is asymptotically optimal. Nonetheless, although the selection of the kernel parameters affects the scale of the estimated entropy, the center-outward ordering induced by $H_{\alpha }(X,K)$, as formally proposed in the next section, is unaffected. In the Supplementary Material, we present relevant experimental results to illustrate this property, which make the method robust in terms of the selection of the kernel and regularization parameters.

3. Minimum Entropy for Anomaly Detection

Anomaly detection is a common task in almost all data analysis context. The unsupervised approach considers a sample $X_1,\dots ,X_n$ of random elements where most instances follow a well-defined pattern and a small proportion, here denoted as $\nu \in [0,1]$, present an abnormal pattern. In recent works (see for instance [10,11,12,13]), the authors propose depth measures and related methods, to deal with functional outliers. In this section, we propose a novel criterion to tackle the problem of anomaly detection with functional data using the ideas and concepts developed in Section 2. For a real-valued d-dimensional random vector Z that admits a continuous density function $f_Z$, define $H_{\alpha }\left(A_Z\right)=\frac{1}{1-\alpha }log\left({\int }_A{f}_Z^{\alpha }\left(\mathbf{z}\right)d\mathbf{z}\right)$ to be the entropy of the Borel-set A with respect to the measure $F_Z$. Then, the $\nu$-Minimal-Entropy Set (MES) is formally defined as:

$${\mathrm{MES}}_{\nu }\left(Z\right):=\{{arg\; min}_{A\subset {\mathbb{R}}^d}{H}_{\alpha }\left(A_Z\right)\mathrm{s}.\mathrm{t}.P\left(A\right)\ge 1-\nu \}.$$

The ${\mathrm{MES}}_{\nu }$ is equivalent [14,15] to a $\nu$-High Density Set (HDS) [16] formally defined as ${\mathrm{HDS}}_{\nu }\left(Z\right)=\{\mathbf{z}\in {\mathbb{R}}^d|f_Z\left(\mathbf{z}\right)>c_{\nu }\}$, where $c_{\nu }$ is the largest constant such that $P\left({\mathrm{HDS}}_{\nu }\left(Z\right)\right)\ge 1-\nu$, for $0<\nu <1$. Therefore, the complement of MES is a suitable set to define outlier data in the sample, considering $\tilde{x}\left(t\right)\notin {\mathrm{MES}}_{\nu }$ as an atypical realization of X. Next, we give two approaches to estimate MES.

3.1. Parametric Approach

Given a random sample of n discrete random paths $\{x_1\left(t_i\right),\dots ,x_n\left(t_i\right)\}$ for $i=1,\dots ,m$, we transform this sample into d-dimensional vectors $S_n=({\mathbf{z}}_1,\dots ,{\mathbf{z}}_n)$ using the representation and truncation method proposed in this work, numerically implemented in Lines 2–8 in Algorithm 1. Assume further that $f_Z(\mathbf{z},\mathit{\theta })$ is a suitable probability model for the random sample ${\mathbf{z}}_1,\dots ,{\mathbf{z}}_n$, then we estimate by Robust Maximum Likelihood (RML) the parameters $\mathit{\theta }$. For instance, in this paper, we consider $f_Z(\mathbf{z},\mathit{\theta })$ to be the normal density, and then, RML estimated parameters are $\widehat{\mathit{\theta }}=(\widehat{\mathit{\mu }},\widehat{\Sigma })$, the robust mean vector and covariance matrix, respectively. For details on robust estimation, we refer to [17]. After the estimation of the distribution parameters, the computation of $H_{\alpha }$ follows by plugging the estimated density $f_Z(\mathbf{z},\widehat{\mathit{\theta }})$ into Equation (1). Moreover, for the normal model, the estimated set ${\mathrm{MES}}_{\nu }$ is defined trough the following expression:

$${\mathrm{MES}}_{\nu }\left(S_n\right)=\{\mathbf{z}\in {\mathbb{R}}^d|{(\mathbf{z}-\widehat{\mathit{\mu }})}^T{\widehat{\Sigma }}^{-1}(\mathbf{z}-\widehat{\mathit{\mu }})\le {\chi }_d^2\left(\nu \right)\},$$

where ${\chi }_d^2\left(\nu \right)$ is the $1-\nu$ quantile of a Chi-square distribution with d-degrees of freedom. Then, if the coefficient ${\mathbf{z}}_i$, representing ${\tilde{x}}_i\left(t\right)$, lies outside this ellipsoid, we say that the functional datum is atypical. When the proportion of outlier $\nu$ in the sample is known a priori, the ${\chi }_d^2\left(\nu \right)$-quantile can be replaced by the corresponding sample $1-\nu$ Mahalanobis distance quantile, as is the case in Section 4.1.

3.2. Non-Parametric Approach

The following are definitions to introduce further non-parametric estimation methods. For the random vector $Z\in {\mathbb{R}}^d$ distributed according to $F_Z$, let $B_Z(\mathbf{z},r_{\delta })\subset {\mathbb{R}}^d$ be the $\mathbf{z}$-centered ball with radius $r_{\delta }$ that fulfills the condition $\delta ={\int }_{B_Z(\mathbf{z},r_{\delta })}{f}_Z\left(\mathbf{z}\right)d\mathbf{z}$, then the $\delta$-neighbors of the point $\mathbf{z}$ comprise the open set ${\Delta }_{\mathbf{z}}={\mathbb{R}}^d\bigcap B(\mathbf{z},r_{\delta })$.

Definition 3 (δ-local α-entropy).

Let $\mathbf{z}\in {\mathbb{R}}^d$, for $\alpha >0$ and $\alpha \ne 1$; the δ-local α-entropy of the r.v. Z is:

$$h_{\alpha }\left({\Delta }_{\mathbf{z}}\right)=\frac{1}{1-\alpha }log\left({\int }_{{\Delta }_{\mathbf{z}}}{f}_Z^{\alpha }\left(\mathbf{z}\right)d\mathbf{z}\right)forall\mathbf{z}\in {\mathbb{R}}^d.$$

Under mild regularity conditions on $f_Z$, the local entropy measure is a suitable metric to characterize the degree of abnormality of every point $\mathbf{z}$ in the support of $F_Z$. Several natural estimators of local entropy measures can be considered, for instance the (average) distance from the point $\mathbf{z}$ to its k-th-nearest neighbor. We estimate MES combining the estimated $\delta$-Local $\alpha$-entropy. As in the parametric case, let $\{x_1\left(t_i\right),\dots ,x_n\left(t_i\right)\}$ for $i=1,\dots ,m$ be a random sample of n discrete random paths; we transform this sample into d-dimensional vectors $S_n=({\mathbf{z}}_1,\dots ,{\mathbf{z}}_n)$ following Lines 2–8 in Algorithm 1. Next, we estimate the local entropy for these data using the estimator ${\widehat{h}}_{\alpha }\left({\Delta }_{{\mathbf{z}}_i}\right)=exp\left({\overline{d}}_k({\mathbf{z}}_i,S_n)\right)$, where ${\overline{d}}_k({\mathbf{z}}_i,S_n)$ is the average distance from ${\mathbf{z}}_i$ to its k-th-nearest neighbor [18], and then estimate ${\mathrm{MES}}_{\nu }$ solving the following optimization problem:

$${\displaystyle \underset{\rho ,{ϵ}_1,\dots ,{ϵ}_n}{max}(1-\nu )\rho -\frac{1}{n}\sum _{i=1}^n{ϵ}_{i}s.t.{\widehat{h}}_{\alpha }\left({\Delta }_{{\mathbf{z}}_i}\right)\ge \rho -{ϵ}_i,{ϵ}_i\ge 0\mathrm{for}i=1,\dots ,n.}$$

(8)

The solution to this problem, ${\rho }^{*}$, leads to the following decision function:

$$D\left(\mathbf{z}\right)=sign({\rho }^{*}-{\widehat{h}}_{\alpha }\left({\Delta }_{\mathbf{z}}\right)),$$

where $D\left(\mathbf{z}\right)=+1$ if $\mathbf{z}$ corresponds to the $(1-\nu )$ proportion of curves projected near the origin, that is the set of curves that belongs to a low entropy (high density) set. The following theorem shows that as the number of available curves increases, the estimation method asymptotically detects the proportion $1-\nu$ of curves belonging to the ${\mathrm{MES}}_{\nu }$.

Theorem 1.

At the solution of the optimization problem stated in Equation 8, the following equality holds:

$${\displaystyle \underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}I\left({\mathbf{z}}_i\right)=1-\nu ,}$$

where $I\left(\mathbf{z}\right)=1$ if ${\widehat{h}}_{\alpha }\left({\Delta }_{\mathbf{z}}\right)\le {\rho }^{*}$ and $I\left(\mathbf{z}\right)=0$ otherwise.

4. Experimental Section

The aim of this section is to illustrate the performance of the proposed methodology to detect abnormal observations in a sample of functional data. In what follows, for the representation of functional data, we consider the Gaussian kernel function $K(t_l,t_k)=e^{-\sigma \parallel t_l-t_k{\parallel }^2}$. The kernel parameter $\sigma$ and the regularization coefficient $\gamma$ in Algorithm 1 were defined through cross-validation.

4.1. Simulation Analysis

In a Monte Carlo study, we investigate the performance of the proposed method over three data configurations (Scenarios A, B and C). Specifically, we consider the following generating processes: a fraction $1-\nu$ of $n=400$ curves are realizations of the following stochastic model:

$$X_l\left(t\right)=\sum _{j=1}^4{\xi }_{j}sin\left(j\pi t\right)+{\epsilon }_l\left(t\right),\mathrm{for}l=1,\dots ,(1-\nu )n,\mathrm{and}t\in [0,1],$$

where $\mathit{\xi }=({\xi }_1,\dots ,{\xi }_4)$ is a normally-distributed multivariate random variable with mean ${\mathit{\mu }}_{\xi }=(4,2,4,1)$ and diagonal co-variance matrix ${\Sigma }_{\xi }=diag(5,2,2,1)$, and ${\epsilon }_l\left(t\right)$ are independent autocorrelated random error functions.

The remaining proportion of data $n\nu$ with $\nu \in \{1\%,5\%,10\%\}$ comprises outliers that contaminate the sample according to the following typical scenarios (see [19]):

(A)

Magnitude outliers: $Y_l\left(t\right)={\sum }_{j=1}^4{\zeta }_{j}sin\left(j\pi t\right)+{\epsilon }_l\left(t\right),\mathrm{for}l=1,\dots ,\nu n,\mathrm{and}t\in [0,1],$ where $\zeta$ is a normally-distributed multivariate r.v. with parameters ${\mathit{\mu }}_{\zeta }=2.5{\mathit{\mu }}_{\xi }$ and ${\Sigma }_{\zeta }={\left(2.5\right)}^2{\Sigma }_{\xi }$.

(B)

Shape outliers: $Y_l\left(t\right)={\sum }_{j=1}^4{\zeta }_{j}sin\left(j\pi t\right)+{\epsilon }_l\left(t\right),\mathrm{for}l=1,\dots ,\nu n,\mathrm{and}t\in [0,1],$ where $\zeta$ is a normally-distributed multivariate r.v. with parameters ${\mathit{\mu }}_{\zeta }=(4,-2,1,3)$ and ${\Sigma }_{\zeta }={\Sigma }_{\xi }$.

(C)

A combination considering $\nu n/2$ outliers from Scenario A and $\nu n/2$ outliers from Scenario B.

To illustrate the generating process, in Figure 2, we show one instance of the simulated paths in Scenario C with $\nu =10\%$. We test our Parametric entropy (PA) and Non-Parametric entropy (NPA) method against several well-known depth measures for functional anomaly detection, namely: the Modified Band Depth (MBD), the H-Mode Depth (HMD), the Random Tukey Depth (RTD) and the Functional Spatial Depth (FSD) (see [10,11,12,13]), respectively, already implemented in the R-package fda-usc [20]. For this experiment, the values of the parameter $\nu$ are assumed known in each scenario. With respect to parameters $\sigma$ and $\gamma$ in Algorithm 1, in this simulation exercise, we chose them with a 10-fold cross-validation procedure using a single set of data, which correspond to the first instance of the simulations. The reference values (which remain fixed throughout the simulation exercise) are $\sigma =10$ and $\gamma ={0.1}^5$.

Figure 2. (Left) Raw data, 400 curves corresponding to Scenario C with $\nu =10\%$. (Right) Functional data, in black (“—”), the sample of regular paths $X\left(t\right)$, and abnormal curves $Y\left(t\right)$ in red (“—”).

Let P and N be the amount of outlier and normal data in the sample, respectively, and let TP = True Positive and TN = True Negative be the respective quantities detected by different methods; in Table 1, we report the following average metrics TPR = TP/P (True Positive Rate or sensitivity), TNR = TN/N (True Negative Rate or specificity) and the area under the ROC curve (aROC) of each method obtained through the $M=1000$ replications in the Monte Carlo study.

Table 1. Simulation analysis: Scenarios and contamination percentages $\nu$ in columns. In rows, different methods and average sensitivities, specificities and the areas under the ROC curves (aROC) (this last on a scale of 10²). The corresponding standard-error is reported in parenthesis.

Method	Metric	Scenario A			Scenario B			Scenario C
		10%	5%	1%	10%	5%	1%	10%	5%	1%
MBD	TPR	74.867	71.010	55.300	48.275	39.395	13.475	67.787	58.365	36.300
		(4.699)	(7.712)	(20.852)	(5.914)	(9.013)	(16.180)	(5.351)	(7.772)	(18.341)
	TNR	97.207	98.474	99.548	94.252	96.810	99.126	96.420	97.808	99.356
		(0.522)	(0.406)	(0.210)	(0.657)	(0.474)	(0.163)	(0.594)	(0.409)	(0.185)
	aROC	96.662	97.375	97.735	89.393	91.693	93.244	95.272	95.444	95.354
		(1.245)	(1.517)	(3.059)	(2.033)	(2.388)	(4.425)	(1.399)	(1.831)	(4.370)
HMD	TPR	92.665	91.545	88.675	66.532	62.780	47.475	79.992	76.765	66.025
		(3.295)	(5.173)	(14.793)	(6.084)	(8.809)	(21.206)	(4.562)	(7.039)	(18.004)
	TNR	99.185	99.555	99.885	96.281	98.041	99.469	97.776	98.777	99.656
		(0.366)	(0.272)	(0.149)	(0.676)	(0.463)	(0.214)	(0.506)	(0.370)	(0.181)
	aROC	99.200	99.256	99.346	94.980	96.153	96.969	97.676	97.924	97.842
		(0.851)	(1.105)	(2.391)	(1.583)	(1.812)	(3.473)	(1.089)	(1.401)	(3.542)
RTD	TPR	83.555	83.045	76.400	50.972	43.940	22.700	71.975	65.225	49.700
		(4.743)	(0.694)	(18.931)	(9.409)	(1.279)	(2.1334)	(7.178)	(9.716)	(1.834)
	TNR	98.174	99.104	99.762	94.544	97.049	99.218	96.889	98.165	99.491
		(0.526)	(0.365)	(0.191)	(1.045)	(0.674)	(0.215)	(0.798)	(0.511)	(0.184)
	aROC	98.187	98.605	98.962	90.426	92.510	94.154	96.156	96.345	96.242
		(1.094)	(1.347)	(2.538)	(2.817)	(2.967)	(4.574)	(1.580)	(1.977)	(4.085)
FSD	TPR	81.472	83.215	81.925	50.275	46.550	27.400	74.775	69.485	53.775
		(3.978)	(5.947)	(16.671)	(5.238)	(8.018)	(19.547)	(4.601)	(6.859)	(16.707)
	TNR	97.941	99.116	99.817	94.475	97.186	99.267	97.197	98.396	99.533
		(0.442)	(0.313)	(0.168)	(0.582)	(0.421)	(0.197)	(0.511)	(0.361)	(0.168)
	aROC	97.934	98.738	99.163	90.059	93.279	95.485	96.777	97.148	97.125
		(1.030)	(1.232)	(2.490)	(1.794)	(2.061)	(3.723)	(1.158)	(1.477)	(3.682)
Entropy-PA	TPR	94.150	93.215	91.725	80.740	77.390	66.925	87.550	84.935	77.650
		(3.078)	(4.817)	(12.591)	(6.250)	(8.550)	(20.330)	(4.632)	(6.604)	(17.015)
	TNR	99.350	99.649	99.916	97.860	98.810	99.664	98.616	99.207	99.774
		(0.342)	(0.253)	(0.127)	(0.694)	(0.450)	(0.205)	(0.514)	(0.347)	(0.171)
	aROC	99.351	99.353	99.374	97.549	97.987	98.301	98.677	98.752	98.641
		(0.788)	(1.078)	(2.474)	(1.364)	(1.495)	(2.785)	(0.944)	(1.208)	(3.081)
Entropy-NPA	TPR	92.725	91.505	89.050	74.215	77.145	71.250	87.225	85.805	79.775
		(3.325)	(5.228)	(14.630)	(6.237)	(7.904)	(19.970)	(4.217)	(6.198)	(16.788)
	TNR	99.191	99.552	99.889	97.135	98.792	99.709	98.586	99.252	99.795
		(0.369)	(0.275)	(0.147)	(0.693)	(0.416)	(0.201)	(0.468)	(0.326)	(0.169)
	aROC	99.243	99.266	99.293	97.240	98.253	98.685	98.782	98.880	98.861
		(0.815)	(1.097)	(2.528)	(1.130)	(1.250)	(2.550)	(0.856)	(1.145)	(2.880)

As can be seen, the PA and NPA entropy methods proposed in this article outperform other recently-proposed depth measures in the three scenarios considered in the experiments when $\nu =\{0.10,0.05\}$. In the remaining case (when $\nu =0.01$), PA and NPA outperform the other methods; however, the standard errors are slightly high to confirm a significant difference between the methods.

When we compare among the proposed methods, the parametric approach seems to be slightly (but consistently) more effective than the non-parametric approach in Scenario A. For Scenarios B and C, both methods provide similar results. It is important to remark that the PA method is especially adequate for Gaussian data, while the NPA method does not assume any distributional hypothesis on the data. In this sense, the simulation results show the robustness of the non-parametric approach even when competing with parametric methods designed for specific distributions.

4.2. Outliers in the Context of Mortality-Rate Curve Analysis

We consider the French mortality rates database, available in the R-package Demography [21], to study age-specific male death rates in a logarithmic scale. In Figure 3 (left), each curve corresponds to one year from 1901–2006 (106 paths in total) and accounts for the number of deaths per 1000 of the mean population in the age group (from 0–101 years) in question. As expected, for low-age cohorts (until 12 years, approximately), the mortality rates present a decreasing trend and then start to grow until late ages, where all cohorts achieve a 100% mortality rate.

Figure 3. French mortality data: On the left, the regular curves in black (“—”) and outliers detected in red (“—”) for $\nu =10\%$. On the right, the first two principal components of the kernel eigenfunctions; the area inside the doted blue ellipsoid (- -) corresponds PA estimation of ${\mathrm{MES}}_{\nu =90\%}$ and the region inside the convex hull in blue (—) to the NPA estimation. The regular curves, represented with black dots (•), lie inside the ${\mathrm{MES}}_{\nu =90\%}$ and detected outliers with a red asterisk (∗) outside of ${\mathrm{MES}}_{\nu =90\%}$.

For some years, the evolution pattern of mortality presents an atypical behavior, mostly coinciding with the first and second World Wars, jointly with the influenza pandemic episode that took place in 1919.

In this experiment, we do not know a priori the proportion of atypical curves. Therefore, after having conducted inference over a wide range of values for $\nu$, as a way to assess the sensitivity and reliability of the inference when determining the number of abnormal curves, we decided to fix $\nu =10\%$. For further details on the way to choose the parameter $\nu$ (and an extended sensitivity analysis on the values of $\nu$), please refer to § 3.2 in the Supplementary Material. In Figure 3 (left), we highlight in red the anomalous detected curves with both the entropy-PA and NPA methods corresponding to the years 1914–1919 and 1940, 1942–1945, which match with men (between 20 and 40 years old) participating in World War I and II. In Figure 3 (right), we use the first two principal components of the kernel eigenfunctions to project the representation coefficients (in this experiment, in ${\mathbb{R}}^{14}$) in two dimensions. As can be seen, the points laying outside the ${\mathrm{MES}}_{\nu =90\%}$, represented with doted-blue ellipses when estimating it with PA (- -) and the convex hull with a continuous blue line (—) when estimating it with NPA, correspond to the the atypical curves in the sample.

5. Discussion

In this article, we propose a definition of entropy for stochastic processes. We provide a reproducing kernel Hilbert space model to estimate entropy from a random sample of realizations of a stochastic process, namely functional data, and introduce two approaches to estimate minimum entropy sets for functional anomaly detection.

In the experimental section, the Monte Carlo simulation illustrates the adequacy of the proposed method in the context of magnitude and shape outliers, outperforming other state of the art methods for functional anomaly detection. In the study of French mortality rates, the parametric and non-parametric approaches for minimum entropy sets estimation show their adequacy to capture anomalous curves, principally associated with the First and Second World Wars and the Influenza episode in 1919.

Regardless of the results presented in the paper, how widely the method can be used in practice, especially with noisier data, is an open question. In this sense, as future work, we will consider testing the performance of the proposed method in other scenarios with different noise assumptions in the observations. Another natural extension for future work entails the study of the asymptotic properties of the ${\mathrm{MES}}_{\nu }$ estimators. The extension of the proposed method from the stochastic process to random fields, useful for several statistical and information science areas, seems straightforward, but a wide range of simulations and numerical experiments must be done in order to stress the performance of entropy methods in comparison to other techniques when dealing with abnormal fields. Another natural avenue for future work entails the study of the connections between entropy for stochastic process, as formally defined here, and the maximum entropy principle when estimating the governing parameters of Gaussian processes.