Shannon Entropy Estimation for Linear Processes

Abstract

In this paper, we estimate the Shannon entropy $S\left(f\right)=-\mathbb{E}[log(f\left(x\right)\left)\right]$ of a one-sided linear process with probability density function $f\left(x\right)$. We employ the integral estimator $S_n\left(f\right)$, which utilizes the standard kernel density estimator $f_n\left(x\right)$ of $f\left(x\right)$. We show that $S_n\left(f\right)$ converges to $S\left(f\right)$ almost surely and in ${Ł}^2$ under reasonable conditions.

1. Introduction

Let $f\left(x\right)$ be the common probability density function of a sequence ${\left\{X_n\right\}}_{n=1}^{\infty }$ of identically distributed observations. The associated Shannon entropy

$$\begin{array}{c}\hfill S\left(f\right)=\mathbb{E}[-logf(X\left)\right]=-\int f\left(x\right)logf\left(x\right)dx\end{array}$$

(1)

of such an observation was first introduced by Shannon (1948). In his 1948 paper, Shannon utilized this tool in his mathematical investigation of the theory of communication. Today, entropy is widely applied in the fields of information theory, statistical classification, pattern recognition and so on, since it is a measure of the amount of uncertainty present in a probability distribution.

In the literature, several estimators for the Shannon entropy have been introduced. See Beirlant et al. (1997) for an overview. Many of these estimators have been studied in cases where the data are independent. In 1976, Ahmad and Lin (1976) obtained results using the resubstitution estimator $H_n=-\frac{1}{n}{\sum }_{i=1}^{n}ln{f}_n\left(X_i\right)$ for independent data ${\left\{X_i\right\}}_{i=1}^n$. In particular, he showed consistency in the first and second mean under certain regularity conditions. Here, $f_n\left(x\right)$ is the kernel density estimator. Dmitriev and Tarasenko (1973) reported results in 1973 for estimating functionals of the type $\int H\left(f\left(x\right),f^{\prime }\left(x\right),\dots ,f^k\left(x\right)\right)dx$, where the common density $f\left(x\right)$ of the independent $X_i$ is assumed to have at least k derivatives. Plugging in kernel density estimators (see their paper and references therein) for the arguments of H and integrating only over the symmetric interval $[-k_n,k_n]$, which is determined by a sequence ${\left\{k_n\right\}}_{n=1}^{\infty }$ of a certain order, they provided a result for the estimation of Shannon entropy using the estimator that Beirlant et al. (1997) refer to as the integral estimator. Their results give conditions for almost sure convergence.

Interestingly enough, Dmitriev and Tarasenko (1973) also provided (because their work is a more general investigation of functionals) a result for the estimation of the quadratic Rényi entropy $Q\left(f\right)=\int f^2\left(x\right)dx$. Conditions are provided specifically for the almost sure convergence of their estimator to the true value $Q\left(f\right)$. The estimation of Rényi entropy for the dependent case is challenging. A dependent case is treated by Sang et al. (2018). They studied the estimation of the quadratic Entropy for the one-sided linear process. Utilizing the Fourier transform along with the projection method, they demonstrate that the kernel entropy estimator satisfies a central limit theorem for short memory linear processes.

To study the Shannon entropy for dependent data is also a challenging problem, and to the best of our knowledge, general results for the Shannon entropy estimation of regular time series data are still unknown. In this paper, we study the Shannon entropy $S\left(f\right)$ for the one-sided linear process

$$\begin{array}{c}\hfill X_n=\sum _{i=0}^{\infty }{a}_i{\epsilon }_{n-i},\end{array}$$

(2)

where the innovations ${\epsilon }_i$ are independent and identically distributed real valued random variables on some probability space $(\Omega ,\mathcal{F},\mathbb{P})$ with mean zero and finite variance ${\sigma }_{\epsilon }^2$ and where the collection $\{a_i:i\ge 0\}$ of real coefficients satisfies $\sum _{i=0}^{\infty }{a}_i^2<\infty$. Additionally, we will require that the common density $f_{\epsilon }\left(x\right)$ of the innovations be bounded. The estimator we utilize employs the kernel method, which was first introduced by Rosenblatt (1956); Parzen (1962). The kernel estimator will be denoted by

$$\begin{array}{c}\hfill f_n\left(x\right)=\frac{1}{n{h}_n}\sum _{i=1}^{n}K\left(\frac{x-X_i}{h_n}\right),\end{array}$$

(3)

where the sequence ${\left\{h_n\right\}}_{n=1}^{\infty }$ provides the bandwidths, and $K:\mathbb{R}\to \mathbb{R}$ is the kernel function which satisfies ${\int }_{\mathbb{R}}K\left(x\right)dx=1$. Typically, the kernel function is a probability density function.

This method has proven to be successful in estimating probability density functions and their derivatives, regression functions, etc., in both the independent and dependent setting. For the independent setting, see the books (Devroye and Györfi (1985); Silverman (1986); Nadaraya (1989); Wand and Jones (1995); Schimek (2000); Scott (2015)) and the references therein. For the dependent setting, we refer the reader to (Tran (1992); Honda (2000); Wu and Mielniczuk (2002); Wu et al. (2010)). Bandwidth selection is an important issue in kernel density estimation, and there is a lot of research in this direction. See, e.g., Duin (1976); Rudemo (1982); Slaoui (2014, 2018).

A few remarks about notation and terms used in the paper follow. Let ${\left\{a_n\right\}}_{n=1}^{\infty }$ and ${\left\{b_n\right\}}_{n=1}^{\infty }$ be real-valued sequences. By $a_n=o\left(b_n\right)$ we understand that $a_n/b_n\to 0$ and $a_n=O\left(b_n\right)$ means that $limsup|a_n/b_n|<C$ for some positive number C. Essentially, this is the standard Landau little oh and big oh notation. When we write, $a_n\ll b_n$, we mean $a_n=o\left(b_n\right)$, and as one might guess, $b_n\gg a_n$ means $a_n\ll b_n$. We also employ the notation $a_n\asymp b_n$ to indicate that $0<{lim\; inf}_{n\to \infty }\frac{a_n}{b_n}\le {lim\; sup}_{n\to \infty }\frac{a_n}{b_n}<\infty$. A function $l:[0,\infty )\to \mathbb{R}$ is referred to as slowly varying (at ∞) if it is positive and measurable on $[A,\infty )$ for some $A\in {\mathbb{R}}^{+}$ such that $\underset{x\to \infty }{lim}l\left(\lambda x\right)/l\left(x\right)=1$ holds for each $\lambda \in {\mathbb{R}}^{+}$. The set of all functions $g:\mathbb{R}\to \mathbb{R}$ which are Hölder continuous of some order r will be denoted as ${\mathcal{C}}^r\left(\mathbb{R}\right)$. That is, for each $g\in {\mathcal{C}}^r\left(\mathbb{R}\right)$ there exists $C_g\in {\mathbb{R}}^{+}$, such that for all $x,x^{\prime }\in \mathbb{R}$, we have $|g\left(x\right)-g\left(x^{\prime }\right)|\le C_g{|x-x^{\prime }|}^r$, and when $r=1$, we recognize this as the well-known Lipschitz condition. The notation ${Ł}^p\left(E\right)$ with $0<p<\infty$ represents the set of all real-valued functions f defined on some measure space $(E,\mathcal{A},\mu )$ having the property that ${\int }_E{\left|f\left(x\right)\right|}^{p}d\mu <\infty$. In the case that $E=\mathbb{R}$ and unless otherwise specified, the measure $\mu$ is tacitly understood to be Lebesgue measure and $\mathcal{A}$ is assumed to contain the Borel sets. ${Ł}^{\infty }\left(E\right)$ refers to the set of real-valued functions defined on E which are bounded almost everywhere. Whenever the domain space of the function is understood, we may simply write ${Ł}^p$.

The following are bandwidth, kernel, and density conditions that we shall refer to throughout this paper:

B.1 

$h_n\asymp {(n^{-1}logn)}^{\frac{1}{5}}$;

K.1 

$K\in {\mathcal{C}}^{\iota }\left(\mathbb{R}\right)$ for some $\iota \in (0,1]$ is bounded with bounded support;

K.2 

$\int uK\left(u\right)du=0$;

D.1 

$f_{\epsilon },f_{\epsilon }^{\prime },f_{\epsilon }^{\prime \prime }\in {Ł}^{\infty }\left(\mathbb{R}\right)$;

D.2 

$f_{\epsilon },f_{\epsilon }^{\prime },f_{\epsilon }^{\prime \prime }\in {Ł}^2\left(\mathbb{R}\right)$;

D.3 

$f^{\prime \prime }\in {Ł}^{\infty }\left(\mathbb{R}\right)$.

Notice that the bandwidth, kernel, and density conditions are prefixed using B, K, and D, respectively.

In this first section, we have provided an introduction to the problem, a survey of past research in this area, and the notation to be used throughout. The main results are reported in Section Two. In Section Three, we present the proofs of the main results. Finally, the Appendix A introduces the reader to foundational results, which will be required in the proof of our main results.

2. Main Results

If $\{{\epsilon }_i:i\in \mathbb{Z}\}$ is a sequence of independent and identically distributed random variables over a common probability space $(\Omega ,\mathcal{F},\mathbb{P})$ in ${Ł}^q(\Omega )$ for some $q>0$, $\mathbb{E}{\epsilon }_i=0$ when $q\ge 1$, and ${\left\{a_i\right\}}_{i=0}^{\infty }$ is a sequence of real coefficients such that ${\displaystyle \sum _{i=0}^{\infty }}{\left|a_i\right|}^{2\wedge q}<\infty$, then the linear process $X_n$ given in (2) exists and is well-defined. For the case $q\ge 2$ where the innovations have finite variance, we say that the process has short memory (short-range dependence) if ${\displaystyle \sum _{i=0}^{\infty }}\left|a_i\right|<\infty$ and ${\displaystyle \sum _{i=0}^{\infty }}{a}_i\ne 0$ and long memory (long-range dependence) otherwise. Throughout, we assume that each ${\epsilon }_i\in {Ł}^q$ with $q\ge 2$.

Let $f\left(x\right)$ be the probability density function of the linear process $X_n={\displaystyle \sum _{i=0}^{\infty }}{a}_i{\epsilon }_{n-i}$, $n\in \mathbb{N}$ defined in (2).In this paper, we estimate the Shannon Entropy $-\int f\left(x\right)logf\left(x\right)dx$ of the linear process. To do this, we employ the integral estimator

$$\begin{array}{c}\hfill S_n\left(f\right)=-{\int }_{A_n}{f}_n\left(x\right)log{f}_n\left(x\right)dx,\end{array}$$

(4)

where $f_n\left(x\right)$ is the standard kernel density estimator defined in (3). The (random) sets $A_n$ are given by

$$\begin{array}{c}\hfill A_n=\{x\in \mathbb{R}:0<{\gamma }_n\le f_n\left(x\right)\},\end{array}$$

(5)

where ${\left\{{\gamma }_n\right\}}_{n=1}^{\infty }$ is an appropriately defined sequence in ${\mathbb{R}}^{+}$ that converges to zero.

Our estimator utilizes the kernel method of density estimation, and we will accordingly require adherence of the kernel to certain conditions. In addition, we impose some conditions on the bandwidths and on some of the densities of the problem. These conditions were listed in the previous section. Based on these conditions, let us consider the properties of the estimator (4). We proceed in a manner similar to the analysis done by Bouzebda and Elhattab (2011) for the independent case.

Theorem 1.

Let $\{X_n:n\in \mathbb{N}\}$ be the linear process given in (2), and assume that it has short memory. Furthermore, assume that $S\left(f\right)$ is finite. If the bandwidth, kernel, and density conditions listed earlier are satisfied, then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}{\left(\frac{n{\gamma }_n^5}{logn}\right)}^{\frac{2}{5}}|S_n\left(f\right)-{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx|\end{array}$$

is bounded almost surely whenever the condition ${\gamma }_n\gg h_n$ is imposed on the sequence ${\left\{{\gamma }_n\right\}}_{n=1}^{\infty }$.

Corollary 1.

If the conditions of Theorem 1 hold, then we have

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}|S_n\left(f\right)-S\left(f\right)|=0\end{array}$$

almost surely.

Theorem 2.

Let $\{X_n:n\in \mathbb{N}\}$ be the linear process given in (2), and assume that it has short memory. Furthermore, assume that $S\left(f\right)$ is finite. If the bandwidth, kernel, and density conditions listed earlier are satisfied, then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}{\left(\frac{n{\gamma }_n^5}{logn}\right)}^{\frac{2}{5}}{∥S_n\left(f\right)-{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx∥}_2\end{array}$$

(6)

is bounded whenever the condition ${\gamma }_n\gg h_n$ is imposed on the sequence ${\left\{{\gamma }_n\right\}}_{n=1}^{\infty }$.

Corollary 2.

If the conditions of Theorem 2 hold, then the mean squared error (MSE) satisfies

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathrm{MSE}\left(S_n\left(f\right)\right)=0.\end{array}$$

Remark 1.

In this paper, we work on the entropy estimation for short memory linear processes by applying the integral method. It is interesting to know whether the similar results hold for long memory linear processes. It is also interesting to know whether the resubstitution method works for dependent data such as linear processes. However, the research in these directions are beyond the scope of this paper. We leave research in these directions for future work.

Remark 2.

In a wide range of disciplines, including finance, geology, and engineering, many time series may be modeled using a linear process. In such instances, our result provides a method for estimating the associated Shannon Entropy. One example is the discriminatory data on the arrival phases of earthquakes and explosions, which were captured at a seismic recording station. Another example is the data about returns on the New York Stock Exchange. See these and many other time series data in the book by Shumway and Stoffer (2011) and other books on time series.

3. Proofs

Lemma 1.

If the conditions of Theorem 1 (or Theorem 2) hold, then

$$\begin{array}{c}\hfill \underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|=O\left({\left(\frac{logn}{n}\right)}^{\frac{2}{5}}\right)\end{array}$$

(7)

almost surely.

Proof. 

This lemma follows from Theorem 2 of Wu et al. (2010) (see their discussion immediately after the statement of Theorem 2 and in the penultimate paragraph of section 4.1). See also the discussion in the Appendix A on fundamental results. □

Lemma 2.

If the conditions of Theorem 1 (or Theorem 2) hold, then

$$\begin{array}{c}\hfill {\gamma }_n^5\gg \frac{logn}{n}.\end{array}$$

(8)

Proof. 

Because $h_n\asymp {(n^{-1}logn)}^{\frac{1}{5}}$, there exists $C\in {\mathbb{R}}^{+}$ such that

$$\frac{h_n^5}{n^{-1}logn}>C$$

for sufficiently large n. Therefore,

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}\frac{{\gamma }_n^5}{n^{-1}logn}& =\underset{n\to \infty }{lim}\frac{{\gamma }_n^5}{h_n^5}·\frac{h_n^5}{n^{-1}logn}\hfill \\ \hfill & \ge C\underset{n\to \infty }{lim}{\left(\frac{{\gamma }_n}{h_n}\right)}^5\to \infty \hfill \end{array}$$

as $n\to \infty$, from which (8) follows. □

Note. Our use of Lemma 2 in the proofs of Theorems 1 and 2 will be tacit.

Lemma 3.

If ν is a finite signed measure that is absolutely continuous with respect to a measure μ, then corresponding to every positive number ε there is a positive number δ such that $\left|\nu \right|\left(E\right)<\epsilon$ whenever E is a measurable set for which $\mu \left(E\right)<\delta$.

Proof. 

This is a basic result from measure theory. See, for example, Theorem B of Halmos (1974) in section 30. □

Proof of Theorem 1.

We begin with the decomposition

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill S_n\left(f\right)-& {\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx\hfill \\ \hfill & =-{\int }_{A_n}{f}_n\left(x\right)log{f}_n\left(x\right)dx+{\int }_{A_n}f\left(x\right)logf\left(x\right)dx\hfill \\ \hfill & =-{\int }_{A_n}{f}_n\left(x\right)log{f}_n\left(x\right)dx+{\int }_{A_n}f\left(x\right)log{f}_n\left(x\right)dx\hfill \\ \hfill & -{\int }_{A_n}f\left(x\right)log{f}_n\left(x\right)dx+{\int }_{A_n}f\left(x\right)logf\left(x\right)dx\hfill \\ \hfill & =I_{n,1}+I_{n,2},\hfill \end{array}\end{array}$$

(9)

where

$$\begin{array}{cc}\hfill I_{n,1}& :=-{\int }_{A_n}\left[f_n\left(x\right)-f\left(x\right)\right]log{f}_n\left(x\right)dx,\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill I_{n,2}& :=-{\int }_{A_n}f\left(x\right)\left[log{f}_n\left(x\right)-logf\left(x\right)\right]dx.\hfill \end{array}$$

First, we consider $I_{n,1}$. Using the inequality

$$\begin{array}{c}\hfill |logz|\le z+\frac{1}{z}\end{array}$$

for $z\in {\mathbb{R}}^{+}$, we notice that for all $x\in A_n$, we have

$$\begin{array}{cc}\hfill |log{f}_n\left(x\right)|& \le f_n\left(x\right)+\frac{1}{f_n\left(x\right)}\hfill \\ \hfill & =\left(1+\frac{1}{{\left(f_n\left(x\right)\right)}^2}\right)f_n\left(x\right)\hfill \\ \hfill & \le \left(1+\frac{1}{{\gamma }_n^2}\right)f_n\left(x\right).\hfill \end{array}$$

It follows that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill |I_{n,1}|& \le \underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|{\int }_{A_n}|log{f}_n\left(x\right)|dx\hfill \\ \hfill & \le \left(1+\frac{1}{{\gamma }_n^2}\right)\underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|,\hfill \end{array}\end{array}$$

(10)

since $f_n\left(x\right)$ integrates to unity over the real line.

Next, we consider $I_{n,2}$. Since the set over which we are integrating may be changed to $A_n\cap \{x:f\left(x\right)>0\}$ without affecting the value of $I_{n,2}$, we may assume that f is positive on $A_n$. Using the inequality

$$\begin{array}{c}\hfill logz\le |z-1|+|z^{-1}-1|\end{array}$$

for $z\in {\mathbb{R}}^{+}$, we notice that for all $x\in A_n$, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill |log{f}_n\left(x\right)-logf\left(x\right)|& =|ln\left(\frac{f_n\left(x\right)}{f\left(x\right)}\right)|\hfill \\ \hfill & \le |\frac{f_n\left(x\right)}{f\left(x\right)}-1|+|\frac{f\left(x\right)}{f_n\left(x\right)}-1|\hfill \\ \hfill & =\left|\frac{f_n\left(x\right)-f\left(x\right)}{f\left(x\right)}\right|+\left|\frac{f\left(x\right)-f_n\left(x\right)}{f_n\left(x\right)}\right|\hfill \\ \hfill & =\left(1+\frac{f_n\left(x\right)}{f\left(x\right)}\right)\left|\frac{f_n\left(x\right)-f\left(x\right)}{f_n\left(x\right)}\right|\hfill \\ \hfill & \le \frac{C}{{\gamma }_n}|f_n\left(x\right)-f\left(x\right)|,\hfill \end{array}\end{array}$$

(11)

if we can justify the existence of $C\in {\mathbb{R}}^{+}$. To that end, define

$$\begin{array}{c}\hfill {\epsilon }_n=\underset{x\in A_n}{sup}|f_n\left(x\right)-f\left(x\right)|,\end{array}$$

and note that for all $x\in A_n$, we have

$$\begin{array}{c}\hfill |1-\frac{f\left(x\right)}{f_n\left(x\right)}|\le \frac{{\epsilon }_n}{f_n\left(x\right)}\le \frac{{\epsilon }_n}{{\gamma }_n}.\end{array}$$

Taking the supremem over $A_n$ yields

$$\begin{array}{cc}\hfill \underset{x\in A_n}{sup}|1-\frac{f\left(x\right)}{f_n\left(x\right)}|& \le \frac{{\epsilon }_n}{{\gamma }_n}={\gamma }_n^{-1}{\epsilon }_n\hfill \\ \hfill & \le C{\gamma }_n^{-1}/{\left(\frac{n}{logn}\right)}^{\frac{2}{5}},\hfill \end{array}$$

by Lemma 1. Note that

$$\begin{array}{c}\hfill {\gamma }_n^{-1}=o\left({\left(\frac{n}{logn}\right)}^{\frac{2}{5}}\right),\end{array}$$

since

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}\frac{{\gamma }_n^{-1}}{{\left(\frac{n}{logn}\right)}^{\frac{2}{5}}}& =\underset{n\to \infty }{lim}\frac{{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}}{{\gamma }_n}\hfill \\ \hfill & =\underset{n\to \infty }{lim}{\left(\frac{\frac{logn}{n}}{{\gamma }_n^5}\frac{\frac{logn}{n}}{1}\right)}^{\frac{1}{5}}\hfill \\ \hfill & =0.\hfill \end{array}$$

This guarantees the existence we sought to establish. We continue with

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|I_{n,2}\right|& \le \frac{C}{{\gamma }_n}\underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|{\int }_{A_n}f\left(x\right)dx\hfill \\ \hfill & \le \frac{C}{{\gamma }_n}\underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|,\hfill \end{array}\end{array}$$

(12)

since $f_n\left(x\right)$ integrates to unity over the real line.

In view of (9), (10) and (12), we have shown that

$$\begin{array}{cc}\hfill |S_n\left(f\right)-& {\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx|\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & \le \left(\frac{1}{{\gamma }_n^2}+\frac{C}{{\gamma }_n}+1\right)\underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|.\hfill \end{array}$$

(14)

Therefore,

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}& {\left(\frac{n{\gamma }_n^5}{logn}\right)}^{\frac{2}{5}}|S_n\left(f\right)-{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx|\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}{\left(\frac{n}{logn}\right)}^{\frac{2}{5}}{\gamma }_n^2\left(\frac{1}{{\gamma }_n^2}+\frac{C}{{\gamma }_n}+1\right)\underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}\left({\gamma }_n^2+C{\gamma }_n+1\right){\left(\frac{n}{logn}\right)}^{\frac{2}{5}}\underset{x\in \mathbb{R}}{sup}|f_n\left(x\right)-f\left(x\right)|,\hfill \end{array}$$

where the last expression is constant almost surely by Lemma 1 and since ${\gamma }_n\to 0$. □

Proof of Corollary 1.

By the triangle inequality

$$\begin{array}{c}\hfill |S_n\left(f\right)-S\left(f\right)|\le J_{n,1}+J_{n,2},\end{array}$$

where

$$\begin{array}{c}\hfill J_{n,1}=|S_n\left(f\right)-{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx|\end{array}$$

and

$$\begin{array}{c}\hfill J_{n,2}=|{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx-S\left(f\right)|.\end{array}$$

Since $J_{n,1}\to 0$ almost surely by Theorem 1, we only need to contend with $J_{n,2}$. That is, we need to show that

$$\begin{array}{c}\hfill {\int }_{A_n^{\mathsf{c}}}f\left(x\right)logf\left(x\right)dx\to 0\end{array}$$

(15)

almost surely as $n\to \infty$.

For any Borel measurable set E, consider

$$\begin{array}{c}\hfill P\left(E\right)={\int }_{E}f\left(x\right)dx,\end{array}$$

and define the signed measure

$$\begin{array}{c}\hfill \nu \left(E\right)=-{\int }_{E}logf\left(x\right)dP.\end{array}$$

Since $\left|S\right(f\left)\right|<\infty$, both ${\nu }^{+}$ and ${\nu }^{-}$ are finite measures, and thus, $\nu$ is a finite signed measure that is absolutely continuous with respect to P. Because of Lemma 3, it suffices for us to demonstrate that

$$\begin{array}{c}\hfill P\left(A_n^{\mathsf{c}}\right)\to 0\end{array}$$

almost surely. For any $x\in A_n^{\mathsf{c}}$, we have $f_n\left(x\right)<{\gamma }_n$. By Lemma 1, there exists $C\in {\mathbb{R}}^{+}$ such that $f\left(x\right)\le f_n\left(x\right)+|f_n\left(x\right)-f\left(x\right)|<{\gamma }_n+C{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}$ almost surely, and hence, we have shown that $A_n^{\mathsf{c}}\subseteq B_n$ almost surely, where

$$B_n:=\left\{x:f\left(x\right)<{\gamma }_n+C{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}\right\}.$$

It is easy to see that

$$\begin{array}{c}\hfill 0\le P\left(A_n^{\mathsf{c}}\right)\le P\left(B_n\right)\to 0\end{array}$$

almost surely, since ${\gamma }_n+C{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}\to 0$ as $n\to \infty$. □

Proof of Theorem 2.

We start with

$$\begin{array}{cc}\hfill \parallel S_n\left(f\right)-& {\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx{\parallel }_2\hfill \\ \hfill & \le {∥S_n\left(f\right)+{\int }_{A_n}{f}_n\left(x\right)logf\left(x\right)dx∥}_2\hfill \\ \hfill & +{∥{\int }_{A_n}{f}_n\left(x\right)logf\left(x\right)dx-{\int }_{A_n}f\left(x\right)logf\left(x\right)dx∥}_2\hfill \\ \hfill & =:K_{n,1}+K_{n,2}.\hfill \end{array}$$

Recall inequality (11) in the proof of Theorem 1. Arguing in a similar manner as before, we can demonstrate the existence of $C_1\in {\mathbb{R}}^{+}$ so that

$$\begin{array}{cc}\hfill K_{n,1}& ={∥{\int }_{A_n}{f}_n\left(x\right)\left[logf\left(x\right)-log{f}_n\left(x\right)\right]dx∥}_2\hfill \\ \hfill & ={∥{\int }_{A_n}{f}_n\left(x\right)log\frac{f\left(x\right)}{f_n\left(x\right)}dx∥}_2\hfill \\ \hfill & \le {∥{\int }_{A_n}{f}_n\left(x\right)|log\frac{f\left(x\right)}{f_n\left(x\right)}|dx∥}_2\hfill \\ \hfill & \le {∥{\int }_{A_n}\frac{C_1}{{\gamma }_n}|f_n\left(x\right)-f\left(x\right)|f_n\left(x\right)dx∥}_2\hfill \\ \hfill & \le \frac{C_1}{{\gamma }_n}{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}{∥{\int }_{x\in A_n}{f}_n\left(x\right)dx∥}_2\hfill \\ \hfill & \le \frac{C_1}{{\gamma }_n}{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}.\hfill \end{array}$$

Notice also that

$$\begin{array}{cc}\hfill K_{n,2}& ={∥{\int }_{A_n}[f_n\left(x\right)-f\left(x\right)]logf\left(x\right)dx∥}_2\hfill \\ \hfill & \le {∥{\int }_{A_n}|f_n\left(x\right)-f\left(x\right)|logf\left(x\right)dx∥}_2\hfill \\ \hfill & \le C_2{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}{∥{\int }_{A_n}\left(f\left(x\right)+\frac{1}{f\left(x\right)}\right)dx∥}_2\hfill \\ \hfill & \le C_2{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}\left[1+{∥{\int }_{A_n}\frac{f_n\left(x\right)}{f\left(x\right)}\frac{1}{f_n^2\left(x\right)}{f}_n\left(x\right)dx∥}_2\right]\hfill \\ \hfill & \le C_2{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}\left(1+\frac{C_3}{{\gamma }_n^2}\right).\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill {\left(\frac{n{\gamma }_n^5}{logn}\right)}^{\frac{2}{5}}& {∥S_n\left(f\right)-{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx∥}_2\hfill \\ \hfill & \le {\left(\frac{n}{logn}\right)}^{\frac{2}{5}}{\gamma }_n^2\left[\frac{C_1}{{\gamma }_n}{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}+C_2{\left(\frac{logn}{n}\right)}^{\frac{2}{5}}\left(1+\frac{C_3}{{\gamma }_n^2}\right)\right]\hfill \\ \hfill & =C_1{\gamma }_n+C_2{\gamma }_n^2+C_2{C}_3,\hfill \end{array}$$

from which the result follows. □

Proof of Corollary 2.

Note the decomposition

$$\begin{array}{cc}\hfill \sqrt{\mathrm{MSE}\left(S_n\left(f\right)\right)}& ={∥S_n\left(f\right)-S\left(f\right)∥}_2\hfill \\ \hfill & \le {∥S_n\left(f\right)-{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx∥}_2\hfill \\ \hfill & +{∥{\int }_{A_n}\left(-logf\left(x\right)\right)f\left(x\right)dx-S\left(f\right)∥}_2\hfill \\ \hfill & =:M_{n,1}+M_{n,2}.\hfill \end{array}$$

By Theorem 2, $M_{n,1}\to 0$. Now, let

$$W_n={\int }_{x\in \mathbb{R}}{\int }_{y\in \mathbb{R}}f\left(x\right)f\left(y\right)logf\left(x\right)logf\left(y\right)\mathcal{I}\left((x,y)\in A_n^{\mathsf{c}}\times A_n^{\mathsf{c}}\right)dxdy$$

and

$$W={\int }_{x\in \mathbb{R}}{\int }_{y\in \mathbb{R}}f\left(x\right)f\left(y\right)|logf\left(x\right)logf\left(y\right)|dxdy.$$

Recall from (15) in the proof of Corollary 1 that $W_n\to 0$ almost surely. Because $\left|S\right(f\left)\right|<\infty$, it follows that $W<\infty$, and moreover, $|W_n|\le W$. Hence,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill M_{n,2}^2& ={∥{\int }_{A_n^{\mathsf{c}}}f\left(x\right)logf\left(x\right)dx∥}_2^2\hfill \\ \hfill & ={∥{\int }_{\mathbb{R}}f\left(x\right)logf\left(x\right)\mathcal{I}(x\in A_n^{\mathsf{c}})dx∥}_2^2\hfill \\ \hfill & =\mathbb{E}\left[W_n\right],\hfill \end{array}\end{array}$$

(16)

and the Lebesgue Dominated Convergence Theorem guarantees that

$$\underset{n\to \infty }{lim}{M}_{n,2}^2=\underset{n\to \infty }{lim}\mathbb{E}\left[W_n\right]=\mathbb{E}\left[\underset{n\to \infty }{lim}{W}_n\right]=\mathbb{E}\left[0\right]=0,$$

thereby proving the corollary. □