Semi-Parametric Estimation Using Bernstein Polynomial and a Finite Gaussian Mixture Model

Abstract

The central focus of this paper is upon the alleviation of the boundary problem when the probability density function has a bounded support. Mixtures of beta densities have led to different methods of density estimation for data assumed to have compact support. Among these methods, we mention Bernstein polynomials which leads to an improvement of edge properties for the density function estimator. In this paper, we set forward a shrinkage method using the Bernstein polynomial and a finite Gaussian mixture model to construct a semi-parametric density estimator, which improves the approximation at the edges. Some asymptotic properties of the proposed approach are investigated, such as its probability convergence and its asymptotic normality. In order to evaluate the performance of the proposed estimator, a simulation study and some real data sets were carried out.

1. Introduction

Density estimation is a widely adopted tool for multiple tasks in statistical inference, machine learning, visualization and exploratory data analysis. Existing density estimation algorithms can be categorized into either parametric, semi-parametric, or non-parametric approaches. In the non-parametric framework, several methods have been set forward for the smooth estimation of density and distribution functions. The most popular one, called kernel method, was introduced by [1]. The advances were carried out by [2] to estimate a density function. The reader is recommended to consult the paper [3] for an introduction to several kernel smoothing techniques. However, kernel methods display estimation problems at the edges, when we have a random variable X with density function f supported on a compact interval. Moreover, if $X_1,\dots ,X_n$ is a sample with the same density f, it is well known, in non-parametric kernel density estimation, that the bias of the standard kernel density estimator

$$\begin{array}{ccc}\hfill {\widehat{f}}_n\left(x\right)& =& \frac{1}{n{h}_n}\sum _{i=1}^{n}K\left(\frac{x-X_i}{h_n}\right),\hfill \end{array}$$

(1)

is of a larger order near the boundary than that in the interior, where K is a kernel (that is, a positive function satisfying $\int K\left(x\right)dx=1$) and ($h_n$) is a bandwidth (that is, a sequence of positive real numbers that goes to zero). Let us now suppose that f has two continuous derivatives everywhere and that, as $n\to \infty$, $h=h_n\to 0$ and $nh\to 0$. Let $x=ph$ for $p>0$. Near the boundary, the expression of the mean and the variance are indicated as

$$\mathbb{E}\left[{\widehat{f}}_n\left(x\right)\right]\simeq f\left(x\right){\int }_{-\infty }^{p}K\left(x\right)dx-f^{\prime }\left(x\right){\int }_{-\infty }^{p}xK\left(x\right)dx+\frac{1}{2}{h}^2{f}^{″}\left(x\right){\int }_{-\infty }^p{x}^{2}K\left(x\right)dx,$$

and

$$Var\left[{\widehat{f}}_n\left(x\right)\right]\simeq {\left(nh\right)}^{-1}f\left(x\right){\int }_{-\infty }^p{K}^2\left(x\right)dx.$$

These bias phenomena are called boundary bias. Numerous authors have elaborated methods for reducing these phenomena, such as data reflection [4], boundary kernels [5,6,7], local linear estimator [8,9], use of beta and gamma kernels [10,11] and bias reduction [12,13]. For a smooth estimator of a density function f with finite known support, there have been several methods, such as Vitale’s method [14], which is based on Bernstein polynomials and expressed as

$$\begin{array}{ccc}\hfill {\tilde{f}}_{1,n,m}\left(x\right)& =& m\sum _{k=0}^{m-1}\left[F_n\left(\frac{k+1}{m}\right)-F_n\left(\frac{k}{m}\right)\right]b_k(m-1,x),\hfill \end{array}$$

(2)

where $F_n$ is the empirical distribution function and $b_k(m,x)=C_m^k{x}^k{(1-x)}^{m-k}$ is the Bernstein polynomial. This estimator was investigated in the literatures [15,16,17,18] and, more recently, by [12,19,20].

Within the parametric framework, it is noteworthy that the Gaussian mixture model can be used to estimate any density function, without any problem of estimation on the edge. This refers to the fact that the set of all normal mixture densities is dense in the set of all density functions under the $L^1$ metric [21]. The investigation of mixture models stands for a full field in modern statistics. It is a probabilistic model introduced by [22] to illustrate the presence of subpopulations within an overall population. It has been developed so far by various authors, such as [23]. It is used for data classification and it provides efficient approaches of model-based clustering. The authors of [24] demonstrated that, when a Gaussian mixture model is used to estimate a density non-parametrically, the density estimator that uses the Bayesian information criterion (BIC) of [25] to select the number of components in the mixture is consistent [26].

However, we obtain the non-parametric kernel estimate of a density if we fit a mixture of n components in equal proportions $1/n$, where n is the size of the observed sample. As a matter of fact, it can be inferred that mixture models occupy an interesting niche between parametric and non-parametric approaches to statistical estimation.

More recently, in the parametric context, [27] proposed a parametric model using Bernstein polynomials with positive coefficients to estimate the unknown density function f; this estimator is defined as follows:

$$\begin{array}{ccc}\hfill f_B(x,p_m)& =& \sum _{i=1}^m{\widehat{p}}_{mi}{\mathcal{B}}_{mi}\left(x\right),\hfill \end{array}$$

(3)

where $B_{mi}\left(x\right)=(m+1)b_i(m,x)$, for $i=0,\dots ,m$, $p_m={(p_{m1},\dots ,p_{mm})}^T$ ($p_{mi}\ge 0$, $i=1,\dots ,m$, ${\sum }_{i=1}^m{p}_{mi}\le 1$) and ${\widehat{p}}_{mi}$ are the estimators of the parameters $p_{mi}$, obtained by the Expectation Maximization (EM) algorithm as follows:

$$p_{mi}^{(s+1)}=\frac{1}{n}\sum _{j=1}^n{\displaystyle \frac{p_{mi}^{\left(s\right)}{\mathcal{B}}_{mi}\left(x_j\right)}{{\sum }_{k=0}^m{p}_{mk}^{\left(s\right)}{\mathcal{B}}_{mk}\left(x_j\right)}},i=0,\dots ,m;s=0,1,\dots$$

with ${\widehat{p}}_{mi}=\underset{s\to \infty }{lim}{p}_{mi}^{\left(s\right)},$ for $i=1,\dots ,m.$ The proposed method gives a consistent estimator in $L^2$ distance under some conditions.

The problem at the edge does not arise for the parametric model. For this reason, the basic idea of this work is to consider a shrinkage method using Bernstein (Vitale’s estimator) and Gaussian mixture estimators, to construct a shrinkage density estimator, in order to improve the approximation at the edge. A shrinkage estimator is a convex combination between estimators [28]. Basically, this implies that a naive or raw estimate is improved by combining it with other information.

The remainder of this paper is organized as follows: In the next section, we recall some intrinsic properties of the classical EM algorithm in the context of the Gaussian mixture parameter estimation. In Section 3, we introduce a new semi-parametric estimation approach based on the shrinkage method using Bernstein polynomials and Gaussian mixture densities. In Section 4, the consistency of the proposed estimator is exhibited, as well as its asymptotic normality.

Section 5 highlights a simulation study that compares the performance of the proposed approach with the Bernstein estimator, the standard Gaussian kernel estimator and Guan’s estimator. The closing Section 6 crowns the whole work, wraps the conclusion and provides new perspectives for future work.

2. Background

The Gaussian Mixture Model and Em Algorithm

Let us consider $X=(X_1,\dots ,X_n)$, a sequence of independent and identically distributed (i.i.d.) with common Gaussian mixture density defined by

$$\begin{array}{ccc}\hfill g\left(x\right|\theta )& =& \sum _{k=1}^K{\pi }_k\mathcal{N}({\mu }_k,{\sigma }_k)\left(x\right),\hfill \end{array}$$

(4)

where

$$\theta =(\pi ,\mu ,\sigma )=({\pi }_1\dots ,{\pi }_K,{\mu }_1\dots ,{\mu }_K,{\sigma }_1,\dots ,{\sigma }_K),$$

satisfies

$$0\le {\pi }_k\le 1,\sum _{k=1}^K{\pi }_k=1,{\mu }_k\in \mathbb{R},{\sigma }_k>0,\mathrm{for}k=1,\dots ,K\mathrm{where}K>0,$$

and

$$\mathcal{N}(\mu ,\sigma )\left(x\right)=\frac{1}{\sigma \sqrt{2\pi }}exp\left(-\frac{{(x-\mu )}^2}{2{\sigma }^2}\right).$$

Finally, for each observed data point $X_i$, we associate a component label vector $Z_i$ in order to manage the data clustering. This random vector $Z_i={\left(Z_{ik}\right)}_{1\le k\le K}$ is defined such that $Z_{ik}=1$ if the considered observation $X_i$ is drawn form the $k^{kt}$ component of the mixture and $Z_{ik}=0$ otherwise. Consequently, $Z_i$ is distributed as a multivariate Bernoulli distribution with vector parameters $({\pi }_1,\dots ,{\pi }_K)$ as follows:

$$\mathbb{P}\left(Z_i=z_i\right)=\prod _{k=1}^K{\pi }_k^{z_{ik}}.$$

The EM algorithm is a popular tool in statistical estimation problems involving incomplete data or problems which can be posed in a similar form, such as the mixture parameters estimation [23,29]. In the EM framework, $(X_1,\dots ,X_n,Z_1,\dots ,Z_n)$ corresponds to the complete data and $(Z_1,\dots ,Z_n)$ stand for the hidden data. Hence, the complete-data log-likelyhood is expressed by

$$\begin{array}{ccc}\hfill L(X_1,\dots ,X_n,Z_1,\dots ,Z_n,\theta )& =& \sum _{i=1}^n\sum _{j=1}^K{Z}_{ij}\left[log\left({\pi }_j\right)+log\left(\mathcal{N}({\mu }_j,{\sigma }_j)\left(X_i\right)\right)\right].\hfill \end{array}$$

(5)

The two steps of the EM algorithm, after l iterations, are the following:

(i) 

E-step: The conditional expectation of the complete-data log-likelyhood given the observed data, using the current fit ${\theta }^{\left(l\right)}$, is defined by

$$\begin{array}{ccc}\hfill \phi \left(\theta |{\theta }^{\left(l\right)}\right)& =& {\mathbb{E}}_{{\theta }^{\left(l\right)}}\left(L(X_1,\dots ,X_n,Z_1,\dots ,Z_n,\theta )|X_1,\dots ,X_n\right).\hfill \end{array}$$

(6)

The posterior probability that $X_i$ belongs to the $j$ component of the mixture at the $l$ iteration, is expressed as

$$\begin{array}{c}\hfill {\tau }_{ij}^{\left(l\right)}={\mathbb{E}}_{{\theta }^{\left(l\right)}}\left(Z_{ij}|X_1,\dots ,X_n\right)=\frac{{\pi }_j^{\left(l\right)}\mathcal{N}\left({\mu }_j^{\left(l\right)},{\left({\sigma }^2\right)}_j^{\left(l\right)}\right)\left(X_i\right)}{{\sum }_{h=1}^K{\pi }_h^{\left(l\right)}\mathcal{N}{\left({\mu }_h^{\left(l\right)},({\sigma }_h^2\right)}^{\left(l\right)}\left(X_i\right)}.\end{array}$$

(7)

Finally, we obtain

$$\begin{array}{ccc}\hfill \phi \left(\theta |{\theta }^{\left(l\right)}\right)& =& \sum _{i=1}^n\sum _{j=1}^K{\tau }_{ij}^{\left(l\right)}\left[log\left({\pi }_j\right)+log\left(\mathcal{N}({\mu }_j,{\sigma }_j)\left(X_i\right)\right)\right].\hfill \end{array}$$

(8)

(ii) 

M-step: It consists of a global maximization of $\phi \left(\theta |{\theta }^{\left(l\right)}\right)$ with respect to $\theta$.

$$\begin{array}{ccc}\hfill {\theta }^{(l+1)}& =& arg\underset{\theta }{max}\phi \left(\theta |{\theta }^{\left(l\right)}\right).\hfill \end{array}$$

(9)

The updated estimates are stated by

$$\begin{array}{c}\hfill {\pi }_j^{(l+1)}=\frac{1}{n}\sum _{i=1}^n{\tau }_{ij}^{\left(l\right)},\end{array}$$

(10)

$$\begin{array}{ccc}\hfill {\mu }_j^{(l+1)}& =& \frac{{\sum }_{i=1}^n{\tau }_{ij}^{\left(l\right)}{X}_i}{{\sum }_{i=1}^n{\tau }_{ij}^{\left(l\right)}},\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\left({\sigma }_j^2\right)}^{(l+1)}& =& \frac{{\sum }_{i=1}^n{\tau }_{ij}^{\left(l\right)}{\left(X_i-{\mu }_j^{(l+1)}\right)}^2}{{\sum }_{i=1}^n{\tau }_{ij}^{\left(l\right)}}.\hfill \end{array}$$

(12)

We repeat these two steps until $\left|\left|{\theta }^{(l+1)}-{\theta }^{\left(l\right)}\right|\right|<ϵ,$ where $ϵ$ is a fixed threshold of convergence. The convergence properties of the EM algorithm have been investigated by [29] and by [30]. Relying upon Jensen’s inequality, it can be noticed that, as $\phi \left(\theta |{\theta }^{\left(l\right)}\right)$ increases, the log-likelihood function also increases [29]. Consequently, the EM algorithm converges within a finite iteration number and gives the parameters’ maximum likelihood estimates. Therefore, under some conditions and according to [29], we have

$$\begin{array}{c}\hfill \underset{l\to \infty }{lim}{\pi }_j^{\left(l\right)}={\widehat{\pi }}_j,\underset{l\to \infty }{lim}{\mu }_j^{\left(l\right)}={\widehat{\mu }}_j\mathrm{and}\underset{l\to \infty }{lim}{\left({\sigma }_j^2\right)}^{\left(l\right)}={\widehat{{\sigma }^2}}_j\mathrm{almost}\mathrm{surely}(\mathrm{a}-\mathrm{s}).\end{array}$$

(13)

In what follows, $\widehat{\theta }=({\widehat{\pi }}_1,\dots ,{\widehat{\pi }}_K,{\widehat{\mu }}_1,\dots ,{\widehat{\mu }}_K,{\widehat{\sigma }}_1,\dots ,{\widehat{\sigma }}_K)$.

3. Proposed Approach

The proposed semi-parametric approach rests upon the shrinkage combination between the Gaussian mixture model and the Bernstein density estimators using the EM algorithm for the parameter estimations. The literature on shrinkage estimation is enormous. From this perspective, it is noteworthy to mention the most relevant contributions. The authors of [28] were the first to introduce the classic shrinkage estimator. The authors of [31] provided theory for the analysis of risk. Oman [32,33] developed estimators which shrink Gaussian density estimators towards linear subspaces. An in-depth investigation of shrinkage theory is displayed in Chapter 5 of [34].

The proposed semi-parametric approach based upon estimating the density function f relies on the same principle of Stein’s works and there are two aspects along this line. The first setting is non-parametric in the sense that we do not assume any parametric form of the density. The non-parametric setting is very important as it allows us to perform statistical inference without making any assumption on the parametric form of the true density f. The second setting is to consider the Gaussian mixture model as a parametric estimator of the unknown density f.

In what follows, we consider $X_1,\dots ,X_n$ a sequence of i.i.d. random variables having a common unknown density function f supported on $[0,1]$. We here develop a shrinkage method to estimate the density function, which is divided into the following three steps:

Step 1

We consider the Bernstein estimator of the density function f, which is defined as

$$\begin{array}{ccc}\hfill {\tilde{f}}_{1,n,m}\left(x\right)& =& m\sum _{i=0}^{m-1}\left[F_n\left(\frac{i+1}{m}\right)-F_n\left(\frac{i}{m}\right)\right]b_i(m-1,x)\hfill \end{array}$$

(14)

Step 2

In view of (13), we consider the Gaussian mixture density as an estimator of the density function f, given by

$$\begin{array}{c}\hfill {\tilde{f}}_{2,n}\left(x\right)=\sum _{k=1}^K{\widehat{\pi }}_k\mathcal{N}({\widehat{\mu }}_k,{\widehat{\sigma }}_k)\left(x\right),\end{array}$$

(15)

where ${\widehat{\mu }}_k$, ${\widehat{\sigma }}_k$ and ${\widehat{\pi }}_k$ are estimated by the EM algorithm defined in (13).

Step 3

We consider the shrinkage density estimator ${\widehat{f}}_{n,m}$ form defined by

$${\widehat{f}}_{n,m}\left(x\right)=\lambda {\tilde{f}}_{1,n,m}\left(x\right)+(1-\lambda ){\tilde{f}}_{2,n}\left(x\right),$$

and we use the EM algorithm to estimate the parameter $\lambda \in [0,1]$ of the proposed model.

By the same way as considered in Section 2, the two steps of the EM algorithm, after t iterations, are denoted in terms of the following:

1. 

E-step: The conditional expectation of the complete-data log-likelihood given the observed data, using the current ${\lambda }^{\left(t\right)}$, is provided by

$$\begin{array}{ccc}\hfill Q\left(\lambda \right|{\lambda }^{\left(t\right)})& =& \sum _{i=1}^n{\mathbb{E}}_{{\lambda }^{\left(t\right)}}\left(W_{i1}\mid X_i\right)log{\tilde{f}}_{1,n,m}\left(X_i\right)+{\mathbb{E}}_{{\lambda }^{\left(t\right)}}\left(W_{i2}\mid X_i\right)log{\tilde{f}}_{2,n}\left(X_i\right),\hfill \end{array}$$

where $W_i=(W_{i1},W_{i2})$ is a discrete random vector, following a multivariate Bernoulli distribution with vector parameters $(\lambda ,1-\lambda )$. Using Bayes’s formula, we obtain the posterior probability in the $t$ iteration denoted by

$$\begin{array}{ccc}\hfill {\overline{\tau }}_{i1}^{\left(t\right)}& =& \frac{{\tilde{f}}_{1,n,m}\left(X_i\right){\lambda }^{\left(t\right)}}{{\lambda }^{\left(t\right)}{\tilde{f}}_{1,n,m}\left(X_i\right)+(1-{\lambda }^{\left(t\right)}){\tilde{f}}_{2,n}\left(X_i\right)},\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill {\overline{\tau }}_{i2}^{\left(t\right)}& =& \frac{{\tilde{f}}_{2,n}\left(X_i\right){\lambda }^{\left(t\right)}}{{\lambda }^{\left(t\right)}{\tilde{f}}_{1,n,m}\left(X_i\right)+(1-{\lambda }^{\left(t\right)}){\tilde{f}}_{2,n}\left(X_i\right)}=1-{\overline{\tau }}_{i1}^{\left(t\right)}.\hfill \end{array}$$

2. 

M-step: It consists of a global maximization of $Q\left(\lambda \right|{\lambda }^{\left(t\right)})$ with respect to $\lambda$.

$${\lambda }^{(t+1)}=arg\underset{\lambda }{max}Q(\lambda \mid {\lambda }^{\left(t\right)}).$$

The updated estimate of $\lambda$ is indicated by

$${\lambda }^{(t+1)}=\frac{1}{n}\sum _{i=1}^n{\overline{\tau }}_{i1}^{\left(t\right)}.$$

The estimation of $\lambda$ is obtained from by iterating the EM algorithm until convergence.

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}{\lambda }^{\left(t\right)}=\widehat{\lambda }.\end{array}$$

(16)

Therefore, the proposed estimator of the density function f is defined by

$$\begin{array}{ccc}\hfill {\widehat{f}}_{n,m}\left(x\right)& =& \widehat{\lambda }{\tilde{f}}_{1,n,m}\left(x\right)+(1-\widehat{\lambda }){\tilde{f}}_{2,n}\left(x\right).\hfill \end{array}$$

(17)

Basically, it is a shrinkage estimator that shrinks the Bernstein estimator towards the Gaussian mixture density by a specified amount of $\lambda$. If $\lambda =1$, the estimator ${\widehat{f}}_{n,m}$ reduces to the Bernstein estimator ${\tilde{f}}_{1,n,n}$.

4. Convergence

In this section, we derive some asymptotic properties of the proposed estimator ${\widehat{f}}_{n,m}$ when the sample size tends to infinity. First, we assume that $\lambda$ and K are fixed. The following proposition gives the probability convergence of the proposed estimator ${\widehat{f}}_{n,m}$.

Proposition 1

(Probability convergence). If $m=o\left(n/log\left(n\right)\right)$, then, for $x\in [0,1]$, we have

$${\widehat{f}}_{n,m}\left(x\right)\underset{n,m\to +\infty }{\stackrel{P}{⟶}}\lambda f\left(x\right)+(1-\lambda )f_2\left(x\right),$$

where $f_2\left(x\right)={\sum }_{j=1}^K{\pi }_j\mathcal{N}({\mu }_j,{\sigma }_j^2)\left(x\right)$, ${\pi }_j=\mathbb{E}\left(Z_{1j}\right),$ ${\mu }_j=\mathbb{E}(X_1\mid Z_{1j}=1),$ ${\sigma }_j^2=Var(X_1\mid Z_{1j}=1)$ for $j=1,\dots ,K$ and $\stackrel{P}{⟶}$ denotes the convergence in probability.

The proof of Proposition 1 necessitates the following technical Lemma.

Lemma 1.

Let ${\left(S_n\right)}_{n\ge 1}$ be a sequence of i.i.d. random variables in the space of square integral functions $L^2$ with a common mean μ and let ${\left(T_n\right)}_{n\ge 1}$ be a sequence of random variables. Hence,

$$\mathbb{E}\left({\overline{S}}_n\right|T_n)\underset{n\to +\infty }{\stackrel{L^2}{⟶}}\mu ,where{\overline{S}}_n=\frac{1}{n}\sum _{i=1}^n{S}_i,$$

where $L^2$ denotes the mean quadratic convergence $L^2$.

The proof of this lemma is reported in [35].

Proof of Proposition 1.

First, using Lemma 1 and following the same steps as the proof of Theorem 4.4 in [35], we prove that ${\widehat{\pi }}_j\underset{n\to +\infty }{\stackrel{P}{⟶}}{\pi }_j,$ $\underset{n\to \infty }{lim}{\widehat{\mu }}_j\underset{n\to +\infty }{\stackrel{P}{⟶}}{\mu }_j$ and ${\widehat{{\sigma }^2}}_j\underset{n\to +\infty }{\stackrel{P}{⟶}}{\sigma }_j^2.$ Then, according to Slutsky’s Theorem, we obtain

$$\begin{array}{c}\hfill \sum _{k=1}^K{\widehat{\pi }}_k\mathcal{N}({\widehat{\mu }}_k,{\widehat{\sigma }}_k)\left(x\right)\underset{n\to +\infty }{\stackrel{P}{⟶}}\sum _{j=1}^K{\pi }_j\mathcal{N}({\mu }_j,{\sigma }_j^2)\left(x\right).\end{array}$$

(18)

Second, based on Theorem 3.1 in [16], we obtain

$$\begin{array}{c}\hfill {\tilde{f}}_{1,n,m}\left(x\right)\underset{n\to +\infty }{\stackrel{P}{⟶}}f\left(x\right)\mathrm{for}x\in [0,1].\end{array}$$

(19)

In addition, referring to (18) and (19) and grounded on the application of Slutsky’s Theorem, we conclude the proof.   □

According to [21], the density $f\left(x\right)$ is a close approximation to the mixture density $f_2\left(x\right)$. Thus, the estimator ${\widehat{f}}_{n,m}\left(x\right)$ provides an approximation to the true density $f\left(x\right)$.

To study the asymptotic normality of the estimator ${\widehat{f}}_{n,m}$ given by (17), we set forward the following assumptions in [36].

(A1) 

For almost $x\in [0,1]$ and for all $i,j,h=1\dots ,\nu$, the partial derivatives $\partial g/\partial {\xi }_i$, ${\partial }^{2}g/\partial {\xi }_i\partial {\xi }_j$ and ${\partial }^{3}g/\partial {\xi }_i\partial {\xi }_j\partial {\xi }_h$ of the density g exist and satisfy that ${\displaystyle \left|\frac{\partial g\left(x\right|\theta )}{\partial {\xi }_i}\right|,}$${\displaystyle \left|\frac{{\partial }^{2}g\left(x\right|\theta )}{\partial {\xi }_i{\xi }_j}\right|}$ and ${\displaystyle \left|\frac{{\partial }^{3}g\left(x\right|\theta )}{\partial {\xi }_i{\xi }_j{\xi }_h}\right|}$ are bounded, respectively, by $J_i$, $J_{ij}$ and $J_{ijh}$, where $J_i$ and $J_{ij}$ are integrable and $J_{ijh}$, satisfies

$${\int }_0^1{J}_{ijh}\left(x\right)g\left(x\right|\widehat{\theta })dx<\infty .$$

(A2) 

The Fisher information matrix $I\left(\theta \right)$ is positively defined at $\widehat{\theta }$.

Proposition 2

(Normality asymptotic). Under the regularity conditions(A1)–(A2), if $f\left(x\right)>0$ for all $x\in [0,1]$, $2\le m\le (n/logn)$ and $\underset{n,m\to \infty }{lim}{n}^{2/3}/m=0$, then, we obtain

$$n^{1/2}{m}^{-1/4}\left[{\widehat{f}}_{n,m}\left(x\right)-\lambda f\left(x\right)-(1-\lambda )f_2\left(x\right)\right]\underset{n,m\to +\infty }{\stackrel{\mathcal{D}}{⟶}}\mathcal{N}\left(0,{\lambda }^2\gamma \left(x\right)\right),$$

where $\gamma \left(x\right)=f\left(x\right){\left(4\pi x(1-x)\right)}^{-1/2},$ for $x\in ]0,1[$, and $\stackrel{\mathcal{D}}{⟶}$ denotes the convergence in distribution.

Proof of Proposition 2.

Using Theorem 3.2 in [16], we obtain

$$n^{1/2}{m}^{-1/4}({\tilde{f}}_{1,n,m}\left(x\right)-f\left(x\right))\underset{n,m\to +\infty }{\stackrel{\mathcal{D}}{⟶}}\mathcal{N}(0,\gamma \left(x\right)).$$

Thus,

$$n^{1/2}{m}^{-1/4}(\lambda {\tilde{f}}_{1,n,m}\left(x\right)-\lambda f\left(x\right))\underset{n,m\to +\infty }{\stackrel{\mathcal{D}}{⟶}}\mathcal{N}(0,{\lambda }^2\gamma \left(x\right)).$$

According to Theorem 3.1 in [36], we obtain ${\displaystyle \sqrt{n}(\widehat{\theta }-\theta )\underset{n\to +\infty }{\stackrel{\mathcal{D}}{⟶}}\mathcal{N}(0,I{\left(\theta \right)}^{-1}).}$ Using the delta method, we obtain

$$\sqrt{n}({\tilde{f}}_{2,n}\left(x\right|\widehat{\theta })-f_2\left(x\right|\theta ))\underset{n\to +\infty }{\stackrel{\mathcal{D}}{⟶}}\mathcal{N}\left(0,D{f}_2\left(x\right|\theta )I{\left(\theta \right)}^{-1}D{f}_2{\left(x\right|\theta )}^T\right),$$

where $D{f}_2\left(x\right|\theta )$ is the Jacobian matrix of $f_2\left(x\right|\theta )=f_2\left(x\right)$ and ${\tilde{f}}_{2,n}\left(x\right|\widehat{\theta })={\tilde{f}}_{2,n}\left(x\right)$. Since $m^{-1/4}\to 0$ if $m\to \infty$, then, using Slutsky’s Theorem, we conclude the proof. □

The following corollary is a consequence of the previous proposition which gives an asymptotic confidence interval of the density f, for a risk $\alpha \in ]0,1[$.

Corollary 1.

The $100(1-\alpha )\%$ asymptotic confidence interval of $f\left(x\right)$ is given by

$$\left({\widehat{f}}_{n,m}\left(x\right)\pm \frac{z_{1-\frac{\alpha }{2}}\lambda \sqrt{\gamma \left(x\right)}}{\sqrt{n}{m}^{-1/4}}\right),$$

where $z_{1-\frac{\alpha }{2}}$ is the normal $(1-\frac{\alpha }{2})$ quantile.

In the next section, we study the performance of the proposed estimator in estimating different distributions by comparing it to the performances of the Bernstein estimator and of the Gaussian kernel estimator.

5. Numerical Studies

5.1. Comparison Study

In this section, we investigate the performance of the proposed estimator given in (17), through estimating different densities by comparing it to the performance of the Bernstein estimator defined in (2), the standard Gaussian kernel estimator defined in (1) and the Guan’s estimator defined in (3). We apply the Bernstein estimator when the sample is concentrated on the interval $[0,1]$. For this purpose, we need to make some suitable transformations in the different cases that are listed as follows:

1. 

Let us suppose that X is concentrated on a finite support $[a,b]$; then, we work with the sample values $Y_1,\dots ,Y_n$, where ${\displaystyle Y_i=(X_i-a)/(b-a)}$.

2. 

For the density functions concentrated on $\mathbb{R}$, we can use the transformed sample $Y_i=1/2+{\pi }^{-1}arctan\left(X_i\right)$, which transforms the range to the interval $(0,1)$.

3. 

For the support ${\mathbb{R}}_{+}$, we can use the transformed sample ${\displaystyle Y_i=X_i/(1+X_i)}$, which transforms the range to the interval $(0,1)$.

If the support is infinite, say, $\mathbb{R}=(-\infty ,\infty )$, we can consider $[x_1,x_t]\subset [a,b]$ as the finite support of f, where $x_1$ and $x_t$ are the minimum and the maximum order, respectively. We choose a and b such that $F\left(a\right)$ and $1-F\left(b\right)$ are of $O\left(n^{-1}\right)$, $a<x_1$, and $b>x_t$, where F is the distribution function [27]. Then, we can use the transformed sample, which transforms $[x_1,x_t]$ to the interval $[0,1]$ mentioned in the case 1.

In the simulation study, three sample sizes were considered, $n=50$, $n=100$, and $n=200$, as well as the following density functions:

(a) 

The beta mixture density $0.5\mathcal{B}(3,9)+0.5\mathcal{B}(9,3)$;

(b) 

The beta mixture density $0.5\mathcal{B}(3,1)+0.5\mathcal{B}(10,10)$;

(c) 

The normal mixture density $1/4\mathcal{N}(2,1)+3/4\mathcal{N}(-3,1)$;

(d) 

The chi-squared ${\chi }_n\left(2\right)$ density.

(e) 

The gamma mixture density $0.5\mathcal{G}(1,6)+0.5\mathcal{G}(6,1)$;

(f) 

The gamma mixture density $0.5\mathcal{G}(1,2)+0.5\mathcal{G}(4,2)$.

Our sample was decomposed into a learning sample of a size of 2/3 of the considered sample, on which the various statistical methods were constructed, and a second sample of a size of 1/3 of the considered sample, on which the predictive performance of the three methods were tested. For each density function f and sample size n, we computed the integrated squared error ($ISE$), the integrated absolute error ($IAE$) and the Kullback–Leibler divergence ($KL$) of the estimator ${\widehat{f}}_{n,m}$ over $N=500$ trials.

$$\widehat{ISE}=\frac{1}{N}\sum _{k=1}^{N}ISE\left({\widehat{f}}_k\right),\widehat{IAE}=\frac{1}{N}\sum _{k=1}^{N}IAE\left({\widehat{f}}_k\right)\mathrm{and}\widehat{KL}=\frac{1}{N}\sum _{k=1}^{N}KL\left({\widehat{f}}_k\right),$$

where ${\widehat{f}}_k$ is the estimator computed from the $k$ sample and

$$\begin{array}{ccc}\hfill ISE\left[{\widehat{f}}_k\right]& =& {\int }_0^1{\left({\widehat{f}}_k\left(x\right)-f\left(x\right)\right)}^{2}dx,IAE\left({\widehat{f}}_k\right)={\int }_0^1|{\widehat{f}}_k\left(x\right)-f\left(x\right)|dx,\hfill \\ \hfill KL\left({\widehat{f}}_k\right|f)& =& {\int }_0^1{\widehat{f}}_k\left(x\right)log\frac{{\widehat{f}}_k\left(x\right)}{f\left(x\right)}dx.\hfill \end{array}$$

Indeed, it is advised to consider a learning sample bigger than a testing sample. In this work, our sample was decomposed into a learning sample of a size of 2/3 of the considered sample, on which the various statistical methods were constructed, and a second sample of a size of 1/3 of the considered sample, on which the predictive performances of the three methods were tested. Each run of the proposed estimator performed the following steps:

- 

We first generated a random sample ${\left(X_i\right)}_{1\le i\le n}$ of size n from the models’ density $\left(a\right)-\left(f\right)$.

- 

We then split the generated data into a training set of a size of $2/3$ of the considered sample and a test set of a size of $1/3$ of the considered sample.

- 

We applied the proposed estimator, using the observed data $X_i$ only from the training set, in order to estimate the density function.

- 

The test set was then used to compute the estimation errors $\widehat{ISE}$, $\widehat{IAE}$ and $\widehat{KL}$.

To select the optimal parameter K, we used the Gap Statistics algorithm [37]. We considered a Monte Carlo experiment to select the optimal choice of the degree m of the Bernstein polynomial and the bandwidth h of the kernel estimator, for each point $x\in [0,1]$. We determined the parameters m (for $1\le m\le 300$) and h (for $h=i/1000$ with $1\le i\le 300$), which minimized the $ISE$, which was approximated by the $\widehat{ISE}$.

We considered $N=500$ random samples of sizes $n=50$, $n=100$ and $n=200$.

Departing from Table 1, Table 2 and Table 3 and Figure 1, we deduce the following:

Table 1. Average $\widehat{ISE}$ for $N=500$ trials of Bernstein estimator, standard Gaussian kernel estimator and the proposed estimator ${\widehat{f}}_{n,m}$, for $n=50$, $n=100$ and $n=200$. The bold values indicate the smallest values of $ISE$.

Density	n	Proposed	Bernstein	Kernel	Guan’s
		Estimator	Estimator	Estimator	Estimator
	50	$\mathbf{0.092242}$	$0.096684$	$0.197497$	$0.140323$
$\left(a\right)$	100	$\mathbf{0.091364}$	$0.092242$	$0.174251$	$0.089129$
	200	$\mathbf{0.075532}$	$0.079299$	$0.148143$	$0.086305$
	50	$1.157827$	$1.215347$	$0.530446$	$\mathbf{0.816906}$
$\left(b\right)$	100	$\mathbf{0.235402}$	$0.306704$	$0.482152$	$0.276573$
	200	$\mathbf{0.199786}$	$0.289870$	$0.474805$	$0.255716$
	50	$\mathbf{0.001423}$	$1.808252$	$2.222369$	$1.035522$
$\left(c\right)$	100	$\mathbf{0.000384}$	$1.410606$	$1.641689$	$1.014602$
	200	$\mathbf{0.000227}$	$1.348292$	$1.077352$	$0.994346$
	50	$0.525192$	$2.812448$	$4.936701$	$0.589465$
$\left(d\right)$	100	$\mathbf{0.492752}$	$2.483141$	$2.331765$	$0.579595$
	200	$\mathbf{0.162917}$	$0.898103$	$1.154646$	$0.507362$
	50	$2.180849$	$2.231986$	$2.424656$	$\mathbf{1.084340}$
$\left(e\right)$	100	$2.050098$	$2.133496$	$2.295932$	$\mathbf{0.835670}$
	200	$2.042379$	$2.086204$	$2.053453$	$\mathbf{0.717715}$
	50	$\mathbf{0.313388}$	$0.896995$	$1.397111$	$0.663889$
$\left(f\right)$	100	$\mathbf{0.253988}$	$0.656400$	$0.762742$	$0.516530$
	200	$\mathbf{0.186290}$	$0.577408$	$0.417980$	$0.472094$

Table 2. Average $\widehat{IAE}$ for $N=500$ trials of Bernstein estimator, standard Gaussian kernel estimator and the proposed estimator ${\widehat{f}}_{n,m}$, for $n=50$, $n=100$ and $n=200$. The bold values indicate the smallest values of $IAE$.

Density	n	Proposed	Bernstein	Kernel	Guan’s
		Estimator	Estimator	Estimator	Estimator
	50	$\mathbf{0.250241}$	$\mathbf{0.250241}$	$0.391072$	$0.251641$
$\left(a\right)$	100	$\mathbf{0.196423}$	$0.207109$	$0.367361$	$0.232399$
	200	$\mathbf{0.180536}$	$0.191673$	$0.348499$	$0.214117$
	50	$0.855008$	$0.823416$	$\mathbf{0.621137}$	$0.747562$
$\left(b\right)$	100	$\mathbf{0.417735}$	$0.457722$	$0.669027$	$0.438088$
	200	$\mathbf{0.386280}$	$0.455983$	$0.583057$	$0.423595$
	50	$\mathbf{0.035161}$	$0.971720$	$1.238839$	$0.948451$
$\left(c\right)$	100	$\mathbf{0.019331}$	$0.960044$	$1.157838$	$0.944000$
	200	$\mathbf{0.013233}$	$0.923259$	$0.953675$	$0.931293$
	50	$0.669269$	$0.807667$	$1.942776$	$\mathbf{0.648617}$
$\left(d\right)$	100	$0.657815$	$1.486974$	$1.415036$	$\mathbf{0.642986}$
	200	$\mathbf{0.351205}$	$1.573886$	$0.881866$	$0.637942$
	50	$0.830317$	$0.834203$	$1.499828$	$\mathbf{0.678362}$
$\left(e\right)$	100	$0.706770$	$0.713225$	$1.374609$	$\mathbf{0.645112}$
	200	$0.681767$	$0.692357$	$1.225502$	$\mathbf{0.620315}$
	50	$\mathbf{0.453948}$	$0.640596$	$1.068483$	$0.608421$
$\left(f\right)$	100	$\mathbf{0.414676}$	$0.714844$	$0.782939$	$0.584572$
	200	$\mathbf{0.383248}$	$0.883320$	$0.516527$	$0.584320$

Table 3. Average $\widehat{KL}$ for $N=500$ trials of Bernstein estimator, standard Gaussian kernel estimator and the proposed estimator ${\widehat{f}}_{n,m}$, for $n=50$, $n=100$ and $n=200$. The bold values indicate the smallest values of $KL$.

Density	n	Proposed	Bernstein	Kernel	Guan’s
		Estimator	Estimator	Estimator	Estimator
	50	$\mathbf{0.025048}$	$0.025048$	$0.289818$	$0.066817$
$\left(a\right)$	100	$\mathbf{0.012086}$	$0.015023$	$0.081830$	$0.058541$
	200	$0.003256$	$\mathbf{0.001788}$	$0.060468$	$0.029419$
	50	$1.088053$	$1.120246$	$\mathbf{0.575298}$	$0.667079$
$\left(b\right)$	100	$\mathbf{0.284795}$	$0.325933$	$0.381406$	$0.659105$
	200	$0.255697$	$0.310759$	$\mathbf{0.150096}$	$0.654338$
	50	$\mathbf{2.689781}$	$3.871172$	$4.732324$	$3.360316$
$\left(c\right)$	100	$\mathbf{0.011844}$	$3.591156$	$4.196295$	$3.356093$
	200	$\mathbf{0.009426}$	$3.505050$	$3.169852$	$3.318666$
	50	$0.976450$	$0.870222$	$4.702359$	$\mathbf{0.878995}$
$\left(d\right)$	100	$0.960633$	$1.572251$	$3.031783$	$\mathbf{0.763044}$
	200	$\mathbf{0.281862}$	$1.584022$	$1.537355$	$0.742696$
	50	$\mathbf{1.549035}$	$1.560172$	$5.538142$	$1.799133$
$\left(e\right)$	100	$\mathbf{1.207084}$	$1.217043$	$3.498273$	$1.557894$
	200	$\mathbf{1.153420}$	$1.169645$	$1.322299$	$1.387843$
	50	$\mathbf{0.528337}$	$1.052789$	$1.952294$	$0.651422$
$\left(f\right)$	100	$\mathbf{0.292017}$	$0.805962$	$1.191070$	$0.537625$
	200	$\mathbf{0.062893}$	$0.589790$	$0.679154$	$0.339442$

Figure 1. Quantitative comparison between the proposed estimator and Guan’s estimator of $0.5\mathcal{B}(3,9)+0.5\mathcal{B}(9,3)$ for $n=50$ (left) and $n=100$ (right).

- 

The results displayed in Table 1, Table 2 and Table 3 show that the $\widehat{ISE}$, $\widehat{IAE}$ and $\widehat{KL}$ decreased as the sample size increased.

- 

Using the proposed estimator, we obtained better results than those given by the other estimators in a large part of the cases.

- 

The Figure 2 and 3 give a better sense of where the error is located.

Figure 2. Quantitative comparison among the mean squared error of the kernel estimator, the Bernstein estimator, the Guan’s estimator and the proposed estimator of $0.5\mathcal{G}(1,6)+0.5\mathcal{G}(6,1)$ for $n=200$.

- 

For the case (e) of the gamma mixture, the average $\widehat{ISE}$ and $\widehat{IAE}$ of Guan’s estimator (1.3) were smaller than those obtained by the proposed density estimator (3.4) and the Bernstein estimator (1.2). However, in all the other cases, using an appropriate choice of the degree m, the average $\widehat{ISE}$ and $\widehat{IAE}$ of the proposed density estimator (3.4) were smaller than what achieved by the kernel estimator (1.1), the Bernstein estimator (1.2) and Guan’s estimator (1.3), even when the sample size was large for same cases.

- 

When we changed the parameters of the gamma mixture density in the sense that we had a smaller bias, our estimator was more competitive than the other approaches and we obtained better results.

- 

Almost in all considered cases, the average $\widehat{KL}$ of the density estimator (17) was smaller than that obtained by the Bernstein estimator defined in (2), that of the kernel estimator defined in (1) and that of Guan’s estimator.

- 

In the considered distribution $0.5\mathcal{B}(3,9)+0.5\mathcal{B}(9,3)$, by choosing the appropriate m, the curve of the proposed distribution estimator (3.4) was closer to the true distribution than that of Guan’s estimator (1.3), even when the sample size was very large.

Referring to Figure 2 and Figure 3, we infer the following:

Figure 3. Quantitative comparison among the mean squared error of the kernel estimator, the Bernstein estimator, the Guan’s estimator and the proposed estimator of $0.25\mathcal{N}(2,1)+0.75\mathcal{N}(-3,1)$ for $n=200$.

- 

None of the estimators for the gamma mixture density $0.5\mathcal{G}(6,1)+0.5\mathcal{G}(1,6)$ had good approximations near $x=0$. However, the $\widehat{ISE}$ of the proposed estimator was closer to zero than that of the Bernstein estimator and the kernel estimator, especially near the edge $x=1$.

- 

Guan’s estimator and the kernel estimator for the normal mixture density $0.25\mathcal{N}(2,1)+0.75\mathcal{N}(-3,1)$ had good approximations near $x=0$. However, the $\widehat{ISE}$ of the proposed estimator was closer to zero than that of the other estimators, especially near the two edges.

Therefore, we note that, for difficult distributions that diverge at the boundaries, the proposed method would fail, but not as badly as the standard methods without shrinkage. In addition, the performed simulations revealed that, on average, the proposed approach could lead to satisfactory estimates near the boundaries, better than the classical Bernstein estimator.

5.2. Real Dataset

5.2.1. COVID-19 Data

In this subsection, we consider the COVID-19 data displayed in the INED website https://dc-covid.site.ined.fr/fr/donnees/france/ (accessed on 16 February 2022). These data concern the numbers of deaths due to COVID-19 in France (daily) from 21 March 2021, for 454 days. These data are such that ${min}_i\left(x_i\right)=605$ and ${max}_i\left(x_i\right)=0$. Then, it is convenient to assume that the density of the numbers of deaths is defined on the interval $[0,605]$ and transform the data into the interval unit. The Monte Carlo procedure was performed and resulted in $h=0.07659612$ for the standard kernel estimator defined in (1.1), $m_1=20$ for the Bernstein estimator defined in (1.2), the proposed estimator, and $m_2=12$ for Guan’s estimator. These estimators are exhibited in Figure 1 (right panel) along with a histogram of the data. All the estimators are smooth and seem to capture the pattern highlighted by the histogram. We record that the proposed estimator outperformed the other estimators near the boundaries.

5.2.2. Tuna Data

The last example concerns the tuna data reported in [38]. The data are derived from an aerial line transect survey of Southern Bluefin Tuna in the Great Australian Bight. An aircraft with two spotters on board flew randomly over allocated line transects. These data correspond to the perpendicular sighting distances (in miles) of 64 detected tuna schools to the transect lines. The survey was conducted in summer when tuna data tend to stay on the surface. The data are such that ${min}_i\left(x_i\right)=0.19$ and ${max}_i\left(x_i\right)=16.26$. The Monte Carlo procedure was performed and resulted in $h=0.1079$ for the standard kernel estimator defined in (1), $m_1=13$ for the Bernstein estimator defined in (2) and the proposed estimator, and $m_2=6$ for Guan’s estimator. These estimators are illustrated in Figure 4 (left panel) along with a histogram of the data. All the estimators are smooth and seem to capture the pattern highlighted by the histogram. We assert that the proposed estimator outperformed the other estimator, especially near the boundaries.

Figure 4. Qualitative comparison among the kernel estimator defined in (1), the Bernstein estimator defined in (2), Guan’s estimator (3) and the proposed density estimator (17) of Tuna data (left) and of COVID-19 data (right).

6. Conclusions

In this paper, we propose a shrinkage estimator of a density function based on the Bernstein density estimator and using a finite Gaussian mixture density. This method rests on three steps. The first step consists of considering the Bernstein estimator ${\tilde{f}}_{1,n,m}$. The second relies upon the Gaussian Mixture density ${\tilde{f}}_{2,n}$ as an estimator of the unknown density f. The last step consists of considering the shrinkage form $\lambda {\tilde{f}}_{1,n,m}+(1-\lambda ){\tilde{f}}_{2,n}$ and EM algorithm in order to estimate the parameter $\lambda$. The asymptotic properties of this estimator were established. Afterwards, we demonstrate the effectiveness of the proposed method using some simulated and real data. We clarify how it can lead to very satisfactory estimates near the boundaries and in terms of $ISE$, $IAE$ and $KL$. Eventually, we would simply assert that our research work is a step that may be taken further, extended and built upon as it lays the ground and paves the way for future works to elaborate a semi-parametric regression estimator using the shrinkage method. We also plan to work on the case where $\lambda$ is a random variable. Another future research direction would be to extend our findings to the setting of serially dependent observations.