Entropy and Divergence Associated with Power Function and the Statistical Application

Abstract

In statistical physics, Boltzmann-Shannon entropy provides good understanding for the equilibrium states of a number of phenomena. In statistics, the entropy corresponds to the maximum likelihood method, in which Kullback-Leibler divergence connects Boltzmann-Shannon entropy and the expected log-likelihood function. The maximum likelihood estimation has been supported for the optimal performance, which is known to be easily broken down in the presence of a small degree of model uncertainty. To deal with this problem, a new statistical method, closely related to Tsallis entropy, is proposed and shown to be robust for outliers, and we discuss a local learning property associated with the method.

1. Introduction

Consider a practical situation in which a data set $\{x_1,\cdots ,x_n\}$ is randomly sampled from a probability density function of a statistical model $\{f_{\theta }\left(x\right):\theta \in \Theta \}$, where θ is a parameter vector and Θ is the parameter space. A fundamental tool for the estimation of unknown parameter θ is the log-likelihood function defined by

$$\begin{array}{c}\hfill \ell \left(\theta \right)=\frac{1}{n}\sum _{i=1}^{n}log{f}_{\theta }\left(x_i\right)\end{array}$$

(1)

which is commonly utilized by statistical researchers ranging over both frequentists and Bayesians. The maximum likelihood estimator (MLE) is defined by

$$\begin{array}{c}\hfill \widehat{\theta }=\underset{\theta \in \Theta }{\mathrm{argmin}}\ell \left(\theta \right)\end{array}$$

(2)

The Fisher information matrix for θ is defined by

$$\begin{array}{c}\hfill I_{\theta }=\int f_{\theta }\left(x\right)\frac{\partial }{\partial \theta }log{f}_{\theta }\left(x\right)\frac{\partial }{\partial {\theta }^{\mathrm{T}}}log{f}_{\theta }\left(x\right)dx\end{array}$$

(3)

where ${\theta }^{\mathrm{T}}$ denotes the transpose of θ. As the sample size n tends to infinity, the variance matrix of $\sqrt{n}(\widehat{\theta }-\theta )$ converges to $I_{\theta }^{-1}$. This inverse matrix exactly gives the lower bound in the class of asymptotically consistent estimators in the sense of matrix inequality, that is,

$$\begin{array}{c}\hfill {\mathrm{AV}}_{\theta }(\tilde{\theta })\ge I_{\theta }^{-1}\end{array}$$

(4)

for any asymptotically consistent estimator $\tilde{\theta }$ of θ, where ${\mathrm{AV}}_{\theta }$ denotes the limiting variance matrix under the distribution with the density $f_{\theta }\left(x\right)$.

On the other hand, the Boltzmann-Shannon entropy

$$\begin{array}{c}\hfill H_0\left(p\right)=\int p\left(x\right)logp\left(x\right)dx\end{array}$$

(5)

plays a fundamental role in various fields, such as statistical physics, information science and so forth. This is directly related with the MLE. Let us consider an underlying distribution with the density function $p\left(x\right)$. The cross entropy is defined by

$$\begin{array}{c}\hfill C_0(p,f_{\theta })=-\int p\left(x\right)log{f}_{\theta }\left(x\right)dx\end{array}$$

(6)

We note that $C_0(p,f_{\theta })={\mathrm{E}}_p\{-\ell \left(\theta \right)\}$, where ${\mathrm{E}}_p$ denotes the expectation with respect to $p\left(x\right)$. Hence, the maximum likelihood principle is equal to the minimum cross entropy principle. The Kulback-Leibler (KL) divergence is defined by

$$\begin{array}{c}\hfill D_0(p,q)=\int p\left(x\right)log\frac{p\left(x\right)}{q\left(x\right)}dx\end{array}$$

(7)

which gives a kind of information distance between p and q. Note that $D_0(p,f_{\theta })=C_0(p,f_{\theta })-C_0(p,p)$

An exponential (type) distribution model is defined by the density form

$$\begin{array}{c}\hfill f_{\theta }\left(x\right)=exp\{{\theta }^{\mathrm{T}}t\left(x\right)-\psi \left(\theta \right)\}\end{array}$$

(8)

where $\psi \left(\theta \right)$ is the cumulant transform defined by $log\int exp\left\{{\theta }^{\mathrm{T}}t\left(x\right)\right\}dx$. Under the assumption of this family, the MLE has a number of convenient properties such as minimal sufficiency, unbiasedness, efficiency [1]. In particular, the MLE for the expectation parameter $\eta ={\mathrm{E}}_{\theta }\left\{t\left(X\right)\right\}$ is explicitly given by

$$\begin{array}{c}\hfill {\widehat{\eta }}_0=\frac{1}{n}\sum _{i=1}^{n}t\left(x_i\right)\end{array}$$

which is associated with a dualistic relation of the canonical parameter θ and the expectation parameter η [2,3]. Thus, the MLE satisfies such an excellent property, which is associated with logarithmic and exponential functions as in (2) and (8).

The MLE has been widely employed in statistics, in which the properties are supported in theoretical discussion, for example, as in [4]. However, the MLE has some inappropriate properties when the underlying distribution does not belong to the model $\{f_{\theta }\left(x\right):\theta \in \Theta \}$. A statistical model is just simulation of the true distribution as Fisher pointed in [1]. The model, which is just used as a working model, is wrong in most practical cases. In such situations, the MLE does not show proper performance because of model uncertainty. In this paper we explore alternative estimation method than the MLE.

2. Power Divergence

The logarithmic transform for observed values is widely employed in data analysis. On the other hand, a power transformation defined by

$$\begin{array}{c}\hfill t_{\beta }\left(x\right)=\frac{x^{\beta }-1}{\beta }\end{array}$$

often gives more flexible performance to get good approximation to normal distribution [5]. In analogy with this transform, the power cross entropy is defined by

$$\begin{array}{c}\hfill C_{\beta }(p,q)=-\int p\left(x\right)\frac{q{\left(x\right)}^{\beta }-1}{\beta }dx+\int \frac{q{\left(x\right)}^{\beta +1}}{\beta +1}dx\end{array}$$

where β is a positive parameter. Thus, it is defined by the power transform of the density. If we take the limit of β to 0, then $C_{\beta }(p,q)$ converges to $C_0(p,q)$, which is given in (6). In fact, the power parameter β is not fixed, so that different β’s give different behaviors of the power entropy. The diagonal power entropy is defined by

$$\begin{array}{c}\hfill H_{\beta }\left(p\right)=\int \frac{(\beta +1)p\left(x\right)-p{\left(x\right)}^{\beta +1}}{\beta (\beta +1)}dx\end{array}$$

which is given by taking $C_{\beta }$ the diagonal. Actually, this is equivalent to Tsallis q-entropy with the relation $\beta =q-1$.

Let $\{x_1,\cdots ,x_n\}$ be a random sample from unknown density function $p\left(x\right)$. Then we define the empirical mean power likelihood by

$$\begin{array}{c}\hfill {\ell }_{\beta }\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n\frac{f_{\theta }{\left(x_i\right)}^{\beta }-1}{\beta }-{\kappa }_{\beta }\left(\theta \right)\end{array}$$

(9)

where ${\kappa }_{\beta }\left(\theta \right)=\int f_{\theta }{\left(x\right)}^{\beta +1}dx/\left(\beta +1\right)$. See [6,7,8,9] for statistical applications. Accordingly, the minus expectation value of ${\ell }_{\beta }\left(\theta \right)$ is equal to $C_{\beta }(p,f_{\theta })$. In general, a relation of cross and diagonal entropy leads to the inequality $C_{\beta }(p,q)\le C_{\beta }(p,p)$, from which we define the power divergence by

$$\begin{array}{c}\hfill D_{\beta }(p,q)=C_{\beta }(p,q)-C_{\beta }(p,p)\end{array}$$

We extend power entropy and divergence defined over the space of all density functions, which are not always assumed to have a total mass one. In particular, this extension is useful for proposing boosting methods [10,11,12,13,14,15,16].

This derivation can be extended by a generator function U. Assume that $U\left(t\right)$ is strictly increasing and convex. The Fencel duality discussion leads to a conjugate convex function of $U\left(t\right)$ defined by

$$\begin{array}{c}\hfill U^{*}\left(s\right)=\underset{t\in \mathbb{R}}{max}\{st-U\left(t\right)\}\end{array}$$

(10)

which reduces to $U^{*}\left(s\right)=s\xi \left(s\right)-U\left(\xi \left(s\right)\right)$, where $\xi \left(s\right)$ is the inverse function of the derivation $\dot{U}$ of U. Then, U-cross entropy is defined by

$$\begin{array}{c}\hfill C_U(\mu ,\nu )=-\int \mu \left(x\right)\xi \left(\nu \left(x\right)\right)dx+\int U\left(\xi \left(\nu \left(x\right)\right)\right)dx\end{array}$$

Similarly U-divergence is defined by

$$\begin{array}{c}\hfill D_U(\mu ,\nu )=\int \{U^{*}\left(\mu \left(x\right)\right)+U\left(\xi \left(\nu \left(x\right)\right)\right)-\mu \left(x\right)\xi \left(\nu \left(x\right)\right)\}dx\end{array}$$

(11)

We note that $D_U(\mu ,\nu )=C_U(\mu ,\nu )-C_U(\mu ,\mu )$. By the definition of $U^{*}$ in (10) we see that the integrand of the right-hand side of (11) is always nonnegative. The power divergence is one example of U-divergence by fixing

$$\begin{array}{c}\hfill U_{\beta }\left(t\right)=\frac{1}{\beta +1}{(1+\beta t)}^{\frac{\beta +1}{\beta }}\end{array}$$

The power divergence can be defined on $\mathcal{M}$ as

$$\begin{array}{c}\hfill D_{\beta }(\mu ,\nu )=\int \left\{\mu \left(x\right)\frac{\mu {\left(x\right)}^{\beta }-\nu {\left(x\right)}^{\beta }}{\beta }+\frac{\nu {\left(x\right)}^{\beta +1}-\mu {\left(x\right)}^{\beta +1}}{\beta +1}\right\}dx\end{array}$$

(12)

for μ and ν of $\mathcal{M}$ [17]. Thus $U_{\beta }\left(t\right)$ is strictly increasing and convex, which implies that the integrand of the right-hand side of (12) is nonnegative.

To explore this, it seems sufficient to restrict the definition domain of $D_{\beta }$ to $\mathcal{P}$. However, we observe that the restriction is not useful for statistical considerations. We discuss the restriction on the projective space as follows. Fix two functions μ and ν in $\mathcal{M}$. We say that μ and ν are projectively equivalent if there exists a positive scalar λ such that

$$\begin{array}{c}\hfill \nu \left(x\right)=\lambda \mu \left(x\right)\left(\mathrm{a}.\mathrm{e}.x\right)\end{array}$$

Thus, we write $\nu \sim \mu$. Similarly, we call a divergence D defined on $\mathcal{M}$ projectively invariant if for all $\lambda >0,\kappa >0$

$$\begin{array}{c}\hfill D(\lambda \mu ,\kappa \nu )=D(\mu ,\nu )\end{array}$$

(13)

We can derive a variant of power divergence as

$$\begin{array}{c}\hfill {\Delta }_{\beta }(\mu ,\nu )=\frac{1}{\beta (\beta +1)}log\int \mu {\left(x\right)}^{\beta +1}dx-\frac{1}{\beta }log\int \mu \left(x\right)\nu {\left(x\right)}^{\beta }dx+\frac{1}{\beta +1}log\int \nu {\left(x\right)}^{\beta +1}dx\end{array}$$

See Appendix 1 for the derivation. Immediately, we observe ${\Delta }_{\beta }$ satisfies (13), or projective invariance. Hereafter, we call ${\Delta }_{\beta }$ the projective power divergence. In this way, for $p\left(x\right)=\mu \left(x\right)/\int \mu \left(x\right)dx$ and $q\left(x\right)=\nu \left(x\right)/\int \nu \left(x\right)dx$, it is obtained that

$$\begin{array}{c}\hfill {\Delta }_{\beta }(p,q)={\Delta }_{\beta }(\mu ,\nu )\end{array}$$

If we take a specific value of β, then

$$\begin{array}{c}\hfill {\Delta }_{\beta =1}(\mu ,\nu )=\frac{1}{2}log\frac{\int \mu {\left(x\right)}^{2}dx\int \nu {\left(x\right)}^{2}dx}{{\left(\int \mu \left(x\right)\nu \left(x\right)dx\right)}^2}\end{array}$$

and

$$\begin{array}{c}\hfill \underset{\beta \to 0}{lim}{\Delta }_{\beta }(\mu ,\nu )=D_0\left(\frac{\mu }{\int \mu \left(x\right)dx},\frac{\nu }{\int \nu \left(x\right)dx}\right)\end{array}$$

where $D_0$ is nothing but the KL divergence (7). We observe that the projective power divergence satisfies information additivity. In fact, if we write p and q as $p(x_1,x_2)=p_1\left(x_1\right)p_2\left(x_2\right)$ and $q(x_1,x_2)=q_1\left(x_1\right)q_2\left(x_2\right)$, respectively, then

$$\begin{array}{c}\hfill {\Delta }_{\beta }(p,q)={\Delta }_{\beta }(p_1,q_1)+{\Delta }_{\beta }(p_2,q_2)\end{array}$$

which means information additivity. We note that this property is not satisfied by the original power divergence $D_{\beta }$. Furthermore, we know that ${\Delta }_{\beta }$ associates with the Pythagorean identity in the following.

Proposition 1. 

Assume that there exist three different points $p,q$ and r in $\mathcal{M}$ satisfying

$$\begin{array}{c}\hfill {\Delta }_{\beta }(p,r)={\Delta }_{\beta }(p,q)+{\Delta }_{\beta }(q,r)\end{array}$$

(14)

Define a path ${\left\{p_t\right\}}_{0\le t\le 1}$ connecting p with q and a path ${\left\{r_s\right\}}_{0\le s\le 1}$ connecting r with q as

$$\begin{array}{c}\hfill p_t\left(x\right)=(1-t)p\left(x\right)+tq\left(x\right),{\left\{r_s\left(x\right)\right\}}^{\beta }=(1-s){\left\{r\left(x\right)\right\}}^{\beta }+s{\left\{q\left(x\right)\right\}}^{\beta }\end{array}$$

Then

$$\begin{array}{c}\hfill {\Delta }_{\beta }(p_t,r_s)={\Delta }_{\beta }(p_t,q)+{\Delta }_{\beta }(q,r_s)\end{array}$$

(15)

holds for all $t(0<t<1)$ and all $s(0<s<1)$.

Proof is given in Appendix 2. This Pytahgorean-type identity is also satisfied with the $D_{\beta }$ [16].

3. Minimum Power Divergence Method

In the previous section we introduce a statistical method defined by minimization of the projective power divergence discussed. By the definition of ${\Delta }_{\beta }$ the cross projective power entropy is led to

$$\begin{array}{c}\hfill {\Gamma }_{\beta }(\mu ,\nu )=-\frac{1}{\beta }log\int \mu \left(x\right)\nu {\left(x\right)}^{\beta }dx+c_{\beta }\left(\theta \right)\end{array}$$

where $c_{\beta }\left(\theta \right)={\left(\beta +1\right)}^{-1}log\{\int f_{\theta }{\left(x\right)}^{\beta +1}dx\}$. We see that ${\Delta }_{\beta }(\mu ,\nu )={\Gamma }_{\beta }(\mu ,\nu )-{\Gamma }_{\beta }(\mu ,\mu )$. Hence, this decomposition leads the empirical analogue based on a given data set $\{x_1,\cdots ,x_n\}$ to

$$\begin{array}{c}\hfill L_{\beta }\left(\theta \right)=\frac{1}{\beta }log\left(\frac{1}{n}\sum _{i=1}^n{f}_{\theta }{\left(x_i\right)}^{\beta }\right)-c_{\beta }\left(\theta \right)\end{array}$$

(16)

which we call the mean power likelihood with the index β. Thus, the minus expectation of $L_{\beta }\left(\theta \right)$ with respect to the unknown density function $p\left(x\right)$ equals to ${\Gamma }_{\beta }(p,f_{\theta })$. The limit of β to 0 leads that $L_{\beta }\left(\theta \right)$ converges to ${\ell }_0\left(\theta \right)$. Assume that $\{x_1,\cdots ,x_n\}$ is a random sample exactly from $f_{\theta }\left(x\right)$. Then the strong law of large number yields that

$$\begin{array}{c}\hfill L_{\beta }\left({\theta }^{\prime }\right)⟶-\Gamma (f_{\theta },f_{{\theta }^{\prime }})\end{array}$$

as n increases to infinity. From the property associated with the projective power divergence it follows that $\Gamma (f_{\theta },f_{{\theta }^{\prime }})\ge \Gamma (f_{\theta },f_{\theta })$, which implies that $\theta ={\mathrm{argmin}}_{{\theta }^{\prime }\in \Theta }\Gamma (f_{\theta },f_{{\theta }^{\prime }})$. Consequently, we conclude that the estimator ${\widehat{\theta }}_{\beta }={\mathrm{argmin}}_{{\theta }^{\prime }\in \Theta }{L}_{\beta }\left({\theta }^{\prime }\right)$ converges to θ almost surely. The proof is similar to that for the MLE in Wald [18]. In general any minimum divergence estimator satisfies the strong consistency in the asymptotical sense.

The estimator ${\widehat{\theta }}_{\beta }$ is associated with the estimating function,

$$\begin{array}{c}\hfill s_{\beta }(x,\theta )=f_{\theta }{\left(x\right)}^{\beta }\left\{s(x,\theta )-\frac{\partial }{\partial \theta }{c}_{\beta }\left(\theta \right)\right\}\end{array}$$

(17)

where $s(x,\theta )$ is the score vector, $(\partial /\partial \theta )log{f}_{\theta }\left(x\right)$. We observe that the estimating function is unbiased in the sense that ${\mathrm{E}}_{\theta }\left\{s_{\beta }(x,\theta )\right\}=0$. This is because

$$\begin{array}{c}\hfill {\mathrm{E}}_{\theta }\left\{s_{\beta }(x,\theta )\right\}=\int f_{\theta }{\left(x\right)}^{\beta +1}s(x,\theta )dx-\int f_{\theta }{\left(x\right)}^{\beta +1}dx\frac{\partial }{\partial \theta }{c}_{\beta }\left(\theta \right)=0\end{array}$$

Thus the estimating equation is given by

$$\begin{array}{c}\hfill S_{\beta }\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n{s}_{\beta }(x_i,\theta )=0\end{array}$$

We see that the gradient vector of $L_{\beta }\left(\theta \right)$ is proportional to $S_{\beta }\left(\theta \right)$ as

$$\begin{array}{c}\hfill \frac{\partial }{\partial \theta }{L}_{\beta }\left(\theta \right)={(\frac{1}{n}\sum _{i=1}^n{f}_{\theta }{\left(x_i\right)}^{\beta })}^{-1}{S}_{\beta }\left(\theta \right)\end{array}$$

Hence, the estimating function (17) exactly leads to the estimator ${\widehat{\theta }}_{\beta }$.

Accordingly, we obtain the following asymptotic normality

$$\begin{array}{c}\hfill \sqrt{n}({\widehat{\theta }}_{\beta }-\theta ){⟶}_{\mathcal{D}}N(0,{\mathrm{AV}}_{\beta }\left(\theta \right))\end{array}$$

where ${⟶}_{\mathcal{D}}$ denotes convergence in law, and $N(\mu ,V)$ denotes a normal distribution with mean vector μ and variance matrix V. Here, the limiting variance matrix is

$$\begin{array}{c}\hfill {\mathrm{AV}}_{\beta }\left(\theta \right)={\left\{\mathrm{E}\frac{\partial s_{\beta }(x,\theta )}{\partial \theta }\right\}}^{-\mathrm{T}}\mathrm{var}\left(s_{\beta }(x,\theta )\right){\left\{\mathrm{E}\frac{\partial s_{\beta }(x,\theta )}{\partial \theta }\right\}}^{-1}\end{array}$$

The inequality (4) implies ${\mathrm{AV}}_{\beta }\left(\theta \right)\ge I_{\theta }^{-1}$ for any β, which implies that any estimator ${\widehat{\theta }}_{\beta }$ is not asymptotically efficient, where $I_{\theta }$ denotes the Fisher information matrix defined in (3). In fact, the estimator ${\widehat{\theta }}_{\beta }$ becomes efficient only when $\beta =0$, which is reduced to the MLE. Hence, there is no optimal estimator except for the MLE in the class ${\{{\widehat{\theta }}_{\beta }\}}_{\beta \ge 0}$ as far as the asymptotic efficiency is concerned.

3.1. Super Robustness

We would like to investigate the influence of the estimator ${\widehat{\theta }}_{\beta }$ against outliers. We consider outliers in a probabilistic manner. An observation $x_{\mathrm{o}}$ is called an outlier if $f_{\theta }\left(x_{\mathrm{o}}\right)$ is very small. Let us carefully look at the estimating equation (17). Then we observe that the larger the value of β is, the smaller $\parallel s_{\beta }(x_o,\theta )\parallel$ for all outliers $x_{\mathrm{o}}$. The estimator ${\widehat{\theta }}_{\beta }$ is solved as

$$\begin{array}{c}\hfill {\widehat{\theta }}_{\beta }=\underset{\theta \in \Theta }{\mathrm{argsolve}}\{\sum _{i=1}^n{s}_{\beta }(x_i,\theta )=0\}\end{array}$$

which implies that, for a sufficiently large β, the estimating equation has little impact from outliers contaminated in the data set because the value of the integral $\int f_{\theta }^{\beta }$ is hardly influenced by the outliers. In this sense, ${\widehat{\theta }}_{\beta }$ is robust for such β [19]. From an empirical viewpoint, we know it is sufficient to fix $\beta \ge 0.1$. In a case that $f_{\theta }\left(x\right)$ is absolutely continuous in ${\mathbb{R}}^p$ we see that ${lim}_{\left|x\right|\to \infty }\left|s_{\beta }(x,\theta )\right|=0$, which is quite contrast with the optimal robust method (cf. [20]). Consider an ϵ-contamination model

$$\begin{array}{c}\hfill f_{\theta ϵ}\left(x\right)=(1-ϵ)f_{\theta }\left(x\right)+ϵ\delta \left(x\right)\end{array}$$

In this context, $\delta \left(x\right)$ is the density for outliers, which departs from the assumed density $f_{\theta }\left(x\right)$ with a large degree. It seems reasonable to suppose that $\int f_{\theta }\left(x\right)\delta \left(x\right)dx\simeq 0$. Thus if the true density function $p\left(x\right)$ equals $f_{\theta ,ϵ}\left(x\right)$, then ${\widehat{\theta }}_{\beta }$ becomes a consistent estimator for θ for all $ϵ,0\le ϵ<1$. In this sense we say ${\widehat{\theta }}_{\beta }$ satisfies super robustness. On the other hand, the mean power likelihood function ${\ell }_{\beta }\left(\theta \right)$ as given in (9) associates with the estimating function

$$\begin{array}{c}\hfill f_{\theta }{\left(x\right)}^{\beta }s(x,\theta )-\frac{\partial }{\partial \theta }{\kappa }_{\beta }\left(\theta \right)\end{array}$$

which is unbiased, but the corresponding estimator does not satisfy such super robustness.

Let us consider a multivariate normal model $N(\mu ,V)$ with mean vector μ and variance matrix V in which the minimum projective power divergence method by (16) is applicable for the estimation of μ and V as follows:

$$({\widehat{\mu }}_{\beta },{\widehat{V}}_{\beta })=\underset{(\mu ,V)\in {\mathbb{R}}^p\times \mathcal{S}}{\mathrm{argmax}}{L}_{\beta }(\mu ,V)$$

where $\mathcal{S}$ denotes the space of all symmetric, positive definite matrices.

Noting the projective invariance, we obtain

$$\begin{array}{c}\hfill L_{\beta }(\mu ,V)=\frac{1}{\beta }log[\frac{1}{n}\sum _{i=1}^{n}exp\{-\frac{\beta }{2}{(x_i-\mu )}^{\mathrm{T}}{V}^{-1}(x_i-\mu )\}]-\frac{1}{\beta +1}logdet(\frac{V}{\beta +1})\end{array}$$

(18)

from which the estimating equation gives the weighted mean and variance as

$$\begin{array}{ccc}\hfill \mu & =& \frac{{\sum }_{i=1}^{n}w{(x_i,\mu ,V)}^{\beta }{x}_i}{{\sum }_{i=1}^{n}w{(x_i,\mu ,V)}^{\beta }},\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill V& =& (\beta +1)\frac{{\sum }_{i=1}^{n}w{(x_i,\mu ,V)}^{\beta }(x_i-\mu ){(x_i-\mu )}^{\mathrm{T}}}{{\sum }_{i=1}^{n}w{(x_i,\mu ,V)}^{\beta }}\hfill \end{array}$$

(20)

where $w(x,\mu ,V)$ is the weight function defined by $exp\{-\frac{1}{2}{(x-\mu )}^{\mathrm{T}}{V}^{-1}(x-\mu )\}$. Although we do not know the explicit solution, a natural iteration algorithm can be proposed that the left-hand sides of (19) and (20), say $({\mu }_{t+1},V_{t+1})$ are both updated by plugging $({\mu }_t,V_t)$ in the right-hand sides of (19) and (20). Obviously, for the estimator $({\widehat{\mu }}_{\beta },{\widehat{V}}_{\beta })$ with $\beta =0$, or the MLE, we need no iteration step but the sample mean vector and sample variance matrix as the exact solution.

3.2. Local Learning

We discuss a statistical idea beyond robustness. Since the expression (16) is inconvenient to investigate the behavior of the mean expected power likelihood function, we focus on

$$\begin{array}{c}\hfill I_{\beta }\left(\theta \right)=\frac{1}{\beta }\left\{\int f_{\theta }{\left(x\right)}^{\beta }p\left(x\right)dx-1\right\}\end{array}$$

as a core term, where $p\left(x\right)$ is the true density function, that is, the underlying distribution generating the data set. We consider K mixture model, while $p\left(x\right)$ is modeled as ϵ-contaminated density function $f_{\theta ,ϵ}\left(x\right)$ in the previous section. Thus, $p\left(x\right)$ is written by K different density functions $p_k\left(x\right)$ as follows:

$$\begin{array}{c}\hfill p\left(x\right)={\pi }_1{p}_1\left(x\right)+\cdots +{\pi }_K{p}_K\left(x\right)\end{array}$$

(21)

where ${\pi }_k$ denotes the mixing ratio. We note that there exists redundancy for this modeling unless $p_k\left(x\right)$s are specified. In fact, the case in which ${\pi }_1=1$ and $p_1\left(x\right)$ is arbitrarily means no restriction for $p\left(x\right)$. However, we discuss $I_{\beta }\left(\theta \right)$ on this redundant model and find that

$$\begin{array}{c}\hfill I_{\beta }(\mu ,V)=\frac{1}{\beta }\left(\sum _{k=1}^K{\pi }_k{\{{\left(2\pi \right)}^{p}det\left(V\right)\}}^{-\frac{\beta }{2}}\int exp\{-\frac{\beta }{2}{(x-\mu )}^{\mathrm{T}}{V}^{-1}(x-\mu )\}{p}_k\left(x\right)dx-1\right)\end{array}$$

(22)

We confirm

$$\begin{array}{c}\hfill I_0(\mu ,V)=-\frac{1}{2}\{\sum _{k=1}^K{\pi }_k\int {(x-\mu )}^{\mathrm{T}}{V}^{-1}(x-\mu )p_k\left(x\right)dx+logdet\left(V\right)\}\end{array}$$

taking the limit of β to 0. It is noted that $I_0(\mu ,V)$ has a global maximizer $(\widehat{\mu },\widehat{V})$ that is a pair of the mean vector and variance matrix with respect to $p\left(x\right)$ since we can write

$$\begin{array}{c}\hfill I_0(\mu ,V)=-\frac{1}{2}\left\{(\mu -\widehat{\mu })V^{-1}(\mu -\widehat{\mu })+\mathrm{trace}(\widehat{V}{V}^{-1})+logdet\left(V\right)\right\}\end{array}$$

This suggests a limitation of the maximum likelihood method. The MLE cannot change $N(\widehat{\mu },\widehat{V})$ as the estimative solution even if the true density function is arbitrarily in (21). On the other hand, if β becomes larger, then the graph of $I_{\beta }(\mu ,V)$ is flexibly changed in accordance with $p\left(x\right)$ in (21). For example, we assume

$$\begin{array}{c}\hfill p\left(x\right)={\pi }_{1}g(x,{\mu }_1,V_1)+\cdots +{\pi }_{K}g(x,{\mu }_K,V_K)\end{array}$$

(23)

where $g(x,{\mu }_k,V_k)$ is a normal density function $N({\mu }_k,V_k)$. Then,

$$\begin{array}{c}\hfill I_{\beta }(\mu ,V)=\frac{1}{\beta }\left(\sum _{k=1}^K{\pi }_k{\{{\beta }^{-p}{\left(2\pi \right)}^{p}det\left(V\right)\}}^{\frac{1-\beta }{2}}\int g\left(x,\mu ,{\beta }^{-1}V\right)g(x,{\mu }_k,V_k)dx-1\right)\end{array}$$

Here, we see a formula

$$\begin{array}{c}\hfill \int g(x,\mu ,V)g(x,{\mu }^{*},V^{*})dx=g(\mu ,{\mu }^{*},V+V^{*})\end{array}$$

(24)

as shown in Appendix 3, from which we get that

$$\begin{array}{c}\hfill I_{\beta }(\mu ,V)=\frac{1}{\beta }\left({\beta }^{-p}{\{{\left(2\pi \right)}^{p}det\left(V\right)\}}^{\frac{1-\beta }{2}}\sum _{k=1}^K{\pi }_{k}g\left(\mu ,{\mu }_k,{\beta }^{-1}V+V_k\right)-1\right)\end{array}$$

In particular, when $\beta =1$,

$$\begin{array}{c}\hfill I_1(\mu ,V)=\sum _{k=1}^K{\pi }_{k}g\left(\mu ,{\mu }_k,V+V_k\right)-1\end{array}$$

which implies that $I_1(\mu ,O)=p\left(\mu \right)-1$, where O is a zero matrix and $p(·)$ is defined as in (23). If the normal mixture model has K modes, $I_1(\mu ,V)$ has the same K modes for sufficiently small $detV$. Therefore, the expected $I_{\beta }(\mu ,V)$ with a large β adaptively behaves according to the true density function. This suggests that the minimum projective power divergence method can improve the weak point of the MLE if the true density function has much degree of model uncertainty. For example, such an adaptive selection for β is discussed in principal component analysis (PCA), which enables us to providing explanatory analysis rather than the conventional PCA.

Consider a problem for extracting principal components in which the data distribution has a density function with multimodality as described in (21). Then we wish to search all the sets of the principal vectors for $V_k$ with $k=1,\cdots ,K$. The minimum projective power divergence method can properly provide the PCA to search the principal vectors for $V_k$ at the centers ${\mu }_k$ separately for $k=1,\cdots ,K$. First we determine the first starting point, say $({\mu }^{\left(1\right)},V^{\left(1\right)})$ in which we employ the iteratively reweighted algorithm (19) and (20) starting from $({\mu }^{\left(1\right)},V^{\left(1\right)})$, so that we get the first estimator $({\widehat{\mu }}^{\left(1\right)},{\widehat{V}}^{\left(1\right)})$. Then the estimator ${\widehat{V}}^{\left(1\right)}$ gives the first PCA with the center ${\widehat{\mu }}^{\left(1\right)}$ by the standard method. Next, we updates the second starting point $({\mu }^{\left(2\right)},V^{\left(2\right)})$ to keep away from the first estimator $({\widehat{\mu }}^{\left(1\right)},{\widehat{V}}^{\left(1\right)})$ by a heuristic procedure based on the weight function $w(x,\mu ,V)$ (see [22] for the detailed discussion). Starting from $({\mu }^{\left(2\right)},V^{\left(2\right)})$, the same algorithm (19) and (20) leads to the second estimator $({\widehat{\mu }}^{\left(2\right)},{\widehat{V}}^{\left(2\right)})$ with the second PCA with the center ${\widehat{V}}^{\left(2\right)}$. In this way, we can make this sequential procedure to explore the multimodal structure with an appropriately determined stopping rule.

4. Concluding Remarks

We focus on that the optimality property of the likelihood method is fragile under model uncertainty. Such weakness frequently appears in practice when we got a data set typically from an observational study rather than a purely randomized experimental study. However, the usefulness of likelihood method is supported as the most excellent method in statistics. We note that the minimum projective power divergence method reduces to the MLE by taking a limit of the index β to 0 since it has one degree of freedom of β as a choice of method. A data-adaptive selection of β is possible by cross validation method. However, an appropriate model selection criterion is requested for faster computation.

Recently novel methods for pattern recognition from machine learning paradigm have been proposed [23,24,25]. These approaches are directly concerned with the true distribution in a framework of probability approximate correct (PAC) learning in the computational learning theory. We need to employ this theory for the minimum projective power divergence method. In statistical physics there are remarkable developments on Tsallis entropy with reference to disequilibrium state, chaos phenomena, scale free network and econophysics. We should explore these developments from the statistical point of view.