Next Article in Journal
Self-Organization during Friction in Complex Surface Engineered Tribosystems
Next Article in Special Issue
Parametric Bayesian Estimation of Differential Entropy and Relative Entropy
Previous Article in Journal
Improvement of Energy Conversion/Utilization by Exergy Analysis: Selected Cases for Non-Reactive and Reactive Systems
Previous Article in Special Issue
Transport of Heat and Charge in Electromagnetic Metrology Based on Nonequilibrium Statistical Mechanics

Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

# Entropy and Divergence Associated with Power Function and the Statistical Application

by
Shinto Eguchi
* and
Shogo Kato
The Institute of Statistical Mathematics, Tachikawa, Tokyo 190-8562, Japan
*
Author to whom correspondence should be addressed.
Entropy 2010, 12(2), 262-274; https://doi.org/10.3390/e12020262
Submission received: 29 December 2009 / Revised: 20 February 2010 / Accepted: 23 February 2010 / Published: 25 February 2010
(This article belongs to the Special Issue Distance in Information and Statistical Physics Volume 2)

## Abstract

:
In statistical physics, Boltzmann-Shannon entropy provides good understanding for the equilibrium states of a number of phenomena. In statistics, the entropy corresponds to the maximum likelihood method, in which Kullback-Leibler divergence connects Boltzmann-Shannon entropy and the expected log-likelihood function. The maximum likelihood estimation has been supported for the optimal performance, which is known to be easily broken down in the presence of a small degree of model uncertainty. To deal with this problem, a new statistical method, closely related to Tsallis entropy, is proposed and shown to be robust for outliers, and we discuss a local learning property associated with the method.

## 1. Introduction

Consider a practical situation in which a data set ${ x 1 , ⋯ , x n }$ is randomly sampled from a probability density function of a statistical model ${ f θ ( x ) : θ ∈ Θ }$, where θ is a parameter vector and Θ is the parameter space. A fundamental tool for the estimation of unknown parameter θ is the log-likelihood function defined by
$ℓ ( θ ) = 1 n ∑ i = 1 n log f θ ( x i )$
which is commonly utilized by statistical researchers ranging over both frequentists and Bayesians. The maximum likelihood estimator (MLE) is defined by
$θ ^ = argmin θ ∈ Θ ℓ ( θ )$
The Fisher information matrix for θ is defined by
$I θ = ∫ f θ ( x ) ∂ ∂ θ log f θ ( x ) ∂ ∂ θ T log f θ ( x ) d x$
where $θ T$ denotes the transpose of θ. As the sample size n tends to infinity, the variance matrix of $n ( θ ^ - θ )$ converges to $I θ - 1$. This inverse matrix exactly gives the lower bound in the class of asymptotically consistent estimators in the sense of matrix inequality, that is,
$AV θ ( θ ˜ ) ≥ I θ - 1$
for any asymptotically consistent estimator $θ ˜$ of θ, where $AV θ$ denotes the limiting variance matrix under the distribution with the density $f θ ( x )$.
On the other hand, the Boltzmann-Shannon entropy
$H 0 ( p ) = ∫ p ( x ) log p ( x ) d x$
plays a fundamental role in various fields, such as statistical physics, information science and so forth. This is directly related with the MLE. Let us consider an underlying distribution with the density function $p ( x )$. The cross entropy is defined by
$C 0 ( p , f θ ) = - ∫ p ( x ) log f θ ( x ) d x$
We note that $C 0 ( p , f θ ) = E p { - ℓ ( θ ) }$, where $E p$ denotes the expectation with respect to $p ( x )$. Hence, the maximum likelihood principle is equal to the minimum cross entropy principle. The Kulback-Leibler (KL) divergence is defined by
$D 0 ( p , q ) = ∫ p ( x ) log p ( x ) q ( x ) d x$
which gives a kind of information distance between p and q. Note that $D 0 ( p , f θ ) = C 0 ( p , f θ ) - C 0 ( p , p )$
An exponential (type) distribution model is defined by the density form
$f θ ( x ) = exp { θ T t ( x ) - ψ ( θ ) }$
where $ψ ( θ )$ is the cumulant transform defined by $log ∫ exp { θ T t ( x ) } d x$. Under the assumption of this family, the MLE has a number of convenient properties such as minimal sufficiency, unbiasedness, efficiency [1]. In particular, the MLE for the expectation parameter $η = E θ { t ( X ) }$ is explicitly given by
$η ^ 0 = 1 n ∑ i = 1 n t ( x i )$
which is associated with a dualistic relation of the canonical parameter θ and the expectation parameter η [2,3]. Thus, the MLE satisfies such an excellent property, which is associated with logarithmic and exponential functions as in (2) and (8).
The MLE has been widely employed in statistics, in which the properties are supported in theoretical discussion, for example, as in [4]. However, the MLE has some inappropriate properties when the underlying distribution does not belong to the model ${ f θ ( x ) : θ ∈ Θ }$. A statistical model is just simulation of the true distribution as Fisher pointed in [1]. The model, which is just used as a working model, is wrong in most practical cases. In such situations, the MLE does not show proper performance because of model uncertainty. In this paper we explore alternative estimation method than the MLE.

## 2. Power Divergence

The logarithmic transform for observed values is widely employed in data analysis. On the other hand, a power transformation defined by
$t β ( x ) = x β - 1 β$
often gives more flexible performance to get good approximation to normal distribution [5]. In analogy with this transform, the power cross entropy is defined by
$C β ( p , q ) = - ∫ p ( x ) q ( x ) β - 1 β d x + ∫ q ( x ) β + 1 β + 1 d x$
where β is a positive parameter. Thus, it is defined by the power transform of the density. If we take the limit of β to 0, then $C β ( p , q )$ converges to $C 0 ( p , q )$, which is given in (6). In fact, the power parameter β is not fixed, so that different β’s give different behaviors of the power entropy. The diagonal power entropy is defined by
$H β ( p ) = ∫ ( β + 1 ) p ( x ) - p ( x ) β + 1 β ( β + 1 ) d x$
which is given by taking $C β$ the diagonal. Actually, this is equivalent to Tsallis q-entropy with the relation $β = q - 1$.
Let ${ x 1 , ⋯ , x n }$ be a random sample from unknown density function $p ( x )$. Then we define the empirical mean power likelihood by
$ℓ β ( θ ) = 1 n ∑ i = 1 n f θ ( x i ) β - 1 β - κ β ( θ )$
where $κ β ( θ ) = ∫ f θ ( x ) β + 1 d x / ( β + 1 )$. See [6,7,8,9] for statistical applications. Accordingly, the minus expectation value of $ℓ β ( θ )$ is equal to $C β ( p , f θ )$. In general, a relation of cross and diagonal entropy leads to the inequality $C β ( p , q ) ≤ C β ( p , p )$, from which we define the power divergence by
$D β ( p , q ) = C β ( p , q ) - C β ( p , p )$
We extend power entropy and divergence defined over the space of all density functions, which are not always assumed to have a total mass one. In particular, this extension is useful for proposing boosting methods [10,11,12,13,14,15,16].
This derivation can be extended by a generator function U. Assume that $U ( t )$ is strictly increasing and convex. The Fencel duality discussion leads to a conjugate convex function of $U ( t )$ defined by
$U * ( s ) = max t ∈ R { s t - U ( t ) }$
which reduces to $U * ( s ) = s ξ ( s ) - U ( ξ ( s ) )$, where $ξ ( s )$ is the inverse function of the derivation $U ˙$ of U. Then, U-cross entropy is defined by
$C U ( μ , ν ) = - ∫ μ ( x ) ξ ( ν ( x ) ) d x + ∫ U ( ξ ( ν ( x ) ) ) d x$
Similarly U-divergence is defined by
$D U ( μ , ν ) = ∫ { U * ( μ ( x ) ) + U ( ξ ( ν ( x ) ) ) - μ ( x ) ξ ( ν ( x ) ) } d x$
We note that $D U ( μ , ν ) = C U ( μ , ν ) - C U ( μ , μ )$. By the definition of $U *$ in (10) we see that the integrand of the right-hand side of (11) is always nonnegative. The power divergence is one example of U-divergence by fixing
$U β ( t ) = 1 β + 1 ( 1 + β t ) β + 1 β$
The power divergence can be defined on $M$ as
$D β ( μ , ν ) = ∫ μ ( x ) μ ( x ) β - ν ( x ) β β + ν ( x ) β + 1 - μ ( x ) β + 1 β + 1 d x$
for μ and ν of $M$ [17]. Thus $U β ( t )$ is strictly increasing and convex, which implies that the integrand of the right-hand side of (12) is nonnegative.
To explore this, it seems sufficient to restrict the definition domain of $D β$ to $P$. However, we observe that the restriction is not useful for statistical considerations. We discuss the restriction on the projective space as follows. Fix two functions μ and ν in $M$. We say that μ and ν are projectively equivalent if there exists a positive scalar λ such that
$ν ( x ) = λ μ ( x ) ( a . e . x )$
Thus, we write $ν ∼ μ$. Similarly, we call a divergence D defined on $M$ projectively invariant if for all $λ > 0 , κ > 0$
$D ( λ μ , κ ν ) = D ( μ , ν )$
We can derive a variant of power divergence as
$Δ β ( μ , ν ) = 1 β ( β + 1 ) log ∫ μ ( x ) β + 1 d x - 1 β log ∫ μ ( x ) ν ( x ) β d x + 1 β + 1 log ∫ ν ( x ) β + 1 d x$
See Appendix 1 for the derivation. Immediately, we observe $Δ β$ satisfies (13), or projective invariance. Hereafter, we call $Δ β$ the projective power divergence. In this way, for $p ( x ) = μ ( x ) / ∫ μ ( x ) d x$ and $q ( x ) = ν ( x ) / ∫ ν ( x ) d x$, it is obtained that
$Δ β ( p , q ) = Δ β ( μ , ν )$
If we take a specific value of β, then
$Δ β = 1 ( μ , ν ) = 1 2 log ∫ μ ( x ) 2 d x ∫ ν ( x ) 2 d x ∫ μ ( x ) ν ( x ) d x 2$
and
$lim β → 0 Δ β ( μ , ν ) = D 0 μ ∫ μ ( x ) d x , ν ∫ ν ( x ) d x$
where $D 0$ is nothing but the KL divergence (7). We observe that the projective power divergence satisfies information additivity. In fact, if we write p and q as $p ( x 1 , x 2 ) = p 1 ( x 1 ) p 2 ( x 2 )$ and $q ( x 1 , x 2 ) = q 1 ( x 1 ) q 2 ( x 2 )$, respectively, then
$Δ β ( p , q ) = Δ β ( p 1 , q 1 ) + Δ β ( p 2 , q 2 )$
which means information additivity. We note that this property is not satisfied by the original power divergence $D β$. Furthermore, we know that $Δ β$ associates with the Pythagorean identity in the following.
Proposition 1
Assume that there exist three different points $p , q$ and r in $M$ satisfying
$Δ β ( p , r ) = Δ β ( p , q ) + Δ β ( q , r )$
Define a path ${ p t } 0 ≤ t ≤ 1$ connecting p with q and a path ${ r s } 0 ≤ s ≤ 1$ connecting r with q as
$p t ( x ) = ( 1 - t ) p ( x ) + t q ( x ) , { r s ( x ) } β = ( 1 - s ) { r ( x ) } β + s { q ( x ) } β$
Then
$Δ β ( p t , r s ) = Δ β ( p t , q ) + Δ β ( q , r s )$
holds for all $t ( 0 < t < 1 )$ and all $s ( 0 < s < 1 )$.
Proof is given in Appendix 2. This Pytahgorean-type identity is also satisfied with the $D β$ [16].

## 3. Minimum Power Divergence Method

In the previous section we introduce a statistical method defined by minimization of the projective power divergence discussed. By the definition of $Δ β$ the cross projective power entropy is led to
$Γ β ( μ , ν ) = - 1 β log ∫ μ ( x ) ν ( x ) β d x + c β ( θ )$
where $c β ( θ ) = ( β + 1 ) - 1 log { ∫ f θ ( x ) β + 1 d x }$. We see that $Δ β ( μ , ν ) = Γ β ( μ , ν ) - Γ β ( μ , μ )$. Hence, this decomposition leads the empirical analogue based on a given data set ${ x 1 , ⋯ , x n }$ to
$L β ( θ ) = 1 β log 1 n ∑ i = 1 n f θ ( x i ) β - c β ( θ )$
which we call the mean power likelihood with the index β. Thus, the minus expectation of $L β ( θ )$ with respect to the unknown density function $p ( x )$ equals to $Γ β ( p , f θ )$. The limit of β to 0 leads that $L β ( θ )$ converges to $ℓ 0 ( θ )$. Assume that ${ x 1 , ⋯ , x n }$ is a random sample exactly from $f θ ( x )$. Then the strong law of large number yields that
$L β ( θ ′ ) ⟶ - Γ ( f θ , f θ ′ )$
as n increases to infinity. From the property associated with the projective power divergence it follows that $Γ ( f θ , f θ ′ ) ≥ Γ ( f θ , f θ )$, which implies that $θ = argmin θ ′ ∈ Θ Γ ( f θ , f θ ′ )$. Consequently, we conclude that the estimator $θ ^ β = argmin θ ′ ∈ Θ L β ( θ ′ )$ converges to θ almost surely. The proof is similar to that for the MLE in Wald [18]. In general any minimum divergence estimator satisfies the strong consistency in the asymptotical sense.
The estimator $θ ^ β$ is associated with the estimating function,
$s β ( x , θ ) = f θ ( x ) β s ( x , θ ) - ∂ ∂ θ c β ( θ )$
where $s ( x , θ )$ is the score vector, $( ∂ / ∂ θ ) log f θ ( x )$. We observe that the estimating function is unbiased in the sense that $E θ { s β ( x , θ ) } = 0$. This is because
$E θ { s β ( x , θ ) } = ∫ f θ ( x ) β + 1 s ( x , θ ) d x - ∫ f θ ( x ) β + 1 d x ∂ ∂ θ c β ( θ ) = 0$
Thus the estimating equation is given by
$S β ( θ ) = 1 n ∑ i = 1 n s β ( x i , θ ) = 0$
We see that the gradient vector of $L β ( θ )$ is proportional to $S β ( θ )$ as
$∂ ∂ θ L β ( θ ) = ( 1 n ∑ i = 1 n f θ ( x i ) β ) - 1 S β ( θ )$
Hence, the estimating function (17) exactly leads to the estimator $θ ^ β$.
Accordingly, we obtain the following asymptotic normality
$n ( θ ^ β - θ ) ⟶ D N ( 0 , AV β ( θ ) )$
where $⟶ D$ denotes convergence in law, and $N ( μ , V )$ denotes a normal distribution with mean vector μ and variance matrix V. Here, the limiting variance matrix is
$AV β ( θ ) = E ∂ s β ( x , θ ) ∂ θ - T var ( s β ( x , θ ) ) E ∂ s β ( x , θ ) ∂ θ - 1$
The inequality (4) implies $AV β ( θ ) ≥ I θ - 1$ for any β, which implies that any estimator $θ ^ β$ is not asymptotically efficient, where $I θ$ denotes the Fisher information matrix defined in (3). In fact, the estimator $θ ^ β$ becomes efficient only when $β = 0$, which is reduced to the MLE. Hence, there is no optimal estimator except for the MLE in the class ${ θ ^ β } β ≥ 0$ as far as the asymptotic efficiency is concerned.

#### 3.1. Super Robustness

We would like to investigate the influence of the estimator $θ ^ β$ against outliers. We consider outliers in a probabilistic manner. An observation $x o$ is called an outlier if $f θ ( x o )$ is very small. Let us carefully look at the estimating equation (17). Then we observe that the larger the value of β is, the smaller $∥ s β ( x o , θ ) ∥$ for all outliers $x o$. The estimator $θ ^ β$ is solved as
$θ ^ β = argsolve θ ∈ Θ { ∑ i = 1 n s β ( x i , θ ) = 0 }$
which implies that, for a sufficiently large β, the estimating equation has little impact from outliers contaminated in the data set because the value of the integral $∫ f θ β$ is hardly influenced by the outliers. In this sense, $θ ^ β$ is robust for such β [19]. From an empirical viewpoint, we know it is sufficient to fix $β ≥ 0 . 1$. In a case that $f θ ( x )$ is absolutely continuous in $R p$ we see that $lim | x | → ∞ | s β ( x , θ ) | = 0$, which is quite contrast with the optimal robust method (cf. [20]). Consider an ϵ-contamination model
$f θ ϵ ( x ) = ( 1 - ϵ ) f θ ( x ) + ϵ δ ( x )$
In this context, $δ ( x )$ is the density for outliers, which departs from the assumed density $f θ ( x )$ with a large degree. It seems reasonable to suppose that $∫ f θ ( x ) δ ( x ) d x ≃ 0$. Thus if the true density function $p ( x )$ equals $f θ , ϵ ( x )$, then $θ ^ β$ becomes a consistent estimator for θ for all $ϵ , 0 ≤ ϵ < 1$. In this sense we say $θ ^ β$ satisfies super robustness. On the other hand, the mean power likelihood function $ℓ β ( θ )$ as given in (9) associates with the estimating function
$f θ ( x ) β s ( x , θ ) - ∂ ∂ θ κ β ( θ )$
which is unbiased, but the corresponding estimator does not satisfy such super robustness.
Let us consider a multivariate normal model $N ( μ , V )$ with mean vector μ and variance matrix V in which the minimum projective power divergence method by (16) is applicable for the estimation of μ and V as follows:
$( μ ^ β , V ^ β ) = argmax ( μ , V ) ∈ R p × S L β ( μ , V )$
where $S$ denotes the space of all symmetric, positive definite matrices.
Noting the projective invariance, we obtain
$L β ( μ , V ) = 1 β log [ 1 n ∑ i = 1 n exp { - β 2 ( x i - μ ) T V - 1 ( x i - μ ) } ] - 1 β + 1 log det ( V β + 1 )$
from which the estimating equation gives the weighted mean and variance as
$μ = ∑ i = 1 n w ( x i , μ , V ) β x i ∑ i = 1 n w ( x i , μ , V ) β ,$
$V = ( β + 1 ) ∑ i = 1 n w ( x i , μ , V ) β ( x i - μ ) ( x i - μ ) T ∑ i = 1 n w ( x i , μ , V ) β$
where $w ( x , μ , V )$ is the weight function defined by $exp { - 1 2 ( x - μ ) T V - 1 ( x - μ ) }$. Although we do not know the explicit solution, a natural iteration algorithm can be proposed that the left-hand sides of (19) and (20), say $( μ t + 1 , V t + 1 )$ are both updated by plugging $( μ t , V t )$ in the right-hand sides of (19) and (20). Obviously, for the estimator $( μ ^ β , V ^ β )$ with $β = 0$, or the MLE, we need no iteration step but the sample mean vector and sample variance matrix as the exact solution.

#### 3.2. Local Learning

We discuss a statistical idea beyond robustness. Since the expression (16) is inconvenient to investigate the behavior of the mean expected power likelihood function, we focus on
$I β ( θ ) = 1 β ∫ f θ ( x ) β p ( x ) d x - 1$
as a core term, where $p ( x )$ is the true density function, that is, the underlying distribution generating the data set. We consider K mixture model, while $p ( x )$ is modeled as ϵ-contaminated density function $f θ , ϵ ( x )$ in the previous section. Thus, $p ( x )$ is written by K different density functions $p k ( x )$ as follows:
$p ( x ) = π 1 p 1 ( x ) + ⋯ + π K p K ( x )$
where $π k$ denotes the mixing ratio. We note that there exists redundancy for this modeling unless $p k ( x )$s are specified. In fact, the case in which $π 1 = 1$ and $p 1 ( x )$ is arbitrarily means no restriction for $p ( x )$. However, we discuss $I β ( θ )$ on this redundant model and find that
$I β ( μ , V ) = 1 β ∑ k = 1 K π k { ( 2 π ) p det ( V ) } - β 2 ∫ exp { - β 2 ( x - μ ) T V - 1 ( x - μ ) } p k ( x ) d x - 1$
We confirm
$I 0 ( μ , V ) = - 1 2 { ∑ k = 1 K π k ∫ ( x - μ ) T V - 1 ( x - μ ) p k ( x ) d x + log det ( V ) }$
taking the limit of β to 0. It is noted that $I 0 ( μ , V )$ has a global maximizer $( μ ^ , V ^ )$ that is a pair of the mean vector and variance matrix with respect to $p ( x )$ since we can write
$I 0 ( μ , V ) = - 1 2 ( μ - μ ^ ) V - 1 ( μ - μ ^ ) + trace ( V ^ V - 1 ) + log det ( V )$
This suggests a limitation of the maximum likelihood method. The MLE cannot change $N ( μ ^ , V ^ )$ as the estimative solution even if the true density function is arbitrarily in (21). On the other hand, if β becomes larger, then the graph of $I β ( μ , V )$ is flexibly changed in accordance with $p ( x )$ in (21). For example, we assume
$p ( x ) = π 1 g ( x , μ 1 , V 1 ) + ⋯ + π K g ( x , μ K , V K )$
where $g ( x , μ k , V k )$ is a normal density function $N ( μ k , V k )$. Then,
$I β ( μ , V ) = 1 β ∑ k = 1 K π k { β - p ( 2 π ) p det ( V ) } 1 - β 2 ∫ g x , μ , β - 1 V g ( x , μ k , V k ) d x - 1$
Here, we see a formula
$∫ g ( x , μ , V ) g ( x , μ * , V * ) d x = g ( μ , μ * , V + V * )$
as shown in Appendix 3, from which we get that
$I β ( μ , V ) = 1 β β - p { ( 2 π ) p det ( V ) } 1 - β 2 ∑ k = 1 K π k g μ , μ k , β - 1 V + V k - 1$
In particular, when $β = 1$,
$I 1 ( μ , V ) = ∑ k = 1 K π k g μ , μ k , V + V k - 1$
which implies that $I 1 ( μ , O ) = p ( μ ) - 1$, where O is a zero matrix and $p ( · )$ is defined as in (23). If the normal mixture model has K modes, $I 1 ( μ , V )$ has the same K modes for sufficiently small $det V$. Therefore, the expected $I β ( μ , V )$ with a large β adaptively behaves according to the true density function. This suggests that the minimum projective power divergence method can improve the weak point of the MLE if the true density function has much degree of model uncertainty. For example, such an adaptive selection for β is discussed in principal component analysis (PCA), which enables us to providing explanatory analysis rather than the conventional PCA.
Consider a problem for extracting principal components in which the data distribution has a density function with multimodality as described in (21). Then we wish to search all the sets of the principal vectors for $V k$ with $k = 1 , ⋯ , K$. The minimum projective power divergence method can properly provide the PCA to search the principal vectors for $V k$ at the centers $μ k$ separately for $k = 1 , ⋯ , K$. First we determine the first starting point, say $( μ ( 1 ) , V ( 1 ) )$ in which we employ the iteratively reweighted algorithm (19) and (20) starting from $( μ ( 1 ) , V ( 1 ) )$, so that we get the first estimator $( μ ^ ( 1 ) , V ^ ( 1 ) )$. Then the estimator $V ^ ( 1 )$ gives the first PCA with the center $μ ^ ( 1 )$ by the standard method. Next, we updates the second starting point $( μ ( 2 ) , V ( 2 ) )$ to keep away from the first estimator $( μ ^ ( 1 ) , V ^ ( 1 ) )$ by a heuristic procedure based on the weight function $w ( x , μ , V )$ (see [22] for the detailed discussion). Starting from $( μ ( 2 ) , V ( 2 ) )$, the same algorithm (19) and (20) leads to the second estimator $( μ ^ ( 2 ) , V ^ ( 2 ) )$ with the second PCA with the center $V ^ ( 2 )$. In this way, we can make this sequential procedure to explore the multimodal structure with an appropriately determined stopping rule.

## 4. Concluding Remarks

We focus on that the optimality property of the likelihood method is fragile under model uncertainty. Such weakness frequently appears in practice when we got a data set typically from an observational study rather than a purely randomized experimental study. However, the usefulness of likelihood method is supported as the most excellent method in statistics. We note that the minimum projective power divergence method reduces to the MLE by taking a limit of the index β to 0 since it has one degree of freedom of β as a choice of method. A data-adaptive selection of β is possible by cross validation method. However, an appropriate model selection criterion is requested for faster computation.
Recently novel methods for pattern recognition from machine learning paradigm have been proposed [23,24,25]. These approaches are directly concerned with the true distribution in a framework of probability approximate correct (PAC) learning in the computational learning theory. We need to employ this theory for the minimum projective power divergence method. In statistical physics there are remarkable developments on Tsallis entropy with reference to disequilibrium state, chaos phenomena, scale free network and econophysics. We should explore these developments from the statistical point of view.

## Acknowledgements

We thank to the anonymous referees for their useful comments and suggestions, in particular, on Proposition 1.

## Appendix 1

We introduce the derivation of $Δ β$ as follows. Consider the minimization for scalar multiplicity as
$κ ( μ , ν ) = argmin κ > 0 D β ( μ , κ ν )$
$∂ ∂ κ D β ( μ , κ ν ) = - κ β - 1 ∫ ν ( x ) β μ ( x ) d x + κ β ∫ ν ( x ) β + 1 d x$
which leads to $κ ( μ , ν ) = ∫ ν ( x ) β μ ( x ) d x / ∫ ν ( x ) β + 1 d x$. Hence
$min κ > 0 D β ( μ , κ ν ) = 1 β ( β + 1 ) { ∫ μ ( x ) β + 1 d x - ∫ μ ( x ) ν ( x ) β d x β + 1 ∫ ν ( x ) β + 1 d x β }$
Taking the ratio as
$Δ β ( μ , ν ) = 1 β ( β + 1 ) log ∫ μ ( x ) β + 1 d x ∫ ν ( x ) β + 1 d x β ∫ μ ( x ) ν ( x ) β d x β + 1 = 1 β ( β + 1 ) log ∫ μ ( x ) β + 1 d x - 1 β log ∫ μ ( x ) ν ( x ) β d x + 1 β + 1 log ∫ ν ( x ) β + 1 d x$
concludes the derivation of $Δ β$ in (13).

## Appendix 2

We give a proof of Proposition 1.
Proof
By definition we get that
$Δ β ( p , r ) - { Δ β ( p , q ) + Δ β ( q , r ) } = 1 β log ∫ p ( x ) q ( x ) β d x ∫ q ( x ) r ( x ) β d x ∫ q ( x ) β + 1 d x ∫ p ( x ) r ( x ) β d x$
which implies
$∫ p ( x ) q ( x ) β d x ∫ q ( x ) r ( x ) β d x ∫ q ( x ) β + 1 d x ∫ p ( x ) r ( x ) β d x = 1$
from (14). Similarly,
$Δ β ( p t , r s ) - { Δ β ( p t , q ) + Δ β ( q , r s ) } = 1 β log ∫ p t ( x ) q ( x ) β d x ∫ q ( x ) r s ( x ) β d x ∫ q ( x ) β + 1 d x ∫ p t ( x ) r s ( x ) β d x$
which is written as
$1 β log ( 1 - t ) ∫ p ( x ) q ( x ) β d x ∫ q ( x ) β + 1 d x + t ( 1 - t ) ∫ p ( x ) r s ( x ) β d x ∫ q ( x ) r s ( x ) β d x + t$
Furthermore, (26) is rewritten as
$1 β log ( 1 - t ) ∫ p ( x ) q ( x ) β d x ∫ q ( x ) β + 1 d x + t ( 1 - t ) ∫ { ( 1 - s ) p ( x ) r ( x ) β + s p ( x ) q ( x ) β } d x ∫ { ( 1 - s ) q ( x ) r ( x ) β + s q ( x ) β + 1 } d x + t$
which is
$1 β log ( 1 - t ) ∫ p ( x ) q ( x ) β d x ∫ q ( x ) β + 1 d x + t ( 1 - t ) ∫ p ( x ) r ( x ) β d x ∫ q ( x ) r ( x ) β d x 1 - s + s ∫ p ( x ) q ( x ) β d x ∫ p ( x ) r ( x ) β d x 1 - s + s ∫ q ( x ) β + 1 d x ∫ q ( x ) r ( x ) β d x + t$
From (25) we can write
$Ξ = ∫ p ( x ) q ( x ) β d x ∫ q ( x ) β + 1 d x = ∫ p ( x ) r ( x ) β d x ∫ q ( x ) r ( x ) β d x$
Then, we conclude that
$Δ β ( p t , r s ) - { Δ β ( p t , q ) + Δ β ( q , r s ) } = 1 β log ( 1 - t ) Ξ + t ( 1 - t ) Ξ ( 1 - s ) + s Ξ ( 1 - s ) + s Ξ + t$
which vanishes for any $s , 0 < s < 1$ and $t , 0 < t < 1$. This completes the proof.   □

## Appendix 3

By writing a p-variate normal density function by
$g ( x , μ , V ) = { ( 2 π ) p det ( V ) } - 1 2 exp { - 1 2 ( x - μ ) T V - 1 ( x - μ ) }$
we have the formula
$∫ g ( x , μ , V ) g ( x , μ * , V * ) d x = g ( μ , μ * , V + V * )$
The proof of this formula is immediate. In fact, the left-hand side of (27) is written by
$( 2 π ) p { det ( V ) det ( V * ) } - 1 2 exp { - 1 2 μ T V - 1 μ - 1 2 μ * T V * - 1 μ * } × ∫ exp { - 1 2 ( x - A - 1 b ) T A ( x - A - 1 b ) } d x$
where
$A = V - 1 + V * - 1 , b = V - 1 μ + V * - 1 μ *$
Hence, we get
${ ( 2 π ) p det ( V ) det ( V * ) det ( V - 1 + V * - 1 ) } - 1 2 exp { 1 2 b T A - 1 b - 1 2 μ T V - 1 μ - 1 2 μ * T V * - 1 μ * }$
Noting that
${ ( 2 π ) p det ( V ) det ( V * ) det ( V - 1 + V * - 1 ) } - 1 2 = { ( 2 π ) p det ( V + V * ) } - 1 2$
and
$exp { 1 2 b T A - 1 b - 1 2 μ T V - 1 μ - 1 2 μ * T V * - 1 μ * } = exp { 1 2 μ T V - 1 ( V - 1 + V * - 1 ) - 1 { I - ( V - 1 + V * - 1 ) V } V - 1 μ + 1 2 μ * T V * - 1 ( V - 1 + V * - 1 ) - 1 { I - ( V - 1 + V * - 1 ) V * } V * - 1 μ * - 1 2 μ T V - 1 ( V - 1 + V * - 1 ) - 1 V * - 1 μ * }$
it is obtained that
$exp { - 1 2 ( μ - μ * ) T ( V + V * ) - 1 ( μ - μ * ) }$
because of $V - 1 ( V - 1 + V * - 1 ) - 1 V * - 1 = ( V + V * ) - 1$. Therefore, (28) and (29) imply (24).   □

## References

1. Fisher, R.A. On the mathematical foundations of theoretical statistics. Philos. Trans. Roy. Soc. London Ser. A 1922, 222, 309–368. [Google Scholar] [CrossRef]
2. Amari, S. Lecture Notes in Statistics. In Differential-Geometrical Methods in Statistics; Springer-Verlag: New York, NY, USA, 1985; Volume 28. [Google Scholar]
3. Amari, S.; Nagaoka, H. Translations of Mathematical Monographs. In Methods of Information Geometry; Oxford University Press: Oxford, UK, 2000; Volume 191. [Google Scholar]
4. Akahira, M; Takeuchi, K. Lecture Notes in Statistics. In Asymptotic Efficiency of Statistical Estimators: Concepts and Higher Order Asymptotic Efficiency; Springer-Verlag: New York, NY, USA, 1981; Volume 7. [Google Scholar]
5. Box, G.E.P.; Cox, D.R. An Analysis of Transformations. J. R. Statist. Soc. B 1964, 26, 211–252. [Google Scholar]
6. Fujisawa, H.; Eguchi, S. Robust estimation in the normal mixture model. J. Stat. Plan Inference 2006, 136, 3989–4011. [Google Scholar] [CrossRef]
7. Minami, M.; Eguchi, S. Robust blind source separation by beta-divergence. Neural Comput. 2002, 14, 1859–1886. [Google Scholar]
8. Mollah, N.H.; Minami, M.; Eguchi, S. Exploring latent structure of mixture ICA models by the minimum beta-divergence method. Neural Comput. 2006, 18, 166–190. [Google Scholar] [CrossRef]
9. Scott, D.W. Parametric statistical modeling by minimum integrated square error. Technometrics 2001, 43, 274–285. [Google Scholar] [CrossRef]
10. Eguchi, S.; Copas, J.B. A class of logistic type discriminant functions. Biometrika 2002, 89, 1–22. [Google Scholar] [CrossRef]
11. Kanamori, T.; Takenouchi, T.; Eguchi, S.; Murata, N. Robust loss functions for boosting. Neural Comput. 2007, 19, 2183–2244. [Google Scholar] [CrossRef] [PubMed]
12. Lebanon, G.; Lafferty, J. Boosting and maximum likelihood for exponential models. In Advances in Neural Information Processing Systems; 2002; Volume 14, pp. 447–454. MIT Press: New York, NY, USA. [Google Scholar]
13. Murata, N.; Takenouchi, T.; Kanamori, T.; Eguchi, S. Information geometry of U-Boost and Bregman divergence. Neural Comput. 2004, 16, 1437–1481. [Google Scholar] [CrossRef] [PubMed]
14. Takenouchi, T.; Eguchi, S. Robustifying AdaBoost by adding the naive error rate. Neural Comput. 2004, 16, 767–787. [Google Scholar] [CrossRef] [PubMed]
15. Takenouchi, T.; Eguchi, S.; Murata, N.; Kanamori, T. Robust boosting algorithm for multiclass problem by mislabelling model. Neural Comput. 2008, 20, 1596–1630. [Google Scholar] [CrossRef] [PubMed]
16. Eguchi, S. Information geometry and statistical pattern recognition. Sugaku Expo. 2006, 19, 197–216. [Google Scholar]
17. Basu, A.; Harris, I.R.; Hjort, N.L.; Jones, M.C. Robust and efficient estimation by minimising a density power divergence. Biometrika 1998, 85, 549–559. [Google Scholar] [CrossRef]
18. Wald, A. Note on the Consistency of the Maximum Likelihood Estimate. Ann. Math. Statist. 1949, 20, 595–601. [Google Scholar] [CrossRef]
19. Fujisawa, H.; Eguchi, S. Robust parameter estimation with a small bias against heavy contamination. J. Multivariate Anal. 2008, 99, 2053–2081. [Google Scholar] [CrossRef]
20. Hampel, F.R.; Ronchetti, E.M.; Rousseeuw, P.J.; Stahel, W.A. Robust Statistics: The Approach Based on Influence Functions; Wiley: New York, NY, USA, 2005. [Google Scholar]
21. Eguchi, S.; Copas, J.A. Class of local likelihood methods and near-parametric asymptotics. J. R. Statist. Soc. B 1998, 60, 709–724. [Google Scholar] [CrossRef]
22. Mollah, N.H.; Sultana, N.; Minami, M.; Eguchi, S. Robust extraction of local structures by the minimum beta-divergence method. Neural Netw. 2010, 23, 226–238. [Google Scholar] [CrossRef] [PubMed]
23. Friedman, J.H.; Hastie, T.; Tibshirani, R. Additive logistic regression: A statistical view of boosting. Annals of Statistics 2000, 28, 337–407. [Google Scholar] [CrossRef]
24. Hastie, T.; Tibishirani, R.; Friedman, J. The Elements of Statistical Learning; Springer: New York, NY, USA, 2001. [Google Scholar]
25. Schapire, R.E.; Freund, Y.; Bartlett, P.; Lee, W.S. Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 1998, 26, 1651–1686. [Google Scholar] [CrossRef]

## Share and Cite

MDPI and ACS Style

Eguchi, S.; Kato, S. Entropy and Divergence Associated with Power Function and the Statistical Application. Entropy 2010, 12, 262-274. https://doi.org/10.3390/e12020262

AMA Style

Eguchi S, Kato S. Entropy and Divergence Associated with Power Function and the Statistical Application. Entropy. 2010; 12(2):262-274. https://doi.org/10.3390/e12020262

Chicago/Turabian Style

Eguchi, Shinto, and Shogo Kato. 2010. "Entropy and Divergence Associated with Power Function and the Statistical Application" Entropy 12, no. 2: 262-274. https://doi.org/10.3390/e12020262