Next Article in Journal / Special Issue
A Paradigm Shift in Biology?
Previous Article in Journal
Using Information Theory to Study Efficiency and Capacity of Computers and Similar Devices

Information 2010, 1(1), 13-27; https://doi.org/10.3390/info1010013

Article
New Information Measures for the Generalized Normal Distribution
Department of Mathematics, Agiou Spyridonos & Palikaridi, Technological Educational Institute of Athens, 12210 Egaleo, Athens, Greece
*
Author to whom correspondence should be addressed.
Received: 24 June 2010; in revised form: 4 August 2010 / Accepted: 5 August 2010 / Published: 20 August 2010

Abstract

:
We introduce a three-parameter generalized normal distribution, which belongs to the Kotz type distribution family, to study the generalized entropy type measures of information. For this generalized normal, the Kullback-Leibler information is evaluated, which extends the well known result for the normal distribution, and plays an important role for the introduced generalized information measure. These generalized entropy type measures of information are also evaluated and presented.
Keywords:
entropy power; information measures; Kotz type distribution; Kullback-Leibler information

1. Introduction

The aim of this paper is to study the new entropy type information measures introduced by Kitsos and Tavoularis [1] and the multivariate hyper normal distribution defined by them. These information measures are defined, adopting a parameter α , as the α -moment of the score function (see Section 2), where α is an integer, while in principle α 2 . One of the merits of this generalized normal distribution is that it belongs to the Kotz-type distribution family [2], i.e., it is an elliptically contoured distribution (see Section 3). Therefore it has all the characteristics and applications discussed in Baringhaus and Henze [3], Liang et al. [4] and Nadarajah [5]. The parameter information measures related to the entropy, are often crucial to the optimal design theory applications, see [6]. Moreover, it is proved that the defined generalized normal distribution provides equality to a new generalized information inequality (Kitsos and Tavoularis [1,7]) regarding the generalized information measure as well as the generalized Shannon entropy power (see Section 3).
In principle, the information measures are divided into three main categories: parametric (a typical example is Fisher’s information [8]), non parametric (with Shannon’s information measure being the most well known) and entropy type [9].
The new generalized entropy type measure of information J α ( X ) , defined by Kitsos and Tavoularis [1], is a function of density, as:
J α ( X ) = p f ( x ) | ln f ( x ) | α d x
From (1), we obtain that J α ( X ) equals:
J α ( X ) = p f ( x ) [ f ( x ) f ( x ) ] α d x = p ( f ( x ) ) 1 α | f ( x ) | α d x
For α = 2 , the measure of information J 2 ( X ) is the Fisher’s information measure:
J ( X ) = p f ( x ) [ f ( x ) f ( x ) ] 2 d x = 4 p | ( f ( x ) ) | 2 d x
i.e., J 2 ( X ) = J ( X ) . That is, J α ( X ) is a generalized Fisher’s entropy type information measure, and as the entropy, it is a function of density.
Proposition 1.1. 
When θ is a location parameter, then J α ( θ ) = J α ( Χ ) .
Proof. Considering the parameter θ as a location parameter and transforming the family of densities to { f ( x ; θ ) = f ( x θ ) } , the differentiation with respect to θ is equivalent to the differentiation with respect to x . Therefore we can prove that J α ( X ) = J α ( θ ) . Indeed, from (1) we have:
J α ( X ) = p f ( x θ ) [ ln f ( x θ ) ] α d x = p f ( x ) [ ln f ( x ) ] α d x = J α ( θ )
and the proposition has been proved.
Recall that the score function is defined as:
U = θ ln f ( X ; θ ) = θ f ( X ; θ ) f ( X ; θ )
with E ( U ) = 0 and E ( U 2 ) = J ( θ ) under some regularity conditions, see Schervish [10] for details. It can be easily shown that when α ,   a > 1 , then J a ( X ) = E ( U α ) . Therefore J α ( X ) behaves as the α -moment of the score function of f ( X ; θ ) . The generalized power is still the power of the white Gaussian noise with the same entropy, see [11], considering the entropy power of a random variable.
Recall that the Shannon entropy H ( X ) is defined as H ( X ) = p f ( x ) ln f ( x ) d x , see [9]. The entropy power N ( X ) is defined through H ( X ) as:
N ( X ) = 1 2 π e e 2 p H ( X )
The definition of the entropy power of a random variable X was introduced by Shannon in 1948 [11] as the independent and identically distributed components of a p-dimensional white Gaussian random variable with entropy H ( X ) .
The generalized entropy power N α ( X ) is of the form;
N α ( X ) = M α e 2 p H ( X )
with the normalizing factor being the appropriate generalization of ( 2 π e ) 1 , i.e.,
M α = ( α 1 α e ) α 1 π α 2 [ Γ ( p 2 + 1 ) Γ ( p α 1 α + 1 ) ] a p = M a ( ξ α p )
is still the power of the white Gaussian noise with the same entropy. Trivially, with α = 2 , the definition in (6) is reduced to the entropy power, i.e., N 2 ( X ) = N ( X ) . In turn, the quantity:
ξ γ p = Γ ( p 2 + 1 ) Γ ( p γ 1 γ + 1 )
appears very often when we define various normalizing factors, under this line of thought.
Theorem 1.1. Generalizing the Information Inequality (Kitsos-Tavoularis [1]). For the variance of X , Var ( X ) and the generalizing Fisher’s entropy type information measure J α ( X ) , it holds:
[ 2 π e p Var ( X ) ] 1 / 2 [ 1 p M α J α ( X ) ] 1 / α 1
with M α as in (7).
Corollary 1.1. When α = 2 then Var ( X ) J 2 ( X ) p 2 , and the Gramer-Rao inequality ([9], Th. 11.10.1) holds.
Proof. Indeed, as M 2 = ( 2 π e ) 1 .
For the above introduced generalized entropy measures of information we need a distribution to play the "role of normal", as in the Fisher's information measure and Shannon entropy. In Kitsos and Tavoularis [12] extend the normal distribution in the light of the introduced generalized information measures and the optimal function satisfying the extension of the LSI. We form the following general definition for an extension of the multivariate normal distribution, the γ -order generalized normal, as follows:
Definition 1.1. The p -dimensional random variable X follows the γ -order generalized Normal, with mean μ and covariance matrix Σ , when the density function is of the form:
K T γ p ( μ , Σ ) = C ( p , γ ) | det Σ | 1 / 2 exp { γ 1 γ Q ( X ) γ 2 ( γ 1 ) }
with Q ( X ) = < ( X μ ) t , Σ 1 ( X μ ) > , where < u t , v > is the inner product of u , v p ) and u t p is the transpose of u . We shall write X K T γ p ( μ , Σ ) . The normality factor C ( p , γ ) is defined as:
C ( p , γ ) = π p / 2 ξ γ p ( γ 1 γ ) p γ 1 γ
Notice that for γ = 2 ,    K T 2 p ( μ , Σ ) is the well known multivariate distribution.
Recall that the symmetric Kotz type multivariate distribution [2] has density:
K o t z m , r , s ( μ , Σ ) = K ( m , r , s ) | det Σ | 1 / 2 Q m 1 exp { r Q s }
where r > 0 ,    s > 0 ,    2 m + n > 2 and the normalizing constant K ( m , r , s ) is given by:
Information 01 00013 i001
see also [1] and [12].
Therefore, it can be shown that the distribution K T γ p ( μ , Σ ) follows from the symmetric Kotz type multivariate distribution for m = 1 , r = γ 1 γ and s = γ 2 ( γ 1 ) , i.e., K T γ p ( μ , Σ ) = K o t z 1 , ( γ 1 ) / γ , γ / 2 ( γ 1 ) ( μ , Σ ) . Also note that for the normal distribution it holds N ( μ , Σ ) = K T 2 p ( μ , Σ ) = K o t z 1 , 1 / 2 , 2 ( μ , Σ ) , while the normalizing factor is C ( p , γ ) = K ( 1 , γ 2 ( γ 1 ) , γ 1 γ ) .

2. The Kullback-Leibler Information for γ –Order Generalized Normal Distribution

Recall that the Kullback-Leibler (K-L) Information for two p -variate density functions f ,   g is defined as [13]:
KLI ( f , g ) = p f ( x ) log f ( x ) g ( x ) d x
The following Lemma provides a generalization of the Kullback-Leibler information measure for the introduced generalized Normal distribution.
Lemma 2.1. The K-L information KLI γ p of the generalized normals K T γ p ( μ 1 , σ 1 2 Ι p ) and K T γ p ( μ 0 , σ 0 2 Ι p ) is equal to:
KLI γ p = C ( p , γ ) σ 1 p [ p 2 ( log σ 0 2 σ 1 2 ) p e q 1 ( x ) d x p e q 1 ( x ) q 1 ( x ) d x + p e q 1 ( x ) q 0 ( x ) d x ]
where q i ( x ) = γ 1 γ ( σ i 1 x μ i ) γ γ 1 ,   x p ,    i = 0 , 1 .
Proof. We have consecutively:
KLI γ p = KLI ( K T γ ( μ 1 , Σ 1 ) , K T γ ( μ 0 , Σ 0 ) ) = p K T γ ( μ 1 , Σ 1 ) ( x ) log K T γ ( μ 1 , Σ 1 ) ( x ) K T γ ( μ 0 , Σ 0 ) ( x ) d x =    p C ( p , γ ) | det Σ 1 | exp { γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) } log C ( p , γ ) | det Σ 1 | exp { γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) } C ( p , γ ) | det Σ 0 | exp { γ 1 γ Q 0 ( x ) γ 2 ( γ 1 ) } d x =    C ( p , γ ) | det Σ 1 | p exp { γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) } [ 1 2 log | det Σ 0 det Σ 1 | γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) + γ 1 γ Q 0 ( x ) γ 2 ( γ 1 ) ] d x
and thus
KLI γ p = C ( p , γ ) | det Σ 1 | ( 1 2 log | det Σ 0 det Σ 1 | p exp { γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) } d x p exp { γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) } γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) d x + p exp { γ 1 γ Q 1 ( x ) γ 2 ( γ 1 ) } γ 1 γ Q 0 ( x ) γ 2 ( γ 1 ) d x )
For Σ 1 = σ 1 2 I p and Σ 0 = σ 0 2 I p , we finally obtain:
KLI γ p = C ( p , γ ) σ 1 p [ 1 2 ( log σ 0 2 p σ 1 2 p ) p e q 1 ( x ) d x p e q 1 ( x ) q 1 ( x ) d x + p e q 1 ( x ) q 0 ( x ) d x ]
where q i ( x ) = γ 1 γ Q i ( x ) γ 2 ( γ 1 ) ,   x p ,    i = 0 , 1 .
Notice that the quadratic forms Q i ( x ) = ( x μ 1 ) t , Σ i 1 ( x μ 1 ) ,    i = 0 , 1 can be written in the form of Q i ( x ) = σ i 2 | x μ 1 | 2 ,    i = 0 , 1 respectively, and thus the lemma has been proved.
Recall now the well known multivariate K-L information measure between two multivariate normal distributions with μ 1 μ 0 is:
KLI 2 p = I p = p 2 [ ( log σ 0 2 σ 1 2 ) 1 + σ 1 2 σ 0 2 + | μ 1 μ 0 | 2 p σ 0 2 ]
which, for the univariate case, is:
KLI 2 1 = I 1 = 1 2 [ ( log σ 0 2 σ 1 2 ) 1 + σ 1 2 σ 0 2 + ( μ 1 μ 0 ) 2 σ 0 2 ]
In what follows, an investigation of the K-L information measure is presented and discussed, concerning the introduced generalized normal. New results are provided that generalize the notion of K-L information. In fact, the following Theorem 2.1 generalizes Ι p   ( = KLI 2 p ) for the γ -order generalized Normal, assuming that μ 1 = μ 0 . Various “sequences” of the K-L information measures KLI γ p are discussed in Corollary 2.1, while the generalized Fisher’s information of the generalized Normal K T γ p ( 0 , Ι p ) is provided in Theorem 2.2.
Theorem 2.1. For μ 1 = μ 0 the K-L information KLI γ p is equal to:
KLI γ p = p 2 ( log σ 0 2 σ 1 2 ) p γ 1 γ [ 1 ( σ 1 σ 0 ) γ γ 1 ]
Proof. We write the result from the previous Lemma 2.1 in the form of:
KLI γ p = C ( p , γ ) σ 1 p [ p 2 ( log σ 0 2 σ 1 2 ) I 1 I 2 + I 3 ]
where:
I 1 = p exp { γ 1 γ ( | x μ 1 | σ 1 ) γ γ 1 } d x  ,  I 2 = p exp { γ 1 γ ( | x μ 1 | σ 1 ) γ γ 1 } γ 1 γ ( | x μ 1 | σ 1 ) γ γ 1 d x
I 3 = p exp { γ 1 γ ( | x μ 1 | σ 1 ) γ γ 1 } γ 1 γ ( | x μ 0 | σ 0 ) γ γ 1 d x
We can calculate the above integrals by writing them as:
I 1 = p exp { [ ( γ 1 γ ) γ 1 γ σ 1 1 | x μ 1 | ] γ γ 1 } d x  ,  I 2 = p exp { [ ( γ 1 γ ) γ 1 γ σ 1 1 | x μ 1 | ] γ γ 1 } [ | x μ 1 | ( γ γ 1 ) γ 1 γ σ 1 ] γ γ 1 d x
I 3 = p exp { [ ( γ 1 γ ) γ 1 γ σ 1 1 | x μ 1 | ] γ γ 1 } [ | x μ 1 | ( γ γ 1 ) γ 1 γ σ 0 ] γ γ 1 d x
and then we substitute z = ( γ 1 γ ) γ 1 γ σ 1 1 ( x μ 1 ) . Thus, we get respectively:
I 1 = ( γ γ 1 ) p γ 1 γ σ 1 p p exp { | z | γ γ 1 } d z  ,  I 2 = ( γ γ 1 ) p γ 1 γ σ 1 p p exp { | z | γ γ 1 } | z | γ γ 1 d z
and:
I 3 = ( γ γ 1 ) p γ 1 γ σ 1 p p exp { | z | γ γ 1 } [ | ( γ γ 1 ) γ 1 γ σ 1 z + μ 1 μ 0 | ( γ γ 1 ) γ 1 γ σ 0 ] γ γ 1 d z ( γ γ 1 ) p γ 1 γ σ 1 p p exp { | z | γ γ 1 } | σ 1 σ 0 z + μ 1 μ 0 ( γ γ 1 ) γ 1 γ σ 0 | γ γ 1 d z
Using the known integrals:
p e | z | β d z = 2 π p / 2 Γ ( p β ) β Γ ( p 2 )  and
p e | z | β | z | β d z = p β p e | z | β d z = 2 p π p / 2 Γ ( p β ) β 2 Γ ( p 2 )
I 1 and I 2 can be calculated as:
I 1 = ( γ γ 1 ) p γ 1 γ σ 1 p 2 π p / 2 Γ ( p γ 1 γ ) γ γ 1 Γ ( p 2 )  and
I 2 = ( γ γ 1 ) p γ 1 γ σ 1 p 2 p π p / 2 Γ ( p γ 1 γ ) ( γ γ 1 ) 2 Γ ( p 2 ) = p γ 1 γ I 1
respectively. Thus, (11) can be written as:
KLI γ p = C ( p , γ ) σ 1 p p I 1 [ 1 2 ( log σ 0 2 σ 1 2 ) γ 1 γ ] + C ( p , γ ) σ 1 p I 3
and by substitution of I 1 from (15) and C ( p , γ ) from definition 1.1, we get:
KLI γ p = 2 p Γ ( p γ 1 γ ) γ γ 1 Γ ( p 2 ) ξ γ p [ 1 2 ( log σ 0 2 σ 1 2 ) γ 1 γ ] + π p / 2 σ 1 p ξ γ p ( γ 1 γ ) p γ 1 γ I 3
Assuming μ 1 = μ 0 , from (12), I 3 is equal to:
I 3 = ( γ γ 1 ) p γ 1 γ σ 1 p ( σ 1 σ 0 ) γ γ 1 p exp { | z | γ γ 1 } | z | γ γ 1 d z
and using the known integral (14), we have:
I 3 = ( γ γ 1 ) p γ 1 γ 2 σ 1 p ( σ 1 σ 0 ) γ γ 1 2 p π p / 2 Γ ( p γ 1 γ ) Γ ( p 2 )
Thus, (16) finally takes the form:
KLI γ p = 2 p Γ ( p γ 1 γ ) γ γ 1 Γ ( p 2 ) ξ γ p [ 1 2 ( log σ 0 2 σ 1 2 ) γ 1 γ ] + 2 p ( γ 1 γ ) 2 ( σ 1 σ 0 ) γ γ 1 Γ ( p γ 1 γ ) Γ ( p 2 ) ξ γ p = 2 p Γ ( p 2 + 1 ) Γ ( p γ 1 γ ) γ γ 1 Γ ( p 2 ) Γ ( p γ 1 γ + 1 ) [ 1 2 ( log σ 0 2 σ 1 2 ) γ 1 γ + γ 1 γ ( σ 1 σ 0 ) γ γ 1 ]
However, Γ ( p 2 + 1 ) = p 2 Γ ( p 2 ) and Γ ( p γ 1 γ + 1 ) = p γ 1 γ Γ ( p γ 1 γ ) , and so (10) has been proved.
For μ 1 μ 0 and for γ = 2 from (16) we obtain:
KLI 2 p = p 2 [ ( log σ 0 2 σ 1 2 ) 1 ] + ( 2 π) p / 2 σ 1 p I 3
where, from (12):
I 3 = 2 p 2 σ 1 p p exp { | z | 2 } | σ 1 σ 0 z + μ 1 μ 0 2 σ 0 | 2 d z = 2 p 2 1 σ 1 p σ 0 2 p exp { | z | 2 } | 2 σ 1 z + μ 1 μ 0 | 2 d z
For γ = 2 we have the only case in which γ 1 γ * , and therefore, the integral of I 3 in (12) can be simplified. Specifically, by setting z = ( z i ) i = 1 p p , μ 0 = ( μ i 0 ) i = 1 p p and μ 1 = ( μ i 1 ) i = 1 p p , from (18) we have consecutively:
I 3 = 2 p 2 1 σ 1 p σ 0 2 p exp { | z | 2 } [ | 2 σ 1 z | 2 + 2 2 σ 1 i = 1 p ( μ i 1 μ i 0 ) z i + | μ 1 μ 0 | 2 ] d z = 2 p 2 σ 1 p + 2 σ 0 2 p exp { | z | 2 } | z | 2 d z + 2 p 2 1 σ 1 p σ 0 2 | μ 1 μ 0 | 2 p exp { | z | 2 } d z + 2 p + 1 2 σ 1 p + 1 σ 0 2 p exp { z 1 2 ... z p 2 } [ i = 1 p ( μ i 1 μ i 0 ) t i ] d z 1 ... d z p
Due to the known integrals (13) and (14), which for β = 2 are reduced respectively to:
p exp { | z | 2 } d z = π p 2  and  p exp { | z | 2 } | z | 2 d z = p 2 π p 2
we obtain:
I 3 = ( 2 π ) p 2 p σ 1 p + 2 2 σ 0 2 + ( 2 π ) p 2 σ 1 p 2 σ 0 2 | μ 1 μ 0 | 2 + 2 p + 1 2 σ 1 p + 1 σ 0 2 ( i = 1 p μ i 1 μ i 0 ) exp { z 1 2 } z 1 d z 1 ... exp { z p 2 } z p d z p = ( 2 π ) p 2 σ 1 p 2 σ 0 2 ( p σ 1 2 + | μ 1 μ 0 | 2 ) + 2 p + 1 2 σ 1 p + 1 σ 0 2 ( i = 1 p μ i 1 μ i 0 ) ( i = 1 p 1 2 exp { z i 2 } d ( z i 2 ) ) = ( 2 π ) p 2 σ 1 p 2 σ 0 2 ( p σ 1 2 + | μ 1 μ 0 | 2 ) + 2 p + 1 2 σ 1 p + 1 σ 0 2 1 2 p ( i = 1 p μ i 1 μ i 0 ) ( i = 1 p lim ε + [ exp { z i 2 } ] z i = ε ε )
i.e.,
I 3 = ( 2 π ) p / 2 σ 1 p 2 σ 0 2 ( p σ 1 2 + | μ 1 μ 0 | 2 ) + 2 1 p 2 σ 1 p + 1 σ 0 2 ( i = 1 p μ i 1 μ i 0 ) 0 0 ... 0 p  times
and hence:
I 3 = ( 2 π ) p / 2 σ 1 p 2 σ 0 2 ( p σ 1 2 + | μ 1 μ 0 | 2 )
Finally, using the above relationship for I 3 , (17) implies:
KLI 2 p = p 2 [ ( log σ 0 2 σ 1 2 ) 1 ] + 1 2 σ 0 2 ( p σ 1 2 + | μ 1 μ 0 | 2 ) = p 2 [ ( log σ 0 2 σ 1 2 ) 1 ] + p σ 1 2 2 σ 0 2 + | μ 1 μ 0 | 2 2 σ 0 2
and the theorem has been proved.
Corollary 2.1. For the Kullback-Leibler information KLI 2 p it holds:
(i)
Provided that σ 0 σ 1 , KLI 2 p has a strict ascending order as dimension p * rises, i.e.,
KLI 2 1 < KLI 2 2 < ... < KLI 2 p < ...
(ii)
Provided that σ 0 σ 1 and μ 1 = μ 0 , and for a given γ , we have:
KLI γ 1 < KLI γ 2 < ... < KLI γ p < ...
(iii)
For KLI p in KLI γ p γ + KLI p , we obtain KLI 1 < KLI 2 < ... < KLI p < ...
(iv)
Given p , σ 0 σ 1 and μ 1 = μ 0 , the K-L information KLI γ p has a strict descending order as γ { 1 } rises, i.e., KLI 2 p > KLI 3 p > ... > KLI γ p > ...
(v)
KLI p is a lower bound of all KLI γ p for γ = 2 , 3 , ...
Proof. From Theorem 2.1 KLI 2 p is a linear expression of p * , i.e.,
KLI 2 p = λ p + | μ 1 μ 0 | 2 2 σ 0 2
with a non-negative slope:
λ = 1 2 [ ( log σ 0 2 σ 1 2 ) 1 + σ 1 2 σ 0 2 ] 0
This is because, from applying the known logarithmic inequality log x x 1 ,    x + * (where equality holds only for x = 1 ) to σ 1 2 / σ 0 2 , we obtain:
( log σ 1 2 σ 0 2 ) σ 1 2 σ 0 2 1     ( log σ 0 2 σ 1 2 ) σ 1 2 σ 0 2 1
and hence λ 0 , where the equality holds respectively only for σ 1 2 / σ 0 2 = 1 , i.e., only for σ 0 = σ 1 . Thus, as the dimension p * rises:
KLI 2 p = λ p + | μ 1 μ 0 | 2 2 σ 0 2
also rises.
In the general p -variate case, KLI γ p is a linear expression of p * , i.e., KLI γ p = λ p , provided that μ 1 = μ 0 , with a non-negative slope:
λ = 1 2 ( log σ 0 2 σ 1 2 ) γ 1 γ [ 1 ( σ 1 σ 0 ) γ γ 1 ] 0
Indeed, λ 0 as:
log ( σ 0 σ 1 ) γ γ 1 = log ( σ 1 σ 0 ) γ γ 1 ( σ 1 σ 0 ) γ γ 1 1
and since γ γ 1 > 1 , we get:
γ 2 ( γ 1 ) log ( σ 0 2 σ 1 2 ) ( σ 1 σ 0 ) γ γ 1 1      1 2 log ( σ 0 2 σ 1 2 ) γ 1 γ [ 1 ( σ 1 σ 0 ) γ γ 1 ]
which implies that λ 0 . The equality holds respectively only for σ 1 2 / σ 0 2 = 1 , i.e., only for σ 0 = σ 1 . Thus, as the dimension p * rises, KLI γ p also rises, i.e., KLI γ 1 < KLI γ 2 < ... , and so KLI 1 < KLI 2 < ... , provided that μ 1 = μ 0 (see Figure 1).
Now, for given p , σ 0 σ 1 and μ 1 = μ 0 , if we choose γ 1 < γ 2 , we have γ 1 γ 1 1 > γ 2 γ 2 1 or γ 1 1 γ 1 < γ 2 1 γ 2 . Thus,
( σ 1 σ 0 ) γ 1 γ 1 1 > ( σ 1 σ 0 ) γ 2 γ 2 1      1 ( σ 1 σ 0 ) γ 1 γ 1 1 < 1 ( σ 1 σ 0 ) γ 2 γ 2 1  ⇒  γ 1 1 γ 1 [ 1 ( σ 1 σ 0 ) γ 1 γ 1 1 ] < γ 2 1 γ 2 [ 1 ( σ 1 σ 0 ) γ 2 γ 2 1 ]
i.e., KLI γ 1 p > KLI γ 2 p . Consequently, KLI 2 p > KLI 3 p > ... and so KLI p < KLI γ p ,   γ = 2 , 3 , ... (see Figure 2).
Figure 1. The graphs of the KLI p , for dimensions p = 1 , 2 , ... , 20 , as functions of the quotient σ 0 / σ 1 (provided that μ 0 = μ 1 ), where we can see that KLI 1 < KLI 2 < ... .
Figure 1. The graphs of the KLI p , for dimensions p = 1 , 2 , ... , 20 , as functions of the quotient σ 0 / σ 1 (provided that μ 0 = μ 1 ), where we can see that KLI 1 < KLI 2 < ... .
Information 01 00013 g001
Figure 2. The graphs of the trivariate KLI γ 3 , for γ = 2 , 3 , ... , and KLI 3 , as functions of the quotient σ 0 / σ 1 (provided that μ 0 = μ 1 ), where we can see that KLI 2 p > KLI 3 p > ... and KLI p < KLI γ p ,   γ = 2 , 3 , ... .
Figure 2. The graphs of the trivariate KLI γ 3 , for γ = 2 , 3 , ... , and KLI 3 , as functions of the quotient σ 0 / σ 1 (provided that μ 0 = μ 1 ), where we can see that KLI 2 p > KLI 3 p > ... and KLI p < KLI γ p ,   γ = 2 , 3 , ... .
Information 01 00013 g002
The Shannon entropy of a random variable X , which follows the generalized normal K T γ p ( μ , Σ ) is:
H ( K T γ p ( μ , Σ ) ) = log | det Σ | 1 / 2 C ( p , γ ) + p γ 1 γ
This is due to the entropy of the symmetric Kotz type distribution (9) and it has been calculated (see [7]) for m = 1 , s = γ 2 ( γ 1 ) , r = γ 1 γ .
Theorem 2.2. The generalized Fisher’s information of the generalized Normal K T γ p ( 0 , Ι p ) is:
J α ( K T γ p ( 0 , Ι p ) ) = p Γ ( α + p ( γ 1 ) γ ) Γ ( 1 + p γ 1 γ ) ( γ 1 γ ) γ α γ
Proof. From:
| f 1 / α | 2 = | 1 α f ( 1 α ) / α f | 2 = ( 1 α ) 2 f 2 1 α α | f | 2
implies:
| f 1 / α | α = ( 1 α ) α f 1 α | f | α
Therefore, from (2), J α ( X ) equals eventually:
J α ( K T γ p ( 0 , Ι p ) ) = C ( p , γ ) α α p ( 1 α ) α | x | α γ 1 exp { γ 1 γ | x | γ γ 1 } d x
Switching to hyperspherical coordinates and taking into account the value of C ( p , γ ) we have the result (see [7] for details).
Corollary 2.2. Due to Theorem 2.2, it holds that J α ( K T α p ( 0 , Ι p ) ) = p and N α ( K T α p ( 0 , Ι p ) ) = 1 .
Proof. Indeed, from (20), J α ( K T α p ( 0 , Ι p ) ) = p and the fact that J α ( X ) N α ( X ) p (which has been extensively studied under the Logarithmic Sobolev Inequalities, see [1] for details and [14]), Corollary 2.2 holds.
Notice that, also from (20):
N α ( K T γ p ( 0 , Ι p ) ) = p J a 1
and therefore, N α ( K T α p ( 0 , Ι p ) ) = 1 .

3. Discussion and Further Analysis

We examine the behavior of the multivariate γ -order generalized Normal distribution. Using Mathematica, we proceed to the following helpful calculations to analyze further the above theoretical results, see also [12].
Figure 3 represents the univariate γ -order generalized Normal distribution for various values of γ : γ = 2 (normal distribution), γ = 5 , γ = 10 , γ = 100 , while Figure 4, represents the bivariate 10-order generalized Normal K T 10 2 ( 0 , Ι 2 ) with mean 0 and covariance matrix Σ = Ι 2 .
Figure 3. The univariate γ -order generalized Normals K T γ 1 ( 0 , 1 ) for γ = 2 , 5 , 10 , 100 .
Figure 3. The univariate γ -order generalized Normals K T γ 1 ( 0 , 1 ) for γ = 2 , 5 , 10 , 100 .
Information 01 00013 g003
Figure 4. The bivariate 10-order generalized normal K T 10 2 ( 0 , 1 ) .
Figure 4. The bivariate 10-order generalized normal K T 10 2 ( 0 , 1 ) .
Information 01 00013 g004
Figure 5 provides the behavior of the generalized information measure for the bivariate generalized normal distribution, i.e., J α ( K T γ ( 0 , Ι 2 ) ) , where ( α , γ ) [ 2 , 20 ] × [ 2 , 10 ] .
Figure 5. J α ( K T 10 5 ( 0 , Ι 5 ) ) ,    1 < a 20
Figure 5. J α ( K T 10 5 ( 0 , Ι 5 ) ) ,    1 < a 20
Information 01 00013 g005
For the introduced generalized information measure of the generalized Normal distribution it holds that:
J α ( K T γ p ( 0 , Ι p ) ) { > p .    f o r    α > γ = p ,    f o r    α = γ < p ,    f o r    α < γ
For α = 2 and γ = 2 , it is the typical entropy power for the normal distribution.
The greater the number of variables involved at the multivariate γ -order generalized normal, the larger the generalized information J α ( K T γ p ( 0 , Ι 2 ) ) , i.e., J α ( K T γ p 1 ( 0 , Ι p 1 ) ) > J α ( K T γ p 2 ( 0 , Ι p 2 ) ) for p 1 < p 2 . In other words, the generalized information measure of the multivariate generalized Normal distribution is an increasing function of the number of the involved variables.
Let us consider α and p to be constants. If we let γ vary, then the generalized information of the multivariate γ -order generalized Normal is a decreasing function of γ , i.e., for γ 1 > γ 2 we have J α ( K T γ 1 p ( 0 , Ι p ) ) > J α ( K T γ 2 p ( 0 , Ι p ) ) except for p = 1 .
Proposition 3.1. The lower bound of J α ( K T γ p ( 0 , Ι p ) ) is the Fisher’s entropy type information.
Proof. Letting α vary and excluding the case p = 1 , the generalized information of the multivariate generalized normal distribution is an increasing function of α , i.e., J α 1 ( K T γ p ( 0 , Ι p ) ) > J α 2 ( K T γ p ( 0 , Ι p ) ) for a 1 > a 2 . That is the Fisher’s information J 2 ( K T γ p ( 0 , Ι p ) ) is smaller than the generalized information measure J α ( K T γ p ( 0 , Ι p ) ) for all α > 2 , provided that p 1 .
When α < γ , the more variables involved at the multivariate generalized Normal distribution the larger the generalized entropy power Ν α ( K T γ p ( 0 , Ι p ) ) , i.e., Ν α ( K T γ p 1 ( 0 , Ι p 1 ) ) > Ν α ( K T γ p 2 ( 0 , Ι p 2 ) ) for p 1 > p 2 . The dual occurs when α > γ . That is, when α < γ the number of involved variables defines an increasing generalized entropy power, while for α > γ the generalized entropy power is a decreasing function of the number of involved variables. Let us consider α and γ to be constants and let p vary. When α < γ the generalized entropy power of the multivariate generalized Normal distribution Ν α ( K T γ p ( 0 , Ι p ) ) is increasing in p .
When α > γ , the generalized entropy power of the multivariate generalized Normal distribution Ν α ( K T γ p ( 0 , Ι p ) ) is decreasing in p . If we let γ vary and let γ 1 > γ 2 > 1 then, for the generalized entropy power of the multivariate generalized Normal distribution, we have Ν α ( K T γ 1 p ( 0 , Ι p ) ) > Ν α ( K T γ 2 p ( 0 , Ι p ) ) for certain α and p , i.e., the generalized entropy power of the multivariate generalized Normal distribution is an increasing function of γ .
Now, letting α vary, Ν α ( K T γ p ( 0 , Ι p ) ) is a decreasing function of α , i.e., Ν α 1 ( K T γ p ( 0 , Ι p ) ) > Ν α 2 ( K T γ p ( 0 , Ι p ) ) provided that α 1 < α 2 . The well known entropy power Ν α ( K T γ p ( 0 , Ι p ) ) provides an upper bound for the generalized entropy power, i.e., 0 < Ν α ( K T γ p ( 0 , Ι p ) ) < Ν 2 ( K T γ p ( 0 , Ι p ) ) , for given γ and p .

4. Conclusions

In this paper new information measures were defined, discussed and analyzed. The introduced γ -order generalized Normal acts on these measures as the well known normal distribution does on the Fisher’s information measure for the well known cases. The provided computations offer an insight analysis of these new measures, which can also be considered as the α -moments of the score function. For the γ -order generalized Normal distribution, the Kullback-Leibler information measure was evaluated, providing evidence that the generalization “behaves well”.

Acknowledgements

The authors would like to thank I. Vlachaki for helpful discussions during the preparation of this paper. The referees’ comments are very much appreciated.

References

  1. Kitsos, C.P.; Tavoularis, N.K. Logarithmic Sobolev inequalities for information measures. IEEE Trans. Inform. Theory 2009, 55, 2554–2561. [Google Scholar] [CrossRef]
  2. Kotz, S. Multivariate distribution at a cross-road. In Statistical Distributions in Scientific Work; Patil, G.P., Kotz, S., Ord, J.F., Eds.; D. Reidel Publishing Company: Dordrecht, The Netherlands, 1975; Volume 1, pp. 247–270. [Google Scholar]
  3. Baringhaus, L.; Henze, N. Limit distributions for Mardia's measure of multivariate skewness. Ann. Statist. 1992, 20, 1889–1902. [Google Scholar] [CrossRef]
  4. Liang, J.; Li, R.; Fang, H.; Fang, K.T. Testing multinormality based on low-dimensional projection. J. Stat. Plan. Infer. 2000, 86, 129–141. [Google Scholar] [CrossRef]
  5. Nadarajah, S. The Kotz-type distribution with applications. Statistics 2003, 37, 341–358. [Google Scholar] [CrossRef]
  6. Kitsos, C.P. Optimal designs for estimating the percentiles of the risk in multistage models of carcinogenesis. Biometr. J. 1999, 41, 33–43. [Google Scholar] [CrossRef]
  7. Kitsos, C.P.; Tavoularis, N.K. New entropy type information measures. In Proceedings of Information Technology Interfaces, Dubrovnik, Croatia, 24 June 2009; Luzar-Stiffer, V., Jarec, I., Bekic, Z., Eds.; pp. 255–259.
  8. Fisher, R.A. Theory of statistical estimation. Proc. Cambridge Phil. Soc. 1925, 22, 700–725. [Google Scholar] [CrossRef]
  9. Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed; Wiley: New York, NY, USA, 2006. [Google Scholar]
  10. Scherrvish, J.M. Theory of Statistics; Springer Verlag: New York, NY, USA, 1995. [Google Scholar]
  11. Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379-423, 623-656. [Google Scholar] [CrossRef]
  12. Kitsos, C.P.; Tavoularis, N.K. On the hyper-multivariate normal distribution. In Proceedings of International Workshop on Applied Probabilities, Paris, France, 7–10 July 2008.
  13. Kullback, S.; Leibler, A. On information and sufficiency. Ann. Math. Statist. 1951, 22, 79–86. [Google Scholar] [CrossRef]
  14. Darbellay, G.A.; Vajda, I. Entropy expressions for multivariate continuous distributions. IEEE Trans. Inform. Theory 2000, 46, 709–712. [Google Scholar] [CrossRef]
Information EISSN 2078-2489 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top