Discrete Versions of Jensen–Fisher, Fisher and Bayes–Fisher Information Measures of Finite Mixture Distributions

Abstract

In this work, we first consider the discrete version of Fisher information measure and then propose Jensen–Fisher information, to develop some associated results. Next, we consider Fisher information and Bayes–Fisher information measures for mixing parameter vector of a finite mixture probability mass function and establish some results. We provide some connections between these measures with some known informational measures such as chi-square divergence, Shannon entropy, Kullback–Leibler, Jeffreys and Jensen–Shannon divergences.

1. Introduction

Over the last seven decades, several different criteria have been introduced in the literature for measuring uncertainty in a probabilistic model. Shannon entropy and Fisher information are the most important information measures that have been used rather extensively. Information theory started with Shannon entropy, introduced in the pioneering work of Shannon [1], based on a study of systems described by probability density (or mass) functions. About two decades earlier, Fisher [2] had proposed another information measure, describing the interior properties of a probabilistic model, that plays an important role in likelihood-based inferential methods. Fisher information and Shannon entropy are fundamental criteria in statistical inference, physics, thermodynamics and information theory. Complex systems can be described by means of their behavior (Shannon) and their architecture (Fisher) information. For more discussions, see Zegers [3] and Balakrishnan and Stepanov [4].

Let X be a discrete random variable with probability mass function (PMF) $\mathit{P}=(p_1,\dots ,p_n)$. Then, the Shannon entropy of random variable X is defined as

$$H\left(X\right)=H\left(\mathit{P}\right)=-\sum _{i=1}^n{p}_{i}log{p}_i,$$

where “log” denotes the natural logarithm. For more details, see Shannon [1]. Following the work of Shannon [1], considerable attention has been paid to providing some extensions of Shannon entropy. Jensen–Shannon (JS) divergence is one such important extension of Shannon entropy that has been widely used; see Lin [5]. The Jensen–Shannon divergence between two probability mass functions $\mathit{P}=(p_1,p_2,\dots ,p_n)$ and $\mathit{Q}=(q_1,q_2,\dots ,q_n)$, for $0\le \alpha \le 1$, is defined as

$$\begin{array}{c}\hfill JS(\mathit{P},\mathit{Q};\alpha )=H(\alpha \mathit{P}+(1-\alpha \left)\mathit{Q}\right)-\alpha H\left(\mathit{P}\right)-(1-\alpha )H\left(\mathit{Q}\right).\end{array}$$

The JS divergence is a smoothed and symmetric version of the most important divergence measure of information theory, namely, Kullback–Leibler divergence. Recently, Jensen–Fisher (JF) and Jensen–Gini (JG) divergence measures have been introduced by Sánchez-Moreno et al. [6] and Mehrali et al. [7], respectively.

In the present paper, motivated by the idea of JS divergence, we consider discrete versions of Fisher information (DFI) and Fisher information distance (DFID), and then develop a new information measure associated with DFI measure. In addition, we provide some results for the Fisher information of a finite mixture probability mass function through a Bayesian perspective. The discrete Fisher information of a random variable X with PMF $\mathit{P}=(p_1,p_2,\dots ,p_n)$ is defined as

$$\begin{array}{c}\hfill \mathcal{I}\left(\mathit{P}\right)=\sum _{i=1}^n\frac{{\left(p_{i+1}-p_i\right)}^2}{p_i},\end{array}$$

(1)

with $p_{n+1}=0$.

The Fisher information in (1) has been made use of in the processing of complex and stationary signals. For example, the discrete version of Fisher information has been used in detecting epileptic seizures in EEG signals recorded in humans and turtles, in detecting dynamical changes in many non-linear models such as logistic map and Lorenz model, and also in the analysis of geoelectrical signals; see Martin et al. [8], Ramírez-Pacheco et al. [9] and Ramírez-Pacheco et al. [10] for pertinent details.

The discrete Fisher information distance (DFID) between two probability mass functions $\mathit{P}=(p_1,p_2,\dots ,p_n)$ and $\mathit{Q}=(q_1,q_2,\dots ,q_n)$ is defined as

$$\begin{array}{c}\hfill \mathcal{D}(\mathit{P},\mathit{Q})=\sum _{i=1}^n{\left(\frac{p_{i+1}}{p_i}-\frac{q_{i+1}}{q_i}\right)}^2{p}_i,\end{array}$$

(2)

where, as above, $p_{n+1}=q_{n+1}=0$. For some of its properties, one may refer to Ramírez-Pacheco et al. [10] and Johnson [11].

With regard to informational properties of finite mixture models, one may refer to Contreras-Reyes and Cortés [12] and Abid et al. [13]. These authors have provided upper and lower bounds for Shannon and Rényi entropies of non-gaussian finite mixtures, skew-normal and skew-t distributions, respectively. Kolchinsky and Tracey [14] have studied the upper and lower bounds for the entropy of Gaussian mixture distributions using the Bhattacharyya and Kullback–Leibler divergences.

The first purpose of this paper is to propose Jensen–Fisher information for discrete random variables $X_1,\dots ,X_n$, with probability mass functions ${\mathit{P}}_1,\dots ,{\mathit{P}}_n,$ respectively. For this purpose, we first define discrete version of Jensen–Fisher information for two PMFs $\mathit{P}$ and $\mathit{Q}$, and then provide some results concerning this new information measure. Then, this idea is extended to the general case of PMFs ${\mathit{P}}_1,\dots ,{\mathit{P}}_n$.

The second purpose of this work is to study Fisher and Bayes–Fisher information measures for the mixing parameter of a finite mixture probability mass function. Let ${\mathit{P}}_1,\dots ,{\mathit{P}}_n$ be n probability mass functions, where ${\mathit{P}}_j=(p_{j1},\dots ,p_{jk})$. Then, a finite mixture probability mass function with mixing parameter vector $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_{n-1})$, for $n\ge 2,$ is given by ${\mathit{P}}_{\theta }=(p_{\mathit{\theta }}^1,\dots ,p_{\mathit{\theta }}^k),$ where

$$\begin{array}{c}\hfill p_{\mathit{\theta }}^j=\frac{1}{n-1}\sum _{i=1}^{n-1}{\theta }_i{p}_{ij}+\left(1-\frac{{\sum }_{i=1}^{n-1}{\theta }_i}{n-1}\right)p_{nj},j=1,\dots ,k,\end{array}$$

(3)

$0\le {\theta }_i\le 1,i=1,\dots ,n-1$ and ${\sum }_{i=1}^{n-1}{\theta }_i\le 1$.

Let X and Y be two discrete random variables with PMFs $\mathit{P}=(p_1,\dots ,p_n)$ and $\mathit{Q}=(q_1,\dots ,q_n)$, respectively. Then, the Kullback–Leibler (KL) distance between X and Y (or $\mathit{P}$ and $\mathit{Q}$) is defined as

$$KL\left(X\right|\left|Y\right)=KL(\mathit{P},\mathit{Q})=\sum _{i=1}^n{p}_{i}log\left(\frac{p_i}{q_i}\right).$$

The Kullback–Leibler discrimination between Y and X can be defined similarly. For more details, see Kullback and Leibler [15]. The chi-square divergence between PMFs $\mathit{P}$ and $\mathit{Q}$ is defined by

$$\begin{array}{c}\hfill {\chi }^2(\mathit{P},\mathit{Q})=\sum _{i=1}^n\frac{{\left(p_i-q_i\right)}^2}{p_i}.\end{array}$$

For pertinent details, see Broniatowski [16] and Cover and Thomas [17].

The rest of this paper is organized as follows. In Section 2, we first consider discrete version of Fisher information and then propose the discrete Jensen–Fisher information (DJFI) measure. We show that DJFI measure can be represented based on the mixture of discrete Fisher information distance measures. In Section 3, we consider a finite mixture probability mass function and establish some results for the Fisher information measure of the mixing parameter vector. We show that the Fisher information of the mixing parameter vector is connected to chi-square divergence. Next, in Section 4, we discuss the Bayes–Fisher information for the mixing parameter vector of probability mass functions under some prior distributions for the mixing parameter. We then show that this measure is connected to Shannon entropy, Jensen–Shannon entropy, Kullback–Leibler and Jeffreys divergence measures. Finally, we present some concluding remarks in Section 5.

2. Discrete Version of Jensen-Fisher Information

In this section, we first give a result for the DFI measure based on the log-convex and log-concave property of the probability mass function. Then, we define the discrete Jensen–Fisher information measure, and establish some interesting properties of it.

Theorem 1.

Let $\mathit{P}=(p_1,p_2,\dots ,p_n)$ be a probability mass function.

(i) 

If $\mathit{P}$ is log-concave, then $\mathcal{I}\left(\mathit{P}\right)\le p_1;$

(ii) 

If $\mathit{P}$ is log-convex, then $\mathcal{I}\left(\mathit{P}\right)\ge p_1.$

Proof. 

$\mathit{P}$ is log-convex (log-concave) if $p_i^2\ge (\le )p_{i-1}{p}_{i+1}\forall i.$ So, from the definition of DFI in (1), we have

$$\begin{array}{ccc}\hfill \mathcal{I}\left(\mathit{P}\right)& =& \sum _{i=1}^n\frac{{\left(p_{i+1}-p_i\right)}^2}{p_i}\ge (\le )p_1.\hfill \end{array}$$

  □

2.1. Discrete Jensen–Fisher Information Based on Two Probability Mass Functions P and Q

We first define a symmetric version of DFID measure in (2), and then propose the discrete Jensen–Fisher information measure involving two probability mass functions.

Definition 1.

Let $\mathit{P}$ and $\mathit{Q}$ be two probability mass functions given by $\mathit{P}=(p_1,p_2,\dots ,p_n)$ and $\mathit{Q}=(q_1,q_2,\dots ,q_n)$. Then, a symmetric version of discrete Fisher information distance in (2) is defined as

$$\begin{array}{ccc}\hfill \mathcal{SD}(\mathit{P},\mathit{Q})& =& \frac{1}{2}\mathcal{D}\left(\mathit{P},\frac{\mathit{P}+\mathit{Q}}{2}\right)+\frac{1}{2}\mathcal{D}\left(\mathit{Q},\frac{\mathit{P}+\mathit{Q}}{2}\right).\hfill \end{array}$$

Definition 2.

Let $\mathit{P}$ and $\mathit{Q}$ be two probability mass functions given by $\mathit{P}=(p_1,p_2,\dots ,p_n)$ and $\mathit{Q}=(q_1,q_2,\dots ,q_n).$ Then, the discrete Jensen–Fisher information is defined as

$$\begin{array}{ccc}\hfill \mathcal{JFI}(\mathit{P},\mathit{Q})& =& \frac{\mathcal{I}\left(\mathit{P}\right)+\mathcal{I}\left(\mathit{Q}\right)}{2}-\mathcal{I}\left(\frac{\mathit{P}+\mathit{Q}}{2}\right).\hfill \end{array}$$

(4)

In the following theorem, we show that the discrete Jensen–Fisher information measure can be obtained based on mixtures of Fisher information distances.

Theorem 2.

Let $\mathit{P}$ and $\mathit{Q}$ be two probability mass functions given by $\mathit{P}=(p_1,p_2,\dots ,p_n)$ and $\mathit{Q}=(q_1,q_2,\dots ,q_n)$. Then,

$$\begin{array}{ccc}\hfill \mathcal{JFI}(\mathit{P},\mathit{Q})& =& \frac{1}{2}\mathcal{D}\left(\mathit{P},\frac{\mathit{P}+\mathit{Q}}{2}\right)+\frac{1}{2}\mathcal{D}\left(\mathit{Q},\frac{\mathit{P}+\mathit{Q}}{2}\right)\hfill \\ & =& \mathcal{SD}(\mathit{P},\mathit{Q}).\hfill \end{array}$$

Proof. 

From the definition of DFID in (2), we get

$$\begin{array}{ccc}\hfill \mathcal{D}\left(\mathit{P},\frac{\mathit{P}+\mathit{Q}}{2}\right)& =& \sum _{i=1}^n{\left(\frac{p_{i+1}}{p_i}-\frac{p_{i+1}+q_{i+1}}{p_i+q_i}\right)}^2{p}_i\hfill \\ & =& \sum _{i=1}^n{\left\{\left(\frac{p_{i+1}}{p_i}-1\right)-\left(\frac{p_{i+1}+q_{i+1}}{p_i+q_i}-1\right)\right\}}^2{p}_i\hfill \\ & =& \sum _{i=1}^n{\left(\frac{p_{i+1}}{p_i}-1\right)}^2{p}_i-2\sum _{i=1}^n\left(\frac{p_{i+1}}{p_i}-1\right)\left(\frac{p_{i+1}+q_{i+1}}{p_i+q_i}-1\right)p_i\hfill \\ & & +\sum _{i=1}^n{\left(\frac{p_{i+1}+q_{i+1}}{p_i+q_i}-1\right)}^2{p}_i\hfill \\ & =& \sum _{i=1}^n\frac{{\left(p_{i+1}-p_i\right)}^2}{p_i}-2\sum _{i=1}^n(p_{i+1}-p_i)\frac{\left(p_{i+1}+q_{i+1}-(p_i+q_i)\right)}{p_i+q_i}\hfill \\ & & +\sum _{i=1}^n\frac{{\left(p_{i+1}+q_{i+1}-(p_i+q_i)\right)}^2}{{(p_i+q_i)}^2}{p}_i.\hfill \end{array}$$

In a similar way, we get

$$\begin{array}{ccc}\hfill \mathcal{D}\left(\mathit{Q},\frac{\mathit{P}+\mathit{Q}}{2}\right)& =& \sum _{i=1}^n\frac{{\left(q_{i+1}-q_i\right)}^2}{q_i}-2\sum _{i=1}^n(q_{i+1}-q_i)\frac{(p_{i+1}+q_{i+1}-(p_i+q_i))}{p_i+q_i}\hfill \\ & & +\sum _{i=1}^n\frac{{(p_{i+1}+q_{i+1}-(p_i+q_i))}^2}{{(p_i+q_i)}^2}{q}_i.\hfill \end{array}$$

Upon adding the above two expressions, we obtain

$$\begin{array}{ccc}\hfill \mathcal{D}\left(\mathit{P},\frac{\mathit{P}+\mathit{Q}}{2}\right)+\mathcal{D}\left(\mathit{Q},\frac{\mathit{P}+\mathit{Q}}{2}\right)& =& \sum _{i=1}^n\frac{{\left(p_{i+1}-p_i\right)}^2}{p_i}+\sum _{i=1}^n\frac{{\left(q_{i+1}-q_i\right)}^2}{q_i}\hfill \\ & & -\sum _{i=1}^n\frac{{(p_{i+1}+q_{i+1}-(p_i+q_i))}^2}{p_i+q_i}\hfill \\ & =& \mathcal{I}\left(\mathit{P}\right)+\mathcal{I}\left(\mathit{Q}\right)-2\mathcal{I}\left(\frac{\mathit{P}+\mathit{Q}}{2}\right)\hfill \\ & =& 2\mathcal{JFI}(\mathit{P},\mathit{Q}),\hfill \end{array}$$

as required.  □

Example 1.

Let

$$X=\left\{\begin{array}{cc}1,\hfill & withprobabilityp,\hfill \\ 0,\hfill & withprobability1-p,\hfill \end{array}\right.$$

and

$$Y=\left\{\begin{array}{cc}1,\hfill & withprobabilityq,\hfill \\ 0,\hfill & withprobability1-q.\hfill \end{array}\right.$$

The corresponding PMFs of variables X and Y are given by $\mathit{P}=(p,1-p)$ and $\mathit{Q}=(q,1-q)$, respectively. From Theorem 2, we then have

$$\begin{array}{ccc}\hfill \mathcal{JFI}(\mathit{P},\mathit{Q})& =& \frac{p+q}{2}{\left(\frac{1-p}{p}-\frac{1-q}{q}\right)}^2.\hfill \end{array}$$

A 3D-plot of this $\mathcal{JFI}(\mathit{P},\mathit{Q})$ is presented in Figure 1.

Figure 1. 3D-plot of the DJFI divergence between the PMFs $\mathit{P}=(p,1-p)$ and $\mathit{Q}=(q,1-q)$.

2.2. Discrete Jensen–Fisher Information Based on n Probability Mass Functions $P_1,\dots ,P_n$

Let ${\mathit{P}}_1,\dots ,{\mathit{P}}_n$ be n probability mass functions, where ${\mathit{P}}_i=(p_{i1},\dots ,p_{ik})$. In the following definition, we extend the discrete Jensen–Fisher information measure in (4) to the case of n probability mass functions.

Definition 3.

Let ${\mathit{P}}_1,\dots ,{\mathit{P}}_n$ be n probability mass functions given by ${\mathit{P}}_i=(p_{i1},p_{i2},\dots ,p_{ik})$, $i=1,2,\dots ,n$, with ${\sum }_{j=1}^k{p}_{ij}=1$, and ${\alpha }_1,\dots ,{\alpha }_n$ be non-negative real numbers such that ${\sum }_{i=1}^n{\alpha }_i=1$. Then, the discrete Jensen–Fisher information (DJFI) based on the n probability mass functions is defined as

$$\begin{array}{ccc}\mathcal{JFI}({\mathit{P}}_1,\dots ,{\mathit{P}}_n;\underline{\alpha })\hfill & =\hfill & \sum _{i=1}^n{\alpha }_i\mathcal{I}\left({\mathit{P}}_i\right)-\mathcal{I}\left(\sum _{i=1}^n{\alpha }_i{\mathit{P}}_i\right)\hfill \\ \hfill & =\hfill & \sum _{i=1}^n{\alpha }_i\sum _{j=1}^k\frac{{(p_{ij+1}-p_{ij})}^2}{p_{ij}}-\sum _{j=1}^k\frac{{({\sum }_{i=1}^n{\alpha }_i{p}_{ij+1}-{\sum }_{i=1}^n{\alpha }_i{p}_{ij})}^2}{{\sum }_{i=1}^n{\alpha }_i{p}_{ij}},\hfill \end{array}$$

(5)

where $\underline{\alpha }=({\alpha }_1,\dots ,{\alpha }_n).$

Theorem 3.

Let ${\mathit{P}}_1,\dots ,{\mathit{P}}_n$ be n probability mass functions given by ${\mathit{P}}_i=(p_{i1},p_{i2},\dots ,p_{ik})$, $i=1,2,\dots ,n$, and ${\sum }_{j=1}^k{p}_{ij}=1$. Then, the DJFI measure can be expressed as a mixture of DFID measures in (2) as follows:

$$\begin{array}{ccc}\hfill \mathcal{JFI}({\mathit{P}}_1,\dots ,{\mathit{P}}_n,\underline{\alpha })& =& \sum _{i=1}^n{\alpha }_i\mathcal{D}({\mathit{P}}_i,{\mathit{P}}_T),\hfill \end{array}$$

where ${\mathit{P}}_T={\sum }_{i=1}^n{\alpha }_i{\mathit{P}}_i$ is the weighted PMF.

Proof. 

From the definition in (5), we get

$$\begin{array}{ccc}\hfill \sum _{i=1}^n{\alpha }_i\mathcal{D}({\mathit{P}}_i,{\mathit{P}}_T)& =& \sum _{i=1}^n{\alpha }_i\sum _{j=1}^k{\left(\frac{p_{ij+1}}{p_{ij}}-\frac{{\sum }_{i=1}^n{\alpha }_i{p}_{ij+1}}{{\sum }_{i=1}^n{\alpha }_i{p}_{ij}}\right)}^2{p}_{ij}\hfill \\ & =& \sum _{i=1}^n{\alpha }_i\sum _{j=1}^k{\left\{\frac{p_{ij+1}}{p_{ij}}-1-\left(\frac{{\sum }_{i=1}^n{\alpha }_i{p}_{ij+1}}{{\sum }_{i=1}^n{\alpha }_i{p}_{ij}}-1\right)\right\}}^2{p}_{ij}\hfill \\ & =& \sum _{i=1}^n{\alpha }_i\sum _{j=1}^k\frac{{(p_{ij+1}-p_{ij})}^2}{p_{ij}}-2\sum _{j=1}^k\frac{{({\sum }_{i=1}^n{\alpha }_i{p}_{ij+1}-{\sum }_{i=1}^n{\alpha }_i{p}_{ij})}^2}{{\sum }_{i=1}^n{\alpha }_i{p}_{ij}}\hfill \\ & & +\sum _{j=1}^k\frac{{({\sum }_{i=1}^n{\alpha }_i{p}_{ij+1}-{\sum }_{i=1}^n{\alpha }_i{p}_{ij})}^2}{{\sum }_{i=1}^n{\alpha }_i{p}_{ij}}\hfill \\ & =& \sum _{i=1}^n{\alpha }_i\sum _{j=1}^k\frac{{(p_{ij+1}-p_{ij})}^2}{p_{ij}}-\sum _{j=1}^k\frac{{({\sum }_{i=1}^n{\alpha }_i{p}_{ij+1}-{\sum }_{i=1}^n{\alpha }_i{p}_{ij})}^2}{{\sum }_{i=1}^n{\alpha }_i{p}_{ij}}\hfill \\ & =& \sum _{i=1}^n{\alpha }_i\mathcal{I}\left({\mathit{P}}_i\right)-\mathcal{I}\left(\sum _{i=1}^n{\alpha }_i{\mathit{P}}_i\right)\hfill \\ & =& \mathcal{JFI}({\mathit{P}}_1,\dots ,{\mathit{P}}_n,\underline{\alpha }),\hfill \end{array}$$

as required. □

3. Fisher Information of a Finite Mixture Probability Mass Function

In this section, we discuss Fisher information for parameter $\mathit{\theta }$ of a finite mixture probability mass function.

Theorem 4.

The Fisher information of PMF in (3) about parameter ${\theta }_i,i=1,\dots ,n-1$, is given by

$$\begin{array}{c}\hfill \mathcal{I}\left({\theta }_i\right)=\frac{1}{{\left({\theta }_i-(n-1)\right)}^2}{\chi }^2({\mathit{P}}_{{\mathit{\theta }}_{-i}},{\mathit{P}}_{\mathit{\theta }}),i=1,\dots ,n-1,\end{array}$$

(6)

where ${\mathit{P}}_{{\mathit{\theta }}_{-i}}=(p_{{\mathit{\theta }}_{-i}}^1,\dots ,p_{{\mathit{\theta }}_{-i}}^k),$

$$p_{{\mathit{\theta }}_{-i}}^j=\frac{n-2}{n-1}{p}_{ij}+\frac{1}{n-1}\sum _{\mathsf{ł}=1,\mathsf{ł}\ne i}^{n-1}{\theta }_{\mathsf{ł}}{p}_{\mathsf{ł}j}+\frac{1}{n-1}\left(1-\sum _{\mathsf{ł}=1,\mathsf{ł}\ne i}^{n-1}{\theta }_{\mathsf{ł}}\right)p_{nj},j=1,\dots ,k,$$

and ${\mathit{\theta }}_{-\mathit{i}}=({\theta }_1,\dots ,{\theta }_{i-1},{\theta }_{i+1},\dots ,{\theta }_{n-1}).$

Proof. 

From the definition of Fisher information in (1) and for $i=1,\dots ,n-1$, we have

$$\begin{array}{ccc}\mathcal{I}\left({\theta }_i\right)\hfill & =\hfill & \sum _{j=1}^k{\left[{\displaystyle \frac{\partial log{\mathit{P}}_{\mathit{\theta }}}{\partial {\theta }_i}}\right]}^2{p}_{\theta }^j\hfill \\ \hfill & =\hfill & {\displaystyle \frac{1}{{\left(n-1\right)}^2}}\sum _{j=1}^k{\displaystyle \frac{{\left(p_{ij}-p_{nj}\right)}^2}{{\left(p_{\mathit{\theta }}^j\right)}^2}}{p}_{\mathit{\theta }}^j\hfill \\ \hfill & =\hfill & {\displaystyle \frac{1}{{\left({\theta }_i-(n-2)\right)}^2}}\sum _{j=1}^k{\displaystyle \frac{(p_{{\mathit{\theta }}_{-i}}^j-p_{\mathit{\theta }}^j{)}^2}{p_{\mathit{\theta }}^j}}\hfill \\ \hfill & =\hfill & {\displaystyle \frac{1}{{\left({\theta }_i-(n-1)\right)}^2}}{\chi }^2({\mathit{P}}_{{\mathit{\theta }}_{-i}},{\mathit{P}}_{\mathit{\theta }}),i=1,\dots ,n-1,\hfill \end{array}$$

(7)

where the third equation follows from the fact that, for $i=1,\dots ,n-1,$

$$p_{ij}-p_{nj}=\frac{n-1}{{\theta }_i-(n-2)}\left(p_{\mathit{\theta }}^j-p_{{\mathit{\theta }}_{-i}}^j\right).$$

  □

4. Bayes–Fisher Information of a Finite Mixture Probability Mass Function

In this section, we discuss Bayes–Fisher information for the mixing parameter vector $\mathit{\theta }$ of the finite mixture probability mass function in (3) under some prior distributions for the mixing parameter vector. We now introduce two notations that will be used in the sequel. Consider the parameter vector $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_{n-1})$, and then define $(0_i,\mathit{\theta })=({\theta }_1,\dots ,{\theta }_{i-1},0,{\theta }_{i+1},\dots ,{\theta }_{n-1})$ and $(1_i,\mathit{\theta })=({\theta }_1,\dots ,{\theta }_{i-1},1,{\theta }_{i+1},\dots ,{\theta }_{n-1}).$

Theorem 5.

The Bayes–Fisher information for parameter ${\theta }_i,i=1,\dots ,n-1$, of the finite mixture PMF in (3), under the uniform prior on $[0,1],$ is given by

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& KL\left({\mathit{P}}_{(1_i,\mathit{\theta })},{\mathit{P}}_{(0_i,\mathit{\theta })}\right)+KL\left({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })}\right)\hfill \\ & =& J\left({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })}\right),\hfill \end{array}$$

where ${\mathit{P}}_{(1_i,\mathit{\theta })}=(p_{(1_i,\mathit{\theta })}^1,\dots ,p_{(1_i,\mathit{\theta })}^n),$ with

$$p_{(1_i,\mathit{\theta })}^j=\frac{1}{n-1}{p}_{ij}+\frac{1}{n-1}\sum _{\mathsf{ł}=1,\mathsf{ł}\ne i}^{n-1}{\theta }_{\mathsf{ł}}+\left(1-\frac{1}{n-1}\left(1+\sum _{\mathsf{ł}=1,\mathsf{ł}\ne i}^{n-1}{\theta }_{\mathsf{ł}}\right)\right)p_{nj},$$

(8)

and ${\mathit{P}}_{(0_i,\mathit{\theta })}=(p_{(0_i,\mathit{\theta })}^1,\dots ,p_{(0_i,\mathit{\theta })}^n),$ with

$$p_{(0_i,\mathit{\theta })}^j=\frac{1}{n-1}\sum _{\mathsf{ł}=1,\mathsf{ł}\ne i}^{n-1}{\theta }_{\mathsf{ł}}{p}_{\mathsf{ł}j}+\left(1-\frac{1}{n-1}\sum _{\mathsf{ł}=1,\mathsf{ł}\ne i}^{n-1}{\theta }_{\mathsf{ł}}\right)p_{nj},$$

(9)

and J corresponds to Jeffreys’ divergence.

Proof. 

By definition and from (7), for $i=1,\dots ,n-1$, we have

$$\begin{array}{ccc}\tilde{\mathcal{I}}\left({\theta }_i\right)\hfill & =\hfill & E\left[\mathcal{I}\left({\mathrm{\Theta }}_i\right)\right]=\frac{1}{{\left(n-1\right)}^2}{\int }_0^1\left\{\sum _{j=1}^k\frac{{\left(p_{ij}-p_{nj}\right)}^2}{p_{\mathit{\theta }}^j}\right\}d{\theta }_i\hfill \\ \hfill & =\hfill & \frac{1}{n-1}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)\left\{{\int }_0^1\frac{1}{n-1}\frac{p_{ij}-p_{nj}}{p_{\mathit{\theta }}^j}d{\theta }_i\right\}\hfill \\ \hfill & =\hfill & \frac{1}{n-1}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)\left\{log\left(p_{\mathit{\theta }}^j\right){|}_0^1\right\}.\hfill \end{array}$$

(10)

On the other hand, we have

$$\begin{array}{c}\hfill p_{(1_i,\mathit{\theta })}-p_{(0_i,\mathit{\theta })}=\frac{1}{n-1}\left(p_{ij}-p_{nj}\right).\end{array}$$

(11)

Hence, upon substituting (11) into (10), we obtain

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& \frac{1}{n-1}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)log\left\{\frac{p_{(1_i,\mathit{\theta })}^j}{p_{(0_i,\mathit{\theta })}^j}\right\}\hfill \\ & =& \sum _{j=1}^k\left(p_{(1_i,\mathit{\theta })}-p_{(0_i,\mathit{\theta })}\right)log\left\{\frac{p_{(1_i,\mathit{\theta })}^j}{p_{(0_i,\mathit{\theta })}^j}\right\}\hfill \\ & =& KL\left({\mathit{P}}_{(1_i,\mathit{\theta })},{\mathit{P}}_{(0_i,\mathit{\theta })}\right)+KL\left({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })}\right)\hfill \\ & =& J\left({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })}\right),\hfill \end{array}$$

as required.  □

Theorem 6.

For the mixture model with PMF in (3), we have the following:

(i) 

The Bayes–Fisher information for ${\theta }_i,i=1,\dots ,n-1$, under $Beta(2,1)$ prior with PMF $\pi \left({\theta }_i\right)=2{\theta }_i,{\theta }_i\in [0,1]$, is

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& 2KL({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })}),i=1,\dots ,n-1;\hfill \end{array}$$

(ii) 

The Bayes-Fisher information for parameter ${\theta }_i,i=1,\dots ,n-1$, under $Beta(1,2)$ prior with PMF $\pi \left({\theta }_i\right)=2(1-{\theta }_i),{\theta }_i\in [0,1]$, is

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& 2KL({\mathit{P}}_{(1_i,\mathit{\theta })},{\mathit{P}}_{(0_i,\mathit{\theta })}),i=1,\dots ,n-1.\hfill \end{array}$$

Proof. 

By definition, and from (7), for $i=1,\dots ,n-1$, we have

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& E\left[\mathcal{I}\left({\mathrm{\Theta }}_i\right)\right]=\frac{1}{{\left(n-1\right)}^2}{\int }_0^1\left\{\sum _{j=1}^k\frac{{\left(p_{ij}-p_{nj}\right)}^2}{p_{\mathit{\theta }}^j}\right\}\pi \left({\theta }_i\right)d{\theta }_i\hfill \\ & =& \frac{2}{n-1}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)\left\{{\int }_0^1\frac{{\theta }_i}{n-1}\frac{p_{ij}-p_{nj}}{p_{\mathit{\theta }}^j}d{\theta }_i\right\}\hfill \\ & =& \frac{2}{n-1}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)\left\{{\int }_0^1\frac{p_{\mathit{\theta }}^j-p_{(0_i,\mathit{\theta })}^j}{p_{\mathit{\theta }}\left(x\right)}d{\theta }_i\right\}\hfill \\ & =& \frac{2}{n-1}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)\left\{1-\frac{(n-1)p_{(0_i,\mathit{\theta })}^j}{p_{ij}-p_{nj}}log\left(p_{\mathit{\theta }}^j\right){|}_0^1\right\}\hfill \\ & =& -2\sum _{j=1}^k{p}_{(0_i,\mathit{\theta })}^{j}log\left\{\frac{p_{(1_i,\mathit{\theta })}^j}{p_{(0_i,\mathit{\theta })}^j}\right\}\hfill \\ & =& 2\sum _{j=1}^k{p}_{(0_i,\mathit{\theta })}^{j}log\left\{\frac{p_{(0_i,\mathit{\theta })}^j}{p_{(1_i,\mathit{\theta })}^j}\right\}\hfill \\ & =& 2KL({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })}),\hfill \end{array}$$

as required for Part (i). Part (ii) can be proved in an analogous manner.

Let us now consider the following general triangular prior for the parameter ${\theta }_i,i=1,\dots ,n-1$:

$${\pi }_{\alpha }\left({\theta }_i\right)=\left\{\begin{array}{cc}\frac{2{\theta }_i}{\alpha },\hfill & 0<{\theta }_i\le \alpha ,\hfill \\ \frac{2(1-{\theta }_i)}{1-\alpha },\hfill & \alpha \le {\theta }_i<1,\hfill \end{array}\right.$$

(12)

for some $\alpha \in (0,1).$  □

Theorem 7.

The Bayes–Fisher information for parameter ${\theta }_i,i=1,\dots ,n-1,$ with the general triangular prior with density ${\pi }_{\alpha }\left({\theta }_i\right)$ in (12), is given by

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& \frac{2}{\alpha (1-\alpha )}\left[\alpha KL({\mathit{P}}_{(1_i,\mathit{\theta })},{\mathit{P}}_{\alpha })+(1-\alpha )KL({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{\alpha })\right]\hfill \\ & =& \frac{2}{\alpha (1-\alpha )}JS\left({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })};\alpha \right),\hfill \end{array}$$

where ${\mathit{P}}_{\alpha }=(p_{\alpha }^1,\dots ,p_{\alpha }^k)$ is a finite mixture PMF, with

$$\begin{array}{ccc}\hfill p_{\alpha }^j& =& \frac{\alpha }{n-1}{p}_{ij}+\frac{1}{n-1}\sum _{l=1,l\ne i}^{n-1}{\theta }_l{p}_{lj}+\left(1-\frac{1}{n-1}\left(\alpha +\sum _{l=1,l\ne i}^{n-1}{\theta }_l\right)\right)p_{nj}\hfill \end{array}$$

and ${\mathit{P}}_{(1_i,\mathit{\theta })}$ and ${\mathit{P}}_{(0_i,\mathit{\theta })}$ are as defined in (8) and (9), respectively.

Proof. 

From the assumptions made, for $i=1,\dots ,n-1$, we have

$$\begin{array}{ccc}\hfill \tilde{\mathcal{I}}\left({\theta }_i\right)& =& E\left[\mathcal{I}\left({\mathrm{\Theta }}_i\right)\right]={\int }_0^{\alpha }\mathcal{I}\left({\theta }_i\right){\pi }_{\alpha }d{\theta }_i+{\int }_{\alpha }^1\mathcal{I}\left({\theta }_i\right){\pi }_{\alpha }d{\theta }_i\hfill \\ & =& \frac{2}{(n-1)\alpha }\sum _{j=1}^k(p_{ij}-p_{nj})\left[{\int }_0^{\alpha }\frac{{\theta }_i}{n-1}\frac{p_{ij}-p_{nj}}{p_{\mathit{\theta }}^j}d{\theta }_i\right]\hfill \\ & & +\frac{2}{(n-1)(1-\alpha )}\sum _{j=1}^k(p_{ij}-p_{nj})\left[{\int }_{\alpha }^1\frac{1-{\theta }_i}{n-1}\frac{(p_{ij}-p_{nj})}{p_{\mathit{\theta }}^j}d{\theta }_i\right]\hfill \\ & =& \frac{2}{(n-1)\alpha }\sum _{j=1}^k(p_{ij}-p_{nj})\left[{\int }_0^{\alpha }\left(1-\frac{p_{(0_i,\mathit{\theta })}^j}{p_{\mathit{\theta }}^j}\right)d{\theta }_i\right]\hfill \\ & & -\frac{2}{(n-1)(1-\alpha )}\sum _{j=1}^k\left(p_{ij}-p_{nj}\right)\left[{\int }_0^{\alpha }\left(1-\frac{p_{(1_i,\mathit{\theta })}^j}{p_{\mathit{\theta }}^j}\right)d{\theta }_i\right]\hfill \\ & =& \frac{2}{\alpha }\sum _{j=1}^k{p}_{(0_i,\mathit{\theta })}^{j}log\left\{\frac{p_{\alpha }^j}{p_{(0_i,\mathit{\theta })}^j}\right\}+\frac{2}{1-\alpha }\sum _{j=1}^k{p}_{(1_i,\mathit{\theta })}^{j}log\left\{\frac{p_{(1_i,\mathit{\theta })}^j}{p_{\alpha }^j}\right\}\hfill \\ & =& \frac{2}{\alpha (1-\alpha )}\left[\alpha KL({\mathit{P}}_{(1_i,\mathit{\theta })},{\mathit{P}}_{\alpha })+(1-\alpha )KL({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{\alpha })\right]\hfill \\ & =& \frac{2}{\alpha (1-\alpha )}JS\left({\mathit{P}}_{(0_i,\mathit{\theta })},{\mathit{P}}_{(1_i,\mathit{\theta })};\alpha \right),\hfill \end{array}$$

as required.  □

5. Concluding Remarks

In this paper, we have introduced the discrete version of Jensen–Fisher information measure, and have shown that this information measure can be expressed as a mixture of discrete Fisher information distance measures. Further, we have considered a finite mixture probability mass function and have derived Fisher information and Bayes–Fisher information for the mixing parameter vector. We have shown that the Fisher information for the mixing parameter is connected to chi-square divergence. We have also studied the Bayes–Fisher information for the mixing parameter of a finite mixture model under some prior distributions. These results have provided connections between the Bayes–Fisher information and some known informational measures such as Shannon entropy, Kullback–Leibler, Jeffreys and Jensen–Shannon divergence measures.