Tsallis and Other Generalised Entropy Forms Subject to Dirichlet Mixture Priors

Abstract

Entropy indicates a measure of information contained in a complex system, and its estimation continues to receive ongoing focus in the case of multivariate data, particularly that on the unit simplex. Oftentimes the Dirichlet distribution is employed as choice of prior in a Bayesian framework conjugate to the popular multinomial likelihood with K distinct classes, where consideration of Shannon- and Tsallis entropy is of interest for insight detection within the data on the simplex. However, this prior choice only accounts for negatively correlated data, therefore this paper incorporates previously unconsidered mixtures of Dirichlet distributions as potential priors for the multinomial likelihood which addresses the drawback of negative correlation. The power sum functional, as the product moment of the mixture of Dirichlet distributions, is of direct interest in the multivariate case to conveniently access the Tsallis- and other generalized entropies that is incorporated within an estimation perspective of the posterior distribution using real economic data. A prior selection method is implemented to suggest a suitable prior for the consideration of the practitioner; empowering the user in future for consideration of suitable priors incorporating entropy within the estimation environment as well as having the option of certain mixture of Dirichlet distributions that may require positive correlation.

1. Introduction

Entropy is a measure of uncertainty, diversity and randomness often adopted for characterizing complex dynamical systems [1], and has seen several expansions over the last few years. The most popular form of entropy is that of Shannon, however, various generalized cases of this entropy exists which relies on the power sum [2,3]. This probability functional has the particular appeal of circumventing occasionally arduous computation of the logarithm of $p_i$ in the expression for the Shannon entropy, and has already been established as a valuable addition and measure in an array of operational problems within information theory [4].

The Dirichlet prior is a popular choice in the Bayesian framework for estimation of entropy when considering a multinomial likelihood [4]. The Dirichlet distribution is a conjugate prior for the multinomial distribution when a Bayes perspective is of interest. The Dirichlet distribution is well known when working with data on the unitary simplex $(0,1)$ and is a multivariate generalization of the beta distribution. Several generalizations of the Dirichlet distribution has been developed and investigated, such as a class of Dirichlet generators as explored in [4], the noncentral Dirichlet construction in [5], as well as the Dirichlet-gamma of [6] in order to strengthen the capability to model different dependence patterns.

The Bayesian framework is a popular choice for complex statistical investigations, specifically with the increase in computational power readily available in personal computers. This framework also allows for more flexibility and intuitive interpretations compared to the frequentist methods [7]. The choice of prior distribution is a crucial aspect of Bayesian analysis and may impact the overall inference. This paper considers three (of which two are previously unconsidered) prior distributions and investigates methods which can be used to select the most appropriate prior. The Dirichlet distribution will be considered as the base prior, followed by the flexible Dirichlet distribution as proposed by [8] which is expressed as a finite mixture of Dirichlet components. The double flexible Dirichlet distribution as proposed by [9] is also considered and is a further generalization of the Dirichlet structure which takes advantage of the finite mixture structure of the flexible Dirichlet distribution. Both these mixtures of Dirichlet distributions are capable of modelling multimodality in data, and so, expert input and opinion regarding potential multimodality in prior behaviour can be captured using these models.

The first of two main contributions of this paper implements and illustrates that using elegant constructs of the complete product moments of the posteriors gives one the comparative advantage of obtaining explicit estimators for three generalised entropy forms (via the power sum functional) subject to these Dirichlet priors. The second shows how these generalised entropy measures can be used as tools to estimate the parameters for fitting these considered distributions (as part of the Bayesian calibration methodology) to data using estimation steps as described in [10] and hence ensuring insightful data fits. These entropy measures as well as prior impact measures can then be used to determine which of the priors will be the best choice for the estimation of the parameters [4].

Interesting research which focuses on Dirichlet forms and entropy measures include (1) ref. [11] who focused on multinomial scaled Dirichlet mixture models with specific application in clustering. The examples evaluated different models, their accuracy, precision, recall and mutual information while applying this on a image classification problem, (2) ref. [12] used multivariate Beta mixture models to proposed a novel variational inference via an entropy-based splitting method. The performance was then evaluated in real-world applications like breast tissue texture classification, cytological breast data analysis and age estimation, and (3) ref. [13] focused on comparing 18 different entropy measures with specific interest in short sequence bits and bytes data. They evaluated the behaviour (means, bias, mean squared error) of these entropy estimators as the sample sizes increased, the correlations between the different entropy estimates and how these estimates were grouped when using logic like hierarchical clustering.

The paper is outlined as follows. In Section 2, the preliminary definitions and properties that are used in the paper are outlined as well as alternative Dirichlet priors as candidates for the Bayesian analysis of the considered generalised entropies. Section 3 derives the resultant posterior models together with their respective complete product moments and estimates for the generalised entropies. In Section 4 an explorative study is performed to obtain optimal values for the parameters of interest and the Wasserstein Impact Measure (WIM) [7] is utilized to determine the impact on entropy via the prior. Section 5 contains concluding remarks.

2. Some Definitions and Properties

In this section, basic notation and definitions relevant for this paper are reviewed. Multivariate count data constrained to add up to a certain constant are commonly modelled using the multinomial distribution, and forms the basis of a countably discrete likelihood in conjunction with our proposed Dirichlet type priors. The fundamental Bayesian relationship between the likelihood function and the prior distribution to form the posterior distribution is given by

$$f\left(\mathbf{p}\right|\mathbf{x})=\frac{f\left(\mathbf{x}\right|\mathbf{p}\left)h\right(\mathbf{p})}{\int f\left(\mathbf{x}\right|\mathbf{p}\left)h\right(\mathbf{p})d\mathbf{p}}.$$

(1)

A multivariate discrete random variable $\mathit{X}=(X_1,\cdots ,X_K)$ follows the multinomial distribution (i.e. with K distinct classes of interest) with parameters $\mathbf{p}=(p_1,p_2,\cdots ,p_K)$ and $n>0$ if its probability mass function (pmf) is given by

$$f\left(\mathbf{x}\right|\mathbf{p})=\frac{n!}{{\prod }_{i=1}^K{x}_i!(n-{\sum }_{i=1}^K{x}_i)!}\prod _{i=1}^K{p}_i^{x_i}{(1-\sum _{i=1}^K{p}_i)}^{n-{\sum }_{i=1}^K{x}_i}.$$

(2)

2.1. Entropy Forms of Interest

The most popular form of entropy is that of Shannon:

$$H\left(P\right)=\sum _{i=1}^{K+1}-p_{i}ln{p}_i.$$

Various generalised versions of this entropy exist, which relies on the power sum:

$$F_{\alpha }\left(P\right)=\sum _{i=1}^{K+1}{p}_i^{\alpha }$$

(3)

where $\alpha >0$ (see [2]). Under the assumption of squared error loss within Bayes estimation, the estimators of both these quantities is given by their expected values:

$$E\left(H\left(P\right)\right)=E\left(\sum _{i=1}^{K+1}-p_{i}ln{p}_i\right)$$

and

$${\widehat{F}}_{\alpha }\left(P\right)=E\left(F_{\alpha }\left(P\right)\right)=E\left(\sum _{i=1}^{K+1}{p}_i^{\alpha }\right)=\sum _{i=1}^{K+1}E\left(p_i^{\alpha }\right).$$

(4)

Since the power sum functional is oftentimes easier to estimate than the Shannon entropy, the power sum is a main consideration in this paper. The entropies of interest considered in this paper are summarised in

Table 1. Entropy measures considered in this paper.

Type	Expression	Estimate Considered
Tsallis	${\displaystyle T=\frac{{\sum }_{i=1}^{K+1}{p}_i^{\alpha }-1}{1-\alpha };\alpha \ge 0,\alpha \ne 1.}$	${\displaystyle E\left(T\right)=\frac{{\widehat{F}}_{\alpha }\left(p\right)-1}{1-\alpha }}$
Generalized Mathai ${}^1$	${\displaystyle GM=\frac{{\sum }_{i=1}^{K+1}{p}_i^{\varphi -\alpha }-1}{\alpha -1};\alpha \le \varphi ,\alpha \ne 1.}$	${\displaystyle E\left(GM\right)=\frac{{\widehat{F}}_{\varphi -\alpha }\left(p\right)-1}{\alpha -1}}$
Abe	${\displaystyle A=-\sum _{i=1}^{K+1}\frac{p_i^{\alpha }-p_i^{{\alpha }^{-1}}}{\alpha -{\alpha }^{-1}};\alpha \in [0,1]}$	${\displaystyle E\left(A\right)=\left(-\frac{{\widehat{F}}_{\alpha }\left(p\right)-{\widehat{F}}_{{\alpha }^{-1}}}{\alpha -{\alpha }^{-1}}\right)}$

¹ For the remainder of the paper we will consider ϕ = 2.

, which are explicitly expressed with the power sum functional. The well known Tsallis entropy, the generalized instance of the Mathai (the generalized Mathai) as well as the symmetrical modification of the Tsallis entropy (the Abe formulation [1]) is considered, and summarised in Table 1.

2.2. Considered Priors

In this section, two alternative Dirichlet formulations as mixtures of the well known usual Dirichlet model will be reviewed and used as priors, together with the usual Dirichlet model. These alternatives are suggested since the Dirichlet type 1 distribution, despite its ease of parameter interpretation [8], is known to be poorly parameterized and cannot model many dependence patterns [9] such as positive correlation. A particular focus of the considered mixtures is to illustrate instances where positive correlation on the constrained unit simplex is achievable for certain parameter structures.

2.2.1. The Dirichlet Distribution

Here, we briefly define our departure model of interest, the well known Dirichlet distribution.

Definition 1.

Suppose $\mathbf{p}$ is distributed as a Dirichlet distribution (of type 1, see [14]) of order $K\ge 2$ and parameters $\Pi =({\pi }_1,{\pi }_2,\cdots ,{\pi }_{K+1})$ for ${\pi }_i>0,i=1,\cdots ,K+1$, with respect to the Lebesgue measure on the Euclidean space ${\mathbb{R}}^K$, then its pdf is given by

$$\begin{array}{ccc}\hfill f(p_1,\cdots ,p_K;\Pi )& =& \frac{\Gamma \left({\pi }_{+}\right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\left(\prod _{i=1}^{K+1}{p}_i^{{\pi }_i-1}\right)\hfill \end{array}$$

(5)

on the K dimensional simplex, defined by

$$\begin{array}{c}\hfill p_1,p_2,\cdots ,p_K>0\\ \hfill p_1+p_2+\cdots +p_K<1\\ \hfill p_{K+1}=1-p_1-\cdots -p_K\end{array}$$

and where $\Gamma (·)$ denotes the usual gamma function with ${\pi }_{+}={\sum }_{i=1}^{K+1}{\pi }_i$ (the space and constraints of this K dimensional simplex is denoted by $\mathcal{A}$).

Figure 1 shows how the changes in ${\pi }_1$ and ${\pi }_2$ affects the correlation for $k=2$. The heatmap here indicates that positive correlation is not feasible for this model, as is known from the literature (see [4]).

2.2.2. The Flexible Dirichlet Distribution

The following prior is represented by the flexible Dirichlet distribution as proposed by [8] and is expressed as a finite mixture of particular Dirichlet components. This distribution models multimodality and has shown to be capable of discriminating among many of the independence concepts relevant for compositional data.

Definition 2.

Suppose $\mathbf{p}$ is distributed as flexible Dirichlet distribution. Then its pdf is given by

$$\begin{array}{ccc}\hfill g(p_1,\cdots ,p_K;\Pi ,\tau ,\mathsf{\beta })& =& \sum _{r=1}^{K+1}{\beta }_{i}f(\mathbf{p},\Pi +\tau {\mathbf{e}}_i)\hfill \\ & =& \frac{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+\tau )}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\left(\prod _{i=1}^{K+1}{p}_i^{{\pi }_i-1}\right)\left[\sum _{r=1}^{K+1}{\beta }_r{p}_r^{\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\right]\hfill \end{array}$$

(6)

where ${\mathbf{e}}_i$ is the vector with elements equal to zero except for the i-th that is equal to 1. The flexible Dirichlet also includes the Dirichlet as a special case if $\tau =1$ and ${\beta }_i=\frac{{\pi }_i}{{\pi }_{+}}$, $i=1,2,\dots ,K$.

Figure 2 aims to show what role each of the parameters play in creating this flexible distribution. The first row of contours can be used as a base to compare the suggested changes against in order to illustrate the effects that each parameter has on the distribution form. The second row shows how increasing $\tau$, from 3 to 7, splits the pdfs into different modes. The last row kept the larger $\tau =7$ but rearranged $\mathsf{\beta }$ (by exchanging ${\beta }_1$ and ${\beta }_2$) and illustrates how this rearrangement flips the concentration of these modes. For this examples ${\beta }_1=0.5,{\beta }_2=0.2$ and ${\beta }_3=0.3$ was used for the first and second row with the third row represented by ${\beta }_1=0.2,{\beta }_2=0.5$ and ${\beta }_3=0.3$. Figure 3 shows how the correlations changes as the values of $\tau$ and $\mathsf{\beta }$ change. Specifically, no positive correlation is observed from this mixture structure of Dirichlet distributions.

2.2.3. The Double Flexible Dirichlet Distribution

The following prior is represented by the double flexible Dirichlet distribution as proposed by [9] and is a generalization of the Dirichlet structure which takes advantage of the finite mixture structure of the flexible Dirichlet distribution and also allows positive covariances. As such, potential positive correlation observed in a prior may be well modelled by this particular prior choice.

Definition 3.

Suppose $\mathbf{p}$ is distributed as a double flexible Dirichlet distribution. Then its pdf is given by

$$\begin{array}{ccc}& & h(p_1,\cdots ,p_K;\Pi ,\tau ,\mathsf{\beta })\hfill \\ & =& \sum _{r=1}^{K+1}\sum _{s=1}^{K+1}{\beta }_{i}f(\mathbf{p},\Pi +\tau ({\mathbf{e}}_r+{\mathbf{e}}_s))\hfill \\ & =& \frac{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+2\tau )}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\left(\prod _{i=1}^{K+1}{p}_i^{{\pi }_i-1}\right)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}& & \left[{\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}{p}_r^{\tau }{p}_s^{\tau }\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}+\sum _{r=1}^{K+1}{\beta }_{rr}{p}_r^{2\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\right]\hfill \end{array}$$

(8)

where ${\mathbf{e}}_i$ is the vector with elements equal to zero except for the i-th that is equal to 1.

Figure 4 shows the role that each of the parameters play in this flexible form while $\mathsf{\beta }$ are similar. The first row of contours can be used as a reference to compare the changes against, while the second row shows how increasing $\tau$ from 3 to 7 splits the pdfs into multiple modes similar to what was seen in the flexible Dirichlet pdf (6). For this example ${\beta }_{11}=0.2;{\beta }_{22}=0.1;{\beta }_{33}=0.1;{\beta }_{12}=0.1;{\beta }_{13}=0.1;{\beta }_{21}=0.1;{\beta }_{23}=0.1;{\beta }_{31}=0.1;$${\beta }_{32}=0.1$. For Figure 5 $\mathsf{\beta }$ were chosen to be less consistent (${\beta }_{11}=0.4;{\beta }_{12}=0.2;$${\beta }_{13}=0.2;{\beta }_{21}=0.09;{\beta }_{22}=0.09;{\beta }_{23}=0.09;{\beta }_{31}=0.01;{\beta }_{32}=0.01;{\beta }_{33}=0.01$) for the first row. For the second row the ${\beta }_{11}=0.09;{\beta }_{12}=0.09;{\beta }_{13}=0.09;{\beta }_{21}=0.4;{\beta }_{22}=0.2;$${\beta }_{23}=0.1;{\beta }_{31}=0.01;{\beta }_{32}=0.01;{\beta }_{33}=0.01$ and the last being ${\beta }_{11}=0.09;{\beta }_{12}=0.09;$${\beta }_{13}=0.09;{\beta }_{21}=0.01;{\beta }_{22}=0.01;{\beta }_{23}=0.01;{\beta }_{31}=0.4;{\beta }_{32}=0.2;{\beta }_{33}=0.1$. By changing the values of $\mathsf{\beta }$ we can see how the concentration of the modes changes.

Figure 6 shows how the correlations changes as the values of $\mathsf{\pi };\tau$ and $\mathsf{\beta }$ change. The first correlation plot (a) speaks to the parameters in the first row of Figure 5 and investigate the effect that the change in $\tau$ has on the correlation. In (b) the parameters in the last row of Figure 5 were used and illustrates and it can be seen that the positive correlation is dependent on $\mathsf{\beta }$ as also discussed in [9].

3. Bayesian Estimation of Entropy

In this section the usual multinomial-Dirichlet setup is enriched with the additional consideration of the flexible Dirichlet- and double flexible Dirichlet mixtures ((6) and (8)) as priors for the multinomial likelihood. In this way, it allows the practitioner to obtain a posterior distribution from where closed form expressions for the entropies under consideration can be obtained, by particularly focussing on the product moment of the posterior model, in order to access the power sum functional under the assumption of squared error loss.

3.1. For the Dirichlet Prior

Theorem 1.

The posterior distribution for the multinomial likelihood as in (2) and the Dirichlet prior distribution in (5) follows a Dirichlet distribution with parameters $\left({\pi }_1+x_1,\dots ,{\pi }_{K+1}+x_{K+1}\right)$ with the form:

$$\begin{array}{ccc}\hfill f\left(\mathbf{p}\right|\mathbf{x})& =& \frac{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i)}{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i)}\left(\prod _{i=1}^{K+1}{p}_i^{{\pi }_i+x_i-1}\right)\hfill \end{array}$$

(9)

where $0<p_i<1$ and ${\pi }_i>0$ for $i=1,\dots ,K+1$.

Proof. 

From (2) and (5) the proof follows directly [4].    □

Since the complete product moments of the posterior distribution is of interest in order to determine the power sum (4) we are interested in $E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)$.

Definition 4.

The definition of the complete product moment of a $\left(K+1\right)$ variable $\mathbf{Y}$ with pdf $f\left(\mathbf{y}\right)$ is given by [4]

$$\begin{array}{ccc}\hfill E\left(\prod _{i=1}^{K+1}{Y}_i^{x_i}\right)& =& {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\prod _{i=1}^{K+1}{y}_i^{x_i}f\left(\mathbf{y}\right)d{y}_1\dots d{y}_{K+1}.\hfill \end{array}$$

(10)

Theorem 2.

Suppose that $\mathbf{p}|\mathbf{x}$ follows a Dirichlet posterior distribution with pdf given in (9). Then the complete product moment is given by

$$\begin{array}{c}\hfill E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)=\frac{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i)}{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i)}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+k_i)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+k_i)}.\end{array}$$

(11)

Proof. 

The result follows directly from (9) and (10) using the law of total probability of the Dirichlet distribution with parameters $\left({\pi }_1+x_1+k_1,\dots ,{\pi }_{K+1}+x_{K+1}+k_{K+1}\right)$ [4].    □

Using the complete product moments derived in (11), the Bayesian estimator for the power sum (4) can be derived by setting $k_i=\alpha$ with $i=1,\dots ,K+1$ and $k_{\ne i}=0$.

Theorem 3.

The Bayesian estimator for the power sum functional under the Dirichlet posterior (9) is given by:

$$\begin{array}{ccc}\hfill {\widehat{F}}_{\alpha }\left(p\right)& =& \sum _{j^{*}=1}^{K+1}E\left(p_{j^{*}}^{\alpha }\right)\hfill \\ & =& \frac{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i)}{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i)}\left[\frac{{\sum }_{j^{*}=1}^{K+1}\Gamma ({\pi }_{j^{*}}+x_{j^{*}}+\alpha ){\prod }_{i\ne j^{*}}\Gamma ({\pi }_i+x_i)}{\Gamma \left(\alpha +{\sum }_{i=1}^{K+1}{\pi }_i+x_i\right)}\right]\hfill \end{array}$$

(12)

Proof. 

From (11) and (4) the proof follows directly [4].    □

3.2. For the Flexible Dirichlet Prior

Theorem 4.

The posterior distribution for the multinomial likelihood as in (2) and the flexible Dirichlet prior distribution in (6) is given by

$$\begin{array}{ccc}\hfill g\left(\mathbf{p}\right|\mathbf{x})& =& \frac{1}{C}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\left[\sum _{r=1}^{K+1}{\beta }_r{p}_r^{\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\right]\hfill \end{array}$$

(13)

where $0<p_i<1$, ${\pi }_i>0$; $0\le {\beta }_i<1$, for $i=1,\dots ,K+1$, $\tau >0$, ${\sum }_{i=1}^{K+1}{\beta }_i=1$ and where

$$\begin{array}{c}\hfill C=\sum _{r=1}^{K+1}{\beta }_r\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\left[\frac{\Gamma ({\pi }_r+x_r+\tau ){\displaystyle {\displaystyle \underset{j\ne r}{{\prod }_{j=1}^{K+1}}}}\Gamma ({\pi }_j+x_j)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau )}\right].\end{array}$$

Proof. 

By applying Bayes’s theorem (1) the numerator will have the following form:

$$\begin{array}{ccc}\hfill f\left(\mathbf{x}\right|\mathbf{p}\left)g\right(\mathbf{p})& =& \frac{n!}{{\prod }_{i=1}^K{x}_i!\left(n-{\sum }_{i=1}^K{x}_i\right)!}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+\tau \right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\hfill \\ & \times & \left[\sum _{r=1}^{K+1}{\beta }_r{p}_r^{\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\right].\hfill \end{array}$$

(14)

The denominator of the posterior distribution will be given by

$$\begin{array}{ccc}& & {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}f\left(\mathbf{x}\right|\mathbf{p})g\left(\mathbf{p}\right)d\mathbf{p}\hfill \\ & =& \frac{n!}{{\prod }_{i=1}^K{x}_i!\left(n-{\sum }_{i=1}^K{x}_i\right)!}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+\tau \right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\hfill \\ & \times & {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\left[\sum _{r=1}^{K+1}{\beta }_r{p}_r^{\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\right]d{p}_1\dots d{p}_{K+1}.\hfill \end{array}$$

(15)

The second line of the above equation can be written as

$$\begin{array}{ccc}& & {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\left[\sum _{r=1}^{K+1}{\beta }_r{p}_r^{\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\right]d{p}_1\dots d{p}_{K+1}\hfill \\ & =& {\beta }_1\frac{\Gamma \left({\pi }_1\right)}{\Gamma ({\pi }_1+\tau )}\frac{\Gamma ({\pi }_1+x_1+\tau ){\displaystyle \underset{i\ne 1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}{\Gamma ({\pi }_1+x_1+\tau ){\displaystyle \underset{i\ne 1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1+\tau -1}{p}_2^{{\pi }_2+x_2-1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +& {\beta }_2\frac{\Gamma \left({\pi }_2\right)}{\Gamma ({\pi }_2+\tau )}\frac{\Gamma ({\pi }_2+x_2+\tau ){\displaystyle \underset{i\ne 2}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}{\Gamma ({\pi }_2+x_2+\tau ){\displaystyle \underset{i\ne 2}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1-1}{p}_2^{{\pi }_2+x_2+\tau -1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & \dots & \hfill \\ & +& {\beta }_{K+1}\frac{\Gamma \left({\pi }_{K+1}\right)}{\Gamma ({\pi }_{K+1}+\tau )}\frac{\Gamma ({\pi }_{K+1}+x_{K+1}+\tau ){\displaystyle \underset{i\ne K+1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}\hfill \\ & \times & [{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}{\Gamma ({\pi }_{K+1}+x_{K+1}+\tau ){\displaystyle \underset{i\ne K+1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & \times & p_1^{{\pi }_1+x_1-1}{p}_2^{{\pi }_2+x_2-1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+\tau -1}d{p}_1\dots d{p}_{K+1}].\hfill \end{array}$$

Each integral is equal to 1, since it corresponds to the total probability of a Dirichlet distribution, hence the denominator will simplify to the following form:

$$\begin{array}{ccc}& & {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}f\left(\mathbf{x}\right|\mathbf{p})g\left(\mathbf{p}\right)d\mathbf{p}\hfill \\ & =& \frac{n!}{{\prod }_{i=1}^K{x}_i!\left(n-{\sum }_{i=1}^K{x}_i\right)!}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+\tau \right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\hfill \\ & \times & \left[\sum _{r=1}^{K+1}{\beta }_r\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\frac{\Gamma ({\pi }_r+x_r+\tau ){\displaystyle \underset{j\ne r}{{\prod }_{j=1}^{K+1}}}\Gamma ({\pi }_j+x_j)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau )}\right].\hfill \end{array}$$

(16)

Combining (14) and (16), the result follows.    □

In the case of the flexible Dirichlet (13) the following theorem derives the complete product moment as defined in (10).

Theorem 5.

Suppose that $\mathbf{p}|\mathbf{x}$ follows a flexible Dirichlet posterior distribution with pdf given in (13). Then the complete product moment is given by

$$\begin{array}{c}\hfill E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)=\frac{{\sum }_{r=1}^{K+1}{\beta }_r\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+k_i+\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+k_i+\tau )}\right]}{{\sum }_{r=1}^{K+1}{\beta }_r\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau )}\right]}.\end{array}$$

(17)

Proof. 

From (13) and (10) it follow that

$$\begin{array}{ccc}& & E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)\hfill \\ & =& \frac{1}{C}{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i+k_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}-1}\left[\sum _{i=1}^{K+1}{\beta }_i{p}_i^{\tau }\frac{\Gamma \left({\pi }_i\right)}{\Gamma ({\pi }_i+\tau )}\right]d{p}_1\dots d{p}_{K+1}\hfill \\ & =& \frac{1}{C}{\beta }_1\frac{\Gamma \left({\pi }_1\right)}{\Gamma ({\pi }_1+\tau )}\hfill \\ & \times & \left[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1+\tau -1}{p}_2^{{\pi }_2+x_2+k_2-1}\dots {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}-1}d{p}_1\dots d{p}_{K+1}\right]\hfill \\ & +& \frac{1}{C}{\beta }_2\frac{\Gamma \left({\pi }_2\right)}{\Gamma ({\pi }_2+\tau )}\hfill \\ & \times & \left[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1-1}{p}_2^{{\pi }_2+x_2+k_2+\tau -1}\dots {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}-1}d{p}_1\dots d{p}_{K+1}\right]\hfill \\ & \dots & \hfill \\ & +& \frac{1}{C}{\beta }_{K+1}\frac{\Gamma \left({\pi }_{K+1}\right)}{\Gamma ({\pi }_{K+1}+\tau )}\hfill \\ & \times & \left[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1-1}{p}_2^{{\pi }_2+x_2+k_2-1}\dots {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}+\tau -1}d{p}_1\dots d{p}_{K+1}\right].\hfill \end{array}$$

(18)

We identify that each integral in the above expression is of the form of the Dirichlet kernel. Using the definition of total probability, the result follows.

   □

Using the complete product moments derived in (17), the Bayesian estimator for the power sum (4) can be derived by setting $k_i=\alpha$ with $i=1,\dots ,K+1$ and $k_{\ne i}=0$.

Theorem 6.

The Bayesian estimator for the power sum functional under the flexible Dirichlet posterior distribution (13) is given by:

$$\begin{array}{ccc}\hfill {\widehat{F}}_{\alpha }\left(p\right)& =& \sum _{j^{*}=1}^{K+1}E\left(p_{j^{*}}^{\alpha }\right)\hfill \\ & =& \sum _{j^{*}=1}^{K+1}\left[\frac{{\sum }_{r=1}^{K+1}{\beta }_r\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\left[\frac{\Gamma ({\pi }_{j^{*}}+x_{j^{*}}+\alpha +\tau {\mathbf{e}}_r){\prod }_{j\ne j^{*}}^{K+1}\Gamma ({\pi }_j+x_j+\tau {\mathbf{e}}_r)}{\Gamma (\alpha +{\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau )}\right]}{{\sum }_{r=1}^{K+1}{\beta }_r\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+\tau )}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau )}\right]}\right].\hfill \end{array}$$

(19)

3.3. For the Double Flexible Dirichlet Prior

Theorem 7.

The posterior distribution for the multinomial likelihood as in (2) and the double flexible Dirichlet prior distribution in (8) is given by

$$\begin{array}{ccc}\hfill h\left(\mathbf{p}\right|\mathbf{x})& =& \frac{1}{C}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\hfill \\ & \times & \left[{\displaystyle \underset{r\ne h}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}{\left(p_r{p}_s\right)}^{\tau }\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}+\sum _{r=1}^{K+1}{\beta }_{rr}{p}_r^{2\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\right]\hfill \end{array}$$

(20)

where $0<p_i<1$, ${\pi }_i>0$; $0\le {\beta }_i<1$, for $i=1,\dots ,K+1$, $\tau >0$, ${\sum }_{r=1}^{K+1}{\beta }_i=1$ and

$$\begin{array}{ccc}\hfill C& =& [{\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+\tau ({\mathbf{e}}_r+{\mathbf{e}}_s))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}\hfill \\ & +& \sum _{r=1}^{K+1}{\beta }_{rr}\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+2\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}].\hfill \end{array}$$

Proof. 

By applying Bayes’s theorem (1) the numerator will have the following form:

$$\begin{array}{ccc}\hfill f\left(\mathbf{x}\right|\mathbf{p}\left)h\right(\mathbf{p})& =& \frac{n!}{{\prod }_{i=1}^K{x}_i!\left(n-{\sum }_{i=1}^K{x}_i\right)!}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+\tau \right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\hfill \\ & \times & \left[{\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}{\left(p_r{p}_s\right)}^{\tau }\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}+\sum _{r=1}^{K+1}{\beta }_{rr}{p}_r^{2\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\right].\hfill \end{array}$$

(21)

The denominator of the posterior will be given by

$$\begin{array}{ccc}& & {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}f\left(\mathbf{x}\right|\mathbf{p})h\left(\mathbf{p}\right)d\mathbf{p}\hfill \\ & =& \frac{n!}{{\prod }_{i=1}^K{x}_i!\left(n-{\sum }_{i=1}^K{x}_i\right)!}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+\tau \right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\hfill \\ & \times & {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\prod _{i=1}^K{p}_i^{{\pi }_i+x_i-1}{\left(1-\sum _{i=1}^K{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}\hfill \\ & \times & \left[{\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}{\left(p_r{p}_s\right)}^{\tau }\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}+\sum _{r=1}^{K+1}{\beta }_{rr}{p}_r^{2\tau }\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\right]d{p}_1\dots d{p}_{K+1}\hfill \\ & =& I_1.\hfill \end{array}$$

(22)

See that

$$\begin{array}{ccc}\hfill I_1& =& {\beta }_{12}\frac{\Gamma \left({\pi }_1\right)\Gamma \left({\pi }_2\right)}{\Gamma ({\pi }_1+\tau )\Gamma ({\pi }_2+\tau )}\frac{\Gamma ({\pi }_1+x_1+\tau )\Gamma ({\pi }_2+x_2+\tau ){\displaystyle \underset{i\ne 1,2}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}\hfill \\ & \times & [{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}{\Gamma ({\pi }_1+x_1+\tau )\Gamma ({\pi }_2+x_2+\tau ){\displaystyle \underset{i\ne 1,2}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1+\tau -1}{p}_2^{{\pi }_2+x_2+\tau -1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +& {\beta }_{13}\frac{\Gamma \left({\pi }_1\right)\Gamma \left({\pi }_3\right)}{\Gamma ({\pi }_1+\tau )\Gamma ({\pi }_3+\tau )}\frac{\Gamma ({\pi }_1+x_1+\tau )\Gamma ({\pi }_3+x_3+\tau ){\displaystyle \underset{i\ne 1,3}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}\hfill \\ & \times & [{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}{\Gamma ({\pi }_1+x_1+\tau )\Gamma ({\pi }_3+x_3+\tau ){\displaystyle \underset{i\ne 1,3}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1+\tau -1}\dots p_3^{{\pi }_3+x_3+\tau -1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & \dots & \hfill \\ & +& {\beta }_{K,K+1}\frac{\Gamma \left({\pi }_K\right)\Gamma \left({\pi }_{K+1}\right)}{\Gamma ({\pi }_K+\tau )\Gamma ({\pi }_{K+1}+\tau )}\frac{\Gamma ({\pi }_K+x_K+\tau )\Gamma ({\pi }_{K+1}+x_{K+1}+\tau ){\displaystyle \underset{i\ne K,K+1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}\hfill \\ & \times & [{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}{\Gamma ({\pi }_K+x_K+\tau )\Gamma ({\pi }_{K+1}+x_{K+1}+\tau ){\displaystyle \underset{i\ne K,K+1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1-1}\dots p_K^{{\pi }_K+x_K+\tau -1}{p}_{K+1}^{{\pi }_{K+1}+x_{K+1}+\tau -1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +& {\beta }_{11}\frac{\Gamma \left({\pi }_1\right)}{\Gamma ({\pi }_1+2\tau )}\frac{\Gamma ({\pi }_1+x_1+2\tau ){\displaystyle \underset{i\ne 1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}\hfill \\ & \times & [{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}{\Gamma ({\pi }_1+x_1+2\tau ){\displaystyle \underset{i\ne 1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1+2\tau -1}{p}_2^{{\pi }_2+x_2-1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +\dots & \hfill \\ & +& {\beta }_{K+1,K+1}\frac{\Gamma \left({\pi }_{K+1}\right)}{\Gamma ({\pi }_{K+1}+2\tau )}\frac{\Gamma ({\pi }_{K+1}+x_{K+1}+2\tau ){\displaystyle \underset{i\ne K+1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau \right)}\hfill \\ & \times & [{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\tau \right)}{\Gamma ({\pi }_{K+1}+x_{K+1}+2\tau ){\displaystyle \underset{i\ne 1}{{\prod }_{i=1}^{K+1}}}\Gamma ({\pi }_i+x_i)}\hfill \\ & & p_1^{{\pi }_1+x_1-1}{p}_2^{{\pi }_2+x_2-1}\dots p_k^{{\pi }_k+x_k-1}{\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+2\tau -1}d{p}_1\dots d_{K+1}]\hfill \end{array}$$

Each integral is equal to 1, since it corresponds to the total probability of a Dirichlet distribution, hence the denominator will simplify to the following form:

$$\begin{array}{ccc}\hfill {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}f\left(\mathbf{x}\right|\mathbf{p})h\left(\mathbf{p}\right)d\mathbf{p}& =& \frac{n!}{{\prod }_{i=1}^K{x}_i!\left(n-{\sum }_{i=1}^K{x}_i\right)!}\frac{\Gamma \left({\sum }_{i=1}^{K+1}{\pi }_i+2\tau \right)}{{\prod }_{i=1}^{K+1}\Gamma \left({\pi }_i\right)}\hfill \\ & \times & [{\displaystyle \underset{r\ne h}{\sum _{r=1}^{K+1}\sum _{h=1}^{K+1}}}{\beta }_{rh}\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_h\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_h+\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+\tau ({\mathbf{e}}_r+{\mathbf{e}}_h))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}\hfill \\ & +& \sum _{r=1}^{K+1}{\beta }_{rr}\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+2\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}].\hfill \end{array}$$

(23)

Combining (21) and (23) the results follow.

   □

Subsequently an expression for $E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)$ will be derived.

Theorem 8.

Suppose that $\mathbf{p}|\mathbf{x}$ follows a double flexible Dirichlet posterior distribution with pdf given in (20). Then the complete product moment is given by

$$\begin{array}{ccc}& & E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)\hfill \\ & =& \frac{A}{B}\hfill \end{array}$$

(24)

where

$$\begin{array}{ccc}\hfill A& =& [{\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+k_i+\tau ({\mathbf{e}}_r+{\mathbf{e}}_s))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+k_i+2\tau )}\right]\hfill \\ & +& \sum _{r=1}^{K+1}{\beta }_{rr}\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+k_i+2\tau \left({\mathbf{e}}_r\right))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+k_i+2\tau )}\right]]\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill B& =& [{\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rh}\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+\tau ({\mathbf{e}}_r+{\mathbf{e}}_s))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}\hfill \\ & +& \sum _{r=1}^{K+1}{\beta }_{rr}\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+2\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}]\hfill \end{array}$$

Proof. 

From (10) and (20) it follows

$$\begin{array}{ccc}& & E\left(p_1^{k_1}{p}_2^{k_2}\dots p_{K+1}^{k_{K+1}}\right)\hfill \\ & =& {\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}\prod _{i=1}^{K+1}{p}_i^{k_i}f\left(\mathbf{p}\right|\mathbf{x})d{p}_1\dots d{p}_{K+1}\hfill \\ & =& \frac{1}{C}{\beta }_{12}\frac{\Gamma \left({\pi }_1\right)\Gamma \left({\pi }_2\right)}{\Gamma ({\pi }_1+\tau )\Gamma ({\pi }_2+\tau )}[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1+\tau -1}{p}_2^{{\pi }_2+x_2+k_2+\tau -1}\dots \hfill \\ & & {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +\dots & \hfill \\ & +& \frac{1}{C}{\beta }_{K,K+1}\frac{\Gamma \left({\pi }_K\right)\Gamma \left({\pi }_{K+1}\right)}{\Gamma ({\pi }_K+\tau )\Gamma ({\pi }_{K+1}+\tau )}[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1-1}\dots p_K^{{\pi }_K+x_K+k_K+\tau -1}\hfill \\ & & {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}+\tau -1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +& \frac{1}{C}{\beta }_{11}\frac{\Gamma \left({\pi }_1\right)}{\Gamma ({\pi }_1+2\tau )}[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1+2\tau -1}{p}_2^{{\pi }_2+x_2+k_2-1}\dots \hfill \\ & & {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}-1}d{p}_1\dots d{p}_{K+1}]\hfill \\ & +\dots & \hfill \\ & +& \frac{1}{C}{\beta }_{K+1,K+1}\frac{\Gamma \left({\pi }_{K+1}\right)}{\Gamma ({\pi }_{K+1}+2\tau )}[{\displaystyle \underset{\mathcal{A}}{\int \cdots \int }}{p}_1^{{\pi }_1+x_1+k_1-1}{p}_2^{{\pi }_2+x_2+k_2-1}\dots \hfill \\ & & {\left(1-\sum _{i=1}^k{p}_i\right)}^{{\pi }_{K+1}+x_{K+1}+k_{K+1}+2\tau -1}d{p}_1\dots d{p}_{K+1}].\hfill \end{array}$$

We identify that each integral in the above expression is of the form of the Dirichlet kernel. Using the definition of total probability, the result follows.    □

Using the complete product moments derived in (24), the Bayesian estimator for the power sum (4) can be derived by setting $k_i=\alpha$ with $i=1,\dots ,K+1$ and $k_{\ne i}=0$.

Theorem 9.

The Bayesian estimator for the power sum functional under the double flexible Dirichlet posterior (20) is given by:

$$\begin{array}{ccc}\hfill {\widehat{F}}_{\alpha }\left(p\right)& =& \sum _{j^{*}=1}^{K+1}E\left(p_{j^{*}}^{\alpha }\right)\hfill \\ & =& \sum _{j^{*}=1}^{K+1}\left[\frac{A}{B}\right]\hfill \end{array}$$

(25)

where

$$\begin{array}{ccc}\hfill A& =& {\displaystyle \underset{r\ne s}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}\frac{{\beta }_{rs}\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}\hfill \\ & \times & \left[\frac{\Gamma ({\pi }_{j^{*}}+x_{j^{*}}+\alpha +\tau ({\mathbf{e}}_r+{\mathbf{e}}_s)){\prod }_{i\ne j^{*}}^{K+1}\Gamma ({\pi }_i+x_i+\tau ({\mathbf{e}}_r+{\mathbf{e}}_s))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\alpha +2\tau )}\right]\hfill \\ & +& \sum _{r=1}^{K+1}\frac{{\beta }_{rr}\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\left[\frac{\Gamma ({\pi }_{j^{*}}+x_{j^{*}}+\alpha +2\tau \left({\mathbf{e}}_r\right)){\prod }_{i\ne j^{*}}^{K+1}\Gamma ({\pi }_i+x_i+2\tau \left({\mathbf{e}}_r\right))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+\alpha +2\tau )}\right]\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill B& =& [{\displaystyle \underset{r\ne h}{\sum _{r=1}^{K+1}\sum _{s=1}^{K+1}}}{\beta }_{rs}\frac{\Gamma \left({\pi }_r\right)\Gamma \left({\pi }_s\right)}{\Gamma ({\pi }_r+\tau )\Gamma ({\pi }_s+\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+\tau ({\mathbf{e}}_r+{\mathbf{e}}_h))}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}\hfill \\ & +& \sum _{r=1}^{K+1}{\beta }_{rr}\frac{\Gamma \left({\pi }_r\right)}{\Gamma ({\pi }_r+2\tau )}\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+2\tau {\mathbf{e}}_r)}{\Gamma ({\sum }_{i=1}^{K+1}{\pi }_i+x_i+2\tau )}].\hfill \end{array}$$

4. Evaluation and Discussion

The following is the exploratory approach to determining potential estimates from the posteriors (9), (13), and (20) by incorporating sample information via the correlation. The following steps [10] were used to determine the optimal values of the parameters of the various priors considered for the multinomial model, in conjunction with the data available and expert judgement. This exploratory approach serves to gain insight into parameter estimation in this Bayes context by utilising sample information via the correlation (which could be positive due to the inclusion of the double flexible Dirichlet distribution as prior).

• Using shape analysis (as in Figure 2, Figure 4 and Figure 5) and visual investigations (trial and error), determine the bands of possible ranges for each of the parameters which need to be estimated.
• Create a grid by specifying all possible values for each parameter within the range specified in step 1. The grid will contain all possible combinations of these parameter options.
• For each step in the grid search calculate the entropy measures (12), (19), and (25) using the Bayesian estimate of the power sum functional.
• For each step in the grid search, calculate the correlation for each parameter combination.
• When selecting the parameters of the prior distribution, choose them such that the parameters ensure a pre-determine range of entropy values. The entropy range can be selected when taking into consideration that lower entropy values are associated with less uncertainty and therefore higher concentrated distributions.
• Ensure that the resultant correlation for the range of possible estimates are in range of what is obtained from the data.
• Visually inspect the selected parameters to ensure a good fit.

The dataset that was considered, obtained from [15], was collected through household budget surveys aimed at studying consumer demand. This dataset was also used in studies such as [8] and reports the household expenditures (in Hong Kong Dollars) on two commodity groups of a sample of 40 individuals. The variables considered are the proportions spent on housing (including fuel and lights) ($p_1$), consumables (including alcohol and tobacco) ($p_2$), and the rest classified as services and other goods (including transport and vehicles, clothing, footwear, and durable goods). The results obtained are reported for $\alpha =0.5$ for Tsallis, Generalized Mathai and Abe (see Table 1) and visually displayed to illustrate the fitted results.

Figure 7 shows how well the estimates, obtained using the Dirichlet prior (5), fitted the dataset.

For equal weight $\mathsf{\beta }$ was chosen as (${\beta }_1=0.3;{\beta }_2=0.34$ and ${\beta }_3=0.36$). Figure 8 shows how well the flexible Dirichlet prior (6) isolates the three modes visible within the data.

Figure 9 shows how the double flexible Dirichlet prior (8) fit the dataset with predetermined weight for the $\mathsf{\beta }$ (${\beta }_{11}=0.15;{\beta }_{22}=0.05;{\beta }_{33}=0.1;{\beta }_{12}=0.2;{\beta }_{13}=0.14;$${\beta }_{21}=0.2;{\beta }_{23}=0.01;{\beta }_{31}=0.14;{\beta }_{32}=0.01$).

The parameters of these different posterior distributions resulted in distributions which compared well to the observed data (some better than others) and also provided similar correlations than those found in the data. As it is known that the prior distribution plays a vital role in Bayesian analysis it is important to be able to quantify the impact of each prior in order to choose between one or more priors [7]. In order to measure the impact of the different priors, the WIM [7] was considered and reported in

Table 2. Wasserstein Impact Measure results for each set of parameters estimated.

Priors Being Compared	WIM (Tsallis)	WIM (Mathai)	WIM (Abe)
Dirichlet vs. flexible Dirichlet	15.60	12.67	13.26
Dirichlet vs. double flexible Dirichlet	16.29	13.38	13.86
Flexible Dirichlet vs. double flexible Dirichlet	1.00	0.99	0.73

. These results were calculated by considering the estimated parameters as reported in Figure 7, Figure 8 and Figure 9 for each of the entropy measures investigated. The posterior distributions were then compared by calculating the WIM using the wasserstein1d function in R (statistical software).

It can be seen that when comparing the WIM for each pair of posteriors considered, the measures resulting from the flexible and double flexible Dirichlet priors yielded large differences when compared to the Dirichlet distribution but almost no difference when comparing the two flexible distributions with each other. From the visual inspections and the WIM results is can be seen that there is value in considering generalizations of the Dirichlet distributions, in particular the considered mixtures of Dirichlet distributions. This may further benefit the practitioner in possible cases of clustering, then multimodal data may be present which the flexible Dirichlet- as well as the double flexible Dirichlet would be able to capture meaningfully.

5. Concluding Remarks

This paper considers key generalized entropy forms via the power sum functional of the posterior distribution, when subject to mixtures of Dirichlet distributions as a prior for the popular multinomial likelihood with K distinct classes. Here, the double flexible Dirichlet distribution offers the potential of positive prior correlation when this might be necessitated by prior information or expert opinion. Bayesian estimators were constructed for generalised entropy functions via this power sum functional, emanating from the product moment of each consider mixture of Dirichlet distribution’s prior, and implemented for parameter estimation when fitting posterior distributions to real economic data. Using the Bayesian estimates of these entropy measures proved to be a useful aid in selecting the prior distribution by consideration of the WIM, however, it is essential to evaluate each scenario separately. A possible further step of action may be to investigate possible other values of $\alpha$ on the estimation. Further work include the potential probability representation of quantum states that can be characterised by the priors considered in this paper [16], as well as future interest in dimension-free estimation of entropy in relevant settings [17].