Consistency of Decision in Finite and Numerable Multinomial Models

Abstract

The multinomial distribution is often used in modeling categorical data because it describes the probability of a random observation being assigned to one of several mutually exclusive categories. Given a finite or numerable multinomial model $\mathcal{M}\left(|n,\mathit{p}\right)$ whose decision is indexed by a parameter $\theta$ and having a cost $c\left(\mathit{\theta },\mathit{p}\right)$ depending on $\theta$ and on $\mathit{p}$, we show that, under general conditions, the probability of taking the least cost decision tends to 1 when n tends to ∞, i.e., we showed that the cost decision is consistent, representing a Statistical Decision Theory approach to the concept of consistency, which is not much considered in the literature. Thus, under these conditions, we have consistency in the decision making. The key result is that the estimator ${\tilde{\mathit{p}}}_n$ with components ${\tilde{p}}_{n,i}={\displaystyle \frac{n_i}{n}},i=1,\cdots$, where $n_i$ is the number of times we obtain the ith result when we have a sample of size n, is a consistent estimator of $\mathit{p}$. This result holds both for finite and numerable models. By this result, we were able to incorporate a more general form for consistency for the cost function of a multinomial model.

1. Introduction

Classical Statistical Inference (cSI) is centered on minimizing the probabilities of errors, but in Statistical Decision Theory (SDT), minimizing the decision costs is the goal; see for instance [1,2]. In minimizing the probability of errors in cSI, or the decision costs in SDT, a desired property is to have our procedure to be consistent, that is, we would like the mass of the distribution of the sequence of the sample for the procedure to converge to the population constant as the sample size gets larger, see [3,4]. As far as we have searched prior to the writing of this manuscript, not many studies have been conducted on the consistency of decision costs for SDT as compared to that of the error probabilities in the case of cSI.

Multinomial distribution is often used in modeling categorical data because it describes the probability of a random observation being assigned to one of several mutually exclusive categories. Thus, having n independent realizations of experiments with a finite or numerable set of incompatible results with probabilities $p_i$, $i=1,\cdots ,\infty$, the probabilities of obtaining $N_i=n_i$ times the ith result, $i=1,\cdots$ follows the multinomial distribution denoted as $\mathcal{M}\left(|n,\mathit{p}\right)$ with parameters n and $\mathit{p}$. The probability density function of this distribution is

$$\mathbb{P}\left[\bigcap _{i=1}^{\infty }\left(N_i=n_i\right)|\mathit{p}\right]=\frac{n!}{{\prod }_{i=1}^{\infty }{n}_i!}\prod _{i=1}^{\infty }{p}_i^{n_i}$$

(1)

where $0^0=1$, $\mathit{p}=\left(p_1,\cdots ,p_i,\cdots \right)$ and $n={\displaystyle \sum _{i=1}^{\infty }}{n}_i$, see [5,6]. To avoid repetitions, we give the expression for the numerable case since the particularization to the finite set is direct.

In what follows, we take a double approach in the treatment of Multinomial Models, both classical and decision theory approaches. In our classical approach, we show that $\tilde{\mathit{p}}=\left(\widetilde{p_1},\cdots ,\widetilde{p_i},\cdots \right)$, with ${\tilde{p}}_{n,i}={\displaystyle \frac{n_i}{n}},i=1,\cdots ,$ is a consistent estimator of $\mathit{p}=\left(p_1,\cdots ,p_i,\cdots \right)$. This result will play a central part in our paper. We point out that we use a finite sample to obtain a numerable family $\left\{{\tilde{p}}_{n,i},i=1,\cdots \right\}$ of jointly consistent estimators. We thus have consistent results in the fold of classical statistical inference.

Now considering a decision problem, let there be a family of possible decisions. For each of these decisions, we have a cost that depends on the results of the experiment. These results will have probabilities $p_i,i=1,\cdots$. We thus have for the ith result the costs $c_i\left(d\right),d\in \mathcal{D},i=1,\cdots$. The average cost for decision $d\in \mathcal{D}$ will be

$$c_{·}\left(d\right)=\sum _{i=1}{p}_i{c}_i\left(d\right),d\in \mathcal{D}.$$

(2)

Assuming the $c_i\left(d\right),d\in \mathcal{D},i=1,\cdots$ is known, we can use the estimators

$${\tilde{p}}_{n,i}={\displaystyle \frac{n_i}{n}},i=1,\cdots ,$$

(3)

where $n_i,i=1,\cdots$ is the number of times that in n independent realizations of the experiment, we get the ith result with cost $c_i\left(d\right),d\in \mathcal{D},i=1,\cdots$. We have the estimators for the costs as

$${\tilde{c}}_{n,·}\left(d\right)=\sum _{i=1}{\tilde{p}}_{n,i}{c}_i\left(d\right),d\in \mathcal{D}.$$

We will show that ${\tilde{p}}_{n,i},i=1,\cdots$, are jointly consistent even when we have a numerable set of possible results, thus the ${\tilde{c}}_{n,·}\left(d\right)$ will also be consistent, $d\in \mathcal{D}$.

If there is a decision $d^{\circ }$ with least average cost and ${\tilde{d}}_n^{\circ }$ is the one with the least estimated average cost where there are n realizations of the experiment, we will show that

$$Pr\left({\tilde{d}}_n^{\circ }=d\right)\underset{n\to \infty }{\to }1$$

so that, see [7], we will have consistency in decision taking into the setup of experiments with finite or numerable set of incompatible results.

In the subsequent sections, we consider the multinomial model and its estimator in Section 2, where by limits distributions we show that the estimators of the multinomial model are consistent. In Section 3, we develop the cost function by statistical decision theory for multinomial models and again show that this function has the property of consistency. A further extension for the cost function is presented in Section 4.

2. Multinomial Models and Estimators

In this section, we obtain and consider estimators for the multinomial model. By limit distributions, see [8], we show that the estimators are consistent.

In $\mathcal{X}$, the space of vectors $\mathit{v}=\left(v_1,\cdots ,v_i,\cdots \right)$ with numerable sets of components such that

$$\parallel \mathit{v}{\parallel }_1=\sum _{i=1}^{\infty }\left|v_i\right|<+\infty .$$

(4)

We can consider $\parallel \mathit{v}{\parallel }_1$ as a norm, see [9]. The sub-space $\mathsf{\Gamma }$ of $\mathcal{X}$ constituted by the vectors $\mathit{p}$ with non-negative components that add up to 1 will be bounded since $\parallel \mathit{p}{\parallel }_1=1$. Given ${\mathit{p}}_1,{\mathit{p}}_2\in \mathsf{\Gamma }$, we have

$$\parallel {\mathit{p}}_1-{\mathit{p}}_2{\parallel }_1\le 2,$$

(5)

and if

$$\parallel {\mathit{p}}_n-\mathit{v}{\parallel }_1\underset{n\to \infty }{\to }0$$

(6)

we have $\mathit{v}\in \mathsf{\Gamma }$ since the components of $\mathit{v}$ will be non-negative and add up to 1. Thus, $\mathsf{\Gamma }$ is compact since it is bounded and closed.

Let us put

$$\left\{\begin{array}{c}p\left(m\right)={\displaystyle \sum _{i=1}^m}{p}_i\hfill \\ \\ p^c\left(m\right)=1-p\left(m\right)\hfill \\ \end{array}\right.$$

(7)

as well as

$$\mathit{p}\left(m\right)=\left(p_1,\cdots ,p_m+p^c\left(m\right),0\cdots \right)$$

(8)

in order to get

$$\begin{array}{cc}\hfill {∥\mathit{p}-\mathit{p}\left(m\right)∥}_1& =p^c\left(m\right)+\sum _{i=m+1}^{\infty }{p}_i\hfill \\ \hfill & =2p^c\left(m\right).\hfill \end{array}$$

(9)

Besides this, we have the vector

$${\tilde{\mathit{p}}}_n=\left({\tilde{p}}_{n,1},\cdots ,{\tilde{p}}_{n,m},\cdots \right)$$

(10)

whose components are the estimators ${\tilde{p}}_{n,i}={\displaystyle \frac{n_i}{n}},i=1,\cdots$. Let us put

$$\left\{\begin{array}{c}{\tilde{p}}_n\left(m\right)={\displaystyle \sum _{i=1}^m}{\tilde{p}}_{n,i}\hfill \\ \\ {\tilde{p}}_n^c\left(m\right)=1-{\tilde{p}}_n\left(m\right)\hfill \\ \end{array}\right.$$

(11)

as well as

$${\tilde{\mathit{p}}}_n\left(m\right)=\left({\tilde{p}}_{n,1},\cdots ,{\tilde{p}}_{n,m}+{\tilde{p}}_n^c\left(m\right),0,\cdots \right)$$

(12)

so that

$${∥{\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)∥}_1=2{\tilde{p}}_n^c\left(m\right).$$

(13)

With

$${\tilde{m}}_n=min\left\{h:0={\tilde{p}}_{n,h+1}={\tilde{p}}_{n,h+2}=\cdots \right\},$$

(14)

for $m\ge {\tilde{m}}_n$, we get

$${∥\left({\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)\right)-\left(\mathit{p}-\mathit{p}\left(m\right)\right)∥}_1={∥\mathit{p}-\mathit{p}\left(m\right)∥}_1=2p^c\left(m\right)$$

(15)

since ${\tilde{\mathit{p}}}_n={\tilde{\mathit{p}}}_n\left(m\right)$ when $m\ge {\tilde{m}}_n$. We also get

$${∥{\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)∥}_1\le 2{∥\mathit{p}-\mathit{p}\left(m\right)∥}_1\le 4p^c\left(m\right).$$

(16)

Now, by representing stochastic convergence by $\underset{n\to \infty }{\overset{s}{\to }}$, we establish Proposition 1:

Proposition 1.

The estimator ${\tilde{\mathit{p}}}_n$ is said to be a consistent estimator for $\mathit{p}$ since

$${\tilde{\mathit{p}}}_n\underset{n\to \infty }{\overset{s}{\to }}\mathit{p},$$

(17)

that is according to the Weak Law of Large numbers, see [10,11].

Proof. 

Taking

$$m\left(\epsilon \right)=min\left\{m:p^c\left(m\right)\le \epsilon \right\},\forall ϵ>0.$$

(18)

so that for $m>m\left(\epsilon \right)$, we have

$$\mathbb{P}\left(p_n^c\left(m\right)\le 2\epsilon \right)\underset{n\to \infty }{\overset{s}{\to }}1,$$

(19)

as well as

$$\mathbb{P}\left({∥{\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)∥}_1\le 4p^c\left(m\right)\right)\underset{n\to \infty }{\overset{s}{\to }}1,$$

(20)

since

$${∥{\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)∥}_1\le 2p^c\left(m\right).$$

(21)

Now

$${∥{\tilde{\mathit{p}}}_n\left(m\right)-\mathit{p}\left(m\right)∥}_1\underset{n\to \infty }{\overset{s}{\to }}0,$$

(22)

since see [7,8,12],

$$\sqrt{n}\left({\tilde{\mathit{p}}}_n\left(m\right)-\mathit{p}\left(m\right)\right)\stackrel{D}{\to }\mathcal{N}\left(\mathbf{0},U\left(\mathit{p}\left(m\right)\right)\right)$$

(23)

where $\stackrel{D}{\to }$ indicates convergence in distribution in this case to the normal distribution with null mean vector and covariance matrix

$$U\left(\mathit{p}\left(m\right)\right)=\mathcal{D}\left(\mathit{p}\left(m\right)\right)-\mathit{p}\left(m\right)\mathit{p}{\left(m\right)}^t,$$

(24)

where $\mathcal{D}\left(\mathit{p}\left(m\right)\right)$ is the diagonal matrix whose principal elements are the components of $\mathit{p}\left(m\right)$. Thus

$$\mathbb{P}\left({∥{\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)∥}_1\le \epsilon \right)\underset{n\to \infty }{\overset{s}{\to }}1,$$

(25)

and so

$$\mathbb{P}\left[\left({∥{\tilde{\mathit{p}}}_n-{\tilde{\mathit{p}}}_n\left(m\right)∥}_1\le 4\epsilon \right)\bigcap \left({∥{\tilde{\mathit{p}}}_n\left(m\right)-\mathit{p}\left(m\right)∥}_1\le \epsilon \right)\right]\underset{n\to \infty }{\overset{s}{\to }}1,$$

(26)

as well as

$$\mathbb{P}\left({∥{\tilde{\mathit{p}}}_n-\mathit{p}\left(m\right)∥}_1\le 5\epsilon \right)\underset{n\to \infty }{\overset{s}{\to }}1,$$

(27)

whenever $m>m\left(\epsilon \right)$.

Now, we also have

$$\parallel \mathit{p}-\mathit{p}\left(m\right){\parallel }_1=2p_n^c\le 2\epsilon ,$$

(28)

if $m>m\left(\epsilon \right)$, then

$$\mathbb{P}\left({∥{\tilde{\mathit{p}}}_n-\mathit{p}∥}_1\le 7\epsilon \right)\underset{n\to \infty }{\overset{s}{\to }}1,$$

(29)

which establishes the thesis. □

Corollary 2.

If $w\left(\mathit{p}\right)$ is a continuous function of $\mathit{p}$, we have

$$w\left({\tilde{\mathit{p}}}_n\right)\underset{n\to \infty }{\overset{s}{\to }}w\left(\mathit{p}\right).$$

(30)

The thesis follows from Proposition 1 and the Slutsky theorem, see [13,14].

Corollary 3.

With ${\mathit{d}}_h\left(\mathit{v}\right)$ the vector of the indexes of the h largest components of ${\mathit{v}}_n\in \mathcal{X}$, we have see [10]

$${\mathit{d}}_h\left({\tilde{\mathit{p}}}_n\right)\underset{n\to \infty }{\overset{s}{\to }}{\mathit{d}}_h\left(\mathit{p}\right)$$

(31)

if the $h+1$ largest components of $\mathit{p}$ are distinct.

Proof. 

When the $h+1$ largest components of $\mathit{p}$ are distinct, $\mathit{p}$ will be a continuity point of ${\mathit{d}}_h(·)$ and the thesis follows from Corollary 3. □

The results in this section belong to the study of consistency in classical Statistical Inference (cSI). These classical inferences are, in most instances, made without regard to the use to which they are to be put. In the next section, we go into consistency for Statistical Decision Theory.

3. Cost Function for Multinomial Models

In Statistical Decision Theory (SDT), the goal is to incorporate more than just sample data in order to arrive at the optimal decision, unlike cSI. The knowledge of the possible consequences of a decision is much incorporated and this knowledge is quantified as the cost incurred for each possible decision that is taken. According to [15], Abraham Wald was the first person to thoroughly examine the inclusion of a cost function in statistical analysis.

The cost function represents the costs associated with taking a particular decision. It is a function that maps every possible decision and outcome to a real-valued cost. The cost function is used to evaluate the performance of various decision rules in terms of their expected cost. The goal of statistical decision theory is to identify the decision rule that minimizes the expected cost, see [16].

Now, we go back to the decision problem we presented in the Introduction and consider the cost function for multinomial models.

Let $c(\mathit{p},{\tilde{\mathit{p}}}_n,d),d\in \mathcal{D}$, be the cost for decision $d\in \mathcal{D}$, where $\mathit{p}$ is the vector of probabilities and ${\tilde{p}}_n$ are the estimated probabilities of the n results. We will assume that this cost is the sum of two components, both non-negative, $c_0(\mathit{p},d)$, that in a given decision $d\in \mathcal{D}$, depends only on $\mathit{p}$, the vector of probabilities, and $c_1(\mathit{p},\widetilde{{\mathit{p}}_n},d)$ that depends on the estimation errors. Namely, we take

$$c\left(\mathit{p},{\tilde{\mathit{p}}}_n,d\right)=c_0\left(\mathit{p},d\right)+k_d\left(\parallel {\tilde{\mathit{p}}}_n-\mathit{p}{\parallel }_1\right),d\in \mathcal{D}$$

(32)

with $k_d>0,d\in \mathcal{D}$, so

$$c_0(\mathit{p},d)=min\left\{c\left(\mathit{p},{\tilde{\mathit{p}}}_n,d\right)\right\},d\in \mathcal{D}$$

since, as we saw

$${\tilde{\mathit{p}}}_n\underset{n\to \infty }{\overset{s}{\to }}\mathit{p}$$

we will have

$$c\left(\mathit{p},{\tilde{\mathit{p}}}_n,d\right)\underset{n\to \infty }{\overset{s}{\to }}{c}_0(\mathit{p},d),d\in \mathcal{D}.$$

(33)

Thus, for every $d\in \mathcal{D}$, the limit cost will be $c_0(\mathit{p},d)$. It is now easy to see that if there is $d\left(\mathit{p}\right)$ such that

$$c_0\left(\mathit{p},d^0\left(\mathit{p}\right)\right)\le c_0(p,d),d\in \mathcal{D},$$

with ${\tilde{d}}_n^0\left(\mathit{p}\right)$ as the decision with the least estimated cost. We have

$$Pr\left({\tilde{d}}_n^0\left(\mathit{p}\right)=d^0\left(\mathit{p}\right)\right)\underset{n\to \infty }{\overset{s}{\to }}1$$

Proposition 4.

We have consistency for the cost function given by Equation (32) whenever Equation (33) holds.

If the $h+1$ largest components of $\mathit{p}$ are obtained, as an alternative to Equation (32), we may take

$$c\left(\mathit{p},{\tilde{\mathit{p}}}_n\right)=c_0\left(\mathit{p}\right)+c_1\left(\parallel {\mathit{d}}_h\left({\tilde{\mathit{p}}}_n\right)-{\mathit{d}}_h\left(\mathit{p}\right){\parallel }_1\right),$$

(34)

reobtaining Proposition 4, since

$$\parallel {\mathit{d}}_h\left({\tilde{\mathit{p}}}_n\right)-{\mathit{d}}_h\left(\mathit{p}\right){\parallel }_1\underset{n\to \infty }{\overset{s}{\to }}0$$

(35)

and so we continue to have

$$c\left(\mathit{p},{\tilde{\mathit{p}}}_n\right)\underset{n\to \infty }{\overset{s}{\to }}{c}_0\left(\mathit{p}\right).$$

(36)

We may also take

$$c_1\left({\tilde{\mathit{p}}}_n-\mathit{p}\right)={\left({\tilde{\mathit{p}}}_n-\mathit{p}\right)}^t\mathit{M}\left({\tilde{\mathit{p}}}_n-\mathit{p}\right)$$

(37)

with $\mathit{M}$ a positive definite matrix, see [8,9], or

$$c_1\left({\tilde{\mathit{p}}}_n-\mathit{p}\right)={\left({\mathit{d}}_h\left({\tilde{\mathit{p}}}_n\right)-{\mathit{d}}_h\left(\mathit{p}\right)\right)}^t\mathit{M}\left({\mathit{d}}_h\left({\tilde{\mathit{p}}}_n\right)-{\mathit{d}}_h\left(\mathit{p}\right)\right).$$

(38)

Thus, there is a wide range of possible cost functions, namely, with $\mathit{g}(·)$, a continuous function

$${\tilde{\mathit{\gamma }}}_n=\mathit{g}\left({\tilde{\mathit{p}}}_n\right)$$

(39)

will, according to the Slutsky theorem, be a consistent estimator of $\mathit{\gamma }=\mathit{g}\left(\mathit{p}\right)$.

Moreover, if we have a cost function

$$c\left(\mathit{\gamma },{\tilde{\mathit{\gamma }}}_n\right)=c_0\left(\mathit{\gamma }\right)+c_1\left(\parallel {\tilde{\mathit{\gamma }}}_n-\mathit{\gamma }{\parallel }_1\right),$$

(40)

where $c_0\left(\mathit{\gamma }\right)$ is the cost that depends on $\mathit{\gamma }$ and $c_1(·)$ is continuous and such that $c_1\left(0\right)=0$, we can use again the Slutsky theorem, to get

$$c\left(\mathit{\gamma },{\tilde{\mathit{\gamma }}}_n\right)\underset{n\to \infty }{\overset{s}{\to }}{c}_0\left(\mathit{\gamma }\right).$$

(41)

We thus extended our previous results on $\mathit{p}$ to any parameter given by a continuous function of $\mathit{p}$, such as the cost function.

4. Extension on Cost Functions

In this section, we develop an extension for the previous Section 3 to incorporate a more general form of our results on consistency for the cost function for multinomial models.

For instance, we consider

$$\gamma \left(\phi \right)=\sum _{j\in \phi }{p}_j$$

(42)

the sum of the probabilities of the results with indexes in $\phi$. These results may be of interest and so we are led to consider their joint probability. A direct extension of this case is given by

$$\gamma ({\phi }_1,\cdots ,{\phi }_k)=\sum _{l=1}^k{a}_l\sum _{j\in {\phi }_l}{p}_j$$

(43)

where we consider k sets of results. The coefficients $a_1,\cdots ,a_k$ value the relevances of the corresponding sets of results.

In general, let us have a succession $\left\{{\mathit{Y}}_n\right\}$ of observation vectors whose distributions depend on a parameter $\mathit{\theta }$ for which we have a consistent estimator ${\tilde{\mathit{\theta }}}_n$. Then, if we have a cost function

$$c\left(\mathit{\theta },{\tilde{\mathit{\theta }}}_n\right)=c_0\left(\mathit{\theta }\right)+c_1\left(\parallel {\tilde{\mathit{\theta }}}_n-\mathit{\theta }{\parallel }_1\right)$$

(44)

with $c_0\left(\mathit{\theta }\right)$ a continuous function of $\mathit{\theta }$ and $c_1(·)$ also continuous, and such that

$$min\left\{c_1{(\parallel z\parallel }_1)\right\}=c_1\left(0\right)=0,$$

(45)

we have

$$c\left(\mathit{\theta },{\tilde{\mathit{\theta }}}_n\right)\underset{n\to \infty }{\overset{s}{\to }}{c}_0\left(\mathit{\theta }\right)$$

(46)

which implies consistency for the cost functions. Thus, the two consistent features display the relation we had already found for multinomial models. Namely, we obtain the following result:

Proposition 5.

If we have a consistent estimator ${\tilde{\mathit{\theta }}}_n$ for a parameter θ, we have consistency for cost functions

$$c\left(\mathit{\theta },{\tilde{\mathit{\theta }}}_n\right)=c_0\left(\mathit{\theta }\right)+c_1\left(\parallel {\tilde{\mathit{\theta }}}_n-\mathit{\theta }{\parallel }_1\right)$$

where $c_1(·)$ is continuous with minimum $c_1\left(0\right)$.

The extension behind this proposition, and getting consistent estimators for a numerable set of parameters, the components of $\mathit{p}$, from a finite sample are maybe the most interesting features of our discussion.

5. Final Remark

In this study, based on limit distribution, by considering the vector of probabilities for the multinomial model, we showed, using classical Statistical Inference, that the estimators for the vector of probabilities are consistent. Due to the limitation of classical Statistical Inference but not incorporating the knowledge of the possible consequences of a decision, we used a Statistical Decision Theory approach to quantify the cost incurred for each possible decision by obtaining a cost function for the vector of probabilities. We showed that the estimators of cost function are consistent.

Our results on having consistency for the estimator of probabilities leads to consistency of decision function; in this, we hope to have opened an interesting line of work on multinomial and other models using Statistical Decision Theory.