Generalizations of Rao–Blackwell and Lehmann–Scheffé Theorems with Applications

Abstract

Our aim in this paper is extending the applicability domain of the Rao–Blackwell theorem, our methodology is using conditional expectation and generalizing sufficient statistics, and one result is a generalization of the Lehmann–Scheffé theorem; as a conclusion, some problems that could not be solved by an earlier version of the Lehmann–Scheffé theorem become solvable by our new generalization.

1. Introduction

A population quantity (for example, the average height of all men) can be estimated in many different ways. An unbiased estimator (see Definition 1) is one that provides zero bias (that is, it estimates the population quantity with zero bias). Some examples of unbiased estimators are estimation for average length of stay in intensive care units in the COVID-19 pandemic (Lapidus et al. [1]); estimation of cumulative incidence incorporating antibody kinetics and epidemic recency (Takahashi et al. [2]); estimation of background distribution for automated quantitative imaging (Silberberg and Grecco [3]); and estimation of target tracking in Doppler radar (Han et al. [4]). The Rao–Blackwell theorem is a generator for unbiased estimators with small variances. In this note, we aim at generalizing this effective theorem for finding those estimators. Fisher [5] introduced sufficient statistics in 1920; see Definition 2.

Definition 1. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. Let $\mathfrak{a}:\mathsf{\Theta }⟶\mathbb{R}$ be a parameter taking real values. An estimator $\delta \left(X\right)$, $\delta :\mathcal{X}⟶\mathbb{R}$, of $\mathfrak{a}$ is unbiased if and only if $E\left[\delta \left(X\right)\right]=\mathfrak{a}$ for every $P_{\theta }\in \mathcal{P}$.

Definition 2 

(Sufficient statistic (Fisher [5])). Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. A statistic $T\left(X\right)$ is said to be sufficient for θ if the conditional probability distribution of X, given the statistic $t=T\left(X\right)$, is independent of θ.

Examples of sufficient statistics can be found in the books Lehmann [6], Casella and Berger [7], and Shao [8]. We should understand sufficient statistics better to derive uniformly minimum variance unbiased estimators (UMVUEs); see Definition 4. Statisticians have cleverly embedded sufficient statistics into estimators, which is the main idea of the Rao–Blackwell theorem; see Rao [9] and Blackwell [10]. UMVUEs can be calculated by complete sufficient statistics (see Definition 5), leading to the Lehmann–Scheffé theorem; see Lehmann and Scheffé [11,12] and Kumar and Vaish [13]. A complete statistic is defined by Definition 3.

Definition 3. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. A statistic $T\left(X\right)$, $T:\mathcal{X}⟶\mathbb{R}$, is said to be complete for $P\in \mathcal{P}$ if and only if, for any Borel-measurable function f from $\mathbb{R}$ to $\mathbb{R}$, $E\left[f\left(T\right)\right]=0$ for all $P\in \mathcal{P}$ implies $f\left(T\right)=0$ almost surely $\mathcal{P}$.

Definition 4. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. An unbiased estimator $T\left(X\right)$ of $\mathfrak{a}$ is a UMVUE if and only if Var$\left[T\left(X\right)\right]\le$ Var$\left[\delta \left(X\right)\right]$ for every $P_{\theta }\in \mathcal{P}$ and every unbiased estimator $\delta \left(X\right)$ of $\mathfrak{a}$.

Definition 5 

(Complete sufficient statistic (Fisher [5])). Let $T\left(X\right)$ be a sufficient statistic for θ. If $E\left[g\left(T\right)\right]=0$ with probability 1, for some function g, then it is said to be a complete sufficient statistic for θ.

Applications of the Rao–Blackwell and Lehmann–Scheffé theorems are still widespread. Application areas have included reliability estimation (Kumar and Vaish [13]), adaptive cluster sampling (Felix-Medina [14]), alchemical free energy calculation (Ding et al. [15]), hazardous source parameter estimation (Ristic et al. [16]) and quantum probability (Sinha [17]).

However, nonconstant functions can be UMVUEs whenever there is not a complete sufficient statistic. Some authors try solving these problems by Theorem 1 by focusing on problem 5.11 in Rao [9] or pages 76–77 in Lehmann and Scheffé [11,12] (which are essentially Example 7 below).

Theorem 1 

(Lehmann–Scheffé theorem (Lehmann and Scheffé [11,12])). Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. If and only if condition for a statistic $T\left(X\right)$ to be UMVUE of its mean is that $E\left[T\left(X\right)U\left(X\right)\right]=0$ for all $\theta \in \mathsf{\Theta }$ and all $U\in {\mathcal{U}}_0$, where ${\mathcal{U}}_0$ denotes the set of all the unbiased estimators of 0.

This theorem can be used whenever there are no complete sufficient statistics. It is a competitor to the Rao–Blackwell theorem.

This fact is hardly pointed out or explained in undergraduate or graduate textbooks; see, for example, Bondesson [18]. The motivation to introduce a new concept of sufficient statistic called an $\mathcal{H}$-sufficient statistic comes from the above discussion. We investigate the properties of $\mathcal{H}$-sufficient statistics and compare them with those of sufficient statistics. Then, the Rao–Blackwell theorem (RBT) and Lehmann–Scheffé theorem (LST) will be generalized in a way that can solve some of the problems where UMVUEs exist but there are no complete sufficient statistics; cf. problem 5.11 in Rao [19], pages 76–77 in Lehmann [6], page 167, Example 3.7 in Shao [8], page 366, Example 10 in Rohatgi and Ehsanes [20], page 377, Section 7.6.1 in Mukhopadhyay [21], page 243 in Peña and Rohatgi [22], page 293, Section 12.4 in Roussas [23] and pages 330–331 in Mood et al. [24]. Some of the theorems are restated and proved by using the newly introduced $\mathcal{H}$-sufficient statistic.

Definition 6 

(Ancillary statistic). Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. A statistic $T\left(X\right)$ is said to be ancillary for θ if its distribution is the same for all $\theta \in \mathsf{\Theta }$.

Boos and Hughes-Oliver [25] state that “If a minimal sufficient statistic is not complete, then by the suggestion of Fisherian tradition we should consider condition on ancillary statistics (see Definition 6) for the purposes of inference. This approach runs into problems because there are many situations where several ancillary statistics exists but there are no maximal “ancillaries”. Of course, when a complete sufficient statistic exists, Basu’s theorem assures us that we need not worry about conditioning on ancillary statistics since they are all independent of the complete sufficient statistic”. We suggest complete $\mathcal{H}$-sufficient statistics for the purposes of inference when there are no complete sufficient statistics. Theorem 1 assures that we need not worry about ancillary statistics since they are uncorrelated regarding complete $\mathcal{H}$-sufficient statistics.

2. The Main Contribution

If the minimal sufficient statistic is not complete, then the RBT and LST will not be of much use, as has been explicitly stated in various books and papers; see, for example, page 46, Section 2, Example 1 of Bondesson [18], page 243 of Peña and Rohatgi [22] page 293, Section 12.4 of Roussas [23], pages 330–331 of Mood et al. [24], page 343, Section 7.3 of Casella and Berger [7], page 86, Example 1.8 of Lehmann and Casella [26], Section 1 of Bahadur [27] and Section 1 of Stigler [28].

The main contribution of this note is a generalization of the RBT and LST, resulting in the use of the newly introduced $\mathcal{H}$-sufficient statistics. This enables us to obtain UMVUEs even when the minimal sufficient statistic is not complete, in which case the RBT and LST are not directly applicable.

Consider a model $\left(\mathcal{X},\mathcal{A},\mathcal{P}=\left\{P_{\theta }:\theta \in \mathsf{\Theta }\right\}\right)$. Let $\left\{P_{\theta }:\theta \in \mathsf{\Theta }\right\}$ denote the set of probability measures on the sample space $\mathcal{X}$. Let $X=\left[X_1,\dots ,X_n\right]$ denote an element in $\mathcal{X}$. $P_{\theta }\in \mathcal{P}$ is the population. $X=\left[X_1,\dots ,X_n\right]$ is a sample. Let $X:$$\mathcal{X}$⟶$\mathcal{X}$, $\left(\mathcal{Y};\mathcal{B}\right)$ and $T:\mathcal{X}⟶\mathcal{Y}$ denote, respectively, the identity mapping, a measurable space and a $\mathcal{A}-\mathcal{B}-$ measurable mapping (that is, $T^{-1}B\in \mathcal{A}$ for all $B\in \mathcal{B}$). $T\left(X\right)$ is a statistic to $\left(\mathcal{Y};\mathcal{B}\right)$, written as $T:\left(\mathcal{X},\mathcal{A}\right)\to \left(\mathcal{Y};\mathcal{B}\right)$. $\mathfrak{a}$ is referred to as a U-estimable parameter if $\mathfrak{a}$ is an unbiased estimator.

Throughout this note, we assume that $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Assume also $\mathfrak{a}$ has an unbiased estimator. Let ${\mathcal{U}}_{\mathfrak{a}}$ denote the class of unbiased estimators $\delta :\mathcal{X}⟶\mathbb{R}$ for $\mathfrak{a}$. All the considered estimators are assumed to have finite variances. The space used in this note is ${\mathbb{R}}^n$ and the elements of $\mathcal{B}$ are Borel sets. For related notation and discussions, see Shao [8].

3. Sufficient Statistics

Sufficient statistics can be used to derive maximum likelihood estimators of a population quantity. Maximum likelihood estimation is a popular method for estimation, so sufficient statistics are important. Sufficient statistics were defined in Definition 2. Two weaker concepts of sufficiency, which are tailored to a given unbiased estimable aspect $\mathfrak{a}:\mathsf{\Theta }⟶\mathbb{R}$, are introduced and discussed in the following. Some properties of these statistics are studied in the sequel.

3.1. $\mathcal{H}$-Sufficient Statistic in Distribution

Definition 7. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. A statistic $T\left(X\right)$ is $\mathcal{H}$-sufficient in distribution for $\mathfrak{a}$ if, for all $\delta \left(X\right)\in {\mathcal{U}}_{\mathfrak{a}}$, there is a Markov kernel $k_{\mathfrak{a},\delta \left(X\right)}:\mathfrak{T}\times \mathcal{B}\left(\mathbb{R}\right)⟶[0,1]$ such that, for every $\theta \in \mathsf{\Theta }$, $k_{\mathfrak{a},\delta }$ is a version of a regular conditional distribution of $\delta \left(X\right)$ given $T\left(X\right)$ under $P_{\theta }$.

Definition 7 introduces a class of statistics that are weaker than sufficient statistics, which is not the main aim of this note. These statistics could be used in Rao–Blackwell and Lehmann–Scheffé theorems. We use this idea in Definition 8.

Example 1 

(Example of Meeden [29]). Let X be Poisson-distributed with $E\left(X\right)=\lambda$, so X belongs to the exponential family. Then, X is a complete sufficient statistic and ${(-1)}^X$ is only unbiased estimator for $e^{-2\lambda }$. By Definition 7, $k{(-1)}^X$ is an $\mathcal{H}$-sufficient statistic in distribution for $e^{-2\lambda }$ for k a constant. We can check that ${(-1)}^X$ is a UMVUE for $e^{-2\lambda }$.

The estimator ${(-1)}^X$ could not be suitable for $e^{-2\lambda }$ in the same way that in the Bernoulli distribution with parameter p the estimator X will not be suitable. Of course, increasing the sample size or varying the loss function remedies this deficiency.

3.2. $\mathcal{H}$-Sufficient Statistic

To derive UMVUEs when there are no complete sufficient statistics, we need to introduce a new concept named an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. It is defined as follows.

Definition 8. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. A statistic $T\left(X\right)$ is an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ if, for all $\delta \left(X\right)\in {\mathcal{U}}_{\mathfrak{a}}$, there is a measurable mapping $h_{\mathfrak{a},\delta }:\mathfrak{T}⟶\mathbb{R}$ such that for every $\theta \in \mathsf{\Theta }$ we have $E_{\theta }\left[\delta \left(X\right)\mid T\right]=h_{\mathfrak{a},\delta }\circ T$ almost surely $P_{\theta }$.

Example 2. 

Let X from $P_{\theta }$ have a discrete distribution with

$$\begin{array}{c}\hfill {\displaystyle P_{\theta }(X=-1)=\theta ,P_{\theta }(X=k)={(1-\theta )}^2{\theta }^k,k=0,1,2,\dots ,}\end{array}$$

where $\theta \in (0,1)$ is unknown. $I_{\left\{0\right\}}\left(X\right)$ is an $\mathcal{H}$-sufficient statistic for ${(1-\theta )}^2$ because

$$\begin{array}{c}\hfill {\displaystyle E_{\theta }\left[I_{\left\{0\right\}}\left(X\right)+\alpha X\mid I_{\left\{0\right\}}\left(X\right)\right]=I_{\left\{0\right\}}\left(X\right)}\end{array}$$

(1)

almost surely $P_{\theta }$ for every $\theta \in (0,1)$ and $\alpha \in \mathbb{R}$. The expectations needed for the left hand side of (1) are

$$\begin{array}{c}\hfill {\displaystyle E_{\theta }\left[X\mid I_{\left\{0\right\}}\left(X\right)=1\right]=E_{\theta }\left[X\mid X=0\right]=0,}\end{array}$$

and

$$\begin{array}{c}\hfill {\displaystyle E_{\theta }\left[X\mid I_{\left\{0\right\}}\left(X\right)=0\right]=E_{\theta }\left[X\mid X\ne 0\right]=0.}\end{array}$$

X is a minimal sufficient statistic for ${(1-\theta )}^2$ because of the LST. However, X is not complete since $E\left(X\right)=0$. Also, $I_{\left\{0\right\}}\left(X\right)$ is not an $\mathcal{H}$-sufficient statistic in distribution for ${(1-\theta )}^2$ since its conditional distribution depends on θ.

For having all the unbiased estimators of ${(1-\theta )}^2$, see the following proof.

For every $g\left(x\right)$, we have

$$\begin{array}{c}\hfill {\displaystyle 0=Eg\left(x\right)=\sum _{x=-1}^{\infty }g\left(x\right)P(X=x)=\theta g(-1)+\sum _{x=0}^{\infty }g\left(x\right){(1-\theta )}^2{\theta }^x.}\end{array}$$

Then, for any $\theta \in (0,1)$,

$$\begin{array}{c}\hfill \sum _{x=0}^{\infty }g\left(x\right){\theta }^x=-\theta g(-1){(1-\theta )}^{-2}.\end{array}$$

We have

$$\begin{array}{c}\hfill {\displaystyle g\left(x\right){\theta }^x=-\theta g(-1)\left(1+2\theta +3{\theta }^2+\cdots \right)=g(-1)\sum _{x=1}^{\infty }x{\theta }^x.}\end{array}$$

Comparing power series coefficients, we have

$$\begin{array}{c}\hfill {\displaystyle g\left(0\right)=0,g\left(x\right)=-g\left(0\right)x,x=1,2,\dots }\end{array}$$

or

$$\begin{array}{c}\hfill {\displaystyle g\left(0\right)=0,g\left(x\right)=\alpha x,x=1,2,\dots ,}\end{array}$$

where $\alpha =-g\left(0\right)$.

Some properties of $\mathcal{H}$-sufficient statistics are in Theorem 2.

Theorem 2. 

Let $\mathcal{P}=\left\{P_{\theta }:\theta \in \mathsf{\Theta }\right\}$. Consider

(i) 

a sufficient statistic for $\mathcal{P}$ (or θ),

(ii) 

an $\mathcal{H}$-sufficient statistic in distribution for $\mathfrak{a}$,

(iii) 

an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$.

Then, we have

(a) 

any sufficient statistic for $\mathcal{P}$ is an $\mathcal{H}$-sufficient statistic in distribution for $\mathfrak{a}$;

(b) 

any $\mathcal{H}$-sufficient statistic in distribution for $\mathfrak{a}$ is an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$;

(c) 

any sufficient statistic for $\mathcal{P}$ is an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$.

Proof. 

(a) follows because the conditional distribution of samples given a sufficient statistic does not depend on $\theta$. (b) follows because the conditional distribution of unbiased estimators given an $\mathcal{H}$-sufficient statistic does not depend on $\theta$. (c) follows because the conditional distribution of samples given a sufficient statistic does not depend on $\theta$. □

Remark 1. 

In general, the converse of none of the three parts of Theorem 2 holds (see Examples 1 and 2).

It is clear from Theorem 2 and Remark 1 that the class of $\mathcal{H}$-sufficient statistics for $\mathfrak{a}$ contains sufficient statistics for $\theta$. Also, we can conclude from Theorem 2 that the jointly sufficient statistics are $\mathcal{H}$-sufficient statistics.

Proposition 1. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. If an unbiased estimator $T\left(X\right)$ is unique for $\mathfrak{a}$, then $T\left(X\right)$ is an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$.

Proof. 

Obviously, $E_{\theta }\left[T\left(X\right)\mid T\left(X\right)\right]=T\left(X\right)$ almost surely $\mathcal{P}$ because of the definition of $\mathcal{H}$-sufficiency; cf. Casella and Berger [7] and Shao [8]. □

Proposition 2. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Let $T\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ such that $S\left(X\right)=g\left(T\left(X\right)\right)$ for $S\left(X\right)$, another statistic, and g, a one-to-one measurable function. Then, $S\left(X\right)$ is an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$.

Proof. 

Let $U\left(X\right)$ denote an unbiased estimator of $\mathfrak{a}$. Then, we have $E_{\theta }$$\left[U\left(X\right)\mid S\left(X\right)\right]$ = $E_{\theta }$$\left[U\left(X\right)\mid T\left(X\right)\right]$ almost surely $\mathcal{P}$, which shows that $E_{\theta }\left[U\left(X\right)\mid S\left(X\right)\right]$ is independent of $\theta$; cf. Casella and Berger [7]. □

Remark 2. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Let $S\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ and $U\left(X\right)$ another statistic such that $S\left(X\right)=g\left(U\left(X\right)\right)$ for a measurable function g. We expect $U\left(X\right)$ to be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$, but, actually, it is not. Consider Example 2 again: Let $S\left(X\right)=I_0\left(X\right)$ and $U\left(X\right)=1$, 0 and 2 for $x=0,-1$ and $x>1$, respectively. Then, verify that (i) $S\left(X\right)$ is an $\mathcal{H}$-sufficient statistic, (ii) $S\left(X\right)$ is a function of $U\left(X\right)$ but (iii) $U\left(X\right)$ is not an $\mathcal{H}$-sufficient statistic.

4. A Generalization of RBT and LST

We now apply the RBT for arbitrary $\mathcal{H}$-sufficient statistics for $\mathfrak{a}$ to obtain a better estimator.

Theorem 3. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Let $H\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. Let $\delta \left(X\right)$ be an unbiased estimator of a U-estimable $\mathfrak{a}$, and the loss function $L(\theta ,\delta (X\left)\right)$ be a strictly convex function of $\delta \left(X\right)$. Then, if $\delta \left(X\right)$ has finite expectation and risk, we have $R\left(\theta ,\delta \left(X\right)\right)=EL\left[\theta ,\delta \left(X\right)\right]<\infty$, and, if $\psi \left(\mathfrak{h}\right)=E\left[\delta \left(X\right)\mid H\left(X\right)=\mathfrak{h}\right]$, then the risk of the estimator $\psi \left(H\left(X\right)\right)$ satisfies $R\left(\theta ,\psi \left(H\left(X\right)\right)\right)<R\left(\theta ,\delta \left(X\right)\right)$ unless $\delta \left(X\right)=\psi \left(H\left(X\right)\right)$ almost surely $\mathcal{P}$.

Proof. 

Since L is convex, by Jensen’s inequality,

$$\begin{array}{c}\hfill {\displaystyle E\left(L\left[\theta ,\delta \left(X\right)\right]\mid H\left(X\right)\right)⩾L\left(\theta ,E\left[\delta \left(X\right)\mid H\left(X\right)\right]\right)}\end{array}$$

and

$$\begin{array}{c}\hfill {\displaystyle R\left[\theta ,\delta \left(X\right)\right]⩾R\left[\theta ,\psi \left(X\right)\right].}\end{array}$$

Hence, the result follows from Definition 8; see Lehmann and Casella [26] for details. □

We now reexpress Lemma 1.10 in Lehmann and Casella [26] within the new framework.

Lemma 1. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Let $H\left(X\right)$ be a complete $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. Then, every U-estimable $\mathfrak{a}$ has one and only one unbiased estimator that is a function of $H\left(X\right)$. Of course, uniqueness here means that any two such functions agree almost surely $\mathcal{P}$.

Proof. 

If $H^{\prime }\left(X\right)$ is another unbiased estimator, then $E\left[H\left(X\right)-H^{\prime }\left(X\right)\right]=0$. By the completeness property, $H\left(X\right)=H^{\prime }\left(X\right)$ with probability one; see Lehmann and Casella [26] for details. □

The generalization of LST (Lehmann and Scheffé [11], Theorem 5.1) by using a complete $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ is as follows.

Theorem 4. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Suppose that $H\left(X\right)$ is a complete $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. Then, we have the following:

(i) 

For every U-estimable $\mathfrak{a}$, there exists an unbiased estimator that uniformly minimizes the risk for any loss function $L(\theta ,\delta )$ that is convex in δ; therefore, the estimator is UMVUE of $\mathfrak{a}$.

(ii) 

The UMVU estimator of (i) is a unique unbiased estimator and is a function of $H\left(X\right)$; it has minimum risk, provided its risk is finite and $L(\theta ,\delta )$ is strictly convex in δ.

Proof. 

(i) If U is unbiased, by Theorem 3, we can consider the estimator of $E\left[U\mid H\left(X\right)\right]$ whose risk is less than the risk of U. (ii) If $U^{\prime }$ is another estimator with minimum risk, then $E\left[U^{\prime }\mid H\left(X\right)\right]$ must have less risk by Theorem 3, which would be impossible. Thus, by Lemma 1, $U=U^{\prime }$. □

Theorem 5. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. Let $T\left(X\right)$ be an unbiased estimator for $\mathfrak{a}$ and $H\left(X\right)$ an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ such that $T\left(X\right)=g\left(H\left(X\right)\right)$ for a measurable function g. Then, if and only if condition for $T\left(X\right)$ to be a UMVUE of $\mathfrak{a}$ is that $E_{\theta }\left[T\left(X\right)U^{*}\left(X\right)\right]=0$ for all $U^{*}\left(X\right)\in {\mathcal{U}}_0\left({\mathcal{H}}_{\mathfrak{a}}\right)$ and $\theta \in \mathsf{\Theta }$, where ${\mathcal{U}}_0\left({\mathcal{H}}_{\mathfrak{a}}\right)$ denotes the set of all unbiased estimators of 0.

Proof. 

Suppose that $U\left(X\right)\in {\mathcal{U}}_0$. The result follows because of $E_{\theta }$$\left[U\left(X\right)\mid H\left(X\right)\right]$∈${\mathcal{U}}_0$$\left({\mathcal{H}}_{\mathfrak{a}}\right)$ and

$$\begin{array}{c}\hfill {\displaystyle E_{\theta }\left[T\left(X\right)U\left(X\right)\right]=E_{\theta }\left\{E_{\theta }\left[g\left(H\left(X\right)\right)U\left(X\right)\mid H\left(X\right)\right]\right\}=E_{\theta }\left\{g\left(H\left(X\right)\right)E_{\theta }\left[U\left(X\right)\mid H\left(X\right)\right]\right\},}\end{array}$$

where $U\left(X\right)$ is an unbiased estimator of 0. $E_{\theta }\left[U\left(X\right)\mid H\left(X\right)\right]$ is a statistic since $E_{\theta }${$T\left(X\right)$−$\left[T\left(X\right)-U\left(X\right)\right]$∣$H\left(X\right)$} is independent of $\theta$. The converse follows because of

$$\begin{array}{c}\hfill {\displaystyle E_{\theta }\left\{g\left(H\left(X\right)\right)E_{\theta }\left[U\left(X\right)\mid H\left(X\right)\right]\right\}=E_{\theta }\left\{E_{\theta }\left[g\left(H\left(X\right)\right)U\left(X\right)\mid H\left(X\right)\right]\right\}=E_{\theta }\left[T\left(X\right)U\left(X\right)\right].}\end{array}$$

□

Theorem 6. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. Let $H\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. In addition, suppose for every unbiased estimator $T\left(X\right)$ for $\mathfrak{a}$ there is a measurable function g such that $T\left(X\right)=g\left(H\left(X\right)\right)$. Then, $T\left(X\right)$ is a UMVUE if $E_{\theta }\left[U\left(X\right)\mid H\left(X\right)\right]=0$ almost surely $P_{\theta }$ for every $U\left(X\right)\in {\mathcal{U}}_0$ and $\theta \in \mathsf{\Theta }$.

Proof. 

For $U\left(X\right)\in {\mathcal{U}}_0$, we have $E_{\theta }\left[T\left(X\right)U\left(X\right)\right]$ = $E_{\theta }\left\{g\left(H\left(X\right)\right)E\left[U\left(X\right)\mid H\left(X\right)\right]\right\}$ = 0 since $E_{\theta }$$\left[U\left(X\right)\mid H\left(X\right)\right]$ = 0 almost surely $P_{\theta }$. Since $E\left[T\left(X\right)U\left(X\right)\right]=0$, $T\left(X\right)$ is a UMVUE by the LST. □

5. Complete $\mathcal{H}$-Sufficient Statistic

We are interested in finding an $\mathcal{H}$-sufficient statistic with the simplest structure. A minimal $\mathcal{H}$-sufficient statistic is an $\mathcal{H}$-sufficient statistic that is a function of any other $\mathcal{H}$-sufficient statistic.

Definition 9 

(Minimal $\mathcal{H}$-sufficient statistics). Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. Let $T\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. A statistic $T\left(X\right)$ is a minimal $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ if and only if, for any other statistic $S\left(X\right)$ that is an $\mathcal{H}$-sufficient for $\mathfrak{a}$, there exists a measurable function ψ such that $T\left(X\right)=\psi \left(S\left(X\right)\right)$ almost surely $\mathcal{P}$.

Theorem 7. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$, $\theta \in \mathsf{\Theta }$. Let $T\left(X\right)$ be a complete sufficient statistic for $\mathcal{P}$ (or θ) such that $T\left(X\right)$, $T:\mathcal{X}⟶\mathbb{R}$, has mean $\mathfrak{a}$. Then, any $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$ is a sufficient statistic for $\mathcal{P}$ (or θ).

Proof. 

Let $H\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. By Theorem 3, var $\left\{E\left[\left(T\right(X)\mid H(X)\right]\right\}$≤ var$\left[T\left(X\right)\right]$. Since $T\left(X\right)$ is a UMVUE, $T\left(X\right)=E\left[T\left(X\right)\mid H\left(X\right)\right]$ almost surely $\mathcal{P}$ because there can be no better estimators. So, we can find a measurable function g such that $T\left(X\right)=g\circ H\left(X\right)$ almost surely $\mathcal{P}$. Hence, $H\left(X\right)$ is a sufficient statistic. □

Thus, we can apply $\mathcal{H}$-sufficient statistics for $\mathfrak{a}$ in case complete sufficient statistics do not exist. Intuitively, an $\mathcal{H}$-sufficient statistic with the complete property will be a minimal $\mathcal{H}$-sufficient statistic. The following theorem, a version of Bahadur’s theorem, see Bahadur [27], states an important property of minimal $\mathcal{H}$-sufficient statistics.

Theorem 8. 

Let $X=\left[X_1,\dots ,X_n\right]$, where $X_1,\dots ,X_n$ are random variables from an unknown population $P_{\theta }\in \mathcal{P}$. If $T\left(X\right)$, $T:\mathcal{X}⟶\mathbb{R}$, is a complete $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$, then $T\left(X\right)$ is a minimal $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$.

Proof. 

Let $S\left(X\right)$ be an $\mathcal{H}$-sufficient statistic for $\mathfrak{a}$. By Theorem 3, $T\left(X\right)=E\left[T\left(X\right)\mid S\left(X\right)\right]$ almost surely $\mathcal{P}$ since $T\left(X\right)$ is a UMVUE. □

We now show that complete $\mathcal{H}$-sufficient statistics may not exist.

Example 3 

(Complete $\mathcal{H}$-sufficient statistics may not exist). Let X be a random variable with $\mathcal{P}=\left\{\mathrm{Bin}(\theta ,0.5):\theta \in \{1,2,\dots \}\right\}$, and then X is not complete. $k{(-1)}^{X+1}$, $k\in \mathbb{R}$ are all of the zero unbiased estimators. Since X is sufficient, X is an $\mathcal{H}$-sufficient statistic for θ (see Theorem 2, part a). However, a complete $\mathcal{H}$-sufficient statistic for θ does not exist. Otherwise, for every $k\in \mathbb{R}$ and some $k_0\in \mathbb{R}$, we would have $E\left[2X+k{(-1)}^{X+1}\mid g\left(X\right)\right]=2X+k_0{(-1)}^{X+1}$ almost surely $\mathcal{P}$, where $g\left(X\right)$ is assumed to be a complete $\mathcal{H}$-sufficient statistic for θ, but this cannot hold since $2X+k_0{(-1)}^{X+1}$ is not UMVUE for θ. Since

$$\begin{array}{c}\hfill {\displaystyle E\left[\left(2X+k{(-1)}^{X+1}\right)\left(k_0{(-1)}^{X+1}\right)\right]=\left\{\begin{array}{cc}{\displaystyle k_0(k+1),}\hfill & if \theta =1,\hfill \\ {\displaystyle k_{0}k,}\hfill & if \theta \ge 2,\hfill \end{array}\right.}\end{array}$$

there is no $k_0$ such that $E\left[\left(2X+k{(-1)}^{X+1}\right)\left(k_0{(-1)}^{X+1}\right)\right]=0$. $2X+k{(-1)}^{X+1}$, $k\in \mathbb{R}$ are all of the unbiased estimators.

6. Some Applications

In this section, some examples are presented for which Theorems 3 and 4 are applicable.

6.1. When the Minimal Sufficient Statistic Is Not Complete

Consider a case where UMVUE exists but the minimal sufficient statistics are not complete. The LST cannot be used to obtain UMVUEs. We illustrate through some examples that we can find a UMVUE without having complete sufficiency. Therefore, some worries in the literature on the inadequacy of the LST and RBT for obtaining UMVUEs can be removed, and seemingly unbeatable obstacles can be overcome by using $\mathcal{H}$-sufficient statistics.

Example 4 

(Example of Lehmann and Scheffé [11]). Let X be a discrete random variable with $P_{\theta }(X=-1)=\theta$ and $P_{\theta }(X=k)={(1-\theta )}^2{\theta }^k$, $k=0,1,2,\dots$, where $\theta \in (0,1)$ is unknown. $I_{\left\{0\right\}}\left(X\right)$ is a complete and minimal $\mathcal{H}$-sufficient statistic for ${(1-\theta )}^2$ because $E_{\theta }\left[I_{\left\{0\right\}}\left(X\right)+\alpha X\mid I_{\left\{0\right\}}\left(X\right)\right]=I_{\left\{0\right\}}\left(X\right)$ almost surely $P_{\theta }$ for every $\theta \in (0,1)$ and $\alpha \in \mathbb{R}$. On the other hand, $I_{\left\{0\right\}}\left(x\right)$ has Bernoulli distribution so is complete. Hence, by Theorem 4, $I_{\left\{0\right\}}\left(X\right)$ is a UMVUE for ${(1-\theta )}^2$, so every function $A{I}_{\left\{0\right\}}\left(X\right)+B$ is also a UMVUE for $A{(1-\theta )}^2+B$.

Alternatively, for every $\theta \in (0,1)$ and $\alpha \in \mathbb{R}$, we have $E_{\theta }\left[\alpha X\mid I_{\left\{0\right\}}\left(X\right)\right]=0$ almost surely $P_{\theta }$, and thus the same result can be obtained by using Theorem 6.

So far, Examples 1 and 2 have shown usefulness of $\mathcal{H}$-sufficiency. However, in both cases, the considered estimation problem is a rather esoteric one. The following examples are more reasonable.

Example 5. 

Let $X_1,\dots ,X_n$ be independent and identical random variables with

$$\begin{array}{c}\hfill {\displaystyle f(x;\mu ,\sigma )=\frac{x-\mu }{{\sigma }^2}{e}^{-\frac{x-\mu }{\sigma }}{I}_{(\mu ,\infty )}\left(x\right),}\end{array}$$

(2)

where $\theta =(\mu ,\sigma )\in \mathbb{R}\times {\mathbb{R}}^{+}$ is an unknown parameter.

Suppose that μ is known. Then, $\overline{X}$ is a complete sufficient statistic for σ since (2) is from the exponential family and σ is a scale parameter (σ does not denote variance, $E\left(X\right)=\mu +2\sigma )$. By the RBT and since $\overline{X}$ is a UMVUE, $\overline{X}$ is an $\mathcal{H}$-sufficient statistic for $\mu +2\sigma$ since $E_{\theta }\left[\delta \left(X\right)|\overline{X}\right]=\overline{X}$ almost surely $P_{\theta }$ for every $\delta \left(X\right)\in {\mathcal{U}}_{\mu +2\sigma }$.

Suppose now μ is unknown. Since $E_{\theta }\left[\delta \left(X\right)|\overline{X}\right]$ is free of parameters, $\overline{X}$ is a complete and minimal $\mathcal{H}$-sufficient statistic for $\mu +2\sigma$. On the other hand, by Theorem 4, $\overline{X}$ is a UMVUE for $\mu +2\sigma$ and so is any function of $\overline{X}$.

Example 6. 

Let $X_1,\dots ,X_n$ be independent and identical random variables with

$$\begin{array}{c}\hfill {\displaystyle f(x;\mu ,\sigma )=2\frac{x-\mu }{{\sigma }^2}{I}_{(\mu ,\mu +\sigma )}\left(x\right),}\end{array}$$

where $\theta =(\mu ,\sigma )\in \mathbb{R}\times {\mathbb{R}}^{+}$ is an unknown parameter. By arguments in Example 5, taking μ to be known, we can see that $max\left(X_1,\dots ,X_n\right)$ is a complete and minimal sufficient statistic for $\mu +\frac{2n}{2n+1}\sigma$ and UMVUE. So, $max\left(X_1,\dots ,X_n\right)$ is a complete and minimal $\mathcal{H}$-sufficient statistic for $\mu +\frac{2n}{2n+1}\sigma$ (see Theorem 2, part a). If μ is unknown, then $max\left(X_1,\dots ,X_n\right)$ is a complete and minimal $\mathcal{H}$-sufficient statistic for $\mu +\frac{2n}{2n+1}\sigma$. Since $E_{\theta }\left[\delta \left(X\right)|max\left(X_1,\dots ,X_n\right)\right]$ is free of parameters, $max\left(X_1,\dots ,X_n\right)$ is a complete and minimal $\mathcal{H}$-sufficient statistic for $\mu +\frac{2n}{2n+1}\sigma$. Hence, by Theorem 4, $max\left(X_1,\dots ,X_n\right)$ is a UMVUE for $\mu +\frac{2n}{2n+1}\sigma$ and so is any function of $max\left(X_1,\dots ,X_n\right)$.

6.2. When a Complete and Sufficient Statistic Is Not Available

Even though complete sufficient statistics do exist in the following examples, namely max$\left[1,max\left(X_1,\dots ,X_n\right)\right]$ and X$I_{\mathbb{N}\setminus \left\{m,m+1\right\}}$$\left(X\right)$, we can apply Theorems 3 and 4 for obtaining their UMVUEs.

Example 7 

(Example of Shao [8]). Let $X_1,\dots ,X_n$ be independent and identical uniform random variables on $(0,\theta )$ with $\mathsf{\Theta }=[1,\infty )$. Then, $X_{\left(n\right)}$ is not complete but sufficient for θ. Thus, the RBT and LST are not applicable. We now apply Theorem 3 to find a UMVUE of θ. Let $U\left(X_{\left(n\right)}\right)$ denote an unbiased estimator of 0 in ${\mathcal{U}}_0\left(X_{\left(n\right)}\right)$.

We can show that $H\left(X_{\left(n\right)}\right)=I_{[0,1]}\left(X_{\left(n\right)}\right)+\frac{n+1}{n}{X}_{\left(n\right)}{I}_{(1,\infty )}\left(X_{\left(n\right)}\right)$ is a complete and $\mathcal{H}$-sufficient statistic for θ, although we need only

$$\begin{array}{c}\hfill {\displaystyle E_{\theta }\left[I_{[0,1]}\left(X_{\left(n\right)}\right)+\frac{n+1}{n}{X}_{\left(n\right)}{I}_{(1,\infty )}\left(X_{\left(n\right)}\right)+U\left(X_{\left(n\right)}\right)\mid H\left(X_n\right)\right]=H\left(X_{\left(n\right)}\right)}\end{array}$$

almost surely $P_{\theta }$ for every $\theta \in \mathsf{\Theta }$. Here, $U\left(X_{\left(n\right)}\right)$ is an arbitrary nonzero unbiased estimator of zero, so its conditional expectation is zero too; that is, $E_{\theta }\left[U\left(X_{\left(n\right)}\right)\mid H\left(X_n\right)\right]=0$.

By definition of zero unbiasedness,

$$\begin{array}{c}\hfill {\displaystyle {\int }_0^{1}U\left(x\right)x^{n-1}dx+{\int }_1^{\theta }U\left(x\right)x^{n-1}dx=0,}\end{array}$$

so

$$\begin{array}{c}\hfill {\displaystyle {\int }_0^{1}U\left(x\right)x^{n-1}dx=0}\end{array}$$

and $U\left(x\right)=0$ for every $\theta \ge 1$. Hence, $H\left(X_{\left(n\right)}\right)$ = $E\left[H\left(X_{\left(n\right)}\right)\mid H\left(X_{\left(n\right)}\right)\right]$+$E\left[U\left(X_{\left(n\right)}\right)\mid \right.$$\left.H\left(X_{\left(n\right)}\right)\right]$ and $I_{[0,1]}\left(X_{\left(n\right)}\right)+\frac{n+1}{n}{X}_{\left(n\right)}{I}_{(1,\infty )}\left(X_{\left(n\right)}\right)$ is a UMVUE for θ.

Example 8 

(Example of Stigler [28]). Let X be a discrete random variable distributed as

$$\begin{array}{c}\hfill {\displaystyle P(X=x)=\left\{\begin{array}{cc}{\displaystyle N^{-1},}\hfill & \mathit{if}x=1,\dots ,N,\hfill \\ {\displaystyle 0,}\hfill & \mathit{otherwise},\hfill \end{array}\right.}\end{array}$$

where N is an unknown parameter.

We have excluded $N=m$ for fixed $m\ge 1$ from $\left\{P_N:N\ge 1\right\}$. Let P= $\left\{P_N:N\ge 1,N\ne m\right\}$. Then, X is not complete but sufficient for N. Consider the following function

$$\begin{array}{c}\hfill {\displaystyle U\left(X\right)=\left\{\begin{array}{cc}{\displaystyle k,}\hfill & \mathit{if}x=m,\hfill \\ {\displaystyle -k,}\hfill & \mathit{if}x=m+1,\hfill \\ {\displaystyle 0,}\hfill & \mathit{otherwise}.\hfill \end{array}\right.}\end{array}$$

It can be easily checked that $E\left[U\left(X\right)\right]=0$. Thus, the RBT and LST are not applicable.

We now apply Theorem 3 to find a UMVUE of N. We can see that

$$\begin{array}{c}\hfill {\displaystyle H\left(X\right)=\left\{\begin{array}{cc}{\displaystyle 2X-1,}\hfill & \mathit{if}X\ne m,X\ne m+1,\hfill \\ {\displaystyle 2m,}\hfill & \mathit{if}X=m,X=m+1\hfill \end{array}\right.}\end{array}$$

is a complete and $\mathcal{H}$-sufficient statistic for N. Here, $H\left(X\right)$ is also a UMVUE for N, which can be proved similarly since $E\left[H\left(X\right)+U\left(X\right)\mid H\left(X\right)\right]=E\left[H\left(X\right)\mid H\left(X\right)\right]+E\left[U\left(X\right)\mid H\left(X\right)\right]=H\left(X\right)+0=H\left(X\right)$.

6.3. A Note on the Structure of UMVUE

We now show that the structure of UMVUEs depends on $\mathcal{H}$-sufficient statistics for $E\left(\mathrm{UMVUE}\right)$.

Theorem 9. 

Let $\mathcal{P}=\left\{P_{\theta }:\theta \in \mathsf{\Theta }\right\}$. Let $S\left(X\right)$ be a sufficient statistic for $\mathcal{P}$ and an $\mathcal{H}$-sufficient statistic $H\left(X\right)$ for $\mathfrak{a}$. For any function such as $\alpha \left(S\left(X\right)\right)$ that is a UMVUE, there exists a function $\beta \left(H\left(X\right)\right)$ so that $\alpha \left(S\left(X\right)\right)=\beta \left(H\left(X\right)\right)$ almost surely $\mathcal{P}$.

Proof. 

The proof is an easy consequence of Theorem 3, where $E_{\theta }$$\left[\mathrm{UMVUE}\mid S\left(X\right)\right]$ and $E_{\theta }$$\left[\mathrm{UMVUE}\mid H\left(X\right)\right]$ are UMVUEs. The proof follows by uniqueness of UMVUEs because $E[$ UMVUE $\mid S\left(X\right)]$ is a function of $S\left(X\right)$ and UMVUE, and $E[$ UMVUE $\mid H\left(X\right)]$ is a function of $H\left(X\right)$ and UMVUE. □

7. Conclusions

Sufficient statistics are of central concern for statisticians. They play a fundamental role in Rao–Blackwell and Lehmann–Scheffé theorems. By Theorem 3, every sufficient statistic is an $\mathcal{H}$-sufficient statistic. The class of $\mathcal{H}$-sufficient statistics contains all of the sufficient statistics and also some statistics that are not necessarily sufficient. So, the factorization theorem, and its corollaries, should not hold generally for $\mathcal{H}$-sufficient statistics. The concepts closest to $\mathcal{H}$-sufficient statistics are those of “partial sufficient” and “sufficient subspace”. However, they are slightly different.

When a complete sufficient statistic is lacking, there may sometimes be nonconstant parametric functions that can be UMVU-estimated. This fact is seldom pointed out and exemplified in undergraduate and graduate textbooks. In this note, we have shown how the concept of $\mathcal{H}$-sufficient statistics can be used to obtain UMVUEs in these contexts.

More research based on the concept of $\mathcal{H}$-sufficiency are under investigation. They are

• Generalizing $\mathcal{H}$-sufficiency to multi-parameter cases.
• How to find $\mathcal{H}$-sufficient statistics.