Efficiency Bound of Local Z-Estimators on Discrete Sample Spaces

Abstract

Many statistical models over a discrete sample space often face the computational difficulty of the normalization constant. Because of that, the maximum likelihood estimator does not work. In order to circumvent the computation difficulty, alternative estimators such as pseudo-likelihood and composite likelihood that require only a local computation over the sample space have been proposed. In this paper, we present a theoretical analysis of such localized estimators. The asymptotic variance of localized estimators depends on the neighborhood system on the sample space. We investigate the relation between the neighborhood system and estimation accuracy of localized estimators. Moreover, we derive the efficiency bound. The theoretical results are applied to investigate the statistical properties of existing estimators and some extended ones.

1. Introduction

For many statistical models on a discrete sample space, the computation of the normalization constant is often intractable. Because of that, the maximum likelihood estimator (MLE) is not of practical use to estimate probability distributions, although the MLE has some nice theoretical properties such as the statistical consistency and efficiency under some regularity conditions [1].

In order to circumvent the computation of the normalization constant, alternative estimators that require only a local computation over the sample space have been proposed. In this paper, estimators on the basis of such a concept are called localized estimators. Examples of localized estimators include pseudo-likelihood [2], composite likelihood [3,4], ratio matching [5,6], proper local scoring rules [7,8], and many others. These estimators are used for discrete statistical models such as conditional random fields [9], Boltzmann machines [10], restricted Boltzmann machines [11], discrete exponential family harmoniums [12], and Ising models [13].

In this paper, we present a theoretical analysis of localized estimators. We use the standard tools in the statistical asymptotic theory. In our analysis, a class of localized estimators including pseudo-likelihood and composite likelihood is treated as M-estimator or Z-estimator which is an extension of the MLE [1]. The localized estimators require local computation around a neighborhood of observed points. Hence, the asymptotic variance of the localized estimator depends on the size of the neighborhood. We investigate the relation between the estimation accuracy and the neighborhood system. A similar result is given by [14], in which asymptotic variances between specific composite likelihoods are compared. In our approach, we consider a stochastic variant of localized estimators, and derive a general result that the larger neighborhood leads to more efficient estimator under a simple condition. The pseudo-likelihood and composite likelihood are obtained as the expectation of a stochastic localized estimator. We derive the exact efficiency bound for the expected localized estimator. As far as we know, the derivation of the efficiency bound is a new result for a class of localized estimators, though upper and lower bounds have been proposed [14] for some localized estimators.

The rest of the paper is organized as follows. In Section 2, we introduce basic concepts such as pseudo-likelihood, composite likelihood, and Z-estimators. Section 3 is devoted to define stochastic local Z-estimator associated with a neighborhood system over the discrete sample space. In Section 4, we study the relation between the neighborhood system and asymptotic efficiency of the stochastic local Z-estimator. In Section 5, we define local Z-estimator as the expectation of the stochastic local Z-estimator, and present its efficiency bound. The theoretical results are applied to study asymptotic properties of existing estimators and some extended ones. Finally, Section 6 concludes the paper with discussions.

2. Preliminaries

M-estimators and Z-estimators were proposed as an extension of the MLE. In practice, M-estimators and Z-estimators are often computationally demanding due to the normalization constant in statistical models. To circumvent computational difficulty, localized estimators have been proposed. We introduce some existing localized estimators especially on discrete sample spaces. In later sections, we consider statistical properties of a localized variant of Z-estimators.

Let us summarize the notations to be used throughout the paper. Let $\mathbb{R}$ be the set of all real numbers. The discrete sample space is denoted as $\mathcal{X}$. The statistical model $p_{\theta }(x)$ for $x\in \mathcal{X}$ with the parameter $\theta \in \mathrm{\Theta }\subset {\mathbb{R}}^d$ is also expressed as $p(x;\theta )$. The vector a usually denotes the column vector, and ${·}^T$ denotes the transposition of vector or matrix. For a linear space T and an integer d, ${(T)}^d$ denotes the d-fold product space of T, and the element $\mathit{c}\in {(T)}^d$ is expressed as $\mathit{c}=(c_1,\dots ,c_d)$. The product space of two subspaces $T_1$ and $T_2$ that are orthogonal to each other is denoted as $T_1\oplus T_2$. For the function $f(\theta )$ of the parameter $\theta \in {\mathbb{R}}^d$, $\nabla f$ denotes the gradient vector ${(\frac{\partial f}{\partial {\theta }_1},\dots ,\frac{\partial f}{\partial {\theta }_d})}^T$. The indicator function is denoted as $\mathbf{1}[A]$ that takes 1 if A is true and 0 otherwise.

2.1. M- and Z-Estimators

Suppose that samples $x_1,\dots ,x_m$ are i.i.d. distributed from the probability $p(x)$ over the discrete sample space $\mathcal{X}$. A statistical model $p_{\theta }(x)=p(x;\theta )$ with the parameter $\theta \in \mathrm{\Theta }\subset {\mathbb{R}}^d$ is assumed to estimate $p(x)$. In this paper, our concern is the statistical efficiency of estimators. Hence, we suppose that the statistical model includes $p(x)$.

The MLE is commonly used to estimate $p(x)$. It uses the negative log-likelihood of the model, $-log{p}_{\theta }(x)$, as the loss function and the estimator is given by the minimum solution of its empirical mean, $-\frac{1}{m}{\sum }_{i=1}^{m}log{p}_{\theta }(x_i)$.

Generally, the estimator obtained by the minimum solution of a loss function is referred to as the M-estimator. The MLE is an example of M-estimators. When the loss function is differentiable, the gradient of the loss function vanishes at the estimated parameter. Instead of minimizing loss function, a solution of the system of equations also provides an estimator of the parameter. Such an estimator is called the Z-estimator [1]. In the MLE, the system of equations is given as

$$\begin{array}{c}\hfill \frac{1}{m}\sum _{i=1}^m\nabla log{p}_{\theta }(x_i)=\mathbf{0},\end{array}$$

where $\mathbf{0}\in {\mathbb{R}}^d$ is the null-vector. The gradient $\nabla log{p}_{\theta }(x)$ is known as the score function of the model $p_{\theta }(x)$. In this paper, the score function is denoted as

$$\begin{array}{c}\hfill {\mathit{u}}_{\theta }(x)=\nabla log{p}_{\theta }(x).\end{array}$$

In general, the Z-estimator is defined as the solution of the system of equations

$$\begin{array}{c}\hfill \frac{1}{m}\sum _{i=1}^m{\mathit{f}}_{\theta }(x_i)=\mathbf{0},\end{array}$$

where the ${\mathbb{R}}^d$-valued function ${\mathit{f}}_{\theta }(x)=\mathit{f}(x;\theta )$ is referred to as the identification function [15,16]. In the M-estimator, the identification function is given as the gradient of the loss function. In general, however, the identification function itself is not necessarily expressed as the gradient of a loss function, if it is not integrable. The identification function ${\mathit{f}}_{\theta }(x)$ is also called Z-estimator with some abuse of terminology.

2.2. Localized Estimators

Below, let us introduce some localized estimators. The statistical model defined on the discrete set $\mathcal{X}$ is denoted by

$$\begin{array}{c}\hfill p_{\theta }(x)=\frac{{\tilde{p}}_{\theta }(x)}{Z_{\theta }}\end{array}$$

(1)

for $x\in \mathcal{X}$, where $Z_{\theta }$ is the normalization constant at the parameter θ, i.e.,

$$\begin{array}{c}\hfill Z_{\theta }=\sum _{x\in \mathcal{X}}{\tilde{p}}_{\theta }(x).\end{array}$$

Throughout the paper, we assume $p_{\theta }(x)>0$ for all $x\in \mathcal{X}$ and all $\theta \in \mathrm{\Theta }\subset {\mathbb{R}}^d$.

Example 1 (Pseudo-likelihood). 

Suppose that $\mathcal{X}$ is expressed as the product space $\mathcal{X}={\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_n$, where ${\mathcal{X}}_1,\dots ,{\mathcal{X}}_n$ are finite sets such as $\{0,1\}$. For $x=(x_1,\dots ,x_n)\in \mathcal{X}$, let $x_{-k}$ be the $n-1$ dimensional vector defined by dropping the k-th element of x. The loss function of the pseudo-likelihood, $S_{\mathrm{PS}}$, is defined as the negative log-likelihood of the conditional probability $p_{\theta }(x_k|x_{-k})$ defined from $p_{\theta }(x)$, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{PS}}(x,p_{\theta })=-\sum _{k=1}^{n}log{p}_{\theta }(x_k|x_{-k})=-\sum _{k=1}^n\{log{\tilde{p}}_{\theta }(x)-log(\sum _{x_k\in {\mathcal{X}}_k}{\tilde{p}}_{\theta }(x))\}.\end{array}$$

(2)

The pseudo-likelihood does not require the normalization constant, and it satisfies the statistical consistency of the parameter estimation [2,17]. The identification function of the corresponding Z-estimator is obtained by the gradient vector of the loss Function (2).

Example 2 (Composite likelihood). 

The composite likelihood was proposed as an extension of the pseudo-likelihood [3]. Suppose that $\mathcal{X}$ is expressed as the product space as in Example 1. For the index subset $A\subset \{1,\dots ,n\}$, let $x_A={(x_i)}_{i\in A}$ be the subvector of $x\in \mathcal{X}$. For each $\ell =1,\dots ,M$, Suppose that $A_{\ell }$ and $B_{\ell }$ are a pair of disjoint subsets in $\{1,\dots ,n\}$, and let $C_{\ell }$ be the complement of the union $A_{\ell }\cup B_{\ell }$, i.e., $C_{\ell }={(A_{\ell }\cup B_{\ell })}^c$. Given positive constants ${\gamma }_1,\dots ,{\gamma }_M$, the loss function of the composite likelihood, $S_{\mathrm{CL}}$, is defined as

$$\begin{array}{c}\hfill S_{\mathrm{CL}}(x,p_{\theta })=-\sum _{\ell =1}^M{\gamma }_{\ell }log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})=-\sum _{\ell =1}^M{\gamma }_{k}log\{\sum _{x_{C_{\ell }}}{\tilde{p}}_{\theta }(x)-\sum _{x_{B_{\ell }^c}}{\tilde{p}}_{\theta }(x)\}.\end{array}$$

The composite likelihood using the subsets $A_{\ell }=\{\ell \},B_{\ell }=A_{\ell }^c$ and positive constant ${\gamma }_{\ell }=1$ for $\ell =1,\dots ,n$ yields the pseudo-likelihood. As well as the pseudo-likelihood, the composite likelihood has the statistical consistency under some regularity condition [4].

Originally, the pseudo and composite likelihoods were proposed to deal with spatial data [2,3]. As a generalization of these estimators, a localized variant of scoring rules works efficiently to the statistical analysis of discrete spatial data [18].

3. A Stochastic Variant of Z-Estimators

In this section, we define a stochastic variant of Z-estimators. For the discrete sample space $\mathcal{X}$, suppose that the neighborhood system N is defined as a subset of the power set $2^{\mathcal{X}}$, i.e., N is a family of subsets in $\mathcal{X}$. Let us define the neighborhood system at $x\in \mathcal{X}$ by $N_x=\{e\in N|x\in e\}$. We assume that $N_x$ is not empty for any x. In some class of localized estimators, the neighborhood system is expressed using an undirected graph on $\mathcal{X}$ [7]. In our setup, the neighborhood system is not necessarily expressed by an undirected graph, and we allow the neighborhood system to possess multiple neighbors at each point x.

Let us define the stochastic Z-estimator. A conditional probability of the set $e\in N$ given $x\in \mathcal{X}$ is denoted as $q(e|x)$. We assume that $q(e|x)=0$ if $e\notin N_x$ throughout the paper. Given a sample x, we randomly generate a neighborhood e from the conditional probability $q(e|x)$. Using i.i.d. copies of $(x,e)$, we estimate $p(x)$. Here, the statistical model $p_{\theta }(x)$ of the form (1) is used. We use the Z-estimator ${\mathit{f}}_{\theta }(x,e)=\mathit{f}(x,e;\theta )\in {\mathbb{R}}^d$ to estimate the parameter $\theta \in \mathrm{\Theta }\subset {\mathbb{R}}^d$. The element of ${\mathit{f}}_{\theta }(x,e)$ is denoted as ${\mathit{f}}_{\theta ,k}(x,e)$ or $f_k(x,e;\theta )$ for $k=1,\dots ,d$. The expectation under the probability $p_{\theta }(x)q(e|x)$ is written as ${\mathbb{E}}_{\theta ,q}[·]$. Suppose that the equality

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[{\mathit{f}}_{\theta }]=\mathbf{0}\end{array}$$

(3)

holds for all $\theta \in \mathrm{\Theta }$. In addition, we assume that the vectors ${\mathbb{E}}_{\theta ,q}[\nabla f_{\theta ,k}],k=1,\dots ,d$ are linearly independent, meaning that ${\mathit{f}}_{\theta }$ depends substantially on the parameter θ [19]. The solution of the system of equations

$$\begin{array}{c}\hfill \frac{1}{m}\sum _{i=1}^m{\mathit{f}}_{\theta }(x_i,e_i)=\mathbf{0}\end{array}$$

(4)

produces a statistically consistent estimator under some regularity condition [1]. In the parameter estimation of the model $p_{\theta }(x)$, the stochastic Z-estimator is defined as the solution of (4) using the identification function satisfying (3). As shown in Section 5, stochastic Z-estimators are useful to investigate statistical properties of the standard pseudo-likelihood and composite likelihood in Examples 1 and 2.

The computational tractability of the stochastic Z-estimator is not necessarily guaranteed. The MLE using the score function ${\mathit{f}}_{\theta }(x,e)={\mathit{u}}_{\theta }(x)$ is regarded as a stochastic Z-estimator for any $q(e|x)$ and it may not be computationally tractable because of the normalizing constant. As a class of computationally efficient estimators, let us define the stochastic local Z-estimator as the stochastic Z-estimator using ${\mathit{f}}_{\theta }(x,e)$ satisfying

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[{\mathit{f}}_{\theta }|e]=\mathbf{0}\end{array}$$

(5)

for any neighborhood $e\in N$, where ${\mathbb{E}}_{\theta ,q}[·|e]$ is the conditional expectation given e. The conditional probability $p(x|e)$ of $p_{\theta }(x)q(e|x)$ can take a positive value only when $x\in e$. Hence, ${\mathit{f}}_{\theta }(x,e)$ depends only on the neighborhood of x and its computation will be tractable.

Example 3 (Stochastic pseudo-likelihood). 

Let us define the stochastic variant of the pseudo-likelihood estimator. On the sample space $\mathcal{X}={\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_n$, the neighborhood system $N_x$ at $x=(x_1,\dots ,x_n)\in \mathcal{X}$ is defined as $N_x=\{e_{x,k}|k=1,\dots ,n\}$, where $e_{x,k}\subset \mathcal{X}$ is given as

$$\begin{array}{c}\hfill e_{x,k}=\{(x_1,\dots ,x_{k-1},z_k,x_{k+1},\dots ,x_n)|z_k\in {\mathcal{X}}_k\}.\end{array}$$

In order to estimate the parameters in complex models such as conditional random fields and Boltzmann machines, the union, ${\cup }_{e\in N_x}e$, is often used as the neighborhood at x [2,9,17]. Let the conditional probability $q(e|x)$ on $N_x$ as $q(e_{x,k}|x)=q_k,k=1,\dots ,n$, where $q_1,\dots ,q_n$ are positive numbers satisfying ${\sum }_{k=1}^n{q}_k=1$. The identification function of the stochastic pseudo-likelihood is defined by

$$\begin{array}{c}\hfill {\mathit{f}}_{\theta }(x,e)=\nabla log\frac{p_{\theta }(x)}{{\sum }_{z\in e}{p}_{\theta }(z)}\end{array}$$

for $e\in N_x$. Then, ${\mathit{f}}_{\theta }(x,e_{x,k})$ is equal to $\nabla log{p}_{\theta }(x_k|x_{-k})$. The conditional probability $p(x|e)$ derived from $p_{\theta }(x)q(e|x)$ is given as

$$\begin{array}{c}\hfill p(x|e_{x,k})=\frac{p_{\theta }(x)q(e_{x,k}|x)}{{\sum }_{z\in e_{x,k}}{p}_{\theta }(z)q(e_{x,k}|z)}=\frac{p_{\theta }(x)q_k}{{\sum }_{z\in e_{x,k}}{p}_{\theta }(z)q_k}=p_{\theta }(x_k|x_{-k}),\end{array}$$

where we used the equality $e_{x,k}=e_{z,k}$ for $z\in e_{x,k}$. Hence, the equality (5) holds for any ${(q_k)}_{k=1,\dots ,n}$. When $q_k$ depends on x, $p(x|e_{x,k})$ is different from $p_{\theta }(x_k|x_{-k})$ in general.

Example 4 (Stochastic variant of composite likelihood). 

Let us introduce a stochastic variant of composite likelihood on the sample space $\mathcal{X}={\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_n$. Below, notations in Example 2 are used. Let us define $e_{x,\ell }^{\prime },\ell =1,\dots ,M$ by the subset $e_{x,\ell }^{\prime }=\{y\in \mathcal{X}|y_{B_{\ell }}=x_{B_{\ell }}\}$, and the neighborhood system $N_x^{\prime }$ by $N_x^{\prime }=\{e_{x,\ell }^{\prime }|\ell =1,\dots ,M\}$. We assume that the map from ℓ to $B_{\ell }$ is one to one. In other words, the disjoint subsets $A_{\ell },B_{\ell },C_{\ell }$ can be specified from the neighborhood $e_{x,\ell }^{\prime }$. Suppose that the conditional probability $q^{\prime }(e^{\prime }|x)$ on $N_x^{\prime }$ is defined as $q^{\prime }(e_{x,\ell }^{\prime }|x)=q_{\ell }^{\prime }$ for $\ell =1,\dots ,M$, where $q_1^{\prime },\dots ,q_M^{\prime }$ are positive numbers satisfying ${\sum }_{\ell =1}^M{q}_{\ell }^{\prime }=1$. As well as Example 3, we see that the conditional probability $q^{\prime }(x|e_{x,\ell }^{\prime })$ defined from $p_{\theta }(x)q^{\prime }(e^{\prime }|x)$ is given as $p_{\theta }(x_{A_{\ell }},x_{C_{\ell }}|x_{B_{\ell }})$. Let us consider the identification function,

$${\mathit{f}}_{\theta }(x,e_{x,\ell }^{\prime })=\nabla log\frac{{\sum }_{x_{C_{\ell }}}{p}_{\theta }(x)}{{\sum }_{x_{B_{\ell }^c}}{p}_{\theta }(z)},$$

(6)

which is nothing but $\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})$. Then, (5) holds under the joint probability $p_{\theta }(x)q^{\prime }(e^{\prime }|x)$. Indeed, we have

$${\mathbb{E}}_{\theta ,q^{\prime }}[{\mathit{f}}_{\theta }|e_{x,\ell }^{\prime }]=\sum _{x_{A_{\ell }},x_{C_{\ell }}}{p}_{\theta }(x_{A_{\ell }},x_{C_{\ell }}|x_{B_{\ell }})\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})=\mathbf{0}$$

for any ${(q_{\ell }^{\prime })}_{\ell =1,\dots ,M}$. In this paper, the Z-estimator using (6) is called the reduced stochastic composite likelihood (reduced-SCL). The stochastic composite likelihood proposed in [20] is a randomized extension of the above ${\mathit{f}}_{\theta }(x,e^{\prime })$. Let $z=(z_1,\dots ,z_M)$ be a binary random vector taking an element of ${\{0,1\}}^M$, and ${\alpha }_1,\dots ,{\alpha }_M$ be positive constants. The SCL is defined as the Z-estimator obtained by

$$\begin{array}{c}\hfill \mathit{f}(x,z;\theta )=\sum _{\ell =1}^M{\alpha }_{\ell }{z}_{\ell }\nabla logp(x_{A_{\ell }}|x_{B_{\ell }};\theta ).\end{array}$$

The statistical consistency and the normality of the SCL is shown in [20].

4. Neighborhood Systems and Asymptotic Variances

We consider the relation between neighborhood systems and statistical properties of stochastic local Z-estimators.

4.1. Tangent Spaces of Statistical Models

At the beginning, let us introduce some geometric concepts to investigate statistical properties of localized estimators. These concepts are borrowed from information geometry [21]. For the neighborhood system N with the conditional probability $q(e|x)$, let us define the linear space $T_{\theta ,q}$ as

$$\begin{array}{c}\hfill T_{\theta ,q}=\{a:\mathcal{X}\times N\to \mathbb{R}|{\mathbb{E}}_{\theta ,q}[a]=0,a(x,e)=0\text{if}q(e|x)=0\}.\end{array}$$

The inner product for $a_1,a_2\in T_{\theta ,q}$ is defined as ${\mathbb{E}}_{\theta ,q}[a_1{a}_2]$. A geometric meaning of $T_{\theta ,q}$ is the tangent space of the statistical model $\{p_{\theta }(x)q(e|x)|\theta \in \mathrm{\Theta }\}$. For any $a\in T_{\theta ,q}$ and sufficiently small $\epsilon >0$, the perturbation of $p_{\theta }(x)q(e|x)$ to the direction $a(x,e)$ leads to the probability function $p_{\theta }(x)q(e|x)(1+\epsilon a(x,e))$. Each element of the score function $u_{\theta ,j}(x),j=1,\dots ,d$ is a member of $T_{\theta ,q}$ by regarding as $u_{\theta ,j}(x)·\mathbf{1}[q(e|x)\ne 0]$.

Let us consider the stochastic Z-estimator derived from ${\mathit{f}}_{\theta }=(f_{\theta ,1},\dots ,f_{\theta ,k})$ satisfying ${\mathit{f}}_{\theta }\in {(T_{\theta ,q})}^d$ for any θ. It leads to a Fisher consistent estimator. Stochastic local Z-estimators use an identification function in the linear subspace

$$\begin{array}{c}\hfill T_{\theta ,q}^L=\{f\in T_{\theta ,q}|{\mathbb{E}}_{\theta ,q}[f|e]=0,\forall e\in N\}.\end{array}$$

(7)

The orthogonal complement of $T_{\theta ,q}^L$ in $T_{\theta ,q}$ is denoted as $T_{\theta ,q}^E$, which is given as

$$\begin{array}{c}\hfill T_{\theta ,q}^E=\{f\in T_{\theta ,q}|f(x,e)\text{does}\text{not}\text{depend}\text{on}x\}.\end{array}$$

Indeed, the orthogonality of $T_{\theta ,q}^L$ and $T_{\theta ,q}^E$ is confirmed by

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[a(x,e)b(e)]={\mathbb{E}}_{\theta ,q}[b(e)\mathbb{E}[a(x,e)|e]]=0\end{array}$$

for any $a\in T_{\theta ,q}^L$ and any $b\in T_{\theta ,q}^E$. In addition, any $f\in T_{\theta ,q}$ can be decomposed into

$$\begin{array}{c}\hfill f=(f-{\mathbb{E}}_{\theta ,q}[f|e])+\mathbb{E}[f|e]\end{array}$$

such that $f-{\mathbb{E}}_{\theta ,q}[f|e]\in T_{\theta ,q}^L$ and ${\mathbb{E}}_{\theta ,q}[f|e]\in T_{\theta ,q}^E$.

The efficient score ${\mathit{u}}_{\theta }^I={(u_{\theta ,k}^I)}_{k=1,\dots ,d}$ is defined as the projection of each element of the score ${\mathit{u}}_{\theta }$ onto $T_{\theta ,q}^L$, i.e.,

$$\begin{array}{c}\hfill {\mathit{u}}_{\theta }^I(x,e)={\mathit{u}}_{\theta }(x)-{\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }|e]=\nabla log\tilde{p}(x;\theta )-{\mathbb{E}}_{\theta ,q}[\nabla log\tilde{p}(x;\theta )|e].\end{array}$$

The efficient score is computationally tractable when the size of the neighborhood e is not exponential order but linear or low-degree polynomial order of n, where n is the dimension of x. The trade-off between the computational and statistical efficiency is presented in Theorems 1 and 2 in Section 4.2.

Another expression of the efficient score is

$$\begin{array}{c}\hfill {\mathit{u}}_{\theta }^I(x,e)=\nabla logp(x|e;\theta ,q),\end{array}$$

(8)

where the conditional probability $p(x|e;\theta ,q)$ is defined from $p_{\theta }(x)q(e|x)$. The above equality is obtained by

$$\begin{array}{cc}\hfill {\mathit{u}}_{\theta }^I(x,e)& =\nabla log{p}_{\theta }(x)-\sum _{x\in e}\frac{p_{\theta }(x)q(e|x)}{{\sum }_{x^{\prime }}{p}_{\theta }(x^{\prime })q(e|x^{\prime })}\nabla log{p}_{\theta }(x)\hfill \\ & =\nabla log(p_{\theta }(x)q(e|x))-\frac{{\sum }_{x\in e}\nabla p_{\theta }(x)q(e|x)}{{\sum }_{x^{\prime }\in e}{p}_{\theta }(x^{\prime })q(e|x^{\prime })}\hfill \\ & =\nabla log\frac{p_{\theta }(x)q(e|x)}{{\sum }_{x^{\prime }}{p}_{\theta }(x^{\prime })q(e|x^{\prime })}=\nabla logp(x|e;\theta ,q).\hfill \end{array}$$

We define $T_{\theta ,q}^I$ as the subspace of $T_{\theta ,q}^L$ spanned by $\{u_{\theta ,k}^I|k=1,\dots ,d\}$, and $T_{\theta ,q}^A$ be the orthogonal complement of $T_{\theta ,q}^I$ in $T_{\theta ,q}^L$. As a result, we obtain

$$\begin{array}{c}\hfill T_{\theta ,q}=T_{\theta ,q}^L\oplus T_{\theta ,q}^E,T_{\theta ,q}^L=T_{\theta ,q}^I\oplus T_{\theta ,q}^A.\end{array}$$

We describe statistical properties of stochastic local Z-estimators using the above tangent spaces.

4.2. Asymptotic Variance of Stochastic Local Z-Estimators

Under a fixed conditional probability $q(e|x)$, we derive the asymptotically efficient stochastic local Z-estimator in the same way to semi-parametric estimation [19,22]. In addition, we consider the monotonicity of the efficiency w.r.t. the size of the neighborhood. Given i.i.d. samples $(x_i,e_i),$ $i=1,\dots ,m$, generated from $p(x)q(e|x)$, the estimator $\widehat{\theta }$ of the parameter in the model (1) is obtained by solving the system of Equation (4), where ${\mathit{f}}_{\theta }\in {(T_{\theta ,q})}^d$ for any $\theta \in \mathrm{\Theta }$. Suppose that the true probability function $p(x)$ is realized by $p_{\theta }(x)$ of the model (1). As shown in [1], the standard asymptotic theory yields that the asymptotic variance of the above Z-estimator is given as

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}m·\mathbb{V}[\widehat{\theta }]={\mathbb{E}}_{\theta ,q}{[{\mathit{f}}_{\theta }{\mathit{u}}_{\theta }^T]}^{-1}{\mathbb{E}}_{\theta ,q}[{\mathit{f}}_{\theta }{\mathit{f}}_{\theta }^T]{\mathbb{E}}_{\theta ,q}{[{\mathit{u}}_{\theta }{\mathit{f}}_{\theta }^T]}^{-1}.\end{array}$$

(9)

The derivation of the asymptotic variance is presented in Appendix for completeness of the presentation.

We consider the asymptotic variance of the stochastic local Z-estimators. A simple expression of the asymptotic variance is obtained using the efficient score ${\mathit{u}}_{\theta }^I$. Without loss of generality, the identification function of the stochastic local Z-estimator, ${\mathit{f}}_{\theta }\in {(T_{\theta ,q}^L)}^d$, is expressed as

$$\begin{array}{c}\hfill {\mathit{f}}_{\theta }(x,e)={\mathit{u}}_{\theta }^I(x,e)+{\mathit{a}}_{\theta }(x,e),\end{array}$$

where ${\mathit{a}}_{\theta }\in {(T_{\theta ,q}^A)}^d$. The reason is briefly shown below. Suppose that ${\mathit{f}}_{\theta }(x,e)$ is decomposed into ${\mathit{f}}_{\theta }(x,e)=B_{\theta }{\mathit{u}}_{\theta }^I(x,e)+{\mathit{a}}_{\theta }(x,e)$, where $B_{\theta }$ is a d by d matrix that does not depend on x and e. The condition that the matrix ${\mathbb{E}}_{\theta ,q}[\nabla {\mathit{f}}_{\theta }]$ is invertible assures that $B_{\theta }$ is invertible, since

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\theta ,q}\left[\frac{\partial }{\partial {\theta }_i}{f}_{\theta ,k}\right]& =\sum _j\frac{\partial B_{\theta ,kj}}{\partial {\theta }_i}{\mathbb{E}}_{\theta ,q}\left[u_{\theta ,j}^I\right]+\sum _j{B}_{\theta ,kj}{\mathbb{E}}_{\theta ,q}\left[\frac{\partial }{\partial {\theta }_i}{u}_{\theta ,j}^I\right]+{\mathbb{E}}_{\theta ,q}\left[\frac{\partial }{\partial {\theta }_i}{a}_{\theta ,j}\right]\hfill \\ & =-\sum _j{B}_{\theta ,kj}{\mathbb{E}}_{\theta ,q}\left[u_{\theta ,i}{u}_{\theta ,j}^I\right]-{\mathbb{E}}_{\theta ,q}\left[u_{\theta ,i}{a}_{\theta ,j}\right]=-\sum _j{B}_{\theta ,kj}{\mathbb{E}}_{\theta ,q}\left[u_{\theta ,i}^I{u}_{\theta ,j}^I\right]\hfill \end{array}$$

holds. In the above equalities, we use the formula

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}\left[\frac{\partial }{\partial {\theta }_i}{f}_{\theta }\right]=-{\mathbb{E}}_{\theta ,q}\left[u_{\theta ,i}{f}_{\theta }\right]\end{array}$$

for $f_{\theta }\in T_{\theta ,q}$ that is obtained by differentiating the identity ${\mathbb{E}}_{\theta ,q}[f_{\theta }]=0$. Clearly, ${\mathit{f}}_{\theta }(x,e)$ provides the same estimator as $B_{\theta }^{-1}{\mathit{f}}_{\theta }(x,e)$. See [19] for details of the standard form of Z-estimators.

Theorem 1. 

Let us define the d by d matrix $G_{\theta }^I$ by

$$\begin{array}{c}\hfill G_{\theta }^I=\mathbb{E}[{\mathit{u}}_{\theta }^I{({\mathit{u}}_{\theta }^I)}^T].\end{array}$$

Then, for a fixed conditional probability $q(e|x)$, the asymptotic variance of any stochastic local Z-estimator $\widehat{\theta }$ satisfies the inequality

$$\begin{array}{c}\hfill \underset{m\to \infty }{lim}m·V[\widehat{\theta }]⪰{(G_{\theta }^I)}^{-1}\end{array}$$

in the sense of the non-negative definiteness. The equality is attained by the Z-estimator using ${\mathit{u}}_{\theta }^I$.

Proof. 

Let us compute each matrix in (9). According to the above argument, without loss of generality, we assume ${\mathit{f}}_{\theta }={\mathit{u}}_{\theta }^I+{\mathit{a}}_{\theta }$ for ${\mathit{a}}_{\theta }\in {(T_{\theta ,q}^A)}^d$. The matrix ${\mathbb{E}}_{\theta ,q}[u_{\theta }{f}_{\theta }^T]$ is then expressed as

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }{\mathit{f}}_{\theta }^T]={\mathbb{E}}_{\theta ,q}[({\mathit{u}}_{\theta }^I+{\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }|e]){({\mathit{u}}_{\theta }^I+{\mathit{a}}_{\theta })}^T]={\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }^I{({\mathit{u}}_{\theta }^I)}^T]=G_{\theta }^I\end{array}$$

due to ${\mathit{u}}_{\theta }^I\in {(T_{\theta ,q}^I)}^d,\mathbb{E}[{\mathit{u}}_{\theta }|e]\in {(T_{\theta ,q}^E)}^d$ and ${\mathit{a}}_{\theta }\in {(T_{\theta ,q}^A)}^d$. Let us define $A=\mathbb{E}[{\mathit{a}}_{\theta }{\mathit{a}}_{\theta }^T]$. Then, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[{\mathit{f}}_{\theta }{\mathit{f}}_{\theta }^T]=G_{\theta }^I+A.\end{array}$$

As a result, we obtain

$$\begin{array}{c}\hfill \underset{m\to \infty }{lim}m·V[\widehat{\theta }]={(G_{\theta }^I)}^{-1}(G_{\theta }^I+A){(G_{\theta }^I)}^{-1}={(G_{\theta }^I)}^{-1}+{(G_{\theta }^I)}^{-1}A{(G_{\theta }^I)}^{-1}⪰{(G_{\theta }^I)}^{-1}.\end{array}$$

When ${\mathit{f}}_{\theta }={\mathit{u}}_{\theta }^I$, the matrix A becomes the null matrix and the minimum asymptotic variance is attained. ☐

The minimum variance of stochastic local Z-estimators is attained by the efficient score. This conclusion agrees to the result of the asymptotically efficient estimator in semi-parametric models including nuisance parameters [19,22].

Remark 1. 

Let us consider the relation between the stochastic pseudo-likelihood $\nabla log{p}_{\theta }(x_k|x_{-k})$ and efficient score ${\mathit{u}}_{\theta }^I(x,e_{x,k})$. Suppose that the neighborhood system $N_x$ and the conditional distribution $q(e|x)$ on $N_x$ are defined as shown in Example 3. Then, we have ${\mathit{u}}_{\theta }^I(x,e_{x,k})=\nabla log{p}_{\theta }(x_k|x_{-k})$. Likewise, we find that the reduced-SCL, $\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})$, is equivalent with the efficient score under the setup in Example 4 when the index subset $A_{\ell }$ is defined as $B_{\ell }^c$.

4.3. Monotonicity of Asymptotic Efficiency

As described in [23], for the composite likelihood estimator with the index pairs $(A_{\ell },B_{\ell })$, $\ell =1,\dots ,M$, it is widely believed that by increasing the size of $A_{\ell }$ (and correspondingly decreasing the size of $B_{\ell }=A_{\ell }^c$), one can capture more dependency relations in the model and increase the accuracy. For the stochastic local Z-estimators, we can obtain the exact relation between the neighborhood system and asymptotic efficiency.

Let us consider two stochastic local Z-estimators; one is defined by $q(e|x)$ on the neighborhood system $e\in N_x$ and the other is given by $q^{\prime }(e^{\prime }|x)$ on the neighborhood system $e^{\prime }\in N_x^{\prime }$. The efficient score are respectively written as ${\mathit{u}}_{\theta }^I(x,e)$ for $q(e|x)$ and ${\mathit{u}}_{\theta }^{\prime I}(x,e^{\prime })$ for $q^{\prime }(e|x)$. In addition, let us define $G_{\theta }^I={\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }^I{({\mathit{u}}_{\theta }^I)}^T]$ and $G_{\theta }^{\prime I}={\mathbb{E}}_{\theta ,q^{\prime }}[{\mathit{u}}_{\theta }^{\prime I}{({\mathit{u}}_{\theta }^{\prime I})}^T]$.

Theorem 2. 

Let $p(x,e,e^{\prime })$ be the joint probability of $(x,e,e^{\prime })\in \mathcal{X}\times N\times N^{\prime }$ and suppose that probability functions, $q(e|x),q^{\prime }(e^{\prime }|x)$ and $p_{\theta }(x)$, are obtained from $p(x,e,e^{\prime })$. We assume that

$$\begin{array}{c}\hfill \mathbb{E}[\mathbb{E}[{\mathit{u}}_{\theta }|e]|e^{\prime }]=\mathbb{E}[{\mathit{u}}_{\theta }|e^{\prime }]\end{array}$$

(10)

holds under the probability distribution $p(x,e,e^{\prime })$. Then, we have

$$\begin{array}{c}\hfill {(G_{\theta }^I)}^{-1}⪰{(G_{\theta }^{\prime I})}^{-1},\end{array}$$

(11)

i.e., the efficiency bound of $N_x^{\prime }$ and $q^{\prime }(x|e)$ is smaller than or equal to that of $N_x$ and $q(x|e)$.

Proof. 

We use the basic formula of the conditional variance

$$\begin{array}{c}\hfill \mathbb{V}[X]=\mathbb{V}[\mathbb{E}[X|Z]]+\mathbb{E}[\mathbb{V}[X|Z]]⪰\mathbb{V}[\mathbb{E}[X|Z]]\end{array}$$

(12)

for random variables X and Z. The above formula is applied to the score ${\mathit{u}}_{\theta }(x)$ and the efficient score ${\mathit{u}}_{\theta }^I(x,e)$. Note that $\mathbb{E}[\mathbb{V}[{\mathit{u}}_{\theta }|e]]=\mathbb{E}[{\mathit{u}}_{\theta }^I{({\mathit{u}}_{\theta }^I)}^T]=G_{\theta }^I$ holds. Then, we have

$$\begin{array}{cc}\hfill \mathbb{V}[{\mathit{u}}_{\theta }]& =\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]+\mathbb{E}[\mathbb{V}[{\mathit{u}}_{\theta }|e]]=\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]+\mathbb{E}[{\mathit{u}}_{\theta }^I{({\mathit{u}}_{\theta }^I)}^T]\hfill \\ & =\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]+G_{\theta }^I=\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e^{\prime }]]+G_{\theta }^{\prime I}.\hfill \end{array}$$

The last equality comes from the fact that the score ${\mathit{u}}_{\theta }(x)$ is common in both setups. Since the equality (10) holds, again the Formula (12) with $X=\mathbb{E}[{\mathit{u}}_{\theta }|e]$ and $Z=e^{\prime }$ yields

$$\begin{array}{c}\hfill \mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]=\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e^{\prime }]]+\mathbb{E}[\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]|e^{\prime }]]⪰\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e^{\prime }]].\end{array}$$

Thus, we obtain

$$\begin{array}{c}\hfill G_{\theta }^I=\mathbb{V}[{\mathit{u}}_{\theta }]-\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]⪯\mathbb{V}[{\mathit{u}}_{\theta }]-\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e^{\prime }]]=G_{\theta }^{\prime I}.\end{array}$$

As a result, we have (11). ☐

A similar inequality is derived in [24] for the mutual Fisher information. The mutual Fisher information is rather similar to $\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]$ than $G_{\theta }^I$. Theorem 13 of [24] corresponds to the one-dimensional version of the inequality $\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e]]⪰\mathbb{V}[\mathbb{E}[{\mathit{u}}_{\theta }|e^{\prime }]]$.

Let us show an example that agrees to (10). We define two neighborhood systems $N=\{N_x|x\in \mathcal{X}\}$ and $N^{\prime }=\{N_x^{\prime }|x\in \mathcal{X}\}$ such that, for any $e\in N_x$, there exists $e^{\prime }\in N_x^{\prime }$ satisfying $e\subset e^{\prime }$. For the joint probability $p(x,e,e^{\prime })$, suppose that x and $e^{\prime }$ are conditionally independent given e and that the conditional probability $r^{\prime }(e^{\prime }|e)$ derived from $p(x,e,e^{\prime })$ is equal to zero unless $e\subset e^{\prime }$. Under these conditions, $q^{\prime }(e^{\prime }|x)$ derived from $p(x,e,e^{\prime })$ takes 0 if $e^{\prime }\notin N_x^{\prime }$. The conditional independence assures that $p(x,e,e^{\prime })$ is expressed as $p(x,e,e^{\prime })=p_{\theta }(x)q(e|x)r^{\prime }(e^{\prime }|e)=p(x|e)r(e|e^{\prime })q(e^{\prime })$. Hence, the conditional probability $p(x|e^{\prime })$ is expressed as ${\sum }_{e\in N}p(x|e)r(e|e^{\prime })$. Thus, we obtain

$$\begin{array}{c}\hfill \mathbb{E}[\mathbb{E}[{\mathit{u}}_{\theta }|e]|e^{\prime }]=\sum _{e\in N}\sum _{x\in \mathcal{X}}{\mathit{u}}_{\theta }(x)p(x|e)r(e|e^{\prime })=\sum _{x\in \mathcal{X}}{\mathit{u}}_{\theta }(x)\sum _{e\in N}p(x|e)r(e|e^{\prime })=\sum _{x\in \mathcal{X}}{\mathit{u}}_{\theta }(x)p(x|e^{\prime }).\end{array}$$

As a result, the better efficiency bound is obtained by the larger neighborhood. A similar result is presented in [25] for the composite likelihood estimators. The relation of the result in [25] and ours is explained in Section 5.3 of this paper.

Example 5. 

Let $N_x$ be a neighborhood system at x endowed with the conditional distribution $q(e|x)$. Another neighborhood system is defined as $N_x^{\prime }=\{\mathcal{X}\}$ for all x, and $q^{\prime }(e^{\prime }|x)=1$ for $e^{\prime }=\mathcal{X}$. Let us define $p(x,e,e^{\prime })=p_{\theta }(x)q(e|x)$ for $e^{\prime }=\mathcal{X}$ and otherwise $p(x,e,e^{\prime })=0$. Since $e^{\prime }$ always takes $\mathcal{X}$, x and $e^{\prime }$ are conditionally independent given e. Thus, we have $G_{\theta }^I⪯G_{\theta }^{\prime I}$. Indeed, $G_{\theta }^{\prime I}$ is the Fisher information matrix of the model $p_{\theta }(x)$.

We compare the stochastic pseudo-likelihood and reduced-SCL. Let $N_x=\{e_{x,k}|k=1,\dots ,n\}$ be the neighborhood system defined in Example 3, and N be ${\cup }_{x\in \mathcal{X}}{N}_x$. The conditional distribution on $N_x$ is given by $q(e_{x,k}|x)=q_k,k=1,\dots ,n$. As shown in Remark 1, the corresponding efficient score is nothing but the stochastic pseudo-likelihood, i.e., $u_{\theta }^I(x,e_{x,k})=\nabla log{p}_{\theta }(x_k|x_{-k})$. Let us define another neighborhood system $N_x^{\prime }$ in the same way as Example 4. For the subsets $B_{\ell }\subset \mathcal{X}$ and $A_{\ell }=B_{\ell }^c,$ $\ell =1,\dots ,M$, we define $e_{x,\ell }^{\prime }$ as $\{y\in \mathcal{X}|y_{B_{\ell }}=x_{B_{\ell }}\}$ and $N_x^{\prime }=\{e_{x,\ell }^{\prime }|\ell =1,\dots ,M\}$. Let $N^{\prime }$ be ${\cup }_{x\in \mathcal{X}}{N}_x^{\prime }$. The conditional distribution on $N_x^{\prime }$ is given as $q^{\prime }(e_{x,\ell }^{\prime }|x)=q_{\ell }^{\prime }$ for $\ell =1,\dots ,M$. Then, the efficient score associated with $N^{\prime }$ and $q^{\prime }$ is equal to the reduced-SCL, i.e., ${\mathit{u}}_{\theta }^{\prime I}(x,e_{x,\ell }^{\prime })=\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})$. As the direct conclusion of Theorem 2 and the above argument about the property of the conditional independence between x and $e^{\prime }\in N^{\prime }$ given $e\in N$, we obtain the following corollary.

Corollary 1. 

We define $N_e^{\prime }$ for $e\in N$ by $N_e^{\prime }=\{e^{\prime }\in N^{\prime }|e\subset e^{\prime }\}$. Let $r^{\prime }(e^{\prime }|e)$ be a conditional probability on $N_e^{\prime }$ given $e\in N$, where $r^{\prime }(e^{\prime }|e)=0$ is assumed for $e^{\prime }\notin N_e^{\prime }$. If the equality $q_{\ell }^{\prime }={\sum }_{k=1}^n{q}_k{r}^{\prime }(e_{x,\ell }^{\prime }|e_{x,k})$ holds, the reduced-SCL with $N^{\prime }$ and $q^{\prime }$ is more efficient than stochastic pseudo-likelihood with N and q.

Example 6. 

Suppose that the size of $N_e^{\prime }$ is the same for all $e\in N$ and that the size of the set $\{e\in N|x\in e\subset e^{\prime }\}$ is the same for any $x\in \mathcal{X}$ and $e^{\prime }\in N^{\prime }$ such that $x\in e^{\prime }$. Let $q(e|x)$ (resp. $q^{\prime }(e^{\prime }|x)$) be the uniform distribution on $N_x$ (resp. $N_x^{\prime }$). Then, the reduced-SCL is more efficient than stochastic pseudo-likelihood. Indeed, the assumption ensures that the sum ${\sum }_{e\in N}q(e|x)r^{\prime }(e^{\prime }|e)$ does not depend on x and $e^{\prime }$. Thus, the uniform distribution $q^{\prime }(e^{\prime }|x)$ meets the condition of the above corollary. For example, let $B_1,\dots ,B_M$ be all subsets of size $n-2$ in $\{1,\dots ,n\}$. Then, we have $M=n(n-1)/2$. The size of $N_e^{\prime }$ is $n-1$, and the size of $\{e\in N|x\in e\subset e^{\prime }\}$ is equal to 2.

5. Local Z-Estimators and Efficiency Bounds

In this section, we define the local Z-estimator as the expectation of a stochastic local Z-estimator, and derive its efficiency bound.

5.1. Local Z-Estimators

Computationally tractable estimators such as pseudo-likelihood and composite likelihood are obtained by the expectation of an identification function in $T_{\theta ,q}^L$. Let us define the local Z-estimator as the Z-estimator using

$$\begin{array}{c}\hfill {\overline{\mathit{f}}}_{\theta }(x)={\mathbb{E}}_{\theta ,q}[{\mathit{f}}_{\theta }|x],\end{array}$$

where ${\mathit{f}}_{\theta }\in {(T_{\theta ,q}^L)}^d$. The conditional expectation given x is regarded as the projection onto the subspace $T_{\theta ,q}^X$ which is defined as

$$\begin{array}{c}\hfill T_{\theta ,q}^X=\{f\in T_{\theta ,q}|f(x,e)\text{does}\text{not}\text{depend}\text{on}e\}.\end{array}$$

Let ${\Pi }_X$ be the projection operator onto $T_{\theta ,q}^X$ and ${\Pi }_X^{\perp }$ be the one onto the orthogonal complement of $T_{\theta ,q}^X$. Then, one can prove ${\Pi }_X[f]=\mathbb{E}[f|x]$ and ${\Pi }_X^{\perp }[f]=f-\mathbb{E}[f|x]$ for $f\in T_{\theta ,q}$. When the number of elements in the neighborhood $N_x$ is reasonable, the computation of the local Z-estimator is tractable.

Below, we show that some estimators are expressed as the local Z-estimator.

Example 7 (Pseudo-likelihood and composite likelihood). 

In the setup of Example 3, the conditional expectation of the efficient score, ${\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }^I|x]$, yields the pseudo-likelihood when $q(e|x)$ is the uniform distribution on $N_x$. In the setup of Example 4, let us assume $A_{\ell }=B_{\ell }^c$ and $q^{\prime }(e_{x,\ell }^{\prime }|x)=q_{\ell }^{\prime }$. Then, the conditional expectation of the efficient score ${\mathit{u}}_{\theta }^I(x,e_{x,\ell }^{\prime })=\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})$ yields

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q^{\prime }}[{\mathit{u}}_{\theta }^I|x]=\sum _{\ell =1}^M{q}_{\ell }^{\prime }\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }}),\end{array}$$

which is the general form of the composite likelihood in Example 2 with ${\gamma }_{\ell }=q_{\ell }^{\prime }$.

5.2. Efficiency Bounds

We derive the efficiency bound of the local Z-estimator. Without loss of generality, the local Z-estimator ${\overline{\mathit{f}}}_{\theta }(x)\in {(T_{\theta ,q}^X)}^d$ is represented as

$$\begin{array}{c}\hfill {\overline{\mathit{f}}}_{\theta }(x)=\mathbb{E}[{\mathit{f}}_{\theta }|x],{\mathit{f}}_{\theta }={\mathit{u}}_{\theta }^I+{\mathit{a}}_{\theta }\in {(T_{\theta ,q}^L)}^d,{\mathit{a}}_{\theta }\in {(T_{\theta ,q}^A)}^d.\end{array}$$

Under the model $p_{\theta }(x)$, we calculate the asymptotic variance (9) of the local Z-estimator $\widehat{\theta }$ using ${\overline{\mathit{f}}}_{\theta }(x)$. The matrix ${\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]$ in (9) is given as

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]={\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }{({\mathit{u}}_{\theta }^I+{\mathit{a}}_{\theta })}^T]=\mathbb{E}[{\mathit{u}}_{\theta }^I{({\mathit{u}}_{\theta }^I)}^T]=G_{\theta }^I.\end{array}$$

Hence, we have

$$\begin{array}{cc}\hfill \underset{m\to \infty }{lim}m·\mathbb{V}[\widehat{\theta }]& ={(G_{\theta }^I)}^{-1}{\mathbb{E}}_{\theta ,q}[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]{(G_{\theta }^I)}^{-1}.\hfill \end{array}$$

Here, the expectation ${\mathbb{E}}_{\theta ,q}[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]$ can be written as the expectation under $p_{\theta }(x)$, i.e., ${\mathbb{E}}_{\theta }[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]$, since ${\mathit{u}}_{\theta }$ and ${\overline{\mathit{f}}}_{\theta }$ depend only on x. The orthogonal decomposition ${\mathit{f}}_{\theta }={\overline{\mathit{f}}}_{\theta }+{\Pi }_X^{\perp }[{\mathit{f}}_{\theta }]$ leads to

$$\begin{array}{c}{(G_{\theta }^I)}^{-1}{\mathbb{E}}_{\theta ,q}[{\mathit{f}}_{\theta }{\mathit{f}}_{\theta }^T]{(G_{\theta }^I)}^{-1}={(G_{\theta }^I)}^{-1}{\mathbb{E}}_{\theta ,q}[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]{(G_{\theta }^I)}^{-1}+{(G_{\theta }^I)}^{-1}{\mathbb{E}}_{\theta ,q}[{\Pi }_X^{\perp }[{\mathit{f}}_{\theta }]{\Pi }_X^{\perp }{[\mathit{f}]}_{\theta }^T]{(G_{\theta }^I)}^{-1}\hfill \\ ⪰{(G_{\theta }^I)}^{-1}{\mathbb{E}}_{\theta ,q}[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]{(G_{\theta }^I)}^{-1},\hfill \end{array}$$

(13)

meaning that the asymptotic variance of the stochastic local Z-estimator using ${\mathit{f}}_{\theta }(x,e)$ is larger then or equal to that of the local Z-estimator using ${\overline{\mathit{f}}}_{\theta }(x)$.

We consider the optimal choice of ${\mathit{a}}_{\theta }\in {(T_{\theta ,q}^A)}^d$ in ${\overline{\mathit{f}}}_{\theta }(x)={\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }^I+{\mathit{a}}_{\theta }|x]$. Let us define the subspace $T_{\theta ,q}^{XA}$ as ${\Pi }_X{T}_{\theta ,q}^A=\{{\Pi }_X[a]|a\in T_{\theta ,q}^A\}$, and ${\Pi }_{XA}$ be the projection operator onto $T_{\theta ,q}^{XA}$. Then, we define $v_{\theta ,j}^I(x)\in T_{\theta ,q}^X$ as the projection of $u_{\theta ,j}^I(x,e)\in T_{\theta ,q}^I$ onto the orthogonal complement of $T_{\theta ,q}^{XA}$ in $T_{\theta ,q}^X$, i.e.,

$$\begin{array}{c}\hfill v_{\theta ,j}^I=({\Pi }_X-{\Pi }_{XA})[u_{\theta ,j}^I]\end{array}$$

for $j=1,\dots ,d$. In this paper, we call $v_{\theta }^I={(v_{\theta ,1}^I,\dots ,v_{\theta ,d}^I)}^T$ the local efficient score.

Theorem 3. 

Let us define d by d matrix $H_{\theta }^I$ as ${\mathbb{E}}_{\theta ,q}[v_{\theta }^I{(v_{\theta }^I)}^T]$. Then, the efficiency bound of the local Z-estimator $\widehat{\theta }$ is given as

$$\begin{array}{c}\hfill \underset{m\to \infty }{lim}m·\mathbb{V}[\widehat{\theta }]⪰{(G_{\theta }^I)}^{-1}{H}_{\theta }^I{(G_{\theta }^I)}^{-1}.\end{array}$$

The equality is attained by the local Z-estimator using the local efficient score ${\mathit{v}}_{\theta }^I=({\Pi }_X-{\Pi }_{XA})[{\mathit{u}}_{\theta }^I]$.

Proof. 

${\overline{\mathit{f}}}_{\theta }(x)={\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }^I+{\mathit{a}}_{\theta }|x]$ has the orthogonal decomposition ${\mathit{v}}_{\theta }^I+{\mathit{b}}_{\theta }$, where ${\mathit{b}}_{\theta }\in {(T_{\theta ,q}^{XA})}^d$. Hence, we obtain ${\mathbb{E}}_{\theta ,q}[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]⪰{\mathbb{E}}_{\theta }[{\mathit{v}}_{\theta }^I{({\mathit{v}}_{\theta }^I)}^T]=H_{\theta }^I$ and

$$\begin{array}{c}\hfill {(G_{\theta }^I)}^{-1}{\mathbb{E}}_{\theta ,q}[{\overline{\mathit{f}}}_{\theta }{\overline{\mathit{f}}}_{\theta }^T]{(G_{\theta }^I)}^{-1}⪰{(G_{\theta }^I)}^{-1}{H}_{\theta }^I{(G_{\theta }^I)}^{-1}.\end{array}$$

The left-hand side of the above inequality is the asymptotic variance of the local Z-estimator. The equality is attained by the local Z-estimator using ${\mathit{v}}_{\theta }^I$. ☐

We consider the relation between the local efficient score ${\mathit{v}}_{\theta }^I(x)$ and the score ${\mathit{u}}_{\theta }(x)$. We define $T_{\theta ,q}^{ML}$ as the subspace spanned by the score $u_{\theta ,j}(x),j=1,\dots ,d$. For any $a\in T_{\theta ,q}^A$, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_{\theta ,q}[u_{\theta ,j}{\mathbb{E}}_{\theta ,q}[a|x]]={\mathbb{E}}_{\theta ,q}[u_{\theta ,j}a]=0,j=1,\dots ,d,\end{array}$$

meaning that $T_{\theta ,q}^{ML}$ and $T_{\theta ,q}^{XA}$ are orthogonal to each other. Hence, $T_{\theta ,q}^X$ is decomposed into

$$\begin{array}{c}\hfill T_{\theta ,q}^X=T_{\theta ,q}^{XA}\oplus T_{\theta ,q}^{ML}\oplus T_{\theta ,q}^{XC},\end{array}$$

where $T_{\theta ,q}^{XC}$ is the orthogonal complement of $T_{\theta ,q}^{XA}\oplus T_{\theta ,q}^{ML}$ in $T_{\theta ,q}^X$. Eventually, subspaces in $T_{\theta ,q}$ satisfy the following relations,

$$\begin{array}{c}\hfill T_{\theta ,q}=T_{\theta ,q}^E\oplus T_{\theta ,q}^I\oplus T_{\theta ,q}^A,T_{\theta ,q}^X=({\Pi }_X{T}_{\theta ,q}^A)\oplus T_{\theta ,q}^{ML}\oplus T_{\theta ,q}^{XC}.\end{array}$$

Let us define $T_{\theta ,q}^{XI}$ as the subspace spanned by the local efficient score $v_{\theta ,j}^I(x),j=1,\dots ,d$. Under a mild assumption, $T_{\theta ,q}^{XI}$ and $T_{\theta ,q}^{ML}$ has the same dimension. Since ${\mathit{v}}_{\theta }^I(x)$ is orthogonal to ${\Pi }_X{T}_{\theta ,q}^A$, $T_{\theta ,q}^{XI}$ is included in $T_{\theta ,q}^{ML}\oplus T_{\theta ,q}^{XC}$. Hence, $T_{\theta ,q}^{XC}$ is interpreted as the subspace expressing the information loss caused by the localization of the score ${\mathit{u}}_{\theta }$.

5.3. Relation to Existing Works

5.3.1. Comparison of Local Z-Estimators

We compare the asymptotic variances of two local Z-estimators that are connected to composite likelihoods.

One estimator is defined from the neighborhood system N which consists of the singleton $N_x=\{e_x\},x\in \mathcal{X}$. Here, we assume that $e_x=e_{x^{\prime }}$ holds for $x^{\prime }\in e_x$ and ${\cup }_{x\in \mathcal{X}}{e}_x=\mathcal{X}$. Such a neighborhood system N is called the equivalence class [25]. An equivalence class corresponds to a partition of the sample space. The conditional probability $q(e|x)$ takes 1 for $e=e_x$ and 0 otherwise. Let ${\mathit{u}}_{\theta }^I(x,e)$ be the efficient score defined from N and $q(e|x)$, and ${\overline{\mathit{u}}}_{\theta }^I(x)$ be the local Z-estimator ${\overline{\mathit{u}}}_{\theta }^I(x)={\mathbb{E}}_{\theta ,q}[{\mathit{u}}_{\theta }^I|x]$.

Another localized estimator is defined from the neighborhood system $N^{\prime }$ which consists of $N_x^{\prime },x\in \mathcal{X}$, where $N_x^{\prime }$ is not necessarily a singleton. Suppose that $e_x\subset e^{\prime }$ holds for any $e^{\prime }\in N_x^{\prime }$. The conditional probability $q^{\prime }(e^{\prime }|x)$ is defined as $q^{\prime }(e^{\prime }|x)=r^{\prime }(e^{\prime }|e_x)$, where $r^{\prime }(e^{\prime }|e_x)$ is a conditional probability of $e^{\prime }\in N_x^{\prime }$ given $e_x$. The corresponding efficient score is denoted as $u_{\theta }^{\prime I}(x,e)$ and let us define ${\overline{\mathit{u}}}_{\theta }^{\prime I}(x)={\mathbb{E}}_{\theta ,q^{\prime }}[{\mathit{u}}_{\theta }^{\prime I}|x]$ as the local Z-estimator associated with $N^{\prime }$ and $q^{\prime }(e^{\prime }|x)$.

From the definition, the joint probability $p_{\theta }(x)q(e|x)r^{\prime }(e^{\prime }|e)$ agrees to $q(e|x)$ an $q^{\prime }(e^{\prime }|x)$. Hence we see that x and $e^{\prime }$ are conditionally independent given e. Hence, Theorem 2 guarantees the inequality ${(G_{\theta }^I)}^{-1}⪰{(G_{\theta }^{\prime I})}^{-1}$.

The efficient score ${\mathit{u}}_{\theta }^I(x,e)$ can take a non-zero value only when $e=e_x$. Hence, ${\mathit{u}}_{\theta }^I(x,e)$ is regarded as the function of x, i.e, ${\mathit{u}}_{\theta }^I(x,e)\in {(T_{\theta ,q}^X)}^d$, and the asymptotic variance of the local Z-estimator obtained by ${\overline{\mathit{u}}}_{\theta }^I(x)={\mathit{u}}_{\theta }^I(x,e_x)$ is ${(G_{\theta }^I)}^{-1}$. On the other hand, the asymptotic variance of the local Z-estimator derived from ${\overline{\mathit{u}}}_{\theta }^{\prime I}(x)$ is less than or equal to ${(G_{\theta }^{\prime I})}^{-1}$ due to (13). Therefore, ${\overline{\mathit{u}}}_{\theta }^{\prime I}$ with $N^{\prime }$ and $q^{\prime }$ provides more efficient estimators than ${\overline{\mathit{u}}}_{\theta }^I$ with N and q.

Liang and Jordan presented a similar result in [25]. In their setup, the larger neighborhood $N_x^{\prime }$ is a singleton $\{e_x^{\prime }\}$ and the smaller one, $N_x$, can have multiple neighborhoods at each x. In such a case, the similar relation holds, i.e., the estimator with $N^{\prime }$ is more efficient. However, their approach is different from ours. In [25], the randomness is introduced over the patterns of the partition of $\mathcal{X}$. Moreover, their identification function corresponding to our ${\overline{\mathit{u}}}_{\theta }^I(x)$ is decomposed into two terms; one is the term conditioned on the partition and the other is its orthogonal complement. On the other hand, our approach uses the decomposition of ${\mathit{u}}_{\theta }^I(x,e)$ into ${\overline{{\mathit{u}}_{\theta }}}^I(x)$ and its orthogonal complement. In their analysis, the simplified expression of the asymptotic variance shown in (9) and the standard expression of the identification function, $\mathit{f}(x,e)={\mathit{u}}_{\theta }^I(x,e)+\mathit{a}(x,e)$, are not used. Hence, the evaluation of the asymptotic variance yields rather a complex dependency on the estimator. As a result, their approach does not show the efficiency bound, though the asymptotic variance of the composite likelihood for exponential families is presented under the misspecified setup.

5.3.2. Closed Exponential Families

The so-called closed exponential family has an interesting property from the viewpoint of localized estimators, as presented in [26]. Let $p_{\theta }(x)=exp\{{\theta }^{T}t(x)-c(\theta )\}$ be the exponential family defined for $x=(x_1,\dots ,x_n)\in \mathcal{X}={\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_n$ with the parameter $\theta \in \mathrm{\Theta }\subset {\mathbb{R}}^d$. The function $t(x)\in {\mathbb{R}}^d$ is referred to as the sufficient statistic. Given disjoint index subsets $A,B\subset \{1,\dots ,n\}$, let $t_B(x)$ be all elements of $t(x)$ that depend just on $x_B$, and $t_{A,B}(x)$ be the other elements. Hence, $t_B(x)$ is expressed as $t_B(x_B)$. The parameter θ is correspondingly decomposed into $\theta =({\theta }_{A,B},{\theta }_B)$. Thus, we have ${\theta }^{T}t(x)={\theta }_{A,B}^T{t}_{A,B}(x)+{\theta }_B^T{t}_B(x_B)$. The exponential family $p_{\theta }(x)$ is called the closed exponential family, when the marginal distribution of $x_B$ is expressed as the exponential family with the sufficient statistic $t_B(x_B)$.

We consider the composite likelihood of the closed exponential family. For the pairs of two disjoint index subsets, $\{A_{\ell },B_{\ell }\},\ell =1,\dots ,M$, suppose that any element of $t(x)$ is included in $t_{A_{\ell },B_{\ell }}(x)$ at least one ℓ. Then, the local Z-estimator using the composite likelihood ${\sum }_{\ell =1}^{M}log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})$ is identical to the MLE [26]. Hence, the composite likelihood of the closed exponential family attains the efficiency bound of the MLE.

For the general statistical model $p_{\theta }(x)$, let us restate the above result in terms of the tangent spaces in $T_{\theta ,q}$. Let us decompose $p_{\theta }(x)$ into

$$\begin{array}{c}\hfill p_{\theta }(x)=p(x_A|x_B;\theta )p(x_B;\theta ).\end{array}$$

We assume that for any index subset B, all elements of $\nabla logp(x_B;\theta )$ are included in $T_{\theta ,q}^{ML}$ that is spanned by the elements of ${\mathit{u}}_{\theta }(x)=\nabla log{p}_{\theta }(x)$. Then, $\nabla logp(x_A|x_B;\theta )$ also lies in ${(T_{\theta ,q}^{ML})}^d$. Thus, $\nabla logp(x_{A_{\ell }}|x_{B_{\ell }};\theta )$ is expressed as $C_{\ell }\nabla log{p}_{\theta }(x)$ using a d by d matrix $C_{\ell }$. If ${\sum }_{\ell =1}^M{C}_{\ell }$ is invertible, the local Z-estimator obtained by ${\sum }_{\ell =1}^M\nabla log{p}_{\theta }(x_{A_{\ell }}|x_{B_{\ell }})$ is identical to the MLE. In this case, ${\Pi }_X{T}_{\theta ,q}^I=T_{\theta ,q}^{ML}$, i.e., $T_{\theta ,q}^{XI}=T_{\theta ,q}^{ML}$ holds. Therefore, there is no information loss caused by the localization. The matrix $C_{\ell }$ for the closed exponential family is given as the projection matrix onto the subspace spanned by $t_{A_{\ell },B_{\ell }}(x)-{\mathbb{E}}_{\theta }[t_{A_{\ell },B_{\ell }}|x_B]$ that is included in $T_{\theta ,q}^{ML}$. The above result implies that the tangent space $T_{\theta ,q}^{XC}$ expressing the information loss will be related to the score of the marginal distribution, $\nabla log{p}_{\theta }(x_B)$.

6. Conclusions

In this paper, some statistical properties of stochastic local Z-estimators and local Z-estimators are investigated. The class of local Z-estimators includes pseudo-likelihood and composite likelihood. For stochastic local Z-estimators, we established the exact relation between neighborhood systems and the efficiency bound under a simple and general condition. In addition, the efficiency bound of the local Z-estimators was presented.

Future works include the study of more general class of localized estimators. Indeed, local Z-estimators do not include the class of proper local scoring rules [7]. It is worthwhile to derive the efficiency bound for more general localized estimators. Exploring nice applications of the efficiency bound will be another interesting direction of our study. In our setup, the local efficient score expressed by the projection of the score attains the efficiency bound among local Z-estimators. An important problem is to develop a computationally tractable method to obtain the projection onto tangent subspaces.

one