On the Convergence Rate of the SCAD-Penalized Empirical Likelihood Estimator

Abstract

This paper investigates the asymptotic properties of a penalized empirical likelihood estimator for moment restriction models when the number of parameters ($p_n$) and/or the number of moment restrictions increases with the sample size. Our main result is that the SCAD-penalized empirical likelihood estimator is $\sqrt{n/p_n}$-consistent under a reasonable condition on the regularization parameter. Our consistency rate is better than the existing ones. This paper also provides sufficient conditions under which $\sqrt{n/p_n}$-consistency and an oracle property are satisfied simultaneously. As far as we know, this paper is the first to specify sufficient conditions for both $\sqrt{n/p_n}$-consistency and the oracle property of the penalized empirical likelihood estimator.

1. Introduction

Recently, sparse regression models have received considerable attention in business, economics, genetics, and various other fields. In these models, the number of possible regressors can be potentially large; however, only a relatively small number of these regressors are relevant.

Penalization is an alternative to a classical subset selection. One of the drawbacks of subset selection is lack of stability due to its discrete nature, meaning that variables are either retained or are dropped from a model. As a result, a small perturbation in a sample may cause a drastic change in the post-selection results (Breiman 1996). Penalization addresses this issue by achieving variable selection and estimation simultaneously, through a continuous process.

Several penalization methods have been advocated for linear regression models. Examples include the bridge penalty (Frank and Friedman 1993), LASSO (Tibshirani 1996), the smoothly clipped absolute deviation (SCAD) penalty (Fan and Li 2001), and the elastic net penalty (Zou and Hastie 2005). However, penalized least squares methods are not applicable when endogeneity exists (Fan and Liao 2014). When endogeneity exists, parameters of interest are identified often by moment restrictions, using instrumental variables.

This study investigates the asymptotic properties of a penalized empirical likelihood (PEL) estimator for moment restriction models, when the number of parameters and/or the number of moment restrictions increases with the sample size. We extend the EL estimator of Qin and Lawless (1994) by employing the SCAD penalty, so that we can achieve estimation and variable selection simultaneously.

Some penalized estimators for moment restriction models have been proposed in the econometric literature. Caner (2009) and Shi (2016b) considered the GMM estimator with a LASSO-type penalty. Caner and Zhang (2014) proposed the adaptive elastic net GMM estimator. Fan and Liao (2014) proposed the penalized focused GMM estimator. Leng and Tang (2012) and Chang et al. (2015) studied the asymptotic properties of the PEL estimator for independent and weakly dependent observations, respectively. Tang et al. (2018) considered a penalized exponential tilting estimator.

This paper shows that the SCAD-penalized EL estimator is $\sqrt{n/p_n}$-consistent, where $p_n$ is the number of parameters. Leng and Tang (2012) showed that the non-penalized EL estimator is $\sqrt{n/p_n}$-consistent under the assumption that $p_n/r_n\to c\in (0,1)$, where $r_n$ is the number of moment restrictions. Thus, essentially, they only proved $\sqrt{n/r_n}$-consistency. Chang et al. (2015) proved $\sqrt{n/p_n}$-consistency of the non-penalized EL estimator without imposing $p_n/r_n\to c\in (0,1)$, but they only obtained $\sqrt{n/r_n}$-consistency for the PEL estimator. We prove $\sqrt{n/p_n}$-consistency of the PEL estimator under a reasonable condition on the regularization parameter of the penalty function. Our result is important because it implies $\sqrt{n}$-consistency of the estimator when $p_n$ is fixed and only $r_n$ increases with the sample size. This is consistent with previous results in the EL literature such as Donald et al. (2003). In contrast, $\sqrt{n/r_n}$-consistency implies that only a slow rate of convergence can be achieved even when $p_n$ is finite and fixed.

This paper also shows that the PEL estimator satisfies the oracle property in the sense of Fan and Peng (2004) when the truth is sparse. That is, if the true parameter vector has some zero components, then they are estimated as zeros with probability approaching one, and the other nonzero components are estimated well, similar to the case when the zero components are known a priori. Although Leng and Tang (2012) and Chang et al. (2015) also discussed the oracle property of the PEL estimator, they obtained their results under high-level assumptions. As far as we know, this paper is the first to specify sufficient conditions for both $\sqrt{n/p_n}$-consistency and the oracle property of the PEL estimator.

Recently, Chang et al. (2018) proposed an alternative PEL estimator that regularizes both parameters and Lagrange multipliers. Their estimator allows the case where $r_n$ and $p_n$ increase at an exponential rate, while our PEL estimator allows a polynomial rate only. Their method is useful when the truth is actually sparse. In contrast, our estimator is valid even when the truth is not sparse because $\sqrt{n/p_n}$-consistency can be established without imposing sparsity.

There is also a large literature on instrument (moment) selection that addresses the problem of selecting/constructing optimal instruments when a large number of instruments are available (e.g., Donald and Newey 2001; Bai and Ng 2009; Kuersteiner and Okui 2010; Belloni et al. 2012; Caner and Fan 2015; Cheng and Liao 2015; Shi 2016a). In contrast to these papers, here we focus on variable selection in a structural model.

This paper is organized as follows. We first show $\sqrt{n/p_n}$-consistency of the SCAD-penalized EL estimator and compare our assumptions with those of Leng and Tang (2012) and Chang et al. (2015). Then, we obtain the asymptotic distribution. Our proofs are new in the EL literature. All the proofs are found in the Appendix A.

2. PEL Estimator and Asymptotic Results

Let $\{y_1,\cdots ,y_n\}$ be a random sample from an unknown distribution on ${\mathbb{R}}^{d_n}$. This study considers the moment restriction model

$$E\left[\mathit{m}(y_i,{\mathbf{\theta }}_0)\right]=\mathbf{0},$$

where ${\mathit{\theta }}_0={({\theta }_{10},\cdots ,{\theta }_{p_{n}0})}^{\prime }\in {\Theta }_n$ is a $p_n$-dimensional true parameter and

$$\mathit{m}(y,\mathbf{\theta })={({\mathit{m}}_1(y,\mathbf{\theta }),\cdots ,{\mathit{m}}_{r_n}(y,\mathbf{\theta }))}^{\prime }$$

is an $r_n$-dimensional moment function. For instance, the model includes the linear instrumental variable model

$$E\left[z_i(y_i-x_i^{\prime }{\mathbf{\theta }}_0)\right]=\mathbf{0},$$

where $z_i$ is an $r_n\times 1$ vector of instrumental variables and $x_i$ is a $p_n\times 1$ vector of explanatory variables. We consider the case where $r_n\ge p_n$. The subscript indicates that $d_n$, $p_n$, and $r_n$ may increase with the sample size.

The PEL estimator for ${\mathit{\theta }}_0$ is

$${\widehat{\mathbf{\theta }}}_n=arg\underset{\mathbf{\theta }\in {\Theta }_n}{min}\underset{\lambda \in {\widehat{\Lambda }}_n\left(\mathbf{\theta }\right)}{max}\left\{\frac{1}{n}\sum _{i=1}^{n}log(1-{\mathbf{\lambda }}^{\prime }\mathit{m}(y_i,\mathbf{\theta }))+\sum _{j=1}^{p_n}{p}_{{\kappa }_n}\left({\theta }_j\right)\right\},$$

where ${\widehat{\Lambda }}_n\left(\mathit{\theta }\right)=\{\mathit{\lambda }\in {\mathbb{R}}^{r_n}:{\mathit{\lambda }}^{\prime }\mathit{m}(y_i,\mathit{\theta })<1,i=1,\cdots ,n\}$ and $p_{\kappa }(·)$ is a penalty function with a regularization parameter $\kappa$. Thus, the estimator is the same as that of Leng and Tang (2012).

For concreteness, we employ the SCAD penalty of Fan and Li (2001):

$$\begin{array}{c}\hfill p_{\kappa }\left(u\right)=\left\{\begin{array}{cc}\kappa \left|u\right|\hfill & \left|u\right|\le \kappa \hfill \\ -(u^2-2a\kappa |u|+{\kappa }^2)/\left[2(a-1)\right]\hfill & \kappa <\left|u\right|\le a\kappa \hfill \\ (a+1){\kappa }^2/2\hfill & \left|u\right|>a\kappa \hfill \end{array}\right.\end{array}$$

for some $a>2$. Similar asymptotic results are obtained also by using a different penalty function, such as the minimax concave penalty of Zhang (2010).

The true model may be sparse, that is, some elements of ${\mathit{\theta }}_0$ may be zero. Let $q_n$ be the number of nonzero elements in ${\mathit{\theta }}_0$. Without loss of generality, we can write ${\mathit{\theta }}_0={({\mathit{\theta }}_{10}^{\prime },{\mathit{\theta }}_{20}^{\prime })}^{\prime }={({{\mathit{\theta }}_{10}}^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ with ${\mathit{\theta }}_1={({\theta }_1,\cdots ,{\theta }_{q_n})}^{\prime }\in {\mathbb{R}}^{q_n}$ and ${\mathit{\theta }}_2={({\theta }_{q_n+1},\cdots ,{\theta }_{p_n})}^{\prime }\in {\mathbb{R}}^{p_n-q_n}$. For now, the sparsity assumption is not crucial. It is possible that $q_n=p_n$.

Let ${\mathit{m}}_i\left(\mathit{\theta }\right)=\mathit{m}(y_i,\mathit{\theta })$ and $M_i\left(\mathit{\theta }\right)=\partial {\mathit{m}}_i\left(\mathit{\theta }\right)/\partial {\mathit{\theta }}^{\prime }$. Also, let ${\mathit{m}}_i={\mathit{m}}_i\left({\mathit{\theta }}_0\right)$ and $M_i=M_i\left({\mathit{\theta }}_0\right)$. We define $Q_n(\mathit{\theta },\mathit{\lambda })=E[log(1-{\mathit{\lambda }}^{\prime }{\mathit{m}}_i\left(\mathit{\theta }\right))]$ and ${\widehat{Q}}_n(\mathit{\theta },\mathit{\lambda })=n^{-1}{\sum }_{i=1}^{n}log(1-{\mathit{\lambda }}^{\prime }{\mathit{m}}_i\left(\mathit{\theta }\right))$. Moreover, we use $\mathit{\lambda }\left(\mathit{\theta }\right)$ and $\widehat{\mathit{\lambda }}\left(\mathit{\theta }\right)$ to denote $arg{max}_{\lambda \in {\Lambda }_n\left(\theta \right)}{Q}_n(\mathit{\theta },\mathit{\lambda })$ and $arg{max}_{\lambda \in {\widehat{\Lambda }}_n\left(\theta \right)}{\widehat{Q}}_n(\mathit{\theta },\mathit{\lambda })$, respectively, where ${\Lambda }_n\left(\mathit{\theta }\right)$ is a subset in ${\mathbb{R}}^{r_n}$, such that $\mathbf{0}\in \mathrm{int}\left({\Lambda }_n\left(\mathit{\theta }\right)\right)$. Let ${\lambda }_{min}\left(A\right)$ and ${\lambda }_{max}\left(A\right)$ denote the minimum and maximum eigenvalues of a matrix A. Also, let $\parallel ·\parallel$ denote the Euclidean (Frobenius) norm.

We impose the following conditions for $\sqrt{n/p_n}$-consistency.

Assumption 1.

(i) The true parameter vector ${\mathit{\theta }}_0$ is the unique minimizer of $Q_n(\mathit{\theta },\mathit{\lambda }\left(\mathit{\theta }\right))$ and belongs to the interior of ${\Theta }_n$; (ii) There are positive functions ${\Delta }_1(r,p)$ and ${\Delta }_2\left(ϵ\right)$ such that for any $ϵ>0$

$$\underset{\{\mathbf{\theta }\in {\Theta }_n:\parallel \mathbf{\theta }-{\mathbf{\theta }}_0\parallel >ϵ\}}{inf}{Q}_n(\mathbf{\theta },\mathbf{\lambda }\left(\mathbf{\theta }\right))\ge {\Delta }_1(r_n,p_n){\Delta }_2\left(ϵ\right)>0,$$

where ${lim\; inf}_{n\to \infty }{\Delta }_1(r_n,p_n)>0$; (iii) ${sup}_{\theta \in {\Theta }_n}\left|{\widehat{Q}}_n(\mathit{\theta },\mathit{\lambda }\left(\mathit{\theta }\right))-Q_n(\mathit{\theta },\mathit{\lambda }\left(\mathit{\theta }\right))\right|=o_p\left({\Delta }_1(r_n,p_n)\right)$.

Assumption 2.

(i) $E[{sup}_{\mathit{\theta }\in {\Theta }_n}(\parallel {\mathit{m}}_i\left(\mathit{\theta }\right)\parallel r_n^{-1/2}{)}^{\alpha }]<\infty$ for some $\alpha >4$; (ii) ${lim}_{n\to \infty }{r}_n^4/n=0$.

Assumption 3.

(i) There exists C such that $0<1/C\le {\lambda }_{min}\left(E\left[{\mathit{m}}_i\left(\mathit{\theta }\right){\mathit{m}}_i{\left(\mathit{\theta }\right)}^{\prime }\right]\right)\le$${\lambda }_{max}\left(E\left[{\mathit{m}}_i\left(\mathit{\theta }\right){\mathit{m}}_i{\left(\mathit{\theta }\right)}^{\prime }\right]\right)<C<\infty$ in a neighborhood of ${\mathit{\theta }}_0$; (ii) There exists C such that ${\lambda }_{max}\left(E{\left[M_i\right]}^{\prime }E\left[M_i\right]\right)<C<\infty$; (iii) There exists C such that ${\lambda }_{max}\left(E\left[M_i\left(\mathit{\theta }\right)M_i{\left(\mathit{\theta }\right)}^{\prime }\right]\right)<C<\infty$ in a neighborhood of ${\mathit{\theta }}_0$.

Assumption 4.

(i) The moment function $\mathit{m}(y,\mathit{\theta })$ is twice continuously differentiable in $\mathit{\theta }$ for all y in a neighborhood of ${\mathit{\theta }}_0$; (ii) There exists C such that ${\lambda }_{min}\left(\frac{d^2{\widehat{Q}}_n(\mathit{\theta },\widehat{\mathit{\lambda }}\left(\mathit{\theta }\right))}{d\mathit{\theta }d{\mathit{\theta }}^{\prime }}\right)\ge C>0$ in a neighborhood of ${\mathit{\theta }}_0$ with probability approaching one.

Assumption 5.

${lim}_{n\to \infty }\sqrt{q_n}{\kappa }_n/{min}_{1\le j\le q_n}\left|{\theta }_{j0}\right|=0$.

Assumption 1 is similar to condition 2.1 of Chang et al. (2015). Assumption 1 (iii) is an extension of the uniform convergence. If we restrict the parameter space such that ${\Theta }_n$ is compact and $E[{sup}_{\mathit{\theta }\in {\Theta }_n}log(1-\mathit{\lambda }{\left(\mathit{\theta }\right)}^{\prime }{\mathit{m}}_i\left(\mathit{\theta }\right))]<\infty$, then Assumption 1 (iii) is satisfied with ${\Delta }_1(r,p)=1$. Assumption 1 is used to show that $\parallel {\widehat{\mathit{\theta }}}_n-{\mathit{\theta }}_0\parallel =o_p\left(1\right)$. Any condition that guarantees consistency of the estimator can replace 1.

Assumptions 2 (i) and (ii) are similar to Assumptions 2 and 4 in Leng and Tang (2012). However, we do not assume that $p_n/r_n\to c\in (0,1)$. Thus, $r_n$ can grow faster than $p_n$. We can allow the case where $p_n$ is fixed and only $r_n$ increases with the sample size.

Assumption 4 states that the objective function of the EL estimator is strictly convex in $\mathit{\theta }$ in a neighborhood of ${\mathit{\theta }}_0$. When $r_n$ and $p_n$ are fixed, this condition is satisfied under fairly weak conditions. We can also relax the condition so that ${\lambda }_{min}\left(\frac{d^2{\widehat{Q}}_n(\mathbf{\theta },\widehat{\mathbf{\lambda }}\left(\mathbf{\theta }\right))}{d\mathbf{\theta }d{\mathbf{\theta }}^{\prime }}\right)\ge {\rho }_n$ with a positive sequence ${\rho }_n$ such that ${\rho }_n\to 0$. In that case, we obtain a different convergence rate of the estimator. Under certain conditions, we have $\parallel {\widehat{\mathit{\theta }}}_n-{\mathit{\theta }}_0\parallel =O_p(\sqrt{p_n/n}/{\rho }_n)$.

Assumption 5 is similar to condition (B2) in Huang and Xie (2007), who obtained the convergence rate of the SCAD-penalized least squares estimator. Assumption 5 states that the minimum of nonzero elements in ${\mathit{\theta }}_0$ may converge to 0, but the convergence rate must be sufficiently slow. If nonzero elements are too small compared to ${\kappa }_n$, then the PEL estimator cannot distinguish between zero and nonzero elements. Following Huang and Xie (2007), we prove $\sqrt{n/p_n}$-consistency of the PEL estimator in two steps. We first prove $\parallel {\widehat{\mathit{\theta }}}_n-{\mathit{\theta }}_0\parallel =O_p(\sqrt{p_n/n}+{\sqrt{q}}_n{\kappa }_n)$ under Assumptions 1–4 and $q_n{\kappa }_n^2\to 0$ (see Lemma A3 in the Appendix A). Then, we improve the convergence rate by using Assumption 5. Notice that if we assume ${\sqrt{q}}_n{\kappa }_n=O\left(\sqrt{p_n/n}\right)$, then $\sqrt{n/p_n}$-consistency of the PEL estimator is obtained immediately from Lemma A3. However, as we will see later, this condition contradicts Assumption 6 (i), which is a key condition for the oracle property. Assumption 5 is imposed so that $\sqrt{n/p_n}$-consistency and the oracle property are satisfied simultaneously.

Theorem 1.

Suppose that Assumptions 1–5 hold. Then, we have $\parallel {\widehat{\mathit{\theta }}}_n-{\mathit{\theta }}_0\parallel =O_p\left(\sqrt{p_n/n}\right)$.

The sparsity assumption is not necessary for this theorem. The same result is obtained even if all elements in ${\mathit{\theta }}_0$ are nonzero. Moreover, because Assumption 5 does not exclude ${\kappa }_n=0$, the theorem also applies to the non-penalized EL estimator, whose $\sqrt{n/p_n}$-consistency has been established by Chang et al. (2015). As we will see in the next theorem, if the truth is sparse, then we obtain $\sqrt{n/q_n}$-consistency of the PEL estimator under certain additional assumptions.

Our convergence rate of the PEL estimator is better than that of Chang et al. (2015). Roughly speaking, different convergence rates are based on different equalities. The asymptotic analyses of Leng and Tang (2012) and Chang et al. (2015) are based on the moment equality $E\left[{\mathit{m}}_i\right]=\mathbf{0}$, which implies $\parallel n^{-1}{\sum }_{i=1}^n{\mathit{m}}_i\parallel =O_p\left(\sqrt{r_n/n}\right)$. Leng and Tang (2012) obtained $\sqrt{n/p_n}$-consistency of the non-penalized EL estimator by assuming $r_n=O\left(p_n\right)$ and hence $\parallel n^{-1}{\sum }_{i=1}^n{\mathit{m}}_i\parallel =O_p\left(\sqrt{p_n/n}\right)$. On the other hand, our asymptotic analysis is based on the first-order condition $E\left[\frac{dlog(1-\mathbf{\lambda }{\left({\mathbf{\theta }}_0\right)}^{\prime }{\mathit{m}}_i\left({\mathbf{\theta }}_0\right))}{d\mathbf{\theta }}\right]=\mathbf{0}$, which implies $∥\frac{d{\widehat{Q}}_n({\mathit{\theta }}_0,{\widehat{\mathit{\lambda }}}_n\left({\mathit{\theta }}_0\right))}{d\mathit{\theta }}∥=O_p\left(\sqrt{p_n/n}\right)$. Therefore, our proof is not a straightforward extension of that of Leng and Tang (2012) and Chang et al. (2015).

To obtain a convergence rate in line with the proof of Leng and Tang (2012) and Chang et al. (2015), we need a rather strong condition on the regularization parameter. For instance, Chang et al. (2015) assumed that $q_n{\kappa }_n{r}_n^{-1}n{M}^{-1}=O\left(1\right)$ to prove $\sqrt{n/r_n}$-consistency, where M is the block length, which is equal to unity when the observations are independent. The condition of Chang et al. (2015) corresponds to the condition that $\sqrt{q_n}{\kappa }_n=o\left(\sqrt{p_n/n}\right)$ in our case. As stated before, although this condition simplifies the proof of $\sqrt{n/p_n}$-consistency, it causes a problem for the oracle property of the estimator.

Next, we show sparsity and asymptotic normality of the PEL estimator. Let ${\widehat{\mathit{\theta }}}_{1n}$ and ${\widehat{\mathit{\theta }}}_{2n}$ be the corresponding estimators of ${\mathit{\theta }}_{10}$ and ${\mathit{\theta }}_{20}$, respectively. Furthermore, let $M_{1i}=\partial {\mathit{m}}_i({\mathit{\theta }}_{10},\mathbf{0})/\partial {\mathit{\theta }}_1^{\prime }$. We define $V_n={\left(E{\left[M_i\right]}^{\prime }E{\left[{\mathit{m}}_i{\mathit{m}}_i^{\prime }\right]}^{-1}E\left[M_i\right]\right)}^{-1}$ and $V_{1n}={\left(E{\left[M_{1i}\right]}^{\prime }E{\left[{\mathit{m}}_i{\mathit{m}}_i^{\prime }\right]}^{-1}E\left[M_{1i}\right]\right)}^{-1}$.

We impose additional conditions.

Assumption 6.

(i) ${lim}_{n\to \infty }\sqrt{n/p_n}{\kappa }_n=\infty$; (ii) ${lim}_{n\to \infty }{r}_n{p}_n^{3/2}/\sqrt{n}=0$

Assumption 7.

There exists $B_{jkl}\left(y\right)$ such that $|{\partial }^2{\mathit{m}}_l(y,\mathit{\theta })/\partial {\theta }_j\partial {\theta }_k|\le B_{jkl}\left(y\right)$ and $E\left[B_{jkl}^2\left(y_i\right)\right]<\infty$ for all $j,k=1,\cdots ,p_n$ and $l=1,\cdots ,r_n$ in a neighborhood of ${\mathit{\theta }}_0$.

Assumption 8.

There exists C such that $0<1/C\le {\lambda }_{min}\left(V_n\right)\le {\lambda }_{max}\left(V_n\right)\le C<\infty$.

Assumption 6 (i) is a key condition for sparsity of the PEL estimator. It requires that the regularization parameter is not too small so that zero elements in ${\mathit{\theta }}_0$ are estimated as zero. The same condition is also employed by Leng and Tang (2012).

Theorem 2.

Suppose that Assumptions 1–8 hold. Let $B_n$ be an $l\times q_n$ matrix such that $B_n{B}_n^{\prime }\to G$, where G is an $l\times l$ matrix with fixed l. Then, the PEL estimator satisfies the following:

1. 

Sparsity: ${\widehat{\mathit{\theta }}}_{2n}=\mathbf{0}$ with probability approaching one.

2. 

$\sqrt{n/q_n}$-consistency: $\parallel {\widehat{\mathit{\theta }}}_{1n}-{\mathit{\theta }}_{10}\parallel =O_p\left(\sqrt{q_n/n}\right)$.

3. 

Asymptotic normality: $\sqrt{n}{B}_n{V}_{1n}^{-1/2}({\widehat{\mathit{\theta }}}_{1n}-{\mathit{\theta }}_{10})\stackrel{d}{\to }N(0,G).$

The selection of the matrix $B_n$ depends on the parameter of interest. For instance, suppose that the parameter of interest is the first element of ${\mathit{\theta }}_{10}$. Let ${\widehat{\theta }}_{1n,1}$ and ${\theta }_{10,1}$ denote first elements of ${\widehat{\mathit{\theta }}}_{1n}$ and ${\mathit{\theta }}_{10}$, respectively. Then, we choose $B_n=(1,0,\cdots ,0)$ and obtain $\sqrt{n}({\widehat{\theta }}_{1n,1}-{\theta }_{10,1})\stackrel{d}{\to }N(0,v_{11})$, where $v_{11}$ is the limit of the first diagonal element of $V_{1n}$.

Although a detailed proof is given in the Appendix A, we give a sketch of the proof for asymptotic normality here. If $\mathit{\lambda }\left(\mathit{\theta }\right)$ were known, then ${\mathit{\theta }}_0$ can be estimated by

$${\tilde{\mathbf{\theta }}}_n=arg\underset{\mathbf{\theta }\in {\Theta }_n}{min}\left\{\frac{1}{n}\sum _{i=1}^{n}log(1-\mathbf{\lambda }{\left(\mathbf{\theta }\right)}^{\prime }{\mathit{m}}_i\left(\mathbf{\theta }\right))+\sum _{j=1}^{p_n}{p}_{{\kappa }_n}\left({\theta }_j\right)\right\},$$

which is a penalized maximum likelihood estimator using a least favorable submodel of the moment restriction model (see Sueishi 2016, for instance). Because ${\tilde{\mathit{\theta }}}_n$ is the penalized maximum likelihood estimator, its distribution can be obtained in a manner similar to Fan and Peng (2004). We derive the asymptotic distribution of ${\widehat{\mathit{\theta }}}_n$ by showing that ${\widehat{\mathit{\theta }}}_n$ is asymptotically equivalent to ${\tilde{\mathit{\theta }}}_n$.

By modifying the proof of Theorem 2, we can obtain easily the asymptotic distribution of the non-penalized EL estimator. Because the asymptotic distribution of the non-penalized EL estimator has already been derived by Leng and Tang (2012), we omit the derivation. We see that the efficiency of the PEL estimator for ${\mathit{\theta }}_{10}$ is the same as that of the non-penalized EL estimator for which it is known a priori that ${\mathit{\theta }}_{20}=\mathbf{0}$. Thus, our estimator satisfies the oracle property in the sense of Fan and Peng (2004).

Theorem 2 is similar to Theorem 3 of Leng and Tang (2012). However, they proved sparsity by assuming that the PEL estimator is $\sqrt{n/p_n}$-consistent. They did not state explicitly the conditions under which the non-penalized and penalized EL estimators have the same convergence rate.

Chang et al. (2015) showed a similar result to Theorem 2 for weakly dependent observations. They obtained $\sqrt{n/r_n}$-consistency and sparsity under two separate ${\kappa }_n$ rate conditions. Specifically, they assume: (i) $q_n{\kappa }_n{r}_n^{-1}n{M}^{-1}=O\left(1\right)$ for $\sqrt{n/r_n}$-consistency and (ii) ${\kappa }_n\sqrt{n/r_n}{M}^{-1}\to \infty$ for sparsity. If condition (ii) is satisfied, however, condition (i) requires that $q_n\sqrt{n/r_n}\to 0$, which is clearly impossible. This causes a trouble because their proof of sparsity requires $\sqrt{n/r_n}$-consistency of the estimator. We relaxed condition (i) and obtained sufficient conditions under which both $\sqrt{n/p_n}$-consistency and sparsity are satisfied.

3. Conclusions

We investigated the asymptotic properties of the PEL estimator when the number of parameters and/or the number of moment restrictions increases with the sample size. In particular, we showed that the PEL estimator is $\sqrt{n/p_n}$-consistent under a reasonable condition on the regularization parameter. Although we cannot compare our results directly to those of Chang et al. (2015) because they allow weakly dependent observations, our convergence rate is improved over the existing ones. In terms of converge rate, our result is even better than Tang et al. (2018) and Chang et al. (2018), because their convergence rates depend also on the number of moment restrictions.

A crucial issue with the PEL estimation concerns selecting the size of the regularization parameter. The asymptotic theory does not tell us how to select the regularization parameter in practice. Although some selection methods are considered by Leng and Tang (2012), Shi (2016b), and Ando and Sueishi (2019), this is still an underdeveloped area of research.