Inference for the Linear IV Model Ridge Estimator Using Training and Test Samples

Abstract

The asymptotic distribution is presented for the linear instrumental variables model estimated with a ridge penalty and a prior where the tuning parameter is selected with a holdout sample. The structural parameters and the tuning parameter are estimated jointly by method of moments. A chi-squared statistic permits confidence regions for the structural parameters. The form of the asymptotic distribution provides insights on the optimal way to perform the split between the training and test sample. Results for the linear regression estimated by ridge regression are presented as a special case.

1. Introduction

This paper contributes to the asymptotic distribution theory for ridge parameter estimates for the linear instrumental variables model. The tuning parameter for the ridge penalty, denoted $\alpha$, is selected by splitting the data into a training sample and a test (or holdout) sample. In [1], the ridge penalty parameter is estimated jointly with the structural parameters and the asymptotic distribution is characterized as the projection of a stochastic process onto a cone. This gives the rate of convergence to the asymptotic distribution but does not provide guidance for inference. To allow inference, the closed form for the asymptotic distribution is presented. These new results allow for the calculation of confidence regions for the structural parameters. When the prior is equal to the population parameter value of the structural parameters, the tuning parameter is not identified. However, the structural parameter estimates are consistent and the asymptotic covariance is smaller than the asymptotic covariance of the two-stage least squares estimator.

A fundamental issue with ridge regression is the selection of the tuning parameter and its resulting impact on inference for the parameters of interest. One approach is to select a deterministic function of the sample size that shrinks to zero fast enough to not impact the asymptotic distribution. The resulting asymptotic distribution is then equivalent to the OLS asymptotic distribution [2]. An alternative approach is to select the tuning parameter with the observed data. The literature contains multiple ways to estimate the tuning parameter [3,4]. For these estimates, inference typically follows the approach stated in [5]. Conditional on a fixed $\alpha$, the ridge estimator’s covariance is a function of $\alpha$. The $\alpha$ is selected using the observed data and substituted into the covariance that was calculated assuming that $\alpha$ is fixed. This covariance is used to create a test statistic. In [5], the resulting tests are correctly referred to as approximate t-tests because the authors appreciate the internal inconsistency of using a covariance obtained assuming a fixed $\alpha$ with an $\alpha$ value estimated with the observed data. (This problem is well known in the literature, see [6,7,8,9,10].) Each estimate for the tuning parameter leads to a different approximate t-test which are typically compared using simulations. This has been the approach for the past 20 years [5,11]. (“When the ridge parameter k is determined from the data, the above arguments are no longer valid. Hence, to investigate the size and power of the approximate ridge-based t-type tests in such cases, a Monte Carlo simulation study is conducted” [5]. “Since a theoretical assessment among the test statistics is not possible, a simulation study has been conducted to evaluate the performance of the suggested test statistics” [11].) For other models, researchers have proposed alternative procedures to obtain hyperparameter (tuning parameter) estimates [12,13]. Like the previous ridge regression literature, these approaches have relied on simulations to demonstrate their behavior. For inference, these procedures would need to be extended to establish their asymptotic distributions. A third approach is followed in this paper. The tuning parameter is selected by splitting the sample into training and test samples. The tuning parameter defines a path from the prior to the IV estimator on the training sample. On this path, the tuning parameter is selected to minimize the prediction error on the test sample. This procedure is written as a method of moments estimation problem where the tuning parameter and the parameters of interest are simultaneously estimated. Inference is then performed using the joint asymptotic distribution.

A related literature concerns the distribution of some empirically selected ridge tuning parameters [14,15,16]. These approaches have relied on strong distribution assumptions (e.g., normal error). In addition, they are built on tuning parameters as functions of the data, where the functions are determined by assuming that the tuning parameter is fixed. This leads to an inconsistency because using the data to select the tuning parameter means that the tuning parameter is no longer fixed. In this paper, the inconsistency is avoided by estimating the structural parameters and the ridge tuning parameter simultaneously. Additionally, the method of moments framework permits weaker assumptions.

In [1], the asymptotic joint distribution for the parameters in the linear model and the ridge tuning parameter is characterized as the projection of a stochastic process onto a cone. This structure occurs because the probability limit of the ridge tuning parameter is on the boundary of the parameters space. (This leads to the same problem of consistently estimating a population parameter that is on the boundary of the parameter space [17,18,19,20].) This leads to a nonstandard asymptotic distribution that depends on the prior and the population parameter value of the structural parameters. When the prior is different from the population parameter value, the asymptotic distribution for the ridge tuning parameter is a mixture with a discrete mass of 1/2 at zero and a truncated normal over the positive reals. In addition, the asymptotic distribution for the structural parameters is normal with a nonzero mean. This mean and variance both contain the population parameter value. This prevents the calculation of t-statistics for individual parameter estimates. However, a hypothesis for the entire set of structural parameters can be tested using a chi-square test and this statistic can be inverted to give accurate confidence regions.

2. Ridge Estimator for Linear IV Model Using a Holdout Sample

Consider the linear instrumental variables model where Y is $n\times 1$, X is $n\times k$, and Z is $n\times m$ with $m\ge k$

$$\begin{array}{ccc}\hfill Y& =& X{\beta }_0+\epsilon \hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill X& =& Z{\Gamma }_0+u\hfill \end{array}$$

(2)

where the $m\times 1$ instruments are $z_i^{\prime }\sim iid$, with full rank $m\times m$ second moments $R_z=E\left[z_i^{\prime }{z}_i\right]$, and conditional on Z,

$$\left[\begin{array}{c}{\epsilon }_i\\ u_i\end{array}\right]\sim iid\left(0,\left[\begin{array}{cc}{\sigma }_{\epsilon }^2& {\Sigma }_{\epsilon u}\\ {\Sigma }_{u\epsilon }& {\Sigma }_U\end{array}\right]\right).$$

(3)

The IV, or 2SLS, estimator

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_{IV}& =& \underset{\beta }{arg\; min}\frac{1}{2n}{(Y-X\beta )}^{\prime }Z{\left(Z^{\prime }Z\right)}^{-1}{Z}^{\prime }(Y-X\beta )={\left(X^{\prime }{P}_{Z}X\right)}^{-1}{X}^{\prime }{P}_{Z}Y\hfill \end{array}$$

(4)

where $P_Z$ is the projection matrix for Z and has the asymptotic distribution $\sqrt{n}\left({\widehat{\beta }}_{IV}-{\beta }_0\right){\sim }^{a}N\left(0,{\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}\right).$

Let $\frac{X^{\prime }{P}_{Z}X}{n}$ have the spectral decomposition $CD{C}^{\prime }$, where C is orthonormal, i.e., $C^{\prime }C=I_k$ and D is a positive semidefinite diagonal $k\times k$ matrix of eigenvalues, ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$. When some of the eigenvectors explain very little variation, i.e., small magnitudes for the corresponding eigenvalues, the objective function is flatter along these dimensions and the resulting covariance estimates are larger because the variance of ${\widehat{\beta }}_{IV}$ is proportional to ${\left(\frac{X^{\prime }{P}_{Z}X}{n}\right)}^{-1}={\left(CD{C}^{\prime }\right)}^{-1}=C{D}^{-1}{C}^{\prime }.$ This leads to a relatively large MSE. The ridge estimator addresses this by shrinking the estimated parameter towards a prior. The ridge objective function augments the usual IV objective function (4) with a quadratic penalty centered at a prior, ${\beta }^p$, weighted by a regularization tuning parameter $\alpha$

$$\begin{array}{c}\hfill \frac{1}{2n}{(Y-X\beta )}^{\prime }{P}_Z(Y-X\beta )+\frac{1}{2}\alpha {(\beta -{\beta }^p)}^{\prime }(\beta -{\beta }^p).\end{array}$$

(5)

Conditional on $\alpha$, the ridge solution is

$${\widehat{\beta }}_{IV}\left(\alpha \right)={\left(\frac{X^{\prime }{P}_{Z}X}{n}+\alpha I_k\right)}^{-1}\left(\frac{X^{\prime }{P}_{Z}Y}{n}+\alpha {\beta }^p\right).$$

Different values of $\alpha$ result in different estimated values for ${\beta }_0$. An optimal value for $\alpha$ can be determined empirically by splitting the data into training and test samples. The training sample is a randomly drawn sample of $\left[\tau n\right]$ observations, denoted, $Y_{\tau n}$, $X_{\tau n}$, and $Z_{\tau n}$. The estimate using the training sample, conditional on $\alpha ,$ is

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_{tr}\left(\alpha \right)& \equiv & \underset{\beta }{arg\; min}\frac{1}{2\left[\tau n\right]}{\left(Y_{\tau n}-X_{\tau n}\beta \right)}^{\prime }{P}_{Z_{\tau n}}\left(Y_{\tau n}-X_{\tau n}\beta \right)+\frac{\alpha }{2}{(\beta -{\beta }^p)}^{\prime }(\beta -{\beta }^p)\hfill \end{array}$$

(6)

$$\begin{array}{ccc}& =& {\left(\frac{X_{\tau n}^{\prime }{P}_{Z_{\tau n}}{X}_{\tau n}}{\left[\tau n\right]}+\alpha I_k\right)}^{-1}\left(\frac{X_{\tau n}^{\prime }{P}_{Z_{\tau n}}{Y}_{\tau n}}{\left[\tau n\right]}+\alpha {\beta }^p\right)\hfill \end{array}$$

(7)

where $P_{Z_{\tau n}}$ is the projection matrix onto $Z_{\tau n}$ and $[·]$ is the greatest integer function. The optimal $\alpha$ is selected to minimize the IV least squares objective function over the remaining $(n-[\tau n\left]\right)$ observations, i.e., the test or holdout sample, denoted $Y_{n(1-\tau )}$, $X_{n(1-\tau )}$ and $Z_{n(1-\tau )}$. The estimated tuning parameter is defined by $\widehat{\alpha }={arg\; min}_{\alpha \in [0,\infty )}{Q}_{n(1-\tau )}\left(\alpha \right)$ where

$$Q_{n(1-\tau )}\left(\alpha \right)=\frac{1}{2(n-[n\tau \left]\right)}{\left(Y_{n(1-\tau )}-X_{n(1-\tau )}{\widehat{\beta }}_{tr}\left(\alpha \right)\right)}^{\prime }{P}_{Z_{n(1-\tau )}}\left(Y_{n(1-\tau )}-X_{n(1-\tau )}{\widehat{\beta }}_{tr}\left(\alpha \right)\right)$$

(8)

and $P_{Z_{n(1-\tau )}}$ is the projection matrix onto $Z_{n(1-\tau )}.$ The ridge regression estimate ${\widehat{\beta }}_{\widehat{\alpha }}\equiv {\widehat{\beta }}_{IV}\left(\widehat{\alpha }\right)$ is then characterized by

$$\begin{array}{ccc}\hfill -\frac{1}{n}{X}^{\prime }{P}_Z\left(Y-X{\widehat{\beta }}_{\widehat{\alpha }}\right)+\widehat{\alpha }({\widehat{\beta }}_{\widehat{\alpha }}-{\beta }^p)& =& 0.\hfill \end{array}$$

Ref. [1] showed how the asymptotic distribution of the ridge estimator can be determined with the method of moments framework using the parameterization

$$\theta ={\left[\begin{array}{ccccccc}\mathrm{vech}{\left(R_{\tau }\right)}^{\prime }& \mathrm{vech}{\left(R_{(1-\tau )}\right)}^{\prime }& \mathrm{vec}{\left(S_{\tau }\right)}^{\prime }& \mathrm{vec}{\left(S_{(1-\tau )}\right)}^{\prime }& {\beta }_{tr}^{\prime }& \alpha & {\beta }^{\prime }\end{array}\right]}^{\prime }$$

where $\mathrm{vec}(·)$ stacks the elements from a matrix into a column vector and $\mathrm{vech}(·)$ stacks the unique elements from a symmetric matrix into a column vector. The population parameter values are

$${\theta }_0={\left[\begin{array}{ccccccc}\mathrm{vech}{\left(R_z\right)}^{\prime }& \mathrm{vech}{\left(R_z\right)}^{\prime }& \mathrm{vec}{\left(R_z{\Gamma }_0\right)}^{\prime }& \mathrm{vec}{\left(R_z{\Gamma }_0\right)}^{\prime }& {\beta }_0^{\prime }& 0& {\beta }_0^{\prime }\end{array}\right]}^{\prime }.$$

The ridge estimator is part of the parameter estimates defined by the just identified system of equations $H_n\left(\theta \right)=\frac{1}{n}{\sum }_{i=1}^n{h}_i\left(\theta \right)=0$ where

$$h_i\left(\theta \right)=\left[\begin{array}{c}{\mathbf{1}}_{\tau }\left(i\right)\mathrm{vech}(R_{\tau }-z_i{z}_i^{\prime })\\ (1-{\mathbf{1}}_{\tau }\left(i\right))\mathrm{vech}(R_{(1-\tau )}-z_i{z}_i^{\prime })\\ {\mathbf{1}}_{\tau }\left(i\right)\mathrm{vec}(S_{\tau }-z_i{x}_i^{\prime })\\ (1-{\mathbf{1}}_{\tau }\left(i\right))\mathrm{vec}(S_{(1-\tau )}-z_i{x}_i^{\prime })\\ {\mathbf{1}}_{\tau }\left(i\right)\left(-S_{\tau }^{\prime }{R}_{\tau }^{-1}{z}_i(y_i-x_i^{\prime }{\beta }_{tr})+\alpha ({\beta }_{tr}-{\beta }^p)\right)\\ (1-{\mathbf{1}}_{\tau }\left(i\right))(y_i-x_i^{\prime }{\beta }_{tr})z_i^{\prime }{R}_{(1-\tau )}^{-1}{S}_{(1-\tau )}{\left(S_{\tau }^{\prime }{R}_{\tau }^{-1}{S}_{\tau }+\alpha I_k\right)}^{-1}({\beta }^p-{\beta }_{tr})\\ -{(\tau S_{\tau }+(1-\tau )S_{1-\tau })}^{\prime }{(\tau R_{\tau }+(1-\tau )R_{1-\tau })}^{-1})z_i(y_i-x_i^{\prime }\beta )+\alpha (\beta -{\beta }^p)\end{array}\right]$$

(9)

and the training and test samples are determined with the indicator function

$${\mathbf{1}}_{\tau }\left(i\right)=\left\{\begin{array}{cc}1,\hfill & i\le \left[\tau n\right]\hfill \\ 0,\hfill & \left[\tau n\right]<i.\hfill \end{array}\right.$$

Using the structure of Equation (9), the system $H_n\left(\theta \right)=\frac{1}{n}{\sum }_{i=1}^n{h}_i\left(\theta \right)=0$ can be seen as seven sets of equations. The first four sets are each self-contained systems of equal numbers of equations and parameters. The fifth set has k equations and introduces the k parameters, ${\beta }_{tr}$. The sixth is a single equation with parameter $\alpha$. The seventh introduces the final k parameters, $\beta$. Identification occurs because the expectation of the gradient is invertible. This is presented in the Appendix A.

3. Asymptotic Behavior

The asymptotic distribution is derived with four high level assumptions.

Assumption 1.

$z_i$ is iid with finite fourth moments and $E\left[z_i{z}_i^{\prime }\right]=R_z$ has full rank.

Assumption 2.

Conditional on Z, ${\left[\begin{array}{cc}{\epsilon }_i& u_i^{\prime }\end{array}\right]}^{\prime }$ are iid vectors with zero mean, full rank covariance matrix with possibly nonzero off-diagonal elements.

Assumptions 1 and 2 imply $E\left[h_i\left({\theta }_0\right)\right]=0$ and $\sqrt{n}{H}_n\left({\theta }_0\right)$ satisfies the CLT.

Assumption 3.

The parameter space Θ is defined as follows: $R_z$ is restricted to a symmetric positive definite matrix with eigenvalues $1/B_1\le e_1\le e_2\le \dots \le e_m\le B_1,$ $\left|{\beta }_j\right|\le B_2$ for $j=1,2,\dots ,k$, ${\Gamma }_0=\left[{\gamma }_{\ell ,j}\right]$ is of full rank with $\left|{\gamma }_{\ell ,j}\right|\le B_3$ for $\ell =1,\dots ,m$, $j=1,2,\dots ,k$ and $\alpha \in [0,B_4]$ where $B_1$, $B_2$, $B_3$, and $B_4$ are positive and finite.

Assumption 4.

The fraction of the sample used for training satisfies $0<\tau <1$.

The tuning parameter selected using a holdout sample is root-n when the prior is different from the population parameter value. When the prior is equal to the population parameter value, the tuning parameter is not identified.

Lemma 1.

Assumptions 1–4imply, when ${\beta }^p\ne {\beta }_0$, (1) $\widehat{\alpha }{\to }^{p}0$ and (2) $\sqrt{n}\widehat{\alpha }=O_p\left(1\right)$, and when ${\beta }^p={\beta }_0$, $\widehat{\alpha }$ converges in distribution to a draw from the distribution for ${\alpha }_{min}$,

$$\begin{array}{ccc}\hfill {\alpha }_{min}& \equiv & \underset{a\in [0,\infty )}{arg\; min}{\left(Z_{(1-\tau )}-R_z^{1/2}{\Gamma }_0{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0+a{I}_k\right)}^{-1}{\Gamma }_0^{\prime }{R}_z^{1/2}{Z}_{\tau }\right)}^{\prime }\hfill \\ & & \times \left(Z_{(1-\tau )}-R_z^{1/2}{\Gamma }_0{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0+a{I}_k\right)}^{-1}{\Gamma }_0^{\prime }{R}_z^{1/2}{Z}_{\tau }\right)\hfill \end{array}$$

where $Z_{(1-\tau )}$ and $Z_{\tau }$ are $\sim iidN(0,I_m)$ and $R_z^{1/2}$ is the symmetric matrix square root of $R_z$.

Proofs are given in the Appendix A. The a-min distribution with, $m\times k$ matrix parameter S, is characterized by

$$\begin{array}{ccc}& & \underset{a\in [0,\infty )}{arg\; min}{\left(Z_1-S{\left(S^{\prime }S+a{I}_k\right)}^{-1}{S}^{\prime }{Z}_2\right)}^{\prime }\left(Z_1-S{\left(S^{\prime }S+a{I}_k\right)}^{-1}{S}^{\prime }{Z}_2\right)\hfill \end{array}$$

where $Z_1$ and $Z_2$ are $\sim iidN(0,I_m)$. When ${\beta }^p={\beta }_0$, $\widehat{\alpha }$ converges in distribution to a draw from the a-min distribution with parameter $R_z^{1/2}{\Gamma }_0$.

When ${\beta }^p={\beta }_0$, $\alpha$ is no longer identified. Recall that $\alpha$ parameterizes a path from the prior to the IV estimator on the training sample. However, when the prior equals ${\beta }_0$ and the IV estimator is consistent for ${\beta }_0$, every value of $\alpha$ will be associated with a consistent estimator for ${\beta }_0$.

Lemma 1 implies that the probability limit for the tuning parameter is zero, ${\alpha }_0=0$, which is on the boundary of the parameter space. This results in a nonstandard asymptotic distribution, which is characterized by the projection of a random vector on a cone (denoted $\Lambda$) that allows for the sample estimate to be on the boundary of the parameter space. The estimation objective function can be expanded into a quadratic approximation about the centered and scaled population parameter values

$$\begin{array}{ccc}& & n{H}_n{\left(\theta \right)}^{\prime }{H}_n\left(\theta \right)\hfill \\ & =& n{H}_n{\left({\theta }_0\right)}^{\prime }{H}_n\left({\theta }_0\right)+2n{H}_n\left({\theta }_0\right)\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}(\theta -{\theta }_0)\hfill \\ & & +n{(\theta -{\theta }_0)}^{\prime }\left\{\frac{\partial H_n{\left({\theta }_0\right)}^{\prime }}{\partial \theta }\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right\}(\theta -{\theta }_0)+o_p\left(1\right)\hfill \\ & =& n{\left(H_n\left({\theta }_0\right)+\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}(\theta -{\theta }_0)\right)}^{\prime }\left(H_n\left({\theta }_0\right)+\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}(\theta -{\theta }_0)\right)+o_p\left(1\right)\hfill \\ & =& {\left({\left(\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right)}^{-1}\sqrt{n}{H}_n\left({\theta }_0\right)+\sqrt{n}(\theta -{\theta }_0)\right)}^{\prime }\left\{\frac{\partial H_n{\left({\theta }_0\right)}^{\prime }}{\partial \theta }\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right\}\hfill \\ & & \times \left({\left(\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right)}^{-1}\sqrt{n}{H}_n\left({\theta }_0\right)+\sqrt{n}(\theta -{\theta }_0)\right)+o_p\left(1\right).\hfill \end{array}$$

This suggests selecting $\widehat{\theta }$ to minimize $H_n{\left(\theta \right)}^{\prime }{H}_n\left(\theta \right)$ results in the asymptotic distribution of $\sqrt{n}(\widehat{\theta }-{\theta }_0)$ being equivalent to the distribution of ${arg\; min}_{\lambda \in \Lambda }{(\mathcal{Z}+\lambda )}^{\prime }{M}_0^{\prime }{M}_0(\mathcal{Z}+\lambda )$, where the random variable is defined as

$$\begin{array}{ccc}& & \mathcal{Z}=\underset{n\to \infty }{lim}{\left(E\left[\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right]\right)}^{-1}\sqrt{n}{H}_n\left({\theta }_0\right),M_0=E\left[\frac{\partial H_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right],\hfill \end{array}$$

and the cone is defined by $\Lambda \equiv \left\{\lambda \in R^{m(m+1)/2+2km+2k+1}:{\lambda }_{m(m+1)/2+2km+k+1}\ge 0\right\}$. The estimator is defined as $\widehat{\theta }={arg\; min}_{\theta \in \mathsf{\Theta }}{H}_n{\left(\theta \right)}^{\prime }{H}_n\left(\theta \right)$ and its asymptotic distribution is characterized in Theorem 1 of [1]. For continuity of presentation, the theorem is repeated here.

Theorem 1.

Assumptions 1–4imply that, when ${\beta }^p\ne {\beta }_0$, the asymptotic distribution of $\sqrt{n}(\widehat{\theta }-{\theta }_0)$ is equivalent to the distribution of

$$\widehat{\lambda }=\underset{\lambda \in \Lambda }{arg\; min}{(\mathcal{Z}+\lambda )}^{\prime }{M}_0^{\prime }{M}_0(\mathcal{Z}+\lambda ).$$

The objective function can be minimized at a value of the tuning parameter in $(0,\infty )$ or possibly at $\alpha =0.$ The asymptotic distribution of the tuning parameter will be composed of two parts, a discrete mass at $\alpha =0$ and a continuous function over $(0,\infty )$. The asymptotic distribution is characterized as the projection of a stochastic process onto a cone. The special structure of the ridge estimator using a holdout sample permits the calculation of the closed form for the asymptotic distribution for the parameter of interest, see Theorem 1 in [17], case 2 after Theorem 2 in [18], Section 3.8 in [19], and Theorem 5 in [20].

Theorem 2.

Assumptions 1–4imply, when ${\beta }^p\ne {\beta }_0$,

(i) $\sqrt{n}(\widehat{\beta }-{\beta }_0)$ converges in distribution to a draw from a normal distribution with mean

$$-{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)\sqrt{\frac{{\sigma }_{\epsilon }^2}{2\pi \tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}}$$

and covariance

$${\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}+\left(\frac{1}{\tau (1-\tau )}\right)\left(\frac{{\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}}{{({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}\right),$$

and

(ii) $\sqrt{n}\widehat{\alpha }$ will converge in distribution to a mixture distribution with discrete mass of 1/2 at zero and over $[0,\infty )$, a truncated normal distribution with zero mean and covariance

$$\frac{{\sigma }_{\epsilon }^2}{\tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}.$$

When the prior is different from the population parameter value, the ridge estimator has an asymptotic bias and its asymptotic variance is larger than for the 2SLS. However, this is restricted to only one dimension. The asymptotic MSE is

$$\begin{array}{ccc}& & {\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}+\left(\frac{(2\pi n+1)}{2\pi \tau (1-\tau )n}\right)\left(\frac{{\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}}{{({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}\right)\hfill \\ & =& {\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1/2}\left\{I_k+\left(\frac{(2\pi n+1)}{2\pi \tau (1-\tau )n}\right)P_{{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1/2}({\beta }_0-{\beta }^p)}\right\}{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1/2}.\hfill \end{array}$$

This is the MSE for the 2SLS estimator plus a term built on $P_{{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1/2}({\beta }_0-{\beta }^p)}$, the projection matrix for ${\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1/2}({\beta }_0-{\beta }^p).$ The ridge estimator using the holdout sample has the same bias, variance, and MSE as the 2SLS estimator, except in the dimension of ${\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1/2}({\beta }_0-{\beta }^p)$. Because $\frac{1}{\tau (1-\tau )}$ takes its minimum at $\tau =0.5$, the optimal sample split to minimize the asymptotic bias, variance, and MSE is when the sample is equally split between the training and the testing (or holdout) sample.

Because the population parameter value enters into both the asymptotic bias and the asymptotic variance, it is not possible to determine individual t-statistics for the parameters. However, under the null hypothesis that $H_0:\beta ={\beta }_0$, the statistic

$$\begin{array}{ccc}& n{\left((\widehat{\beta }-{\beta }_0)+{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)\sqrt{\frac{{\sigma }_{\epsilon }^2}{n2\pi \tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}}\right)}^{\prime }& \\ & \times {\left({\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}+\left(\frac{1}{\tau (1-\tau )}\right)\left(\frac{{\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}}{{({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}\right)\right)}^{-1}& \\ & \times \left((\widehat{\beta }-{\beta }_0)+{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)\sqrt{\frac{{\sigma }_{\epsilon }^2}{n2\pi \tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right)}^{-1}({\beta }_0-{\beta }^p)}}\right)\end{array}$$

will converge in distribution to a chi-square with k degrees of freedom. This statistic can be inverted to create accurate confidence regions.

The asymptotic behavior is different when ${\beta }^p={\beta }_0$.

Theorem 3.

Assumptions 1–4imply, when ${\beta }^p={\beta }_0,$$\sqrt{n}(\widehat{\beta }-{\beta }_0)$, conditional on $\widehat{\alpha }$, converges in distribution to

$$N\left(0,{\sigma }_{\epsilon }^2{\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0+\widehat{\alpha }{I}_k\right)}^{-1}\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0\right){\left({\Gamma }_0^{\prime }{R}_z{\Gamma }_0+\widehat{\alpha }{I}_k\right)}^{-1}\right)$$

where $\widehat{\alpha }$ converges in distribution to a draw from an a-min distribution with parameter $\mathbf{S}=R_z^{1/2}{\Gamma }_0$.

In the unlikely event that the prior is selected equal to the population parameter value, the asymptotic covariance is smaller than or equal to the 2SLS asymptotic covariance. In terms of implementation, the covariance and bias associated with ${\beta }^p\ne {\beta }_0$ should be used because it is asymptotically correct for all priors except for ${\beta }^p={\beta }_0$, where it leads to conservative confidence regions.

Linear Regression

A special case is the linear regression model where Y is $n\times 1$ and X is $n\times k$

$$\begin{array}{ccc}\hfill Y& =& X{\beta }_0+\epsilon \hfill \end{array}$$

(10)

with full rank $k\times k$ second moments $R_x=E\left[x_i^{\prime }{x}_i\right]$, and conditional on X, ${\epsilon }_i\sim iid\left(0,{\sigma }_{\epsilon }^2<\infty \right)$. The estimation equations for the ridge regression estimate where the tuning parameter, $\alpha \ge 0$, is selected with a holdout sample can be written in the method of moments framework using the parameterization

$$\theta ={\left[\begin{array}{cccc}\mathrm{vech}{\left(R_{x\tau }\right)}^{\prime }& {\beta }_{tr}^{\prime }& \alpha & {\beta }^{\prime }\end{array}\right]}^{\prime }with{\theta }_0={\left[\begin{array}{cccc}\mathrm{vech}{\left(R_x\right)}^{\prime }& {\beta }_0^{\prime }& 0& {\beta }_0^{\prime }\end{array}\right]}^{\prime }.$$

The ridge estimator is part of the parameter estimates defined by the just identified system of equations ${\mathfrak{H}}_n\left(\theta \right)=\frac{1}{n}{\sum }_{i=1}^n{\mathfrak{h}}_i\left(\theta \right)=0$, where

$${\mathfrak{h}}_i\left(\theta \right)=\left[\begin{array}{c}{\mathbf{1}}_{\tau }\left(i\right)\mathrm{vech}(R_{x\tau }-x_i{x}_i^{\prime })\\ {\mathbf{1}}_{\tau }\left(i\right)\left(-x_i(y_i-x_i^{\prime }{\beta }_{tr})+\alpha ({\beta }_{tr}-{\beta }^p)\right)\\ (1-{\mathbf{1}}_{\tau }\left(i\right))(y_i-x_i^{\prime }{\beta }_{tr})x_i^{\prime }{\left(R_{x\tau }+\alpha I_k\right)}^{-1}({\beta }^p-{\beta }_{tr})\\ -x_i(y_i-x_i^{\prime }\beta )+\alpha (\beta -{\beta }^p)\end{array}\right].$$

(11)

Along with Assumption 4, the following three assumptions are sufficient to obtain the asymptotic results.

Assumption 5.

$x_i$ is iid with finite fourth moments and $E\left[x_i{x}_i^{\prime }\right]=R_x$ has full rank.

Assumption 6.

Conditional on X, ${\epsilon }_i\sim iid(0,{\sigma }_{\epsilon }^2<\infty )$.

Assumption 7.

The parameter space Θ is defined as follows: $R_x$ is restricted to a symmetric positive definite matrix with eigenvalues $1/B_1\le e_1\le e_2\le \dots \le e_k\le B_1,$ $\left|{\beta }_j\right|\le B_2$ for $j=1,2,\dots ,k$, and $\alpha \in [0,B_3]$ where $B_1$, $B_2$, and $B_3$ are positive and finite.

Lemma 2 gives the rate of convergence for the tuning parameter when the prior is different from the population parameter value and characterizes its asymptotic distribution when the prior is equal to the population parameter value.

Lemma 2.

Assumptions 4–7imply

(i) if ${\beta }^p\ne {\beta }_0$, (1) $\widehat{\alpha }{\to }^{p}0$ and (2) $\sqrt{n}\widehat{\alpha }=O_p\left(1\right)$; (ii) if ${\beta }^p={\beta }_0$, $\widehat{\alpha }$ converges in distribution to a draw from the a-min distribution with parameter $\mathbf{S}=R_x^{1/2}$, the symmetric matrix square root of $R_x$.

The asymptotic distribution of $\sqrt{n}(\widehat{\theta }-{\theta }_0)$ is equivalent to the distribution of ${arg\; min}_{\lambda \in \Lambda }{(\mathcal{Z}+\lambda )}^{\prime }{M}_0^{\prime }{M}_0(\mathcal{Z}+\lambda )$ where the random variable is defined as

$$\begin{array}{ccc}& & \mathcal{Z}=\underset{n\to \infty }{lim}{\left(E\left[\frac{\partial {\mathfrak{H}}_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right]\right)}^{-1}\sqrt{n}{\mathfrak{H}}_n\left({\theta }_0\right),M_0=E\left[\frac{\partial {\mathfrak{H}}_n\left({\theta }_0\right)}{\partial {\theta }^{\prime }}\right],\hfill \end{array}$$

and the cone is defined by $\Lambda \equiv \left\{\lambda \in R^{k(k+1)/2+2k+1}:{\lambda }_{k(k+1)/2+k+1}\ge 0\right\}$. The estimator is defined as

$$\widehat{\theta }=\underset{\theta \in \mathsf{\Theta }}{arg\; min}{\mathfrak{H}}_n{\left(\theta \right)}^{\prime }{\mathfrak{H}}_n\left(\theta \right).$$

Theorem 4.

Assumptions 4–7imply, when ${\beta }^p\ne {\beta }_0$, the asymptotic distribution of $\sqrt{n}(\widehat{\theta }-{\theta }_0)$ is equivalent to the distribution of

$$\widehat{\lambda }=\underset{\lambda \in \Lambda }{arg\; min}{(\mathcal{Z}+\lambda )}^{\prime }{M}_0^{\prime }{M}_0(\mathcal{Z}+\lambda ).$$

This Theorem characterizes the asymptotic distribution of the estimator as the projection of a stochastic process onto a cone. The special structure of this problems allows for the analytic derivation of the asymptotic distribution.

Theorem 5.

Assumptions 4–7imply, when ${\beta }^p\ne {\beta }_0$,

(i) $\sqrt{n}(\widehat{\beta }-{\beta }_0)$ is asymptotically normally distributed with mean

$$-R_x^{-1}({\beta }_0-{\beta }^p)\sqrt{\frac{{\sigma }_{\epsilon }^2}{2\pi \tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}}$$

and covariance

$${\sigma }_{\epsilon }^2{R}_x^{-1}+\left(\frac{1}{\tau (1-\tau )}\right)\left(\frac{{\sigma }_{\epsilon }^2{R}_x^{-1}({\beta }_0-{\beta }^p){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}}{{({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}\right),$$

and

(ii) $\sqrt{n}\widehat{\alpha }$ asymptotically has a mixture distribution with a discrete mass of 1/2 at zero and over $[0,\infty )$, a truncated normal distribution with zero mean and covariance

$$\frac{{\sigma }_{\epsilon }^2}{\tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}.$$

When ${\beta }^p\ne {\beta }_0$, the ridge estimator has an asymptotic bias and its asymptotic variance is larger than that of the OLS estimator. However, this is only in one dimension. The asymptotic MSE is

$${\sigma }_{\epsilon }^2{R}_x^{-1}+\left(\frac{(2\pi n+1)}{2\pi \tau (1-\tau )n}\right)\left(\frac{{\sigma }_{\epsilon }^2{R}_x^{-1}({\beta }_0-{\beta }^p){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}}{{({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}\right).$$

This is the MSE for the OLS estimator plus a constant times the projection matrix for $R_x^{-1/2}({\beta }_0-{\beta }^p).$ The ridge estimator using the holdout sample has the same bias, variance, and MSE as the OLS estimator except in the dimension of the $R_x^{-1/2}({\beta }_0-{\beta }^p)$. In order to minimize bias, variance, and MSE of the estimator, $\tau =0.5$ should be selected. The population parameter value in both the asymptotic bias and the asymptotic variance does not allow for individual t-statistics. However, under the null hypothesis that $H_0:\beta ={\beta }_0$, the statistic

$$\begin{array}{ccc}& n{\left((\widehat{\beta }-{\beta }_0)+R_x^{-1}({\beta }_0-{\beta }^p)\sqrt{\frac{{\sigma }_{\epsilon }^2}{n2\pi \tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}}\right)}^{\prime }& \\ & \times {\left({\sigma }_{\epsilon }^2{R}_x^{-1}+\left(\frac{1}{\tau (1-\tau )}\right)\left(\frac{{\sigma }_{\epsilon }^2{R}_x^{-1}({\beta }_0-{\beta }^p){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}}{{({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}\right)\right)}^{-1}& \\ & \times \left((\widehat{\beta }-{\beta }_0)+R_x^{-1}({\beta }_0-{\beta }^p)\sqrt{\frac{{\sigma }_{\epsilon }^2}{n2\pi \tau (1-\tau ){({\beta }_0-{\beta }^p)}^{\prime }{R}_x^{-1}({\beta }_0-{\beta }^p)}}\right)\end{array}$$

will converge in distribution to a draw from a chi-square with k degrees of freedom and can be used to create confidence regions.

When ${\beta }^p={\beta }_0$, a different asymptotic distribution occurs.

Theorem 6.

Assumptions 4–7imply, when ${\beta }^p={\beta }_0$, $\sqrt{n}(\widehat{\beta }-{\beta }_0)$, conditional on $\widehat{\alpha }$, converges in distribution to a draw from

$$N\left(0,{\sigma }_{\epsilon }^2{\left(R_x+\widehat{\alpha }{I}_k\right)}^{-1}{R}_x{\left(R_x+\widehat{\alpha }{I}_k\right)}^{-1}\right)$$

where $\widehat{\alpha }$ will converge in distribution to a draw from an a-min distribution with parameter $\mathbf{S}=R_x^{1/2}$.

If ${\beta }^p={\beta }_0$, the asymptotic covariance is smaller than or equal to the OLS estimator’s asymptotic covariance. Again, the covariance and bias associated with ${\beta }^p\ne {\beta }_0$ should be used for inference.

4. Small Sample Properties

The behavior in finite samples is investigated next by simulating the model in Equations (1) and (2) with $k=2$ and $m=4$ (Because the ridge estimator and the 2SLS estimator will be compared using MSE, two moments need to exist. This is ensured by having four instruments to estimate the two parameters. See [21].). To standardize the model, set $z_i\sim iidN(0,I_4)$ and ${\beta }_0={(0,0)}^{\prime }$. Create endogeneity with

$$\left[\begin{array}{c}{\epsilon }_i\\ u_i\end{array}\right]\sim iidN\left(0,\left[\begin{array}{ccc}1& 0.7& 0.7\\ .7& 1& 0\\ 0.7& 0& 1\end{array}\right]\right).$$

(12)

The strength of the instrument is controlled by the parameter $\delta$ with ${\Gamma }_0={\left[\begin{array}{cccc}1& 0& 1& 1\\ 0& \delta & 0& 0\end{array}\right]}^{\prime }.$ If $\delta =0$, the second element of ${\beta }_0$ is not identified.

The ridge parameter estimate $\alpha$ is determined in two steps. In the first step, the objective function is evaluated on a grid of values to determine a starting value. In the second step, the objective function is evaluated over a finer grid (10,000 points) centered at the best value obtained from the first step. A value of $\widehat{\alpha }=0$ in the second step corresponds to the ridge estimator ignoring the prior in favor of the data, whereas a value $\widehat{\alpha }={10}^7$ corresponds to “infinite regularization” implying that the ridge estimator ignores the data in favor of the prior.

4.1. Coverage Probabilities

The chi-square test is performed for a range of parameterizations including different priors, different strengths of the instrument, and different sample sizes. Each model is simulated 10,000 times and a size of ten percent is used for each test. The results are presented in Table A1. The observed coverage probabilities agree with the theoretical values. As expected, the approximations are best in cases where the sample size is large, the correlation between instruments and covariates is higher, and the prior is closer to the population parameter value.

4.2. MSE for the Ridge Estimator

The ridge estimator is compared with the 2SLS estimator to demonstrate settings where the ridge estimator can be expected to give more accurate results in small samples. The simulated models differ in three dimensions: sample size, strength of the instruments, and the prior on the structural parameters. For smaller sample sizes, the ridge estimator should have better properties, whereas, for larger sample sizes, 2SLS should perform better. Sample sizes of $n=25$, 50, 250 and 500 are considered. As noted above, the instrument signal strength decreases with $\delta$. For smaller signal strengths, the ridge estimator should perform better. Values of $\delta =0.01$, 0.05, 0.1, and 0.5 are considered. For prior values further from to the population parameter values, the ridge estimator should perform worse. Three different values of ${\beta }^p$ are considered ${\left(\frac{1}{2},\frac{1}{2}\right)}^{\prime }$, ${\left(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}\right)}^{\prime },$ and ${(1,1)}^{\prime }$. A total of 48 model specifications are simulated: four sample sizes n, four values of the precision parameter $\delta$, and three values of the prior ${\beta }^p$. Each specification is simulated $10,000$ times and estimated with 2SLS and the ridge estimator. $\tau =0.5$ is used to split the sample between training and test samples for the ridge estimator.

Table A2, Table A3 and Table A4 compare the performance of the 2SLS estimator with the ridge estimator for different precision levels and sample sizes when the prior is fixed at ${\beta }^p={(\frac{1}{2},\frac{1}{2})}^{\prime }$, ${\beta }^p={(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})}^{\prime }$, and ${\beta }^p={(1,1)}^{\prime }$, respectively. The estimators are compared on the basis of bias, standard deviation of the estimates, MSE values of the estimates, and sum of MSE values of ${\widehat{\beta }}_1$ and ${\widehat{\beta }}_2$. All three tables demonstrate expected results. The ridge estimator dominates in models with smaller sample sizes, weaker instrument strength, and when the prior is closer to the population parameter values.

Overall, the simulations demonstrate that, for some models’ specifications, the ridge estimator using a holdout sample has better small sample performance relative to the 2SLS estimator. The simulations agree with the asymptotic distributions. As the sample size increases, the 2SLS estimator performs better than the ridge estimator.

5. Conclusions

Inference has always been a weakness of ridge regression. This paper presents a methodology and results to help address some of its weaknesses. Theoretically accurate inferences can be performed with the asymptotic distribution of the ridge estimates of the linear IV model when the tuning parameter is empirically selected by a holdout sample. It is well known that the distribution of the estimates of the structural parameters is affected by empirically selected tuning parameters. This is addressed by simultaneously estimating both the parameters of interest and the ridge regression tuning parameter in the method of moments framework. When the prior is different from the population parameter value, the estimator accounts for the probability limit of the tuning parameter being on the boundary of the parameter space. The asymptotic distribution for the tuning parameter is a nonstandard mixed distribution. The asymptotic distribution for the estimates of the structural parameters is normal but with a nonzero mean. The ridge estimator of the structural parameters has asymptotic bias and the asymptotic covariance is larger than the asymptotic covariance for the 2SLS estimator; however, the bias and larger covariance only apply to one dimension of the parameter space. The dependence of the asymptotic mean and variance on the population parameter values, prevents the calculation of t-statistics for individual parameters. Fortunately, a chi-square statistic provides accurate confidence regions for the structural parameters.

If the prior is equal to the population parameter value, the ridge estimator is consistent and the asymptotic covariance is smaller than the 2SLS asymptotic covariance. The asymptotic distribution provides insights on how to perform estimation with a holdout sample. The minimum bias, variance, and MSE for the structural parameters occur when the sample is equally split into a training sample and a test (or holdout) sample.

This paper’s approach can be useful in determining the asymptotic behavior for other empirical procedures that select tuning parameters. Two natural extensions would be to generalize cross-validation (see [22]) and K-fold cross-validation, where the entire dataset would be used to select the tuning parameter. This paper has focused on strong correlations between the instruments and the regressors. Another important extension would be the asymptotic behavior in models with weaker correlation between the instruments and the regressors, see [23].