Model Selection with Missing Data Embedded in Missing-at-Random Data

Abstract

When models are built with missing data, an information criterion is needed to select the best model among the various candidates. Using a conventional information criterion for missing data may lead to the selection of the wrong model when data are not missing at random. Conventional information criteria implicitly assume that any subset of missing-at-random data is also missing at random, and thus the maximum likelihood estimator is assumed to be consistent; that is, it is assumed that the estimator will converge to the true value. However, this assumption may not be practical. In this paper, we develop an information criterion that works even for not-missing-at-random data, so long as the largest missing data set is missing at random. Simulations are performed to show the superiority of the proposed information criterion over conventional criteria.

1. Introduction

In data analysis, all variables are not equally important in the final model; some variables are more important than others. Thus, what matters in practice is model selection. For model selection, several variants of the Kullback-Leibler (KL) information criteria have been developed. In the case of complete data, the criteria include Akaike’s information criterion (AIC; [1]) and Takeuchi’s information criterion (TIC; [2]). These all measure the “distance” from the true distribution to the model distribution in the sense of Kullback-Leibler information and find the model that minimizes the information.

Applying these information criteria to real data is not straightforward, as nearly all real data have missing values. In practical cases, missing values are an obstacle to applying statistical methods since the methods implicitly assume the availability of complete data. To handle such missing values with information criteria, modifications have been proposed by authors such as [3,4,5,6,7]. The first modification of the information criterion is attributed to [3]. The paper [4] derived the information criterion for missing data in a straight way, and the paper [5] used the symmetric KL divergence for the derivation. The paper [6] used the Hermite expansion with the Markov Chanin Monte Carlo method to approximate the H-function of the EM algorithm, and hence it needed much computation in application. The paper [7] also used the EM algorithm to develop the information criterion for the settings with missing covariates.

However, despite modification, these information criteria still face a major problem in their practical application. The common problem is their implicit or explicit assumption that any set of data selected by them produces consistent parameters in the model. This assumption is simply not realistic. In reality, these criteria can exclude a variable that causes missingness, resulting in inconsistent parameters, as the missing data are not missing at random (NMAR; [8]). For the parameters to be consistent, the missing-data mechanism requires modeling [9]. However, this is practically difficult since the data necessary for modeling are missing.

To overcome the problems described above, we propose a new information criterion for missing data. The primary advantage of our information criterion is that it can handle situations where a subset of the data is NMAR so long as the largest set of missing data is missing at random (MAR). In computing our criterion, we compute the parameters with the largest data set and extract the parameters necessary for the model under consideration. These two steps allow us to obtain an estimator that is consistent with the true value for any possible model. Since the existing criteria have no such feature, they are unable to produce a proper value for the information criterion when the data are NMAR.

The remainder of the paper is organized as follows. In the next section, we develop the asymptotic theory for a subset of MAR data. In Section 3, we derive our information criterion for a subset of MAR data. In Section 4, the results of two simulations are presented to demonstrate the effectiveness of the proposed information criterion in the practical situations. We conclude the paper with suggestions for future research.

2. Inference of Maximum Likelihood Estimator with Missing Data

We will represent the variables of the entire data set as a vector $\mathit{x}$ and partition it to $(\mathit{y},\mathit{z})$, where $\mathit{y}$ is the variable of interest and $\mathit{z}$ is not. The model density functions with respect to these variables are written as $f(·)$ in common; however, they can be distinguished by the context. If the values of $\mathit{x}$ are observed completely, we would denote this as $X_{\mathrm{c}}$ in matrix form. However, since $\mathit{x}$ contains missing values, $X_{\mathrm{c}}$ consists of an observed part, $X_{\mathrm{o}}$, and a missing part, $X_{\mathrm{m}}$. For $Y_{\mathrm{c}}$, $Y_{\mathrm{o}}$ and $Y_{\mathrm{m}}$ are defined in the same way. We denote the parameters of the distributions of $\mathit{x}$ and $\mathit{y}$ as $\mathit{\theta }$ and $\mathit{\alpha }$, respectively. The relationship between $\mathit{\theta }$ and $\mathit{\alpha }$ is assumed to be $\mathit{\alpha }=S\mathit{\theta }$, where S is a matrix to extract the necessary part of the parameter $\mathit{\theta }$, and $\mathit{\theta }={({\mathit{\alpha }}^{\prime },{\mathit{\beta }}^{\prime })}^{\prime }$. The rest of $\mathit{\theta }$ is written as $\mathit{\beta }$. Note that although we could theoretically develop $\mathit{\alpha }=g\left(\mathit{\theta }\right)$ for a nonlinear function, in this paper we confine our focus to the linear function that can be expressed as $\mathit{\alpha }=S\mathit{\theta }$ for simplicity.

In real data analysis, we often encounter a situation in which a subset of data is embedded in MAR data. With the MAR mechanism, the missingness of data is caused by observed values for variables included in the analysis; on the other hand, for NMAR data (i.e., data that are not MAR), the missingness is caused by variables not included in the analysis and/or by missing values of the included variables [8]. It follows that using more variables in the analysis makes it more likely that the missing data will be MAR (or close to MAR). One reason for this is that variables added into the analysis are likely to include those variables causing missingness. Another reason is that more variables are likely to include some variables related to missing values, recovering information lost by the missingness. These phenomena have been partly confirmed in simulations by [10,11,12]. Thus, MAR is a more plausible missing-data mechanism for data with a larger set of variables.

An example is given in which a subset of the missing data is NMAR but the entire data is MAR [9]. An investigation is conducted with an aim to estimate the correlation between the self-esteem and the sexual behavior for the teenagers. However, the question of the sexual behavior is so sensitive that the question is only asked for over 15. The resulting data are missing at random (MAR), since the question on the sexual behavior is missing for those less than 15 years of age. As described before, our interest is the relationship between the self-esteem and the sexual behavior for teenagers. If the variables for analysis are limited to the self-esteem and the sexual behavior, the data are NMAR since the data set excludes “age” which causes missingness, and thus the correlation estimate is biased. On the other hand, if the age is used as an additional variable, the data become MAR, and the likelihood estimation using the data on these three variables creates the consistent maximum likelihood estimator of the correlation coefficient between the self-esteem and the sexual behavior.

For missing data, an exact distribution such as the t-distribution for complete data is rarely obtained other than in special cases such as monotonic missing data [13,14]. Hence, asymptotic theory is preferable for deriving the distribution of an estimator with missing data. In asymptotic theory, the most important properties of an estimator are consistency to the true value and asymptotic normality. Of these two properties, the former is more important in practice. That an estimator has consistency means that the estimated parameter converges to the true value of the assumed model and that the estimated parameter is close enough to the true value in a large sample. For an analyst, this is a highly desirable property. In fact, an estimator without such consistency is essentially useless to the analyst as a means of approximating the parameter that he/she wishes to estimate. However, even if an estimator lacks consistency, asymptotic normality still might hold [15]. Notably, it has been shown that a maximum likelihood (ML) estimator under MAR is a good estimator, since it will have both consistency and asymptotic normality [16].

We can now provide more detail regarding the estimation methods when a subset of MAR data is used to produce the ML estimator. It has been shown that the ML estimator based on the maximum likelihood method has consistency [16]. Expressing this in the form of an equation, we have

$$\begin{array}{c}\hfill \widehat{\mathit{\theta }}=arg\underset{\mathit{\theta }}{max}\frac{1}{n}logf\left(X_{\mathrm{o}}\right|\mathit{\theta })\stackrel{\mathrm{a}.\mathrm{s}.}{⟶}{\mathit{\theta }}_0\stackrel{\mathrm{def}}{=}arg\underset{\mathit{\theta }}{max}E[logf\left(\mathit{x}\right|\mathit{\theta })],\end{array}$$

(1)

where n is the sample size, and the arrow indicates that the left-side term converges almost surely to the right-side term as n increases to infinity. What matters here is that the ML estimator is based on missing data $X_{\mathrm{o}}$, and it is a function of the data and the corresponding missing-data indicator. This ML estimator converges to the value which maximizes $E[logf(\mathit{x}\left|\mathit{\theta }\right)]$ (where the expectation is taken with respect to $\mathit{x}$), the same convergence point as the maximizer of $n^{-1}logf\left(X_{\mathrm{c}}\right|\mathit{\theta })$, which, in actuality, is not available under the presence of missing data. The asymptotic distribution of $\sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ is normal, with mean zero and variance $V\left(\mathit{\theta }\right)={\left(E[-{\nabla }^{2}logf\left(X_{\mathrm{o}}\right|\mathit{\theta })]\right)}^{-1}$, where ∇ is the first-derivative operator with respect to the parameter $\mathit{\theta }$. On the other hand, the ML estimator

$$\begin{array}{c}\hfill \tilde{\mathit{\alpha }}=arg\underset{\mathit{\alpha }}{max}\frac{1}{n}logf\left(Y_{\mathrm{o}}\right|\mathit{\alpha })\end{array}$$

(2)

would generally not be guaranteed to converge to the true value, ${\mathit{\alpha }}_0$ in ${\mathit{\theta }}_0$, since the set of data corresponding to $\mathit{\alpha }$ might be NMAR. Hence, to estimate $\mathit{\alpha }$, we construct an estimator by extracting the part corresponding to $\mathit{\alpha }$ from the ML estimator $\widehat{\mathit{\theta }}$, and set $\widehat{\mathit{\alpha }}=S\widehat{\mathit{\theta }}$.

3. Information Criterion for Missing Data

In this section, we propose an information criterion (IC) for missing data that are MAR as a whole but might be NMAR as a subset. In selecting a model among multiple candidate models, the standard IC underlies the discrepancy from the true distribution $g\left(\mathit{y}\right)$ to the model $f\left(\mathit{y}\right|\mathit{\theta })$. This is measured using the KL information, which is defined as

$$\begin{array}{c}\hfill E_{g\left(\mathit{y}\right)}\left[log\frac{f\left(\mathit{y}\right|\mathit{\alpha })}{g\left(\mathit{y}\right)}\right]=E_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|\mathit{\alpha })\right]-E_{g\left(\mathit{y}\right)}\left[logg\left(\mathit{y}\right)\right]\end{array}$$

Since the second term is constant, the first term determines the KL information. The first term works as a measure of a model’s goodness. When we estimate the parameter $\mathit{\alpha }$ using the ML method and obtain its estimate $\tilde{\mathit{\alpha }}$, a naive measure of goodness of fit is $E_{g\left(\mathit{y}\right)}[logf\left(\mathit{y}\right|\tilde{\mathit{\alpha }})]$. The problem is that the true distribution g is unknown. If the complete data are available and are a random sample, we can use $n^{-1}{\sum }_{j=1}^{n}logf\left({\mathit{y}}_j\right|\tilde{\mathit{\alpha }})$ as an estimated measure to the true distribution, where ${\mathit{y}}_j$ ($j=1,\dots ,n$) are the data used to estimate $\tilde{\mathit{\alpha }}$. However, the problem with using thisempirical likelihood is that the estimates depend on the data, and hence $n^{-1}{\sum }_{j=1}^{n}logf\left({\mathit{y}}_j\right|\tilde{\mathit{\alpha }})$ is anasymptotically biased estimator of $E_{g\left(\mathit{y}\right)}[logf\left(\mathit{y}\right|\tilde{\mathit{\alpha }})]$. The likelihood with the corrected bias is known as the IC. Some ICs have a special name, such as AIC [1], TIC [2] and GIC [17]. In the correction and derivation of the IC, asymptotic properties play a central role. Of the various desirable properties of an ML estimator, the most important are asymptotic normality and the consistency of the ML estimator with respect to the true value. See also Konishi and Kitagawa [17] for the technical details.

For missing data, an IC can be computed in a similar spirit. However, considerable caution must be exercised in constructing the criterion, as different subsets of the missing data may have different types of missing-data mechanisms. Even if a subset of the missing data, call it $Y_{\mathrm{o}}$, is MAR, which would mean that a naive ML estimator based on $logf\left(Y_{\mathrm{o}}\right|\mathit{\alpha })$ would have consistency, a smaller subset of $Y_{\mathrm{o}}$ might be NMAR because the variables causing missingness are excluded from the model under consideration, in which case the ML would not have consistency. This means that a straightforward extension of the conventional IC could not be effectively applied to such missing-data cases.

To derive the proposed IC for cases with missing data, we will assume that the data as a whole, represented as $X_{\mathrm{o}}$, are MAR. As shown in [16], the ML estimator based on $X_{\mathrm{o}}$ has consistency and asymptotic normality. However, a subset of MAR data $X_{\mathrm{o}}$, call it $Y_{\mathrm{o}}$, might be NMAR, and thus the ML estimator based on $Y_{\mathrm{o}}$ would be asymptotically biased. To ensure that the ML estimator is consistent (i.e., converges to the true value), it is necessary to perform ML estimation with the entire data set, even when we are interested in models for a subset of $X_{\mathrm{o}}$, $Y_{\mathrm{o}}$. Let the ML estimator based on the entire data set $X_{\mathrm{o}}$ be $\widehat{\mathit{\theta }}$, and let the part of $\mathit{\theta }$ associated with the model of interest be $\mathit{\alpha }$. In this normality case, there is a selection matrix S such that $S\mathit{\theta }=\mathit{\alpha }$. The ML estimates of $\mathit{\alpha }$ can be written as $\widehat{\mathit{\alpha }}=S\widehat{\mathit{\theta }}$. In this setting, we can evaluate the bias.

We will use $logf\left(Y_{\mathrm{o}}\right|\mathit{\theta })$ to denote a log-likelihood function. With the regularity conditions and the MAR assumption, the ML estimator is proved to have consistency and asymptotic normality [16].

We can now evaluate the bias of $logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})$. The bias is given as

$$\begin{array}{c}\hfill b\left(G\right)=E\left[logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})-n{E}_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|\widehat{\mathit{\alpha }})\right]\right].\end{array}$$

Decomposing the bias, we have

$$\begin{array}{ccc}\hfill b\left(G\right)& =& E\left[logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})-logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)\right]\hfill \\ & & +E\left[logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)-n{E}_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|{\mathit{\alpha }}_0)\right]\right]\hfill \\ & & \phantom{+}+E\left[n{E}_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|{\mathit{\alpha }}_0)\right]-n{E}_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|\widehat{\mathit{\alpha }})\right]\right].\hfill \end{array}$$

Let the terms be, in order, $D_1$, $D_2$, and $D_3$.

In this paper, we have assumed that $X_{\mathrm{o}}$ is MAR and that $\widehat{\mathit{\theta }}$ is asymptotically normally distributed. First, we evaluate $D_1$. ∇ and ${\nabla }_{\mathit{\alpha }}$ indicate the first derivative operator with respect to $\mathit{\theta }$, and to $\mathit{\alpha }$, respectively. Assume that the ML estimator $\widehat{\mathit{\theta }}$ takes the form

$$\begin{array}{c}\hfill \sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)=V_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\sqrt{n}\frac{1}{n}\nabla logf\left(X_{\mathrm{o}}\right|{\mathit{\theta }}_0)+o_p\left(1\right),\end{array}$$

(3)

where ${\mathit{\theta }}_0$ is the true value of the parameter, and $V_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)$ is the variance defined as $V_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)=E{[\nabla logf\left(X_{\mathrm{o}}\right|{\mathit{\theta }}_0)\nabla logf{\left(X_{\mathrm{o}}\right|{\mathit{\theta }}_0)}^{\prime }]}^{-1}$. In addition, assume that $\sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ is asymptotically distributed as normal with mean zero and variance $V_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)$. By $\sqrt{n}(\widehat{\mathit{\alpha }}-{\mathit{\alpha }}_0)=S\sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$, $-{\nabla }_{\mathit{\alpha }}^{2}logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)\approx V_{Y_{\mathrm{o}}}{\left({\mathit{\theta }}_0\right)}^{-1}/n$, we obtain

$$\begin{array}{ccc}& & logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})-logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)\hfill \\ & & ={\nabla }_{\mathit{\alpha }}logf{\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)}^{\prime }(\widehat{\mathit{\alpha }}-{\mathit{\alpha }}_0)+\frac{1}{2}{(\widehat{\mathit{\alpha }}-{\mathit{\alpha }}_0)}^{\prime }{\nabla }_{\mathit{\alpha }}^{2}logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)(\widehat{\mathit{\alpha }}-{\mathit{\alpha }}_0)+o_p\left(1\right)\hfill \\ & & ={\left(\sqrt{n}\frac{1}{n}\nabla logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)\right)}^{\prime }{S}^{\prime }S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\sqrt{n}\frac{1}{n}\nabla logf\left(X_{\mathrm{o}}\right|{\mathit{\theta }}_0)\hfill \\ & & -\frac{1}{2}\sqrt{n}{(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)}^{\prime }{S}^{\prime }{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}S\sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)+o_p\left(1\right).\hfill \end{array}$$

(4)

The expected first term in Equation (4) can be written as

$$\begin{array}{c}\hfill \mathrm{tr}\left\{S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)E\left[\sqrt{n}\frac{1}{n}\nabla logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)\sqrt{n}\frac{1}{n}\nabla logf{\left(Z_{\mathrm{o}}\right|Y_{\mathrm{o}};{\mathit{\beta }}_0)}^{\prime }\right]S^{\prime }+S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)S^{\prime }{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\},\end{array}$$

where tr$\left\{\right\}$ is the trace of the matrix in $\left\{\right\}$. Because $E_{g\left(Z_{\mathrm{o}}\right|Y_{\mathrm{o}})}[\nabla logf\left(Z_{\mathrm{o}}\right|Y_{\mathrm{o}};{\mathit{\beta }}_0)|Y_{\mathrm{o}}]=\mathbf{0}$, the first term in this trace is zero. Therefore, the expected first term is $\mathrm{tr}\left\{S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)S^{\prime }{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\}$. Next, we calculate the expected second term in Equation (4) as

$$\begin{array}{ccc}& & -\frac{1}{2}\mathrm{tr}\left\{E\left[\sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)\sqrt{n}{(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)}^{\prime }\right]S^{\prime }{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}S\right\}\hfill \\ & & =-\frac{1}{2}\mathrm{tr}\left\{S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)S^{\prime }{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\}.\hfill \end{array}$$

In summary, ${\displaystyle D_1=\frac{1}{2}\mathrm{tr}\left\{S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)S^{\prime }{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\}+o_p\left(1\right)}$.

Second, we evaluate $D_2$.

$$\begin{array}{ccc}\hfill D_2& =& E\left[logf\left(Y_{\mathrm{o}}\right|{\mathit{\alpha }}_0)\right]-n{E}_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|{\mathit{\alpha }}_0)\right]\hfill \\ & =& E_{g\left(\mathit{y}\right)}\left[logf\left(Y_{\mathrm{c}}\right|{\mathit{\alpha }}_0)-logf\left(Y_{\mathrm{m}}\right|Y_{\mathrm{o}};{\mathit{\alpha }}_0)\right]-n{E}_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|{\mathit{\alpha }}_0)\right]\hfill \\ & =& -E\left[E\left[\left.logf\left(Y_{\mathrm{m}}\right|Y_{\mathrm{o}};{\mathit{\alpha }}_0)\right|Y_{\mathrm{o}}\right]\right]\hfill \\ & =& -E\left[H\left({\mathit{\alpha }}_0\right|{\mathit{\alpha }}_0)\right],\hfill \end{array}$$

where $H\left({\mathit{\alpha }}_0\right|{\mathit{\alpha }}_0)$ is the H-function of the EM algorithm based on $logf\left(Y_{\mathrm{o}}\right|\mathit{\alpha })$ evaluated at $\mathit{\alpha }={\mathit{\alpha }}_0$.

Finally, we evaluate $D_3$. Define $\mathit{\eta }\left(\mathit{\alpha }\right)=E_{g\left(\mathit{y}\right)}\left[logf\left(\mathit{y}\right|\mathit{\alpha })\right]$. Since $\nabla \mathit{\eta }\left({\mathit{\alpha }}_0\right)=\mathbf{0}$,

$$\begin{array}{ccc}\hfill \mathit{\eta }\left(\widehat{\mathit{\alpha }}\right)-\mathit{\eta }\left({\mathit{\alpha }}_0\right)& =& -\frac{1}{2}{(\widehat{\mathit{\alpha }}-{\mathit{\alpha }}_0)}^{\prime }{\left(V_{Y_{\mathrm{c}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}(\widehat{\mathit{\alpha }}-{\mathit{\alpha }}_0)+o_p\left(1\right)\hfill \\ & =& -\frac{1}{2}{(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)}^{\prime }{S}^{\prime }{\left(V_{Y_{\mathrm{c}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}S(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)+o_p\left(1\right),\hfill \end{array}$$

where $V_{Y_{\mathrm{c}}}\left({\mathit{\theta }}_0\right)={\left(-E[{\nabla }_{\mathit{\alpha }}^{2}logf\left(Y_{\mathrm{c}}\right|{\mathit{\alpha }}_0)]\right)}^{-1}$. Taking the expectation of both sides, we have

$$\begin{array}{ccc}\hfill D_3& =& \frac{1}{2}\mathrm{tr}\left\{SE\left[\sqrt{n}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)\sqrt{n}{(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)}^{\prime }\right]S^{\prime }{\left(V_{Y_{\mathrm{c}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\}\hfill \\ & =& \frac{1}{2}\mathrm{tr}\left\{S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)S^{\prime }{\left(V_{Y_{\mathrm{c}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\}.\hfill \end{array}$$

Therefore, ignoring the term $o_p\left(1\right)$, we obtain an unbiased estimator for $E[logf\left(\mathit{y}\right|\widehat{\mathit{\alpha }})]$ as

$$\begin{array}{c}\hfill logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})-E\left[H\left({\mathit{\alpha }}_0\right|{\mathit{\alpha }}_0)\right]-\frac{1}{2}\mathrm{tr}\left[S{V}_{X_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)S^{\prime }\left\{{\left(V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}+{\left(V_{Y_{\mathrm{c}}}\left({\mathit{\theta }}_0\right)\right)}^{-1}\right\}\right].\end{array}$$

(5)

The bias cannot be computed since unknown quantities such as $\mathit{\alpha }$ are involved.

We can construct an estimated version of the IC given in Equation (5). By replacing the unknown parameter with the consistent ML estimator and eliminating the expectation from the second term in Equation (5), we can substitute the first two terms with

$$\begin{array}{ccc}\hfill logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})-H\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }})& =& Q_Y\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }}).\hfill \end{array}$$

This is a natural estimator for the first two terms in Equation (5). The penalty term is estimated by replacing ${\mathit{\theta }}_0$ with its ML estimator, which can be consistently estimated under the assumption that the data as a whole are MAR. As a result, the following Equation (6) is obtained. As is conventional in deriving the AIC, we double the bias, obtaining the corrected bias as

$$\begin{array}{c}\hfill IC=-2Q_Y\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }})+\mathrm{tr}\left[S{\widehat{V}}_{X_{\mathrm{o}}}\left(\widehat{\mathit{\theta }}\right)S^{\prime }\left\{{\widehat{V}}_{Y_{\mathrm{o}}}{\left(\widehat{\mathit{\theta }}\right)}^{-1}+{\widehat{V}}_{Y_{\mathrm{c}}}{\left(\widehat{\mathit{\theta }}\right)}^{-1}\right\}\right],\end{array}$$

(6)

where $Q_Y\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }})$ is the Q-function of the EM algorithm based on $logf\left(Y_{\mathrm{c}}\right|\mathit{\alpha })$ evaluated at $\mathit{\alpha }=\widehat{\mathit{\alpha }}$ and ${\widehat{V}}_{•}$ is the sample variance of $V_{•}$ for each suffix •. Under the regularity condition assumed in [16], the estimators of the variance-covariance matrices in (6) converges to their population version (i.e., ${\widehat{V}}_{Y_{\mathrm{o}}}\left(\widehat{\mathit{\theta }}\right)$ converges to $V_{Y_{\mathrm{o}}}\left({\mathit{\theta }}_0\right)$ in probability as $n\to \infty$).

This IC is an extension of the AIC for missing data. It becomes the original AIC [1] when $X_{\mathrm{o}}=X_{\mathrm{c}}=Y_{\mathrm{o}}=Y_{\mathrm{c}}$ and $\mathit{\theta }=\mathit{\alpha }$ without missing data. Similar to the original AIC, our IC shares the starting point $E[logf\left(\mathit{y}\right|\widehat{\mathit{\theta }})]$ with [3,4]. However, the first and second terms of our IC differ from those of [3]. Our IC shares the first term with that of [4] and that of [6], but not the second term. This difference arises from the way in which the bias is approximated. The former [4] approximated the bias under the setting that all the subsets are MAR. The latter [6] approximated the bias by using the Hermite expansion and the Markov Chain Monte Carlo methods.

4. Simulations

To confirm the validity of our proposed IC, two simulations were performed.

4.1. Selection of the Parameter Structure

We will confirm through simulations that our IC selects the correct model when there is missing data for variables $(y_1,y_2)$ in $\mathit{x}={(y_1,y_2,z)}^{\prime }$. In so doing, we compare our IC with a slightly modified AIC, since the original AIC proposed by [1] cannot deal with missing data:

$$\begin{array}{c}\hfill \mathrm{AIC}=-2logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})+2p,\end{array}$$

where p is the dimension of vector $\widehat{\mathit{\alpha }}$. This has been used as a competitor to the information criterion proposed by [4]. As an additional competitor, we use AIC${}_{\mathrm{cd}}$, which Cavanaugh and Shumway [4] developed for missing data:

$$\begin{array}{c}\hfill {\mathrm{AIC}}_{\mathrm{cd}}=-2Q\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }})+2\mathrm{tr}\left({\widehat{V}}_{Y_{\mathrm{c}}}{\left(\widehat{\mathit{\alpha }}\right)}^{-1}\left\{\frac{1}{2}{\nabla }_{\mathit{\alpha }}^{2}logf\left(Y_{\mathrm{o}}\right|\widehat{\mathit{\alpha }})\right\}\right).\end{array}$$

Four candidate models were considered, all of which are based on a trivariate normal distribution with variables $\mathit{x}=(y_1,y_2,z)$. The respective models restrict the means of $(y_1,y_2)$, and/or the variances of $(y_1,y_2)$, or impose no restrictions at all. Model 1 restricts the two expectations to be equal and the two variances to be equal. Model 2 restricts that only the expectations be equal. Model 3 restricts that only the two variances be equal. Model 4 has no restrictions. In the simulations, one of the four models is the true data-generating model, and a model is selected according to each of the three information criteria: AIC, ${\mathrm{AIC}}_{\mathrm{cd}}$, and our proposed IC.

With respect to model selection, it should be noted that all the criteria assume that the true model must be included as a special case of any of the candidate models. Model 1, for example, is a special case of the other three candidate models. On the other hand, Model 2 is not a special case of Model 3, since Model 2 has no restrictions on the variances. Theoretically, models 2, 3, and 4 cannot be compared using an information criterion. For each model, the models that can or cannot be theoretically compared are shown in

Table 1. True models and the corresponding candidate models. A number in parentheses indicates the number of a model which is theoretically inappropriate to use for comparison but compared in practice.

True Model	Candidate Models
1	1, 2, 3, 4
2	1, 2, (3), 4
3	1, (2), 3, 4
4	(1), (2), (3), 4

. A number in parentheses is that of a model that cannot be theoretically compared. Nevertheless, we have included in the simulation these theoretically un-comparable models in the set of candidate models for comparison because such a comparison is often conducted in practice, as pointed out in [18].

The simulation was conducted using the following procedure. First, a complete data set of a given size was generated from a trivariate normal distribution $N(\mathit{\mu },\Sigma )$ for variables ${(y_1,y_2,z)}^{\prime }$, where $\mathit{\mu }={({\mu }_1,{\mu }_2,{\mu }_3)}^{\prime }$ and $\Sigma ={\left[{\sigma }_{ab}\right]}_{a,b=1,2,3}$. The mean and variance for the data generation are shown in

Table 2. True values of parameters. The covariances are common.

True Model	${\mathit{\mu }}_{\mathbf{1}}$	${\mathit{\mu }}_{\mathbf{2}}$	${\mathit{\mu }}_{\mathbf{3}}$	${\mathit{\sigma }}_{\mathbf{11}}$	${\mathit{\sigma }}_{\mathbf{22}}$	${\mathit{\sigma }}_{\mathbf{33}}$	${\mathit{\sigma }}_{\mathit{ab}}$ ($\mathit{a}\ne \mathit{b}$)
1	0	0	0	1	1	1	0.7
2	0	0	0	1	2	1	0.7
3	0.5	−0.5	0	1	1	1	0.7
4	0.5	−0.5	0	1	2	1	0.7

, where the covariances are all set to $0.7$. The model that is assumed to be true is varied.

Second, denoting the missing indicators for $(y_1,y_2)$ as $(r_1,r_2)$, we generated missing values according to the following two missing-data mechanisms. The first mechanism is, for $i=1,2$,

$$\begin{array}{ccc}\hfill P(r_i=1|z)& =& \left\{\begin{array}{cc}{\psi }_i\hfill & (z\ge 0),\hfill \\ 1\hfill & (z<0).\hfill \end{array}\right.\hfill \end{array}$$

(7)

This is a case where the missing-data mechanism is non-smooth. We set ${\psi }_1$ at $0.3$, and ${\psi }_2$ at $0.7$. The second of the two missing-data mechanisms used the binomial distribution, with the observing probability given as, for $i=1,2$,

$$\begin{array}{c}\hfill P(r_i=1|\mathit{z})=1-\frac{2{\omega }_i}{exp\left(\mathit{z}\right)+exp\left(-\mathit{z}\right)}.\end{array}$$

(8)

This is a smooth missing-data mechanism. We set ${\omega }_1$ at $0.3$, and ${\omega }_2$ at $0.7$. With the first missing-data mechanism, the values for $(y_1,y_2)$ beyond the threshold $z=0$ are always missing, while those below $z=0$ are always observed. In the second mechanism, the missingness of $(y_1,y_2)$ is stochastically determined based on the inverse of the hyperbolic cosine. Notice that the missing data for $(y_1,y_2,z)$ are MAR for both mechanisms and that those for $(y_1,y_2)$ are NMAR since z, which is not included in the models, causes the missingness of $y_1$ and $y_2$.

Third, for the missing data, the model that gives the minimum IC value is selected for each of the three ICs. The procedure is repeated 1000 times, and the number of times that each of the four models is selected is recorded.

The results for the first missing-data mechanism are summarized in

Table 3. Performance comparison under the missing-data mechanism (7).

		True Model
		1				2				3				4
n	criterion	Selected				Selected				Selected				Selected
		1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4
	Proposed	754	117	106	23	67	630	87	216	0	0	855	145	0	0	131	869
100	${\mathrm{AIC}}_{\mathrm{cd}}$	506	163	243	88	6	538	3	453	0	0	749	251	0	0	8	992
	AIC	597	134	211	58	10	578	4	408	0	0	808	192	0	0	13	987
	Proposed	743	99	143	15	12	652	38	298	0	0	892	108	0	0	46	954
200	${\mathrm{AIC}}_{\mathrm{cd}}$	441	142	295	122	0	397	0	603	0	0	724	276	0	0	0	1000
	AIC	559	111	255	75	0	432	0	568	0	0	811	189	0	0	0	1000
	Proposed	672	104	197	27	1	638	15	346	0	0	881	119	0	0	17	983
400	${\mathrm{AIC}}_{\mathrm{cd}}$	298	118	399	185	0	205	0	795	0	0	685	315	0	0	0	1000
	AIC	429	93	345	133	0	247	0	753	0	0	752	248	0	0	0	1000
	Proposed	652	83	234	31	0	592	5	403	0	0	893	107	0	0	6	994
800	${\mathrm{AIC}}_{\mathrm{cd}}$	190	58	466	286	0	40	0	960	0	0	646	354	0	0	0	1000
	AIC	270	58	464	208	0	51	0	949	0	0	727	273	0	0	0	1000

. As can be seen in the table, our IC is generally superior to its two competitors. In the cases where Models 1 to 3 are true, the proposed IC outperforms the other criteria for any of the sample sizes. In the case where Model 4 is true, the proposed IC underperforms the other criteria when the sample size is small. However, as the sample size increases from 200 to 800, the proposed IC gradually selects the correct model as often as the other criteria. In addition to the above findings, the AIC is found to be more robust against the violation of the missing-data mechanism than ${\mathrm{AIC}}_{\mathrm{cd}}$.

The results for the second missing-data mechanism are given in

Table 4. Performance comparison under the missing-data mechanism (8).

		True Model
		1				2				3				4
n	criterion	Selected				Selected				Selected				Selected
		1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4
	Proposed	829	88	74	9	134	743	9	114	0	0	914	86	0	1	121	878
100	${\mathrm{AIC}}_{\mathrm{cd}}$	484	200	232	84	73	638	24	265	0	0	716	284	0	0	98	902
	AIC	670	165	138	27	154	661	17	168	0	0	808	192	0	0	170	830
	Proposed	820	99	74	7	28	852	3	117	0	0	895	105	0	0	28	972
200	${\mathrm{AIC}}_{\mathrm{cd}}$	465	210	212	113	8	710	3	279	0	0	675	325	0	0	12	988
	AIC	662	166	134	38	20	800	4	176	0	0	799	201	0	0	24	976
	Proposed	795	124	70	11	2	882	0	116	0	0	867	133	0	0	1	999
400	${\mathrm{AIC}}_{\mathrm{cd}}$	418	251	193	138	0	710	1	289	0	0	611	389	0	0	1	999
	AIC	598	239	120	43	0	813	1	186	0	0	717	283	0	0	1	999
	Proposed	765	158	60	17	0	885	0	115	0	0	827	173	0	0	0	1000
800	${\mathrm{AIC}}_{\mathrm{cd}}$	375	303	163	159	0	718	0	282	0	0	536	464	0	0	0	1000
	AIC	549	279	106	66	0	824	0	176	0	0	656	344	0	0	0	1000

. Here, again, our IC is shown to be generally superior to its competitors. While all the information criteria are capable of selecting the true model, as the sample size increases, our IC is more likely to select the true model. For any true model, our IC outperforms the other information criteria except when Model 4 is true. Even in that case, our IC is equivalent to the others. In sum, our IC nearly consistently outperforms its major competitors and selects the correct model in all of the cases.

4.2. Selection of Linear Regression Models

In this simulation, we demonstrate how our IC works for linear regression models. Since the derivation given above does not consider the covariate variables, our IC does not directly apply to a regression model. However, it is in the practical analysis of a regression model that we need information criteria to select from the various candidate models. Thus, we naively applied our IC in a simulation and monitored its performance.

In this simulation, we compared the performance of our proposed IC with that of the AIC and the ${\mathrm{AIC}}_{\mathrm{cd}}$ criteria. Four regression models were used in the comparison. From here on, for simplicity, we will use more common notation. Let y be the dependent variable and $\mathit{x}={(x_1,x_2)}^{\prime }$ be independent variables. The regression model is

$$\begin{array}{c}\hfill y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ,\end{array}$$

where $ϵ$ has mean zero and a finite variance conditional on $(x_1,x_2)$. The four models considered are Model 5 with ${\beta }_1={\beta }_2=0$, Model 6 with ${\beta }_1$ nonzero and ${\beta }_2=0$, which is the true model, Model 7 with ${\beta }_1=0$ and ${\beta }_2$ nonzero, and Model 8 with both ${\beta }_1$ and ${\beta }_2$ nonzero. For example, the data-generating model of Model 6 with $({\beta }_0,{\beta }_1,{\beta }_2)=(1,\frac{1}{\sqrt{2}},0)$ is that the joint distribution of $(y,x_1,x_2)$ is a trivariate normal with mean ${[0,0,0]}^{\prime }$ and variance

$$\begin{array}{c}\hfill \left[\begin{array}{ccc}1& \frac{1}{\sqrt{2}}& 0\\ \frac{1}{\sqrt{2}}& 1& 0\\ 0& 0& 1\end{array}\right].\end{array}$$

These four models do not necessarily have a nested/nesting relationship. Model 5 is a special case of Models 6 to 8. Models 6 and 7 are a special case of Model 8, but they do not nest with one another. Models 6 and 7 are not in a nesting relationship. Thus, theoretically speaking, these four models cannot be compared with the AIC-type information criteria. However, in practice, the AIC criterion has been commonly used for the comparison of nonnested/nonnesting models [18]. Thus, we compare the four models according to this convention.

The missingness of y and $x_1$ depends only on $x_2$. To represent the missing-data mechanism in a more formal way, we introduce missing-data indicators $(r_y,r_{x_1})$ for $(y,x_1)$, each of which takes a value of 1 when the corresponding variable is observed and 0 when it is not. The missing-data mechanisms that were used are

$$\begin{array}{ccc}\hfill P(r_y=1|x_2)& =& 1-\frac{2{\omega }_y}{exp\left(x_2\right)+exp(-x_2)},\hfill \\ \hfill P(r_{x_1}=1|x_2)& =& 1-\frac{2{\omega }_{x_1}}{exp\left(x_2\right)+exp(-x_2)},\hfill \end{array}$$

with setting ${\omega }_y$ at $0.7$, and ${\omega }_{x_1}$ at $0.3$.

The simulation was conducted according to the following procedure. First, a sample without missing values is created by drawing random sample of size n from the joint normal distribution of $(y,x_1,x_2)$ which is specified above. Second, the missing values in the sample are created through either of the missing-data mechanisms for $(y,x_1)$ as introduced above. Finally, the proposed IC and the competitor ICs are computed with such a sample for Models 1 to 4, and each is used to select the model corresponding to the minimum value. We repeated this procedure 1000 times, counting the number of times that each model is selected. The counts are used as an indicator of how well each of the ICs work in selecting the best model.

The results of the simulation are shown in

Table 5. Performance comparison with the regression model.

		True Model
		5				6				7				8
		Selected				Selected				Selected				Selected
n	Criterion	5	6	7	8	5	6	7	8	5	6	7	8	5	6	7	8
	Proposed	912	17	36	35	0	983	0	17	0	0	969	31	5	21	115	859
200	${\mathrm{AIC}}_{\mathrm{cd}}$	703	153	144	0	0	1000	0	0	0	0	1000	0	0	271	729	0
	AIC	157	205	213	425	0	439	0	561	0	0	442	558	0	6	19	975
	Proposed	915	18	31	36	0	982	0	18	0	0	980	20	0	0	3	997
400	${\mathrm{AIC}}_{\mathrm{cd}}$	706	138	156	0	0	1000	0	0	0	0	1000	0	0	188	812	0
	AIC	160	223	201	416	0	430	0	570	0	0	452	548	0	0	1	999
	Proposed	890	30	30	50	0	987	0	13	0	0	976	24	0	0	0	1000
800	${\mathrm{AIC}}_{\mathrm{cd}}$	699	162	139	0	0	1000	0	0	0	0	1000	0	0	112	888	0
	AIC	178	217	180	425	0	420	0	580	0	0	401	599	0	0	0	1000
	Proposed	887	39	31	43	0	976	0	24	0	0	971	29	0	0	0	1000
1600	${\mathrm{AIC}}_{\mathrm{cd}}$	705	141	154	0	0	1000	0	0	0	0	1000	0	0	28	972	0
	AIC	180	191	177	452	0	425	0	575	0	0	445	555	0	0	0	1000

. Regardless of which model is the actual true model, our proposed IC appears to work well. In the case where Model 5 is true, our proposed IC outperforms the other two competitors for any sample size. Here, ${\mathrm{AIC}}_{\mathrm{cd}}$ works much better than AIC, but falls behind our IC. In the case where either Models 6 or 7 is true, ${\mathrm{AIC}}_{\mathrm{cd}}$ shows the best performance for any sample size. The second best is our IC. AIC selects the wrong model, Model 8, more than half the time. Interestingly, all three IC’s select neither Model 5 that is in a nesting relationship, nor Model 7(6) that is not in a nesting relationship when Model 6(7) is true. In the case in which Model 8 is true, AIC performs best. However, while for a sample size of 200, AIC selects the true model more often than our proposed IC, for sample sizes of more than 200, our proposed IC is comparable. ${\mathrm{AIC}}_{\mathrm{cd}}$ always selects the wrong models. In summary, our proposed IC performed best or, at worst, second best throughout the simulation. Even in the second-best case, the performance of our proposed IC is very close to that of its higher-performing competitor.

Note that even for different values of the parameters in the missing-data mechanisms, we observed almost the same trend.

5. Conclusions

This paper considered the practical situation in which a set of missing data may not be MAR and developed a new information criterion to handle such a situation. This contrasts with previous ICs, which implicitly consider only the unrealistic case in which the data are MAR as a whole and subsets of the data are MAR as well. Our proposed IC uses the largest data set to estimate model parameters and circumvents the problem that the conventional ICs ignore. Using numerical simulations, it was shown that our new IC works better than, or at worst, equivalently to, its competitors.

The study of missing-data information criterion is far from complete. The present paper requires refinement. Although we applied our IC to regression, this was done without a rigorous basis, as noted in the body of the paper. A solid foundation for the comparison of regression models needs to be developed. Furthermore, application of the proposed IC can be extended to cases in which the parameters of the model are not a linear function of the parameters for the entire data. The approach used to develop our IC should not be limited to regression models; rather, it can be applied to other conventional statistical models such as the generalized linear regression and the explanatory factor model. Further refinements of our IC is also possible. Currently our IC uses $H\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }})$ as the estimator of the H-function $H\left(\widehat{\mathit{\alpha }}\right|\widehat{\mathit{\alpha }})$. That is, implementing the MCMC-based estimator of the H-function presented in [6] to our IC can improve the performance of model selection.

Further, the framework of our proposed IC might be applied to the data taken under the two-phase design. Under the two-phase design, an efficient estimation of the regression parameter has been extensively developed [19,20]. In the two-phase design, some part of the first-phase variables are observed for all subjects and the second-phase variables are observed only for some of the subjects who are selected based on the first-phase variables to avoid high cost. Since the missingness of the two-phase variables are caused by the values of the first-phase variables, the entire data are MAR. However, the variables of interest vary from analysis to analysis, and in some cases, the subset of the data on the model become NMAR. Hence, the variable selection becomes necessary. In such a case, our proposed IC might contribute to select variables. It is necessary to develop our IC to deal with such a two-phase design.