Selection Criteria for Overlapping Binary Models—A Simulation Study

Abstract

This paper deals with the problem of choosing the optimum criterion for selecting the best model out of a set of overlapping binary models. The criteria we studied were the well-known AIC and SBIC, and a third one called $C_2$. Special attention was paid to the setting where neither of the competing models was correctly specified. This situation has not been studied very much but it is the most common case in empirical works. The theoretical study we carried out allowed us to conclude that, in general terms, all criteria perform well. A Monte Carlo exercise corroborated those results.

1. Introduction

This work focused on the analysis of model selection criteria within the framework of binary choice models (BCM), where the endogenous variable is binary $\left(Y_i\right)$, representing the choice of the decision-maker (i) between two options which are quantified by the values 1 and 0. These models are usually expressed as $p_i=F\left(x_i^{\prime }\beta \right)$, F being the cumulative distribution function (c.d.f.), $x_i$ the regressors vector and $p_i$ the probability that $Y_i=1$. The c.d.f. can be normal or logistic, leading to a probit model or a logit model, respectively. Although the common analysis procedure for these models is to apply the maximum likelihood estimation (MLE) method, they can also be implemented from a Bayesian framework using Gibbs Sampling Markov Chain Monte Carlo (MCMC) methods [1,2]. Nevertheless, in this work, we considered the conventional context; thus, the MLE procedure was used.

This paper compares several models in order to select the “best” of them. In a general context, and following [3], the compared models can be nested, overlapping and non-nested models. In the specific framework of BCM, two binary models are nested if they possess the same c.d.f. (both probit or both logit) and the regressors of one of the models are included in the other one. Two binary models are overlapping if both possess the same c.d.f. (both probit or both logit) with some common explanatory variables and other specific variables. Finally, the compared models are non-nested if they possess only specific regressors. Moreover, the models are also non-nested when they possess different c.d.f. (probit versus logit), even if there are some common regressors. Many works define nested and non-nested models, and describe the way of working in every situation [4,5,6], while the overlapping models have been the least analyzed. In this paper, we compared overlapping models, and we found that they were equivalent or that one of them was better than the other (non-equivalent).

Although the hypothesis testing procedures (HTP) are widely used to discriminate between models, we can only use them to choose between pairs of models. In comparison, the selection criteria allowed us to select the best model from quite a large set. This is an important advantage in empirical econometric works. The latter approach allows researchers to express their objectives in the form of a loss function, or by using the discrepancy concept. As [7] established, the discrepancy concept is a particular case of loss function. For non-linear regression models (our framework), the procedures developed by [3,8,9] belongs to the first category (HTP). The second category involves the well-known AIC [10] and SBIC [11], where the discrepancy was obtained from the Kullback–Leibler distance. Additionally, the use of the mean square error (MSE) of prediction as a discrepancy enabled us to derive another criterion, denoted as $C_2$ (see [12]).

Many works have studied the behaviour of selection procedures in linear regression models. However, this subject has been less analysed in a non-linear regression context, and the nested framework is nearly always assumed [13,14,15]. The performance of some selection procedures has also been studied in phylogenetics, where partitioned models were used [16,17,18,19]. Specific references for discrete choice models are [12] for nested models, and [20] for non-nested models.

In this paper, the competing models we selected from were overlapping models. The purpose was to investigate the discriminatory power of certain model selection criteria assuming two situations: (i) at least one of the models was correctly specified; (ii) neither of the models was correctly specified. According to [21], a well-specified model can include irrelevant variables together with the set of regressors of the data generating process (DGP). In our opinion, situation (ii) is the most interesting in practice but the least studied in the literature. Given that, in this case, no model was well-specified, we could not consider consistency as the condition that makes a given selection criterion adequate. The requirement we proposed is that the criterion selects the closest model to the DGP.

The article is organised as follows. In Section 2, we establish the general context and the methodology. Section 3 is dedicated to study the theoretical behaviour of the criteria. Section 4 presents and discusses the results from a Monte Carlo experiment. Conclusions are presented in Section 5.

2. Materials and Methods

Consider the following DGP:

$$M0:p_i=F\left(x_i^{\prime }{\beta }^0\right)=F\left({\beta }_0+{\beta }_1{x}_{1i}+{\beta }_2{x}_{2i}\right)\hspace{1em}\hspace{1em}\hspace{1em}(i=1, \dots , N)$$

(1)

and a pair of overlapping models which, in general terms, are defined as follows:

$$\begin{array}{l}M1:p_i=F\left(a_i^{\prime }\gamma \right)\\ M2:p_i=F\left(b_i^{\prime }\delta \right)\end{array}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(i=1,\dots , N)$$

(2)

where F(·) is the cumulative distribution function (c.d.f), which can be normal or logistic, leading to the probit model or the logit model, respectively. The two competing models have the same c.d.f.; $a_i^{\prime }$ and $b_i^{\prime }$ are the $1\times k_1$ and $1\times k_2$ explanatory variables vectors of M1 and M2, for the i-observation; $\gamma$ and $\delta$ are the corresponding parameter vectors. Given the definition of overlapping models, $a^{\prime }\not\subset b^{\prime }$ and $b^{\prime }\not\subset a^{\prime }$ are satisfied, and both vectors have some common variables.

In order to describe the relationship of each of the competing models with the true model (DGP) we used the Kullback–Leibler distance (KLIC) to the DGP:

$$\mathrm{KLIC} (M0, M\mathrm{j})=E_0 \ln \left[\frac{f_0}{f_j}\right]\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(j=1,2)$$

(3)

being $f_0$ the density function of the DGP and $f_j$ corresponding to model Mj.

From (3), we can write:

$$\mathrm{KLIC} (M0, M1)-\mathrm{KLIC}(M0,M2)=E_0 [\ln f (y | b, {\delta }^{\ast }) ]-E_0 [ \ln f (y | a, {\gamma }^{\ast }) ]=E_0\left[{\ell }_2^{\ast }-{\ell }_1^{\ast }\right]$$

(4)

where ${\gamma }^{\ast }$ and ${\delta }^{\ast }$ are the corresponding pseudo-true parameter vectors (see [22]).

It is well-known that if this statistic (expression (4)) is positive, then M2 is the preferred model, M1 being preferred if (4) is negative. If it is null, the two models are equivalents.

Given the DGP of (1), and following [21], any model which is correctly specified can be written as:

$$p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{x}_{2i}+{\displaystyle \sum _{j=3}^{k_j}{\gamma }_j{d}_{ji}}\right)$$

where d_j are additional regressors, including the particular case, where $d_j$ do not exist.

It is worth noting that, for each competing model, the maximum likelihood estimation of the parameter vectors satisfies:

$$\begin{array}{c}\widehat{\gamma }\stackrel{p}{\to }{\gamma }^{\ast }\\ \widehat{\delta }\stackrel{p}{\to }{\delta }^{\ast }\end{array}$$

(5)

We can distinguish two cases: the case where at least one of the competing models was correctly specified, and the case where neither of them was correct.

2.1. Case 1: At Least One of the Competing Models Is Correctly Specified

Then, the situations we considered are:

• Case 1.1: Both models were well-specified (or both models included the DGP):

  $$\begin{array}{l}M1:p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{x}_{2i}+{\displaystyle \sum _{j=3}^{k_1}{\gamma }_j{d}_{ji}}\right)\\ M2:p_i=F\left({\delta }_0+{\delta }_1{x}_{1i}+{\delta }_2{x}_{2i}+{\delta }_3{z}_i\right)\end{array}$$

  (6)

  with $z\ne d_j \forall j$.

• Case 1.2: Only one of them was well-specified (or only one of them included the DGP):

$$\begin{array}{l}M1:p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{x}_{2i}+{\displaystyle \sum _{j=3}^{k_1}{\gamma }_j{d}_{ji}}\right)\\ M2:p_i=F\left({\delta }_0+{\delta }_1{x}_{1i}+{\delta }_2{z}_i\right)\end{array}$$

(7)

Let ${\beta }^{0+}$ be the parameter vector extended with elements equal to zero in the places corresponding to the variables that are not included in the DGP, that is, ${\left({\beta }^{0+}\right)}^{\prime }$ = ${\left({\beta }^0|0\right)}^{\prime }$. From the convergence result (5), and according to [23], in case 1.1, the equality ${\gamma }^{\ast }={\delta }^{\ast }={\beta }^{0+}$ held, implying that both models are equivalent. However, in case 1.2 ${\gamma }^{\ast }={\beta }^{0+}$ but ${\delta }^{\ast }\ne {\beta }^{0+}$, M1 being better than M2.

2.2. Case 2: Neither of the Models Is Correctly Specified (or Neither of the Models Includes the DGP)

In this situation the compared models are:

$$\begin{array}{c}M1:p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{w}_i\right)\\ M2:p_i=F\left({\delta }_0+{\delta }_1{x}_{2i}+{\delta }_2{w}_i\right)\end{array}$$

(8)

We again used the convergence result (5) to conclude that, in this case, ${\gamma }^{\ast }\ne {\beta }^{0+}$ and ${\delta }^{\ast }\ne {\beta }^{0+}$, it being possible that the competing models are equivalent or not. Specifically, according to [3], there are two possible situations:

• Case 2.1: $f\left(y_i;{\gamma }^{\ast },a_i\right)=f\left(y_i;{\delta }^{\ast },b_i\right)$ that is, the density functions of $y_i$ in M1 and M2, evaluated at the corresponding pseudo-true parameter vectors, were observationally identical. It implies that M1 and M2 are equivalent specifications.
• Case 2.2: $f\left(y_i;{\gamma }^{\ast },a_i\right)\ne f\left(y_i;{\delta }^{\ast },b_i\right)$. In this situation the models can be:

  (a)

  Equivalent, which means that $E_0[{\ell }_1^{\ast }]=E_0[{\ell }_2^{\ast }]$.

  (b)

  Non-equivalent, or $E_0[{\ell }_1^{\ast }]\ne E_0[{\ell }_2^{\ast }]$.

Now, we present the selection criteria, whose behaviour was the aim of our paper. Specifically, we are discussing the well-known information criteria (IC) of Akaike (AIC) and Schwarz (SBIC), and another criterion we call $C_2$. To obtain them, we adopted the discrepancy concept (see [7]). As we can see in [12], “A discrepancy measures the lack of fit between the proposed model and the DGP, in the aspect which the researcher considers the most relevant”. Then, the discrepancy for model M1 could be written as $\Delta \left(F_1,F_0\right)$, and we wished to minimize the “overall discrepancy”, expressed as $\Delta \left(F_{\widehat{\gamma }},F_0\right)$, or equivalently $\Delta \left(\widehat{\gamma }\right)$, with $F_{\widehat{\gamma }}$ the estimated model M1 (that is, ${\widehat{p}}_i=F\left({a^{\prime }}_i\widehat{\gamma }\right)$). The estimation of the expected overall discrepancy, ${\widehat{E}}_0\Delta \left(\widehat{\gamma }\right)$, constitutes the selection criterion. Details about this procedure can be found in [7] and [12]. We assume two discrepancies, called ${\Delta }_1$ and ${\Delta }_2$. The first one is the Kullback–Leibler distance. For a model Mj, it is expressed as ${\Delta }_1\left(F_j,F_0\right)=KLIC(Mj,M0)$, and leads to the information criteria (IC) AIC and SBIC:

$$IC\left(M\mathrm{j}\right)=-\frac{{\widehat{\ell }}_j}{N}+\frac{K_N\left(M\mathrm{j}\right)}{N}$$

(9)

where ${\widehat{\ell }}_j \left(j=1,2\right)$ denotes the log-likelihood of model Mj, evaluated at the corresponding vector of estimates, $k_j$ is the number of parameters of Mj, $K_N\left(M\mathrm{j}\right)$ is the correction factor ($k_j$ for AIC and $\frac{k_j\log N}{2}$ for SBIC).

The second discrepancy is the mean square error (MSE) of prediction. For model M1, this discrepancy is ${\Delta }_2\left(F_1,F_0\right)=E_0{\left(Y_{N+1}-F\left({a^{\prime }}_{N+1}\gamma \right)\right)}^2$, with “N + 1” indicating an out-sample observation. For any Mj model, and following the previously mentioned procedure, the expression of the criterion is:

$$C_2\left(Mj\right)=\frac{SS{D}_j}{N}\left(1+\frac{2k_j}{N}\right)$$

(10)

and $SS{D}_j={\displaystyle \sum _{i=1}^N{\left(Y_i-{\widehat{F}}_{ji}\right)}^2}$, the squared sum of the differences between the binary variable and the estimated probability with model Mj. The proof of (10) is developed in [24].

The model having the criterion with the lowest value was chosen, so different criteria could have led to a different choice. Nevertheless, we were interested in analysing if the criteria worked well, that is, if the selection was correct, in the sense we define in the following section.

3. Theoretical Results

In this section, we study the theoretical behaviour of the criteria, in order to prove if they perform well. All proofs of the results we present in this section can be seen, in detail, in [24].

We carry out an asymptotic analysis, which needs a set of initial assumptions, the results and definitions that we state below.

Assumption 1.

The$x_i^{\prime }$, $a_i^{\prime }$ and $b_i^{\prime }$regressor vectors of the models specified in (1) and (2) are non stochastic. The variables of these vectors have sample means and variances with finite limits.

Lemma 1.

Let be$y_i$a variable, which is not i.i.d., but heterogeneous (non-identical means and non-identical variances). Then:

$$N^{-1}{\displaystyle \sum _{i=1}^{N}a(y_i,\tilde{\theta }) \stackrel{p}{\to }} E\left[\frac{1}{N}{\displaystyle \sum _{i=1}^{N}a(y_i,{\theta }_0)}\right]$$

(11)

Proof. 

The proof of (11) is based on the law of large numbers for heterogeneous variables, together with a lemma of [25].

The law of large numbers for heterogeneous variables is expressed in the following terms [26]: “Let the sequence {$y_i-{\mu }_i$} be independent with E ($y_i-{\mu }_i$) = 0. If E $|y_i-{\mu }_i{|}^{1+\delta }$≤ B < ∞ ∀i with δ > 0, then $\frac{1}{N}{\displaystyle \sum _{i=1}^N}(y_i-{\mu }_i)$$\stackrel{p}{\to }$0”.

The lemma of [25] (p. 2156) is expressed as follows: “If $z_i$ is i.i.d., a (z, $\theta$) is continuous at ${\theta }_0$ with probability one, and there is a neighbourhood $\mathrm{\Gamma }$ of ${\theta }_0$ such that E [${\sup }_{\theta \in \mathrm{\Gamma }}‖a(z,\theta )‖$] < ∞, then for any $\tilde{\theta } \stackrel{p}{\to }$ ${\theta }_0$, $N^{-1}{\displaystyle \sum _{i=1}^{N}a(z_i,\tilde{\theta }) \stackrel{p}{\to }} E[a(z,{\theta }_0)]$”. This lemma, together with the law of large numbers, allows us to write (11). □

Definition 1.

M1 and M2 are equivalent models if

$$E_0\left[\log \frac{f\left(y_i;a_i,{\gamma }^{\ast }\right)}{f\left(y_i;b_i,{\delta }^{\ast }\right)}\right]=0$$

(12)

which leads to:

$$\left[F_{0i}\log \frac{F_{1i}}{F_{2i}}+\left(1-F_{0i}\right)\log \frac{1-F_{1i}}{1-F_{2i}}\right]=0$$

(13)

where$F_{0i}=F\left(x_i^{\prime }\beta \right)$,$F_{1i}=F\left(a_i^{\prime }{\gamma }^{\ast }\right)$and$F_{2i}=F\left(b_i^{\prime }{\delta }^{\ast }\right)$.

Definition 2.

M1 is closer to the DGP than M2 if:

$$E_0\left[\log \frac{f\left(y_i;a_i,{\gamma }^{\ast }\right)}{f\left(y_i;b_i,{\delta }^{\ast }\right)}\right]>0$$

(14)

which leads to:

$$\left[F_{0i}\log \frac{F_{1i}}{F_{2i}}+\left(1-F_{0i}\right)\log \frac{1-F_{1i}}{1-F_{2i}}\right] >0$$

(15)

Definition 3.

Let $R\left(·\right)$ be a model selection criterion. It is said that $R\left(·\right)$ is adequate when:

(i) 

If M1 and M2 are equivalent, plim [R(M1)] = plim [R(M2)].

(ii) 

If M1 is closer than M2 to the DGP, then plim [R(M1)] < plim [R(M2)].

Now, for every case we enuntiated in the previous section, we must prove whether the definition of “adequate criterion” is satisfied.

Result 1.

The IC criteria behave well in all settings.

Proof. 

The basic tool for achieving this result is the comparison of Definitions 1 and 2 with Definition 3. In this sense, Definitions 1 and 2 establish the condition that must be met when the compared models are equivalent or non-equivalent, respectively. On the other hand, Definition 3 tells us the requirements for determining if a specific criterion is adequate in each context (of equivalence or not).

Expression (9) can be written as:

$$IC\left(Mj\right)=-\frac{1}{N}{\displaystyle \sum _{i=1}^N\left[y_i\log {\widehat{F}}_{ji}+\left(1-y_i\right)\log \left(1-{\widehat{F}}_{ji}\right)\right]}+\frac{K_N\left(Mj\right)}{N}\hspace{1em}j=1,2$$

(16)

with ${\widehat{F}}_{1i}=F\left(a_i^{\prime }\widehat{\gamma }\right)$ and ${\widehat{F}}_{2i}=F\left(b_i^{\prime }\widehat{\delta }\right)$.

Using Lemma 1 and the convergences given in (5) for the first term, we obtain:

$$\frac{{\widehat{\ell }}_j}{N}\stackrel{p}{\to }\frac{1}{N}{\displaystyle \sum _{i=1}^N{E}_0\left[y_i\log F_{ji}+\left(1-y_i\right)\log \left(1-F_{ji}\right)\right]}\hspace{1em}j=1,2$$

(17)

The correction factor $\frac{K_N\left(M\mathrm{j}\right)}{N}$ converges to zero for every model.

When the competing models are equivalent (cases 1.1, 2.1 and 2.2.(a)), the IC criteria will be adequate if equality of Definition 3 (i) holds, which, using (17), leads to the following expression:

$$\frac{1}{N}{\displaystyle \sum _{i=1}^N\left[F_{0i}\log \frac{F_{1i}}{F_{2i}}+\left(1-F_{0i}\right)\log \frac{\left(1-F_{1i}\right)}{\left(1-F_{2i}\right)}\right]}=0$$

(18)

This result is always satisfied, given Definition 1, so we can say that the IC criteria performed well.

When the models are non-equivalent, and assuming M1 is always better than M2 (cases 1.2 and 2.2.(b)), the IC criteria will be adequate if equality of Definition 3 (ii) holds, which, using (17), leads to:

$$\frac{1}{N}{\displaystyle \sum _{i=1}^N\left[F_{0i}\log \frac{F_{1i}}{F_{2i}}+\left(1-F_{0i}\right)\log \frac{\left(1-F_{1i}\right)}{\left(1-F_{2i}\right)}\right]} > 0$$

(19)

this result being identical to Definition 2. Thus, the IC criteria performed well. □

It should be noted that $\frac{K_N\left(M\mathrm{j}\right)}{N}$ converges to zero faster for the model with a lower $k_j$. Additionally, expression (19) shows that AIC and SBIC are asymptotically identical, which is not strictly true when $k_1\ne k_2$. In this situation, for a given pair of competing models, the difference between both criteria is due to the different rate of convergence to zero between $AIC(M1)-AIC(M2)$ and $SBIC(M1)-SBIC(M2)$. This difference is caused by the corrector factor.

Specifically, we can write:

$$\frac{SBIC(M1)-SBIC(M2)}{AIC(M1)-AIC(M2)}=\frac{O\left(\frac{\log N}{N}\right)}{O\left(\frac{1}{N}\right)}=O\left(\log N\right)$$

(20)

which means that, when N increases, the distance between the convergence rate of the numerator and the denominator of expression (20) becomes larger. This implies that SBIC will tend toward one of the models more than AIC. Which model? It is evident that, if $k_1>k_2$, the tendency will be toward M2, given that both AIC and SBIC selects the model with a lower value of the criterion. It is important to remark that $plim\left(IC\left(M1\right)-IC\left(M2\right)\right)=0$ is not contradictory with a higher tendency to the model that is more parsimonious, given that both models are equivalent.

Result 2.

The$C_2$criterion is adequate, except in a specific situation.

Proof. 

Expression (10) can be written as:

$$C_2\left(Mj\right)=\frac{{\displaystyle \sum _{i=1}^N{\left(y_i-{\widehat{F}}_{ji}\right)}^2}}{N}\left(1+\frac{2k_j}{N}\right)\hspace{1em}j=1, 2$$

(21)

Applying Lemma 1 together with convergences (5) we obtain:

$$\frac{SS{D}_j}{N}=\frac{1}{N}{{\displaystyle \sum _{i=1}^N\left(y_i-{\widehat{F}}_{ji}\right)}}^2\stackrel{p}{\to }\hspace{1em}\frac{{\displaystyle \sum _{i=1}^N{E}_0{\left(y_i-F_{ji}\right)}^2}}{N}\hspace{1em}j=1, 2$$

(22)

Additionally, the term $\left(1+\frac{2k_j}{N}\right)$ (j = 1, 2) converges to 1 when N → ∞.

When the compared models are equivalent and well-specified (case 1.1), the probability limit (22) is the same for both models. It implies that Definition 3 i) is satisfied, in other words, the $C_2$ criterion performed well. Note that the convergence rate is different between M1 and M2 when $k_1\ne k_2$. The term $\frac{2k_j}{N}$ is $O\left(\frac{1}{N}\right)$ and converges to zero faster for models with a lower $k_j$.

When the compared models are non-equivalent and only M1 is well-specified (case 1.2), the probability limit (22) is different for each model:

$$\frac{SS{D}_1}{N}\stackrel{p}{\to }\frac{{\displaystyle \sum _{i=1}^N{F}_{0i}\left(1-F_{0i}\right)}}{N}=h_1$$

(23)

$$\frac{SS{D}_2}{N}\stackrel{p}{\to }\frac{{\displaystyle \sum _{i=1}^N{F}_{0i}\left(1-F_{0i}\right)}}{N}+\frac{{\displaystyle \sum _{i=1}^N{\left(F_{0i}-F_{2i}\right)}^2}}{N}=h_1+h_2$$

(24)

being $h_1$ and $h_2$ positive terms. It is straightforward to see that Definition 3 (ii) is satisfied, so the $C_2$ criterion performed adequately.

If neither of the competing models is correctly specified (case 2), the probability limits of (22) for each model can be written as:

$$\frac{SS{D}_1}{N}\stackrel{p}{\to }\frac{{\displaystyle \sum _{i=1}^N{F}_{´0i}\hspace{0.17em}\left(1-F_{0i}\right)}}{N}+\frac{{\displaystyle \sum _{i=1}^N{\left[F_{0i}-F_{1i}\right]}^2}}{N}=h_1+h_3$$

(25)

$$\frac{SS{D}_2}{N}\stackrel{p}{\to }\frac{{\displaystyle \sum _{i=1}^N{F}_{0i}\hspace{0.17em}\left(1-F_{0i}\right)}}{N}+\frac{{\displaystyle \sum _{i=1}^N{\left[F_{0i}-F_{2i}\right]}^2}}{N}=h_1+h_2$$

(26)

with $h_i$ (i = 1, 2, 3) being positive constants.

Now, the final conclusions depend on the relationship between the density functions. Then, in Case 2.1, where the density functions were observationally identical (equivalent models), $F_{1i}=F_{2i}$ is satisfied. It implies that $h_3=h_2$, so Definition 3 is verified, and the $C_2$ criterion behaved well.

In Case 2.2. (a), with non-observationally identical density functions and equivalent models, the only possibility for achieving $h_3=h_2$ is that, on average, $\left[F_{0i}-F_{1i}\right]=-\left[F_{0i}-F_{2i}\right]$, or, equivalently, $2F_{0i}=F_{1i}+F_{2i}$. Therefore, there can be empirical works where the criterion $C_2$ did not behave well. The Monte Carlo experiment will allow us a more specific analysis of the behaviour of the criterion.

Finally, in Case 2.2.(b), where the competing models were non-equivalent, we assumed that M1 was better than M2. In order to study the power of the criterion, we applied a strategy similar to that used in the IC criteria. That is, we related Definitions 2 and 3 (ii). Definition 2 can be written as:

$${\left(\frac{F_{1i}}{F_{2i}}\right)}^{F_{0i}}>{\left(\frac{1-F_{2i}}{1-F_{1i}}\right)}^{1-F_{0i}}$$

(27)

We wanted to find the combinations of $F_{0i}$, $F_{1i}$ and $F_{2i}$ that satisfy (27). The results we obtained are summarized in

Table 1. Combinations of $F_0$, $F_1$ and $F_2$ which leads to M1 be better than M2.

Value of $F_0$	Condition Satisfied When M1 Is Better than M2
$F_{0i}=0$	$F_{1i}<F_{2i}$ and $F_{1i}\ne 0$
$F_{0i}$∈ (0, 0.5]	$\begin{array}{c}{F}_{1i}<F_{2i}\hspace{1em}\mathrm{and}\hspace{1em}{F}_{1i}+F_{2i}>1\hspace{1em}\\ F_{1i}>F_{2i}\hspace{1em}\mathrm{and}\hspace{1em}{F}_{1i}+F_{2i}<1\hspace{1em}\mathrm{and}\hspace{1em}{F}_{0i}\cong 0.5\hspace{1em}\\ F_{1i}<F_{2i}\hspace{1em}\mathrm{and}\hspace{1em}{F}_{1i}+F_{2i}<1\hspace{1em}\mathrm{and}\hspace{1em}{F}_{0i}\approx 0\end{array}$
$F_{0i}$∈ (0.5, 1)	$\begin{array}{c}{F}_{1i}>F_{2i}\hspace{1em}\mathrm{and}\hspace{1em}{F}_{1i}+F_{2i}<1\hspace{1em}\\ F_{1i}>F_{2i}\hspace{1em}\mathrm{and}\hspace{1em}{F}_{1i}+F_{2i}>1\hspace{1em}\mathrm{and}\hspace{1em}{F}_{0i}\approx 1\hspace{1em}\\ F_{1i}<F_{2i}\hspace{1em}\mathrm{and}\hspace{1em}{F}_{1i}+F_{2i}>1\hspace{1em}\mathrm{and}\hspace{1em}{F}_{0i}\approx 0.5\end{array}$
$F_{0i}=1$	$F_{1i}>F_{2i}$ and $F_{1i}\ne 1$

.

Definition 3 (ii) establishes that the $C_2$ criterion behaves well if $h_3<h_2$, that is to say:

$$\frac{{\displaystyle \sum _{i=1}^N\left[\left(F_{1i}-F_{2i}\right)\left(F_{1i}+F_{2i}-2F_{0i}\right)\right]}}{N}<0$$

(28)

For every combination presented in Table 1, we get the previous result, so the $C_2$ criterion is adequate. □

4. Simulation Study and Discussion

The objective of the Monte Carlo experiments is twofold: confirm the theoretical results and assess the performance of all criteria with finite samples.

The generation of the binary variable $y_i$ is based on the latent linear model that underlies any binary model:

$$y_i^{\ast }=x_i^{\prime }\beta +u_i$$

(29)

where $y_i^{\ast }$ is a latent (unobservable) variable which generates $y_i$ through:

$$y_i=\left\{\begin{array}{c}1\\ 0\end{array}\right.\begin{array}{c}if\\ if\end{array}\begin{array}{c}{y}_i^{\ast }>0\\ y_i^{\ast }\le 0\end{array}$$

(30)

Under the assumption established in Section 3, and following the procedure of [27], we obtained the values of $y_i$. We considered different sets of parameter values and different kinds of explanatory variables (continuous and dummy) and the standard normal distribution function was chosen for the error term, implying exclusive focus on probit models. Two sample sizes N = 200 and 2000 were used; we carried out 500 replications for each experiment. Additionally, the intercept was fixed at a value of −2, in order to avoid a non-balanced number of ones in the sample of $y_i$, which would lead to problems when estimating and interpreting results.

In each of the 500 replications, we estimated M1 and M2, and calculated the value of the IC and $C_2$ criteria in each replication. The corresponding tables for every experiment show the number of times that each criterion selected M1. Note that we only present tables for N = 2000 and comment the differences from N = 200 if such differences exist. In all cases, the DGP is $p_i=F\left({\beta }_0+{\beta }_1{x}_{1i}+{\beta }_2{x}_{2i}\right)$.

4.1. Montecarlo Exercise When Both Models Are Correctly Specified (Case 1.1)

We consider the following well-specified models M1 and M2:

$$\begin{array}{l}M1\equiv p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{x}_{2i}+{\gamma }_3{w}_i+{\gamma }_4{s}_i\right)\\ M2\equiv p_i=F\left({\delta }_0+{\delta }_1{x}_{1i}+{\delta }_2{x}_{2i}+{\delta }_3{z}_i\right)\end{array}$$

Firstly, we assumed ${\gamma }_4=0$, so we chose between models with the same number of parameters and, afterwards, we assumed ${\gamma }_4\ne 0$. These settings are called A and B, respectively; the results are presented in

Table 2. Behaviour of the selection criteria in Case1.1 N = 2000.

	DGP			Specified Models
	${\mathit{\beta }}^{\mathbf{\prime }}$	${\mathit{x}}_{\mathbf{1}}$	${\mathit{x}}_{\mathbf{2}}$	$\mathit{w}$	z	s	AIC	SBIC	${\mathit{C}}_{\mathbf{2}}$
Setting A	(−2,3,1)	U	U	U	N(3,1)		273	273	275
	(−2,1,3)	U	U	U	N(3,1)		236	236	240
	(−2,1,1)	U	U	U	N(3,1)		234	234	244
	(−2,3,1)	${\chi }^2$	${\chi }^2$	U	N(3,1)		240	240	252
	(−2,1,3)	${\chi }^2$	${\chi }^2$	U	N(3,1)		273	273	261
	(−2,1,1)	${\chi }^2$	${\chi }^2$	U	N(3,1)		236	236	238
	(−2,3,1)	U	${\chi }^2$	U	N(3,1)		247	247	245
	(−2,1,3)	U	${\chi }^2$	U	N(3,1)		231	231	245
	(−2,1,1)	U	${\chi }^2$	U	N(3,1)		243	243	260
	(−2,3,1)	U	dummy	U	N(3,1)		259	259	269
	(−2,1,3)	U	dummy	U	N(3,1)		231	231	245
	(−2,1,1)	U	dummy	U	N(3,1)		242	242	261
	(−2,3,1)	${\chi }^2$	dummy	U	N(3,1)		255	255	251
	(−2,1,3)	${\chi }^2$	dummy	U	N(3,1)		248	248	257
	(−2,1,1)	${\chi }^2$	dummy	U	N(3,1)		251	251	234
Setting B	(−2,3,1)	U	U	U	N(3,1)	${\chi }^2$	144	10	179
	(−2,1,3)	U	U	U	N(3,1)	${\chi }^2$	133	10	151
	(−2,1,1)	U	U	U	N(3,1)	${\chi }^2$	126	2	144
	(−2,3,1)	${\chi }^2$	${\chi }^2$	U	N(3,1)	${\chi }^2$	128	12	215
	(−2,1,3)	${\chi }^2$	${\chi }^2$	U	N(3,1)	${\chi }^2$	140	10	230
	(−2,1,1)	${\chi }^2$	${\chi }^2$	U	N(3,1)	${\chi }^2$	130	11	185
	(−2,3,1)	U	${\chi }^2$	U	N(3,1)	${\chi }^2$	128	9	165
	(−2,1,3)	U	${\chi }^2$	U	N(3,1)	${\chi }^2$	106	2	192
	(−2,1,1)	U	${\chi }^2$	U	N(3,1)	${\chi }^2$	120	5	161
	(−2,3,1)	U	dummy	U	N(3,1)	${\chi }^2$	144	12	177
	(−2,1,3)	U	dummy	U	N(3,1)	${\chi }^2$	145	7	153
	(−2,1,1)	U	dummy	U	N(3,1)	${\chi }^2$	141	7	162
	(−2,3,1)	${\chi }^2$	dummy	U	N(3,1)	${\chi }^2$	147	10	200
	(−2,1,3)	${\chi }^2$	dummy	U	N(3,1)	${\chi }^2$	136	8	185
	(−2,1,1)	${\chi }^2$	dummy	U	N(3,1)	${\chi }^2$	143	12	185

.

For setting A, we can see that the number presented in each cell was around 250 (50% of 500 times), which is the correct behaviour for equivalent models. However, if we consider setting B, where M1 had more irrelevant regressors than M2, we could see that the criteria tended to select the model with fewer parameters, that is, it selected the more parsimonious model (M2), and that this tendency grew with the sample size. This behaviour was also correct given that both specifications were equivalent. Specifically, the most parsimonious criterion is SBIC, so the Monte Carlo exercise corroborated this theoretical aspect of the previous section. Additionally, we observed that neither the kind of variables nor the set of values of the DGP parameter vector seemed to affect the behaviour of the criteria.

4.2. Montecarlo Exercise When Only One of the Models Is Correctly Specified (Case 1.2)

In these experiments, M1 and M2 are expressed as follows:

$$\begin{array}{l}M1\equiv p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{x}_{2i}+{\gamma }_3{w}_i\right)\\ M2\equiv p_i=F\left({\delta }_0+{\delta }_1{x}_{1i}+{\delta }_2{z}_i\right)\end{array}$$

The theoretical results were corroborated, so all the criteria tended to select M1 for whatever kind of explanatory variables. The corresponding table has been omitted, given that the value in all cells was 500.

However, for N = 200 the results were not so evident, although they tended towards adequate behaviour. Specifically, differences were found when the variables $x_1$ and $x_2$ were both uniform and the weight of $x_2$ was not greater than that of $x_1$; this difference was more evident for the SBIC criterion.

4.3. Montecarlo Experiment When Neither of the Models Is Correctly Specified (Case 2)

We needed to analyse each of the situations defined in Case 2 of Section 2 separately. The two compared models are:

$$\begin{array}{c}M1\equiv p_i=F\left({\gamma }_0+{\gamma }_1{x}_{1i}+{\gamma }_2{w}_i\right)\\ M2\equiv p_i=F\left({\delta }_0+{\delta }_1{x}_{2i}+{\delta }_2{w}_i\right)\end{array}$$

The implementation of the experiments for 2.1 and 2.2.(a) required using the relationship between the true parameter vector (${\beta }_{}^0$) and each of the pseudo-true parameter vectors (${\gamma }^{\ast }$ and ${\delta }^{\ast }$). In case 2.1, we should have been able to obtain the value of ${\beta }_{}^0$ from them, satisfying the equality of density functions. In other terms, if we had ${\gamma }^{\ast }=m_1({\beta }^0)$ and ${\delta }^{\ast }=m_2({\beta }^0)$, we were interested in obtaining ${\beta }_{}^0$ that makes $f(Y_i;m_1\left({\beta }^0\right),a_i)=f(Y_i;m_2\left({\beta }^0\right),b_i)$. The same idea could be used in 2.2.(a) in order to make E(${\ell }_1^{\ast }$) and E(${\ell }_2^{\ast }$) equal, that is, $E\left[{\ell }_1\left(m_1\left({\beta }^0\right)\right)\right]=E\left[{\ell }_2\left(m_2\left({\beta }^0\right)\right)\right]$. Nevertheless, the non-linear equation system that we needed to solve possessed insurmountable problems. Given that we could not obtain the exact relationship, we needed to approximate the equalities of densities and likelihoods. To this end, we generated several DGPs modifying the value of the parameter vector ${\beta }_{}^0$ and the kind of explanatory variables. Again, the intercept was fixed at a value of −2, while ${\beta }_1$ and ${\beta }_2$ took values in the range of (−2, 2) counting by 0.5 s; additionally, they took values 3, 4, 5 and 7. As a result of this strategy, we generated 132 different DGPs. Each of the outlined DGPs lead to a specific relationship between models M1 and M2: equivalent models (with identical or non-identical densities) or non-equivalent models.

In order to classify the 132 experiments into the two categories, we used the following indicators:

$$absel=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left|de{l}_i\right|}$$

(31)

$$\mathrm{AM}1\equiv \mathrm{times} (\mathrm{of} \mathrm{the} \mathrm{N} \mathrm{observations}) \mathrm{that} de{l}_i>0$$

(32)

$$absdifden=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left|difde{n}_i\right|}$$

(33)

with $de{l}_i=E\left({\ell }_{1i}^{\ast }\right)-E\left({\ell }_{2i}^{\ast }\right)$ and $difde{n}_i=f\left(y_i;{\gamma }^{\ast },x_1,x_2\right)-f\left(y_i;{\delta }^{\ast },x_1,x_2\right)$.

Firstly, we classified the experiments into “containing equivalent models”, or “containing nonequivalent models”. Secondly, in the first group, we distinguished identical from non-identical densities. Finally, we classified the non-equivalent models depending on their closeness to the DGP. The following three stages were carried out:

Step 1.

The two requirements for considering the models as equivalents are:

(R.1)

A value of absel close to zero.

(R.2)

A value of AM1 close to $N/2$.

Taking into account absel, two models will tend to be equivalent if, for most of the observations, $de{l}_i\approx 0$, which should lead to $absel\approx 0$. Could we have used only this measure to affirm that the models were equivalent? The answer is no, because we could find $absel\approx 0$ but with most of the observations satisfying $E\left({\ell }_{1i}^{\ast }\right)>E\left({\ell }_{2i}^{\ast }\right)$, which means that M1 was closer to the DGP. Using AM1 instead of absel, two models will tend to be equivalent if $AM1\approx N/2$. Could we have used only AM1 to classify the models? Again, the answer is no, because it could happen that $AM1\approx N/2$ but with a large value of absel, due to large values of $\left|de{l}_i\right|$. Then, we need the two requirements (R.1) and (R.2).

Step 2.

To consider that two equivalent models have identical densities, a value of difden close to zero is required.

Step 3.

If model M1 is better than M2, $N/2<AM1<N$ must be satisfied, while M2 will be better if $0<AM1<N/2$.

Each experiment was numbered from 1 to 132, and was classified in type A, B or C, as we can see in

Table 3. Number of every experiment in Case 2, and types of x in the DGP (A,B,C) ¹.

Number	DGP	Number	DGP	Number	DGP	Number	DGP
A,B,C	${\displaystyle \left({\mathit{\beta }}_{\mathbf{1}}\mathbf{,}{\mathit{\beta }}_{\mathbf{2}}\right)}$	A,B,C	${\displaystyle \left({\mathit{\beta }}_{\mathbf{1}}\mathbf{,}{\mathit{\beta }}_{\mathbf{2}}\right)}$	A,B,C	${\displaystyle \left({\mathit{\beta }}_{\mathbf{1}}\mathbf{,}{\mathit{\beta }}_{\mathbf{2}}\right)}$	A,B,C	${\displaystyle \left({\mathit{\beta }}_{\mathbf{1}}\mathbf{,}{\mathit{\beta }}_{\mathbf{2}}\right)}$
1,45,89	(1,−2)	12,56,100	(1,7)	23,67,111	(3,5)	34,78,122	(7,1)
2,46,90	(1,−1.5)	13,57,101	(3,−2)	24,68,112	(3,7)	35,79,123	(−2,3)
3,47,91	(1,−1)	14,58,102	3,−1.5)	25,69,113	(−2,1)	36,80,124	(−1.5,3)
4,48,92	(1,−0.5)	15,59,103	(3,−1)	26,70,114	(−1.5,1)	37,81,125	(−1,3)
5,49,93	(1,0.5)	16,60,104	(3,−0.5)	27,71,115	(−1,1)	38,82,126	(−0.5,3)
6,50,94	(1,1)	17,61,105	(3,0.5)	28,72,116	(−0.5,1)	39,83,127	(0.5,3)
7,51,95	(1,1.5)	18,62,106	(3,1)	29,73,117	(0.5,1)	40,84,128	(1.5,3)
8,52,96	(1,2)	19,63,107	(3,1.5)	30,74,118	(1.5,1)	41,85,129	(2,3)
9,53,97	(1,3)	20,64,108	(3,2)	31,75,119	(2,1)	42,86,130	(4,3)
10,54,98	(1,4)	21,65,109	(3,3)	32,76,120	(4,1)	43,87,131	(5,3)
11,55,99	(1,5)	22,66,110	(3,4)	33,77,121	(5,1)	44,88,132	(7,3)

¹ In type A both variables are U(0,1), in B $x_1$ $\sim$ U(0,1) and $x_2$ $\sim$ ${\chi }_1^2$, and C has both variables ${\chi }_1^2$.

.

Those experiments with an extreme percentage of zeros in the sample of the binary variable were omitted.

Table 4. Experiments sequenced by each of the three indicators (absel, AM1 and absdifden).

Experiments with the Lowest Value of absel			Experiments with AM1 around 1000			Experiments with the Lowest Value of absdifden
Exp.	absel	AM1	Exp.	AM1	absel	Exp.	absdifden
3	0.00716535	1001	57	1011	0.17489579	27	0.0224
27	0.00716886	1003	27	1003	0.00716886	3	0.0227
28	0.00731291	505	3	1001	0.00716535	28	0.0257
4	0.00737039	1498	6	1000	0.01854382	4	0.0261
5	0.01279404	1514	63	995	0.25329468	48	0.0325
29	0.0127966	510	21	991	0.13463303	47	0.0341
48	0.01515962	1035	109	980	0.44042798	46	0.0349
						45	0.0354
						29	0.0510

presents the experiments with the lowest value of absel, the values of AM1 closer to 1000, and the values of absdifden near zero.

We concluded that experiments 27, 3 and 6 included equivalent models, 27 and 3 having identical densities. The rest of the experiments corresponded to non-equivalent models, and we needed to classify them according to their closeness to the DGP. Given that AM1 was the adequate indicator,

Table 5. Behaviour of the selection criteria in Case 2. N = 2000 ¹.

Exp.	AM1	IC	C2	Exp.	AM1	IC	C2	Exp.	AM1	IC	C2	Exp.	AM1	IC	C2
104	1868	500	500	20	1393	500	500	129	805	0	0	84	315	0	0
34	1861	500	500	76	1366	500	500	47	786	2	22	37	315	0	0
17	1846	500	500	119	1365	500	500	65	764	0	0	50	261	0	0
16	1834	500	500	30	1359	499	499	112	719	0	0	123	245	0	0
33	1827	500	500	13	1342	500	500	128	708	0	0	124	232	0	0
92	1825	500	500	107	1321	500	500	96	705	0	0	51	226	0	0
103	1809	500	500	88	1292	500	500	22	701	0	0	10	224	0	0
91	1807	500	500	42	1288	500	500	66	686	0	0	114	209	0	0
90	1794	500	500	118	1268	500	500	7	685	2	2	125	208	0	0
102	1779	500	500	59	1252	500	500	46	668	0	5	52	204	0	0
89	1779	500	500	132	1219	500	500	35	662	0	0	11	195	0	0
101	1753	500	500	108	1188	500	500	41	636	0	0	113	194	0	0
32	1751	500	500	131	1156	500	500	67	601	0	0	69	190	0	0
105	1691	500	500	62	1154	67	244	45	589	0	2	115	180	0	0
18	1691	500	500	94	1112	441	451	117	583	0	0	53	170	0	0
15	1675	500	500	58	1112	352	494	97	576	0	0	39	170	0	0
78	1641	500	500	130	1093	500	500	23	537	0	0	38	170	0	0
122	1606	500	500	87	1067	447	496	68	518	0	0	116	167	0	0
44	1582	500	500	48	1035	72	195	98	513	0	0	54	162	0	0
121	1556	500	500	57	1011	41	322	29	510	1	1	70	162	0	0
19	1540	500	500	27	1003	245	240	28	505	10	13	126	161	0	0
120	1538	500	500	3	1001	250	245	8	504	0	0	79	157	0	0
31	1524	500	500	6	1000	249	252	36	494	0	0	12	137	0	0
77	1516	500	500	63	995	0	0	74	491	0	0	71	133	0	0
5	1514	499	498	21	991	240	223	99	486	0	0	56	131	0	0
93	1510	500	500	109	980	357	324	40	475	0	0	55	129	0	0
14	1504	500	500	86	932	0	2	85	444	0	0	80	128	0	0
4	1498	484	484	110	891	0	0	100	414	0	0	73	104	0	0
60	1492	500	500	64	877	0	0	49	390	0	0	81	97	0	0
106	1474	500	500	95	835	0	0	24	385	0	0	72	84	0	0
43	1438	500	500	111	813	0	0	127	329	0	0	83	80	0	0
61	1429	500	500	75	813	0	0	9	325	0	0	82	57	0	0

¹ Experiments 1, 2, 25 and 26 are omitted, due to extreme percentage of ones/zeros in the samples.

shows all the experiments sequenced from the highest to the lowest value of this measure.

The variable w is always generated as N(3,1) and IC groups AIC and SBIC together, because the results of both criteria were identical.

We find the experiments with equivalent models at the middle of this table. Above them (the upper part of the table), we see the experiments where M1 was better and, below them (the lower part), those where M2 was the best model. The results showed that, in the upper end of the table, the values in columns IC and C2 tended towards 500 and, in the lower end, tended towards zero, corroborating the theoretical conclusions.

The experiments that contained equivalent models with identical densities (27 and 3) corroborated the theoretical results. On the other hand, in experiment 6 (equivalent models with non-identical densities), both IC and $C_2$ performed well. Nevertheless, the theoretical results for $C_2$ concluded that this criterion was adequate only in some situations. We can affirm that experiment 6 belonged to one of these situations, characterized by uniform distribution of the DGP variables, and similar (but not oversized) weights for both variables. We think that these characteristics could cause $C_2$ to behave well.

Finally, we observed a non-adequate behaviour of the criteria in some experiments. In the upper part of the table, experiments 62, 48 and 57 showed values far less than 500 and large differences between the values of the criteria columns. In the lower part of the table, we find that experiments 21 and 109 have IC and C2 column values far from 0. Could this atypical behaviour be due to anomalous observations? Taking into account that $de{l}_i$ is the main element which underlies the indicators used to classify the experiments, we studied whether extreme values of $\left|de{l}_i\right|$ were always associated with the same selection (same sign of $de{l}_i$). We found that this happened in all the experiments, except in 21. Eliminating these extreme values, the behaviour of the criteria became adequate, as we show in

Table 6. Atypical experiments in Case 2. Behaviour of the selection criteria.

Experiment	$\mathbf{(}{\mathit{\beta }}_{\mathbf{1}}\mathbf{,}{\mathit{\beta }}_{\mathbf{2}}\mathbf{)}$	Type of x	N	AM1	absdel	IC	C2
62	(3,1)	B	1800	1144	0,1517	500	500
48	(1,−0.5)	B	1750	1035	0,0097	377	499
57	(3,−2)	B	1800	1003	0,12	499	500
109	(3,3)	C	1925	909	0,4	0	0

.

As a final comment, we observed a greatly reduced number of experiments with equivalent models. We understood that this was logical, because the experiments of Case 2 corresponded to pairs of models where the DGP was not nested in M1 or M2. Given that model M1 contained the variable $x_1$ and model M2 contained $x_2$ ($x_1$ and $x_2$ being the only DGP variables), it was very difficult to find cases where both the M1 and M2 models were equivalent.

When we re-executed the analysis for a sample size of 200, the results were similar in general terms, although the tendency toward correct behaviour of the criteria was slower. Nevertheless, we could affirm that the three criteria performed quite well for finite sample sizes.

5. Conclusions

Within the framework of overlapping binary models, we have studied the power of model selection criteria: the well-known information criteria AIC and SBIC, and the $C_2$ criterion, based on the mean square error of prediction.

As we previously mentioned, two binary models are overlapping if both have the same functional form (both probit or both logit), with some common explanatory variables and some specific variables. In this article, we distinguished two cases: i) at least one of the competing models is well-specified, and ii) neither of them is correctly specified. This last case is an important aspect of our work because it is not commonly considered in empirical works.

From a theoretical point of view, we have classified the competing models as equivalent or non-equivalent. Once this classification had been carried out, the task was to define the requirement that a given criterion must satisfy to be considered as adequate. Specifically, if two models are equivalent, the probability limits of a given criterion must be the same in both models. However, if one of them is better, its corresponding limit must be lower than that of the other model. The theoretical analysis carried out has confirmed that all the criteria performed well in every situation. Only $C_2$ did not, sometimes, behave well in a specific alternative.

These theoretical results have been corroborated by a Monte Carlo experiment. The most complicated situation to simulate was, as we expected, when neither of the two models were well-specified. This situation can lead to three possibilities: equivalent models with identical densities, equivalent models with non-identical densities, and non-equivalent models.

In order to develop this part of the Monte Carlo exercise, we had to generate 132 different DGPs, leading to 132 different experiments. Each of the experiments corresponded to one of the three theoretical relationships mentioned above. To establish the specific relationship, we have defined three indicators:

(a)

The average of the absolute differences between the expected log-likelihoods (at the pseudo-trues) of both models. We have denoted it as absel.

(b)

The number of observations in the sample where the expected log-likelihood in model M1 is larger than in M2. Note that we have assumed that M1 is the closest to the DGP. This indicator is called AM1.

(c)

The average of the absolute differences between the density functions (at the pseudo-trues) of both models. We have denoted it as absdifden.

The general conclusion is that the three criteria behaved well for overlapping binary models: when neither of the two competing models was well-specified, the criteria tended to choose the best of them, that is, the closest to the DGP. In the most commonly studied case, where at least one of the competing models was correct, our conclusion was that the criteria also performed well, as we expected. Furthermore, when both models were correct, the criteria tended to choose the most parsimonious model.

It is important to note that these criteria are also used when we compared an extensive set of models, correctly specified or misspecified. The criteria AIC, SBIC and $C_2$ allow us to order them, being in the first places those correctly specified, which will be equivalent to each other. Among them, the first one (the selected model) will be the most parsimonious if we use the SBIC criterion. The misspecified models will be at the bottom of the ranking.

This paper has been focused on the restricted framework of overlapping binary models. In order to complete this analysis, a future work should study the behaviour of AIC, SBIC and $C_2$ in the non-nested framework. Moreover, the wider context of multinomial dependent variables could be the aim of future research. Given that the MLE procedure was also applied to estimate these models, the formal expression of the IC would be quite straight, while $C_2$ would require a deeper analysis.