Model Selection Criteria Using Divergences

In this note we introduce some divergence-based model selection criteria. These criteria are defined by estimators of the expected overall discrepancy between the true unknown model and the candidate model, using dual representations of divergences and associated minimum divergence estimators. It is shown that the proposed criteria are asymptotically unbiased. The influence functions of these criteria are also derived and some comments on robustness are provided.


Introduction
The minimum divergence approach is a useful technique in statistical inference.In recent years, the literature dedicated to the divergence-based statistical methods has grown substantially and the monographs of Pardo [1] and Basu et al. [2] are important references that present developments and applications in this field of research.Minimum divergence estimators and related methods have received considerable attention in statistical inference because of their ability to reconcile efficiency and robustness.Among others, Beran [3], Tamura and Boos [4], Simpson [5,6] and Toma [7] proposed families of parametric estimators minimizing the Hellinger distance between a nonparametric estimator of the observations density and the model.They showed that those estimators are both asymptotically efficient and robust.Generalizing earlier work based on the Hellinger distance, Lindsay [8] and Basu and Lindsay [9] have investigated minimum divergence estimators, for both discrete and continuous models.Some families of estimators based on approximate divergence criteria have also been considered; see Basu et al. [10].Broniatowski and Keziou [11] have introduced a minimum divergence estimation method based on a dual representation of the divergence between probability measures.Their estimators, called minimum dual divergence estimators, are defined in a unified way for both continuous and discrete models.They do not require any prior smoothing and include the classical maximum likelihood estimators as a benchmark.Robustness properties of these estimators have been studied in [12,13].
In this paper we apply estimators of divergences in dual form and corresponding minimum dual divergence estimators, as presented by Broniatowski and Keziou [11], in the context of model selection.
Model selection is a method for selecting the best model among candidate models.A model selection criterion can be considered as an approximately unbiased estimator of the expected overall discrepancy, a nonnegative quantity that measures the distance between the true unknown model and a fitted approximating model.If the value of the criterion is small, then the approximated candidate model can be chosen.
Many model selection criteria have been proposed so far.Classical model selection criteria using least square error and log-likelihood include the C p -criterion, cross-validation (CV), the Akaike information criterion (AIC) based on the well-known Kullback-Leibler divergence, Bayesian information criterion (BIC), a general class of criteria that also estimates the Kullback-Leibler divergence (GIC).These criteria have been proposed by Mallows [14], Stone [15], Akaike [16], Schwarz [17] and Konishi and Kitagawa [18], respectively.Robust versions of classical model selection criteria, which are not strongly affected by outliers, have been firstly proposed by Ronchetti [19], Ronchetti and Staudte [20].Other references on this topic can be found in Maronna et al. [21].Among the recent proposals for model selection we recall the criteria presented by Karagrigoriou et al. [22], the divergence information criteria (DIC) introduced by Mattheou et al. [23].The DIC criteria use the density power divergences introduced by Basu et al. [10].
In the present paper, we apply the same methodology used for AIC, and also for DIC, to a general class of divergences including the Cressie-Read divergences [24] in order to obtain model selection criteria.These criteria also use dual forms of the divergences and minimum dual divergence estimators.We show that the criteria are asymptotically unbiased and compute the corresponding influence functions.
The paper is organized as follows.In Section 2 we recall the duality formula for divergences, as well as the definitions of associated dual divergence estimators and minimum dual divergence estimators, together with their asymptotic properties, all these being necessary in the next section where we define new criteria for model selection.In Section 3, we apply the same methodology used for AIC to the divergences in dual form in order to develop criteria for model selection.We define criteria based on estimators of the expected overall discrepancy and prove their asymptotic unbiasedness.The influence functions of the proposed criteria are also derived.In Section 4 we present some conclusions.

Examples of Divergences
Let ϕ be a non-negative convex function defined from (0, ∞) onto [0, ∞] and satisfying ϕ(1) = 0. Also extend ϕ at 0 defining ϕ(0) = lim x↓0 ϕ(x).Let (X , B) be a measurable space and P be a probability measure (p.m.) defined on (X , B).Following Rüschendorf [25], for any p.m. Q absolutely continuous (a.c.) w.r.t.P , the divergence between Q and P is defined by When Q is not a.c.w.r.t.P, we set D(Q, P ) = ∞.We refer to Liese and Vajda [26] for an overview on the origin of the concept of divergence in statistics.
A commonly used family of divergences is the so-called "power divergences" or Cressie-Read divergences.This family is defined by the class of functions for γ ∈ R \ {0, 1} and ϕ 0 (x) : the modified Kullback-Leibler (KL m ) to ϕ 0 , the χ 2 divergence to ϕ 2 , the modified χ 2 divergence (χ 2 m ) to ϕ −1 and the Hellinger distance to ϕ 1/2 .We refer to [11] for the modified versions of χ 2 and KL divergences.

Dual Form of a Divergence and Minimum Divergence Estimators
Let {F θ , θ ∈ Θ} be an identifiable parametric model, where Θ is a subset of R p .We assume that for any θ ∈ Θ, F θ has density f θ with respect to some dominating σ-finite measure λ.Consider the problem of estimating the unknown true value of the parameter θ 0 on the basis of an i.i.d.sample X 1 , . . ., X n with the law F θ 0 .
In the following, D(f θ , f θ 0 ) denotes the divergence between f θ and f θ 0 , namely Using a Fenchel duality technique, Broniatowski and Keziou [11] have proved a dual representation of divergences.The main interest on this duality formula is that it leads to a wide variety of estimators, by a plug-in method of the empirical measure evaluated to the data set, without making use of any grouping, nor smoothing.
We consider divergences, defined through differentiable functions ϕ, that we assume to satisfy (C.0)There exists Condition (C.0) holds for all power divergences, including KL and KL m divergences.Assuming that D(f θ , f θ 0 ) is finite and that the function ϕ satisfies the condition (C.0), the dual representation holds where φ is the notation for the derivative of ϕ, the supremum in Equation ( 5) being uniquely attained in α = θ 0 , independently on θ.
We mention that the dual representation Equation ( 5) of divergences has been obtained independently by Liese and Vajda [30].
Naturally, for fixed θ, an estimator of the divergence D(f θ , f θ 0 ) is obtained by replacing Equation ( 5) by its sample analogue.This estimator is exactly the supremum being attained for Formula ( 8) defines a class of estimators of the parameter θ 0 called dual divergence estimators.Further, since and since the infimum in the above display is unique, a natural definition of estimators of the parameter θ 0 , called minimum dual divergence estimators, is provided by For more details on the dual representation of divergences and associated minimum dual divergence estimators, we refer to Broniatowski and Keziou [11].

Asymptotic Properties
Broniatowski and Keziou [11] have proved both the weak and the strong consistency, as well as the asymptotic normality for the classes of estimators α(θ), α( θ) and θ.Here, we shortly recall those asymptotic results that will be used in the next sections.The following conditions are considered.(a) for any positive ε, there exists some positive η such that for any α ∈ Θ with α − θ 0 > ε and for all θ ∈ Θ it holds that m(α, θ, x)f θ 0 (x)dx < m(θ 0 , θ, x)f θ 0 (x)dx − η.
(C.4)There exists a neighborhood N θ 0 of θ 0 such that the first and the second order partial derivatives with respect to α and θ of φ f θ (x) fα(x) f θ (x) are dominated on N θ 0 × N θ 0 by some λ-integrable functions.The third order partial derivatives with respect to α and θ of m(α, θ, x) are dominated on N θ 0 × N θ 0 by some P θ 0 -integrable functions (where P θ 0 is the probability measure corresponding to the law F θ 0 ).

Model Selection Criteria
In this section, we apply the same methodology used for AIC to the divergences in dual form in order to develop model selection criteria.Consider a random sample X 1 , . . ., X n from the distribution with density g (the true model) and a candidate model f θ from a parametric family of models (f θ ) indexed by an unknown parameter θ ∈ Θ, where Θ is a subset of R p .We use divergences satisfying (C.0) and denote for simplicity the divergence D(f θ , g) between f θ and the true density g by W θ .

The Expected Overall Discrepancy
The target theoretical quantity that will be approximated by an asymptotically unbiased estimator is given by where θ is a minimum dual divergence estimator defined by Equation (10).The same divergence is used for both W θ and θ.The quantity E[W θ ] can be viewed as the average distance between g and (f θ ) and it is called the expected overall discrepancy between g and (f θ ).
The next Lemma gives the gradient vector and the Hessian matrix of W θ and is useful for evaluating the expected overall discrepancy E[W θ ] through Taylor expansion.We denote by ḟθ and fθ the first and the second order derivative of f θ with respect to θ, respectively.We assume the following conditions allowing derivation under the integral sign.
(C.6)There exists a neighborhood N θ of θ such that (C.7)There exists a neighborhood N θ of θ such that Lemma 1. Assume that conditions (C.6) and (C.7) hold.Then, the gradient vector ∂ ∂θ W θ of W θ is given by When the true model g belongs to the parametric model (f θ ), hence g = f θ 0 , the gradient vector and the Hessian matrix of W θ evaluated in θ = θ 0 simplify to The hypothesis that the true model g belongs to the parametric family (f θ ) is the assumption made by Akaike [16].Although this assumption is questionable in practice, it is useful because it provides the basis for the evaluation of the expected overall discrepancy (see also [23]).
Proposition 2. When the true model g belongs to the parametric model (f θ ), assuming that conditions (C.6) and (C.7) are fulfilled for g = f θ 0 and θ = θ 0 , the expected overall discrepancy is given by where R n = o( θ − θ 0 2 ) and θ 0 is the true value of the parameter.
Proof.By applying a Taylor expansion to W θ around the true parameter θ 0 and taking θ = θ, on the basis of Equations ( 22) and ( 23), we obtain Then Equation ( 24) is proved.

Estimation of the Expected Overall Discrepancy
In this section we construct an asymptotically unbiased estimator of the expected overall discrepancy, under the hypothesis that the true model g belongs to the parametric family (f θ ).
For a given θ ∈ Θ, a natural estimator of W θ is using the sample analogue of the dual representation of the divergence.
The following conditions allow derivation under the integral sign for the integral term of Q θ .(C.8)There exists a neighborhood N θ of θ such that (C.9)There exists a neighborhood N θ of θ such that Lemma 2. Under (C.8) and (C.9), the gradient vector and the Hessian matrix of Q θ are Proof.Since derivation yields Note that, by its very definition, α(θ) is a solution of the equation taken with respect to α, therefore On the other hand, Proposition 3.Under conditions (C.1)-(C.3)and (C.8)-(C.9)and assuming that the integrals A Taylor expansion of Using the fact that ∂ 2 θ m(θ 0 , θ 0 , x) f θ 0 (x)dx is finite, the weak law of large numbers leads to Then, since ( α( θ) − θ 0 ) = o P (1) and ( θ − θ 0 ) = o P (1), and taking into account that ∂ 2 θ∂α m(θ 0 , θ 0 , x) f θ 0 (x)dx and ∂ 3 θ m(θ 0 , θ 0 , x) f θ 0 (x)dx are finite, we deduce that Thus we obtain Equation (39).

Influence Functions
In the following, we compute the influence function of the statistics Q θ .As it is known, the influence function is a useful tool for describing the robustness of an estimator.Recall that a map T defined on a set of distribution functions and parameter space valued is a statistical functional corresponding to an estimator θ of the parameter θ, if θ = T (F n ), where F n is the empirical distribution function associated to the sample.The influence function of T at F θ is defined by where F εx := (1 − ε)F θ + εδ x , ε > 0, δ x being the Dirac measure putting all mass at x. Whenever the influence function is bounded with respect to x, the corresponding estimator is called robust (see [31]).Since the statistical functional corresponding to Q θ , which we denote by U (•), is defined by where T θ (F ) is the statistical functional associated to the estimator α(θ) and V (F ) is the statistical functional associated to the estimator θ.
On the other hand, according to the results presented in [12], the influence function of the minimum dual divergence estimator is IF(x; V, F θ 0 ) = I(θ 0 ) −1 ḟθ 0 (x) Consequently, we obtain Equation (60).
Note that, for Cressie-Read divergences, it holds IF(x; U, F θ 0 ) = ḟθ 0 (x) irrespective of the used divergence, since φγ (1) = 1, for any γ.Generally, IF(x; U, F θ 0 ) is not bounded, therefore the robustness of the statistics Q θ , as measured by the influence function, does not hold.

Conclusions
The dual representation of divergences and corresponding minimum dual divergence estimators are useful tools in statistical inference.The presented theoretical results show that, in the context of model selection, these tools provide asymptotically unbiased criteria.These criteria are not robust in the sense of the bounded influence function, but this fact does not exclude the stability of the criteria with respect to other robustness measures.The computation of Q θ could lead to serious difficulties, for example when considering various regression models to choose from.Such difficulties are implied by the double optimization in the criterion.Therefore, from the computation point of view, some other existing model selection criteria could be preferred.On the other hand, some performant computation techniques, involving such a double optimization, could arrive in the favor of using these new criteria also.These problems represent the topic of future research.