## 1. Introduction

This paper develops Bayesian analogs of minimum Hellinger distance methods. In particular, we aim to produce methods that enable a Bayesian analysis to be both robust to unusual values in the data and to retain their asymptotic precision when a proposed parametric model is correct.

All statistical models include assumptions which may or may not be true of the mechanisms producing a given data set. Robustness is a desired property in which a statistical procedure is relatively insensitive to deviations from these assumptions. For frequentist inference, concerns are largely associated with distributional robustness: the shape of the true underlying distribution deviates slightly from the assumed model. Usually, this deviation represents the situation where there are some outliers in the observed data set; see [

1] for example. For Bayesian procedures, the deviations may come from the model, prior distribution, or utility function, or some combination thereof. Much of the literature on Bayesian robustness has been concerned with the prior distribution or utility function. By contrast, the focus of this paper is robustness with respect to outliers in a Bayesian context, a relatively understudied form of robustness for Bayesian models. For example, we know that Bayesian models with heavy tailed data distributions are robust with respect to outliers for the case of one single location parameter estimated by many observations. However, as a consequence of the Crámer–Rao lower bound and the efficiency of the MLE, modifying likelihoods to account for outliers will usually result in a loss of precision in parameter estimates when they are not necessary. The methods we propose, and the study of their robustness properties, will provide an alternative means of making any i.i.d. data distribution robust to outliers that do not lose efficiency when no outliers are present. We speculate that they can be extended beyond i.i.d. data as in [

2], but we do not pursue this here.

Suppose we are given the task of estimating

${\theta}_{0}\in \mathsf{\Theta}$ from independent and identically distributed univariate random variables

${X}_{1},\dots ,{X}_{n}$, where we assume each

${X}_{i}$ has density

${f}_{{\theta}_{0}}\in \mathcal{F}=\{{f}_{\theta}:\theta \in \mathsf{\Theta}\}$. Within the frequentist literature, minimum Hellinger distance estimates proceed by first estimating a kernel density

${\widehat{g}}_{n}\left(x\right)$ and then choosing

$\theta $ to minimize the Hellinger distance

$h({f}_{\theta},{g}_{n})={[\int {\{{f}_{\theta}^{1/2}\left(x\right)-{\widehat{g}}_{n}^{1/2}\left(x\right)\}}^{2}dx]}^{1/2}$. The minimum Hellinger distance estimator was shown in [

3] to have the remarkable properties of being both robust to outliers and statistically efficient, in the sense of asymptotically attaining the information bound, when the data are generated from

${f}_{{\theta}_{0}}$. These methods have been generalized to a class of minimum disparity estimators, based on alternative measures of the difference between a kernel density estimate and a parametric model, which have been studied since then, e.g., [

4,

5,

6,

7,

8]. While some adaptive M-estimators can be shown to retain both robustness and efficiency, e.g., [

9], minimum disparity methods are the only generic methods we are aware of that retain both properties and can also be readily employed within a Bayesian context. In this paper, we only consider Hellinger distance in order to simplify the mathematical exposition; the extension to more general disparity methods can be made following similar developments to those in [

5,

7].

Recent methodology proposed in [

2] suggested the use of disparity-based methods within Bayesian inference via the construction of a “disparity likelihood” by replacing the likelihood function when calculating the Bayesian posterior distribution; they demonstrated that the resulting expected

a posteriori estimators retain the frequentist properties studied above. These methods first obtain kernel density estimates from data and then calculate the disparity between the estimated density function and the corresponding density functions in the parametric family.

In this paper, we propose the use of Bayesian non-parametric methods instead of the classical kernel methods in applying the minimum Hellinger distance method. One method we proposed is just to replace the kernel density estimate used in classical minimum Hellinger distance estimate by the Bayesian nonparametric expected a posteriori density, which we denote by MHB (minimum Hellinger distance method using a Bayesian nonparametric density estimate). The second method combines the minimum Hellinger distance estimate with the Bayesian nonparametric posterior to give a posterior distribution of the parameter of interest. This latter method is our main focus. We show that it is more robust than usual Bayesian methods and demonstrate that it retains asymptotic efficiency, hence the precision of the estimate is maintained. So far as we are aware, this is the first Bayesian method that can be applied generically and retain both robustness and (asymptotic) efficiency. We denote it by BHM (Bayesian inference using a minimum Hellinger distance).

To study the properties of the proposed new methods, we treat both MHB and BMH as special cases of semi-parametric models. The general form of a semi-parametric model has a natural parametrization

$(\theta ,\eta )\mapsto {P}_{\theta ,\eta}$, where

$\theta \in \mathsf{\Theta}$ is a Euclidean parameter and

$\eta \in H$ belongs to an infinite-dimensional set. For such models,

$\theta $ is the parameter of primary interest, while

$\eta $ is a nuisance parameter. Asymptotic properties of some of Bayesian semi-parametric models have been discussed in [

10]. Our disparity based methods involve parameters in Euclidean space and Hilbert space, with the former being of most interest. However, unlike many semi-parametric models in which

${P}_{\theta ,\eta}\in \mathcal{P}$ is specified jointly by

$\theta $ and

$\eta $, in our case, the finite dimensional parameter and the nonparametric density functions are parallel specifications of the data distribution. Therefore, standard methods to study asymptotic properties of semi-parametric models will not apply to the study of disparity-based methods. Nevertheless, considering the problem of estimating

$\psi \left(P\right)$ of some function

$\psi :\mathcal{P}\mapsto {\mathbb{R}}^{d}$, where

$\mathcal{P}$ is the space of the probability models

P, semi-parametric models and disparity-based methods can be unified into one framework.

The MHB and BMH methods are introduced in detail in

Section 2, where we also discuss some related concepts and results, such as tangent sets, information, consistency, and the specific nonparametric prior that we employ. In

Section 3, both MHB and BMH are shown to be efficient, in the sense that asymptotically the variance of the estimate achieves the lower bound of the Cramér–Rao theorem. For MHB, we show that asymptotic normality of the estimate holds, where the asymptotic variance is the inverse of the Fisher information. For BMH, we show that the Bernstein von Mises (BvM) theorem holds. The robustness property and further discussion of these two methods are given in

Section 4 and

Section 5, respectively. A broader discussion is given in

Section 6.

## 2. Minimum Hellinger Distance Estimates

Assume that random variables ${X}_{1},\dots ,{X}_{n}$ are independent and identically distributed (iid) with density belonging to a specified parametric family $\mathcal{F}=\{{f}_{\theta}:\theta \in \mathsf{\Theta}\}$, where all the ${f}_{\theta}$ in the family have the same support, denoted by $supp\left(f\right)$. For simplicity, we use ${\mathbb{X}}_{n}$ to denote the random variables ${X}_{1},\dots ,{X}_{n}$. More flexibly, we model ${\mathbb{X}}_{n}\sim {g}^{n}$, where g is a probability density function with respect to the Lebesgue measure on $supp\left(f\right)$. Let $\mathcal{G}$ denote the collection of all such probability density functions. If the parametric family contains the data-generating distribution, then $g={f}_{\theta}$ for some $\theta $. Formally, we can denote the probability model of the observations in the form of a semi-parametric model $(\theta ,g)\mapsto {P}_{\theta ,g}$. We aim at estimating $\theta $ and consider g as a nuisance parameter, which is typical of semi-parametric models.

Let

$\pi $ denote a prior on

$\mathcal{G}$, and for any measurable subset

$B\subset \mathcal{G}$, the posterior probability of

$g\in B$ given

${\mathbb{X}}_{n}$ is

Let

${g}_{n}^{*}=\int g\pi (dg\mid {\mathbb{X}}_{n})$ denote the Bayesian nonparametric expected

a posteriori estimate. Our first proposed method can be described formally as follows:

MHB: Minimum Hellinger distance estimator with Bayesian nonparametric density estimation:

This estimator replaces the kernel density estimate in the classical minimum Hellinger distance method introduced in [

3] by the posterior expectation of the density function.

For this method, we will view

${\widehat{\theta}}_{1}$ as the value at

${g}_{n}^{*}$ of a functional

$T:\mathcal{G}\mapsto \mathsf{\Theta}$, which is defined via

where

$\parallel \xb7\parallel $ denotes the

${L}_{2}$ metric. We can also write

${\widehat{\theta}}_{1}$ as

$T\left({g}_{n}^{*}\right)$.

In a more general form, what we estimate is the value $\psi \left(P\right)$ of some functional $\psi :\mathcal{P}\mapsto {\mathbb{R}}^{d}$, where the P stands for the common distribution from which data are generated, and $\mathcal{P}$ is the set of all possible values of P, which also denotes the corresponding probability model. In the setting of minimum Hellinger distance estimation, the model $\mathcal{P}$ is set as $\mathcal{F}\times \mathcal{G}$, P can be specified as ${P}_{\theta ,g}$, and $\psi \left(P\right)=\psi \left({P}_{\theta ,g}\right)=\theta $. For the methods we proposed in this paper, we will focus on the functional $T:\mathcal{G}\mapsto \mathsf{\Theta}$, for a given $\mathcal{F}$, as defined above. Note that the constraint associated with the family $\mathcal{F}$ is implicitly applied by T.

Using functional T, we can also propose a Bayesian method, which assigns nonparametric prior on the density space and gives inference on the unknown parameter $\theta $ of a parametric family as follows:

BMH: Bayesian inference with minimum Hellinger distance estimation:

A nonparametric prior

$\pi $ on the space

$\mathcal{G}$ and the observation

${\mathbb{X}}_{n}$ leads to the posterior distribution

$\pi (g\mid {\mathbb{X}}_{n})$, which can then be converted to the posterior distribution of the parameter

$\theta \in \mathsf{\Theta}$ through the functional

$T:\mathcal{G}\mapsto \mathsf{\Theta}$.

In the following subsections, we discuss properties associated with the functional

T as well as the consistency of MHB and BHM, and we provide a detailed example of the random histogram prior that we will employ and its properties that will be used for the discussion of efficiency in

Section 2.1.

#### 2.1. Tangent Space and Information

In this subsection, we obtain the efficient influence function of the functional T on the linear span of the tangent set on ${g}_{0}$ and show that the local asymptotic normality (LAN) expansion related to the norm of the efficient influence function attains the Caramér–Rao bound. These results play important roles in showing that BvM holds for the BMH method in the next section.

Estimating the parameter by $T\left(g\right)$ under the assumption $g\in \mathcal{G}$ uses less information than estimating this parameter for $g\in {\mathcal{G}}^{*}\subset \mathcal{G}$. Hence, the lower bound of the variance of $T\left(g\right)$ for $g\in \mathcal{G}$ should be at least the supremum of the lower bounds of all parametric sub-models ${\mathcal{G}}^{*}=\{{G}_{\lambda}:\lambda \in \mathsf{\Lambda}\}\subset \mathcal{G}$.

To use mathematical tools such as functional analysis to study the properties of the proposed methods, we introduce some notations and concepts below. Without loss of generality, we consider one-dimensional sub-models

${\mathcal{G}}^{*}$, which pass through the “true” distribution, denoted by

${G}_{0}$ with density function

${g}_{0}$. We say a sub-model indexed by

t,

$\{{g}_{t}:0<t<\u03f5\}\subset \mathcal{G}$, is differentiable in quadratic mean at

$t=0$ if we have that, for some measurable function

$q:supp\left({g}_{0}\right)\mapsto \mathbb{R}$,

where

${G}_{t}$ is the cumulative distribution function associated with

${g}_{t}$. Functions

$q\left(x\right)$s are known as the score functions associated with each sub-model. The collection of these score functions, which is called a tangent set of the model

$\mathcal{G}$ at

${g}_{0}$ and denoted by

${\dot{\mathcal{G}}}_{{g}_{0}}$, is induced by the collection of all sub-models that are differentiable at

${g}_{0}$.

We say that

T is differentiable at

${g}_{0}$ relative to a given tangent set

${\dot{\mathcal{G}}}_{{g}_{0}}$, if there exists a continuous linear map

${\dot{T}}_{{g}_{0}}:{L}_{2}\left({G}_{0}\right)\mapsto \mathbb{R}$ such that for every

$q\in {\dot{\mathcal{G}}}_{{g}_{0}}$ and a sub-model

$t\mapsto {g}_{t}$ with score function

q, there is

where

${L}^{2}\left({G}_{0}\right)=\{q:supp\left({g}_{0}\right)\mapsto \mathbb{R},\int {q}^{2}\left(x\right){g}_{0}\left(x\right)dx<\infty \}$. By the Riesz representation theorem for Hilbert spaces, the map

${\dot{T}}_{{g}_{0}}$ can always be written in the form of an inner product with a fixed vector-valued, measurable function

${\tilde{T}}_{{g}_{0}}:supp\left({g}_{0}\right)\mapsto \mathbb{R}$,

Let

${\tilde{T}}_{{g}_{0}}$ denote the unique function in

$\overline{lin}{\dot{\mathcal{G}}}_{{g}_{0}}$, the closure of the linear span of the tangent set. The function

${\tilde{T}}_{{g}_{0}}$ is the efficient influence function and can be found as the projection of any other “influence function” onto the closed linear span of the tangent set.

For a sub-model

$t\mapsto {g}_{t}$ whose score function is

q, the Fisher information about

t at 0 is

${G}_{0}{q}^{2}=\int {q}^{2}d{G}_{0}$. In this paper, we use the notation

$Fg$ to denote

$\int gdF$ for a general function

g and distribution

F. Therefore, the “optimal asymptotic variance” for estimating the functional

$t\mapsto T\left({g}_{t}\right)$, evaluated at

$t=0$, is greater than or equal to the Caramér–Rao bound

The supremum of the right-hand side (RHS) of the above expression over all elements of the tangent set is a lower bound for estimating

$T\left(g\right)$ given model

$\mathcal{G}$, if the true model is

${g}_{0}$. The supremum can be expressed in the norm of the efficient influence function

${\tilde{T}}_{{g}_{0}}$ by Lemma 25.19 in [

11]. The lemma and its proof is quite neat, and we reproduce it here for the completeness of the argument.

**Lemma** **1.** Suppose that the functional $T:\mathcal{G}\mapsto \mathbb{R}$ is differentiable at ${g}_{0}$ relative to the tangent set ${\dot{\mathcal{G}}}_{{g}_{0}}$. Then **Proof.** This is a consequence of the Cauchy–Schwarz inequality ${\left({G}_{0}{\tilde{T}}_{{g}_{0}}q\right)}^{2}\le {G}_{0}{\tilde{T}}_{{g}_{0}}^{2}{G}_{0}{q}^{2}$ and the fact that, by definition, the efficient influence function, ${\tilde{T}}_{{g}_{0}}$, is contained in the closure of $lin{\dot{\mathcal{G}}}_{{G}_{0}}$. □

Now we show that functional T is differentiable under some mild conditions and construct its efficient influence function in the following theorem.

**Theorem** **1.** For the functional T defined in Equation (2), and for $t\in \mathsf{\Theta}\subset \mathbb{R}$, let ${s}_{t}\left(x\right)$ denote ${f}_{\theta}^{1/2}\left(x\right)$ for $\theta =t$. We assume that there exist ${\dot{s}}_{t}\left(x\right)$ and ${\ddot{s}}_{t}\left(x\right)$ both in ${L}_{2}$, such that for α in a neighborhood of zero,where ${u}_{\alpha}$ and ${v}_{\alpha}$ converge to zero as $\alpha \to 0$. Assuming $T\left({g}_{0}\right)\in int\left(\mathsf{\Theta}\right)$, the efficient influence function of T iswhere ${a}_{t}$ converges to 0 as $t\to 0$. In particular, for ${g}_{0}={f}_{\theta}$, **Proof.** Let the

t-indexed sub-model be

where

$q\left(x\right)$ satisfies

$\int q\left(x\right){g}_{0}\left(x\right)dx=0$ and

$q\in {L}_{2}\left({g}_{0}\right)$. By direct calculation, we see that

q is the score function associated with such a sub-model at

$t=0$ in the sense of Equation (

4) and thus the collection of

q is the maximal tangent set.

By the definition of

T,

$T\left({g}_{0}\right)$ maximizes

$\int {s}_{t}\left(x\right){g}_{0}^{1/2}\left(x\right)dx$. From Equation (

6), we have that

Since

$T\left({g}_{0}\right)\in int\left(\mathsf{\Theta}\right)$, we have that

Similarly,

$\int {\dot{s}}_{T\left({g}_{t}\right)}\left(x\right){g}_{t}^{1/2}\left(x\right)dx=0$. Using Equation (7) to substitute

${\dot{s}}_{T\left({g}_{t}\right)}$, we have that

where

${v}_{t}\left(x\right)$ converge in

${L}_{2}$ to zero as

$t\to 0$ since

$T\left({g}_{t}\right)\to T\left({g}_{0}\right)$. Thus,

Since by the definition of

$\tilde{T}$, which requires

$\int {\tilde{T}}_{{g}_{0}}{g}_{0}\left(x\right)dx=0$, we have that

By the same argument we can show that, when

${g}_{0}={f}_{\theta}$, Equation (

9) holds. □

Some relatively accessible conditions under which Equations (

6) and (7) hold are given by Lemmas 1 and 2 in [

3]. We do not repeat them here.

Now we can expand

T at

${g}_{0}$ as

where

$\tilde{T}$ is given in Theorem 1 and

$\tilde{r}=0$.

#### 2.2. Consistency of MHB and BMH

Since

$T\left(g\right)$ may have more than one value, the notation

$T\left(g\right)$ is used to denote any arbitrary one of the possible values. In [

3], the existence, continuity in Hellinger distance, and uniqueness of functional

T are ensured under the following condition:

**A1** (i) $\mathsf{\Theta}$ is compact, (ii) ${\theta}_{1}\ne {\theta}_{2}$ implies ${f}_{{\theta}_{1}}\ne {f}_{{\theta}_{2}}$ on a set of positive Lebesgue measures, and (iii), for almost every x, ${f}_{\theta}\left(x\right)$ is continuous in $\theta $.

When a Bayesian nonparametric density estimator is used, we assume the posterior consistency:

**A2** For any given $\u03f5>0$, $\pi \{g:h(g,{f}_{{\theta}_{0}})>\u03f5\mid {\mathbb{X}}_{n}\}\to 0$ in probability.

Under Conditions A1 and A2, consistency holds for MHB and BMH.

**Theorem** **2.** Suppose that Conditions A1 and A2 hold, then

- 1.
$\parallel {g}_{n}^{*1/2}-{f}_{{\theta}_{0}}^{1/2}{\parallel}^{2}\to 0$ in probability, $T\left({g}_{n}^{*}\right)\to T\left({f}_{{\theta}_{0}}\right)$ in probability, and hence ${\widehat{\theta}}_{1}\to {\theta}_{0}$ in probability;

- 2.
For any given $\u03f5>0$, $\pi \left(\right|\theta -{\theta}_{0}|>\u03f5\mid {\mathbb{X}}_{n})\to 0$ in probability.

**Proof.** Part 1: To show that

$\parallel {g}_{n}^{*1/2}-{f}_{{\theta}_{0}}^{1/2}{\parallel}^{2}\to 0$ in probability, which is equivalent to showing that

$\int {\left(\int g\pi {(dg\mid {\mathbb{X}}_{n})}^{1/2}-{f}_{{\theta}_{0}}^{1/2}\right)}^{2}dx\to 0$ in probability, it is sufficient to show that

$\int \left|\int g\pi (dg\mid {\mathbb{X}}_{n})-{f}_{{\theta}_{0}}\right|dx\to 0$ in probability, since

${h}^{2}(f,g)\le {\parallel f-g\parallel}_{1}$. We have that

Note that the change of order of integration is due to Fubini’s theorem and the last inequality is due to

${\parallel f-g\parallel}_{1}\le \sqrt{2}h(f,g)$. Split the integral on the right-hand side of the above expression into two parts:

where

$\mathcal{A}=\{g:h(g,{f}_{{\theta}_{0}})\le \u03f5\}$ for any given

$\u03f5>0$. The first term is bounded by

$\u03f5$ by construction. By Condition A1, the posterior of measure of

${\mathcal{A}}^{c}$ to 0 in probability as

$n\to \infty $. Since Hellinger distance is bounded by 2, so does the second term above. This completes the proof for

$\parallel {g}_{n}^{*1/2}-{f}_{{\theta}_{0}}^{1/2}{\parallel}^{2}\to 0$ in probability.

To show

$T\left({g}_{n}^{*}\right)\to T\left({f}_{{\theta}_{0}}\right)$ and

${\widehat{\theta}}_{1}\to {\theta}_{0}$ in probability, we need that the functional

T is continuous and unique at

${f}_{{\theta}_{0}}$, which is proved by Theorem 1 in [

3] under Condition A1.

Part 2: By Condition A1 and Theorem 1 in [

3], the functional

T is continuous and unique at

${f}_{{\theta}_{0}}$. Hence, for any given

$\u03f5>0$, there exist

$\delta >0$ such that

$|T\left(g\right)-T\left({f}_{{\theta}_{0}}\right)|<\u03f5$ when

$h(g,{f}_{{\theta}_{0}})<\delta $. By Condition A2, we have that

$\pi (h(g,{f}_{{\theta}_{0}})<\delta )\to 1$, which implies that

$\pi \left(\right|\theta -{\theta}_{0}|<\u03f5)\to 1$ in probability. □

It should be noted that, if we change the

$\u03f5$ in Condition A2 to

${\u03f5}_{n}$, a sequence converging to 0, then we can apply the results for the concentration rate of the Bayesian nonparametric density estimation here. However, such an approach cannot lead to the general “efficiency” claim, no matter in the form of rate of concentration or asymptotic normality. There are two reasons for this. First, the rate of concentration for Bayesian nonparametric posterior is about

${n}^{-2/5}$ for a rather general situation and

${(logn)}^{a}\times {n}^{-1/2}$, where

$a>0$, for some special cases (see [

12,

13,

14]). This concentration rate is not sufficient in many situations to directly imply that the concentration of the corresponding parametric estimates achieves the lower bound of the variance given in the Cramér–Rao theorem. Second, the Hellinger distances between pairs of densities as functions of parameters vary among different parametric families. Therefore, obtaining the rate of concentration in parameters from the rate of convergence in the densities cannot be generally applied to different distribution families.

It should also be noted that, although

$\mathsf{\Theta}$ is required to be compact in Condition A1, Theorem 2 is useful for a

$\mathsf{\Theta}$ that is not compact, as long as the parametric family

${f}_{\theta}:\theta \in \mathsf{\Theta}$ can be re-parameterized where the space of new parameters can be embedded within a compact set. An example of re-parameterizing a general location-scale family with parameters

$\mu \in \mathbb{R}$ and

$\sigma \in {\mathbb{R}}^{+}$ to a family with parameters

${t}_{1}=ta{n}^{-1}\left(\mu \right)$ and

${t}_{2}=ta{n}^{-1}\left(\sigma \right)$, where

${\mathsf{\Theta}}_{({t}_{1},{t}_{2})}=(-\pi /2,\pi /2)\times (0,\pi /2)$ and

$\mathsf{\Theta}\subset \overline{\mathsf{\Theta}}=[-\pi /2,\pi /2]\times [0,\pi /2]$, is discussed in [

3], and the conclusions of Theorem 1 in [

3] is still valid for a location-scale family. Therefore, Theorem 2 remains valid for the same type of the families, whose parameter space may not be compact and for the same reasons; the compactness requirement stated in the theorem is mainly for mathematical simplicity.

#### 2.3. Prior on Density Functions

We introduce a random histogram as an example for priors used in Bayesian nonparametric density estimation. It can be seen as a simplified version of a Dirichlet process mixture (DPM) prior, which is commonly used in practice. Both DPM and random histogram are mixture densities. While DPM uses a Dirichlet process to model the weights within an infinite mixture of kernels, the random histogram prior only has a finite number of components. Another difference is that, although we specify the form of the kernel function for DPM, the kernel function could be any density function in general, while the random histogram uses only the uniform density as its mixing kernel. Nevertheless, the limit on the finite number of the mixing components is not that important in practice, since the Dirichlet process will always be truncated in computation. In the next section, we will verify that the random histogram satisfies the conditions that are needed for our proposed methods to be efficient. On the other hand, although we believe that DPM should also lead to efficiency, the authors are unaware of the theoretical results or tools required to prove it. This is mostly due to the flexibility of DPM, which in turn significantly increases the mathematical complexity of the analysis.

For any $k\in \mathbb{N}$, denote the set of all regular k bin histograms on $[0,1]$ by ${\mathcal{H}}_{k}=\{f\in {L}^{2}\left([0,1]\right):m\left(x\right)={\sum}_{j=1}^{k}{f}_{j}1\phantom{\rule{-0.166667em}{0ex}}{\mathrm{l}}_{{I}_{j}}\left(x\right),{f}_{j}\in \mathbb{R},j=1,\dots ,k\},$ where ${I}_{j}=[(j-1)/k,j/k)$. Denote the unit simplex in ${\mathbb{R}}^{k}$ by ${\mathcal{S}}_{k}=\{\omega \in {[0,1]}^{k}:{\sum}_{j=1}^{k}{\omega}_{j}=1\}$. The subset of ${\mathcal{H}}_{k}$, ${\mathcal{H}}_{k}^{1}=\{f\in {L}^{2}\left(\mathbb{R}\right),f\left(x\right)={f}_{\omega ,k}=k\xb7{\sum}_{j=1}^{k}{\omega}_{j}1\phantom{\rule{-0.166667em}{0ex}}{\mathrm{l}}_{{I}_{j}}\left(x\right),({\omega}_{1},\dots ,{\omega}_{k})\in {\mathcal{S}}_{k}\}$, denotes the collection of densities on $[0,1]$ in the form of a histogram.

The set ${\mathcal{H}}_{k}$ is a closed subset of ${L}_{2}[0,1]$. For any function $f\in {L}_{2}[0,1]$, denote its projection in the ${L}_{2}$ sense on ${\mathcal{H}}_{k}$ by ${f}_{\left[k\right]}$, where ${f}_{\left[k\right]}=k{\sum}_{j=1}^{k}1\phantom{\rule{-0.166667em}{0ex}}{\mathrm{l}}_{{I}_{j}}{\int}_{{I}_{j}}f$.

We assign priors on

${\mathcal{H}}_{k}^{1}$ via

k and

$({\omega}_{1},\dots ,{\omega}_{k})$ for each

k. A degenerate case is to let

$k={K}_{n}=o\left(n\right)$. Otherwise, let

${p}_{k}$ be a distribution on positive integers, where

for all

k large enough and some

$0<{b}_{1}<{b}_{2}<\infty $. For example, Condition (

13) is satisfied by the Poisson distribution, which is commonly used in Bayesian nonparametric models.

Conditionally on

k, we consider a Dirichlet prior on

$\omega =\{{\omega}_{1},\dots ,{\omega}_{k}\}$:

for some fixed constants

$a,{c}_{1},{c}_{2}>0$ and any

$1\le j\le k$. For posterior consistency, we need the following condition:

where

${\mathcal{K}}_{n}\subset \{1,2,\dots ,\lfloor n/{(logn)}^{2}\rfloor \}$.

The consistency result of this prior is given by Proposition 1 in the supplement to [

15]. For

$n\ge 2,k\ge 1,M>0$, let

where

${\u03f5}_{n,k}^{2}=klogn/n$ denote a neighborhood of

${g}_{0,\left[k\right]}$, and we have that

(a) there exist

c,

$M>0$ such that

(b) Suppose

${g}_{0}\in {\mathcal{C}}^{\beta}$ with

$0<\beta \le 1$, if

${k}_{n}\left(\beta \right)={(nlogn)}^{1/(2\beta +1)}$ and

${\u03f5}_{n}\left(\beta \right)={k}_{n}{\left(\beta \right)}^{-\beta}$, then, for

${k}_{1}$ and a sufficiently large

M,

where

${\mathcal{C}}^{\beta}$ denotes the class of

$\beta $-Hölder functions on

$[0,1]$.

This means that the posterior of the density function concentrates around the projection

${g}_{0\left[k\right]}$ of

${g}_{0}$ and around

${g}_{0}$ itself in terms of the Hellinger distance. We can easily conclude that

$\pi ({\mathcal{K}}_{n}\mid {\mathbb{X}}_{n})=1+o\left(1\right)$ from Equation (

18) for

${g}_{0}\in {\mathcal{C}}^{\beta}$.

It should be noted that, although the priors we defined above are on the densities on

$[0,1]$, this is for mathematical simplicity, which could easily be extended to the space of probability densities on any given compact set. Further, transformations of

${\mathbb{X}}_{n}$, similar to those discussed at the end of

Section 2.2, can extend the analysis to the real line (refer to [

3,

16] for more example and details).

## 4. Robustness Properties

In frequentist analysis, robustness is usually measured by the influence function and breakdown point of estimators. These have been used to study robustness in minimum Hellinger distance estimators in [

3] and in more general minimum disparity estimators in [

2,

7].

In Bayesian inference, robustness is labeled “outlier rejection” and is studied under the framework of the “theory of conflict resolution”. There is a large literature on this topic, e.g., [

20,

21,

22]. While the results of [

22] are only about symmetric distributions, [

23] provides corresponding results covering a wider class of distributions with tails in the general exponential power family. These results provide a complete theory for the case of many observations and a single location parameter.

We examine the behavior of the methods MHB and BMH under a mixture model for gross errors. Let ${\delta}_{z}$ denote the uniform density of the interval $(z-\u03f5,z+\u03f5)$, where $\u03f5>0$ is small, and let ${f}_{\theta ,\alpha ,z}=(1-\alpha ){f}_{\theta}+\alpha {\delta}_{z}$, where $\theta \in \mathsf{\Theta}$ and $\alpha \in [0,1)]$ and z is a real number. The density ${f}_{\theta ,\alpha ,z}$ models a situation, where $100(1-\alpha )\%$ observations are distributed from ${f}_{\theta}$, and $100\alpha \%$ of the observations are the gross errors located near z.

**Theorem** **5.** For every $\alpha \in (0,1)$ and every $\theta \in \mathsf{\Theta}$, denote the mixture model for gross errors by ${f}_{\theta ,\alpha ,z}$. We then have that $li{m}_{z\to \infty}li{m}_{n\to \infty}T\left({g}_{n}^{*}\right)=\theta $, under the assumptions of Theorem 3 and that, for the BMH method, $\pi (T\left(g\right)\mid {\mathbb{X}}_{n})\to \varphi (\theta ,\parallel {\tilde{T}}_{{f}_{\theta ,\alpha ,z}}{\parallel}_{L}^{2})$ in the distribution as $n\to \infty $ and $z\to \infty $, where ϕ denotes the probability function of the normal distribution, when conditions in Theorem 4 are satisfied.

**Proof.** By Theorem 7 in [

3], for functional

T, as we defined and under the conditions in this theorem, we have that

We also have that, for MHB, under conditions of Theorem 3,

$li{m}_{n\to \infty}T\left({g}_{n}^{*}\right)\to T\left({f}_{\theta ,\alpha ,z}\right)$ in probability. Combining the two results,

$li{m}_{z\to \infty}li{m}_{n\to \infty}T\left({g}_{n}^{*}\right)=\theta $, when the data is generated from a contaminated distribution as

${f}_{\theta ,\alpha ,z}$. Similarly, by Theorem 4, we have that

$\pi (T\left(g\right)\mid {\mathbb{X}}_{n})\to \varphi (T\left({f}_{\theta ,\alpha ,a}\right),\parallel {\tilde{T}}_{{f}_{\theta ,\alpha ,z}}{\parallel}_{L}^{2})$ in distribution as

$n\to \infty $, and which converges to

$\varphi (\theta ,\parallel {\tilde{T}}_{{f}_{\theta ,\alpha ,z}}{\parallel}_{L}^{2})$, as

$z\to \infty $. □

## 5. Demonstration

We provide a demonstration of both BMH and MHB methods on two data sets: the classical Newcomb light speed data (see [

24,

25]), in which 2 out of 66 values are clearly negative oultiers, and a bivariate simulation containing 10% contamination in two asymmetric locations.

We have implemented the BMH and MHB methods using two Bayesian nonparametric priors:

We obtain posteriors for both using BUGS. We have elected to use BUGS here as opposed to the package `DPpackage` within R despite the latter’s rather efficient MCMC algorithms because our BMH method requires direct access to samples from the posterior distribution as opposed to the expected a posteriori estimate. The R package `distrEx` is then used to construct the sampled density functions and calculated the Hellinger distance between the sampled densities from the nonparametric model and the assumed normal distribution. The R package `optimx` is also used to find the minima of the Hellinger distances. The time cost of our methods are dominated by the optimization step rather than by the obtaining of samples from the posterior density.

We first apply BMH and MHB on the Simon Newcomb’s measurements to measure the speed of light. The data contains 66 observations. For this example, we specify the parameters and hyper-parameters of the DPM as

$\alpha =1$,

${m}_{2}=0,{s}_{2}=1000$,

${\tau}_{1}=1,{\tau}_{2}=100$, and

${\nu}_{2}=2,{\psi}_{2}=1$. We plot the data and a bivariate contour of the BMH posterior for both the mean and variance of the assumed normal in

Figure 1, where, despite outliers, the BvM result is readily apparent.

Table 1 summarizes these estimates. We report the estimated mean and variance with and without the obvious outliers as well as the same quantities estimated using both MHB and BMH methods with the last of these being the expected

a posteriori estimates. Quantities in parentheses given the “natural” standard error for each quantity: likelihood estimates correspond to standard normal theory—dividing the estimated standard error by

$\sqrt{n}$, and BMH standard errors are obtained from the posterior distribution. For MHB, we used a bootstrap and note that, while the computational cost involved in estimating MHB is significantly lower than BMH when obtaining a point estimate, the standard errors require and MCMC chain for each bootstrap, significantly raising the cost of obtaining these estimates. We observe that both prior specifications result in parameter estimates that are identical to two decimal places and very close to those obtained after removing outliers.

To examine the practical implementation of methods that go beyond our theoretical results, we applied these methods to a simulated two-dimensional data set of 100 data points generated from a standard normal with two contamination distributions. Specifically, our data distribution comes from

where exactly five points were generated from each of the second-two Gaussians. Our DP prior used the same hyper-parameters as above with the exception that

${\mathsf{\Psi}}_{1}$ was obtained from the empirical variance of the (contaminated) data, and

$({m}_{2},{S}_{2})$ were extended to their 2-dimensional form as

$\left({(0,0)}^{T},diag(1000,1000)\right)$.

Figure 2 plots these data along with the posterior for the two means.

Figure 3 provides posterior distributions for the components of the variance matrix.

Table 2 presents estimation results for the full data and those with the contaminating distributions removed as well as from the BMH method. Here we again observe that BMH yields results that are very close to those obtained using the uncontaminated data. There is some more irregularity in our estimates, particularly in

Figure 3, which we speculate is due to poor optimization. There is considerable scope to improve the numerics of minimum Hellinger distance methods more generally, but this is beyond the scope of this paper.

## 6. Discussion

This paper investigates the use of minimum Hellinger distance methods that replace kernel density estimates with Bayesian nonparametric models. We show that simply substituting the expected

a posteriori estimator will reproduce the efficiency and robustness properties of the classical disparity methods first derived in [

3]. Further, inducing a posterior distribution on

$\theta $ through the posterior for

g results in a Bernstein von Mises theorem and a distributional robustness result.

There are multiple potential extensions of this work. While we have focused on the specific pairing of Hellinger distance and random histogram priors, both of these can be generalized. A more general class of disparities was examined in [

7], and we believe the extension of our methods to this class are straightforward. More general Bayesian nonparametric priors are discussed in [

14], where the Dirichlet process prior has been particularly popular. Extensions to each of these priors will require separate analysis (e.g., [

26]). Extensions of disparities to regression models were examined in [

27] using a conditional density estimate, where equivalent Bayesian nonparametrics are not as well developed. Other modeling domains such as time series may require multivariate density estimates, resulting in further challenges.

Our results are a counterpoint to the Bayesian extensions of Hellinger distance methods in [

2] where the kernel density was retained for

${g}_{n}$ but a prior was given for

$\theta $ and the disparity treated as a log likelihood. Combining both these approaches represents a fully Bayesian implementation of disparity methods and is an important direction of future research.