Projection Pursuit Through ϕ-Divergence Minimisation

Abstract

In his 1985 article (“Projection pursuit”), Huber demonstrates the interest of his method to estimate a density from a data set in a simple given case. He considers the factorization of density through a Gaussian component and some residual density. Huber’s work is based on maximizing Kullback–Leibler divergence. Our proposal leads to a new algorithm. Furthermore, we will also consider the case when the density to be factorized is estimated from an i.i.d. sample. We will then propose a test for the factorization of the estimated density. Applications include a new test of fit pertaining to the elliptical copulas.

1. Outline of the Article

The objective of projection pursuit is to generate one or several projections providing as much information as possible about the structure of the data set regardless of its size:

Once a structure has been isolated, the corresponding data are transformed through a Gaussianization. Through a recursive approach, this process is iterated to find another structure in the remaining data, until no further structure can be evidenced in the data left at the end.

Friedman [1] and Huber [2] count among the first authors to have introduced this type of approaches for evidencing structures. They each describe, with many examples, how to evidence such a structure and consequently how to estimate the density of such data through two different methodologies each. Their work is based on maximizing Kullback–Leibler divergence.

For a very long time, the two methodologies exposed by each of the above authors were thought to be equivalent but Zhu [3] showed it was in fact not the case when the number of iterations in the algorithms exceeds the dimension of the space containing the data, i.e., in case of density estimation. In the present article, we will therefore only focus on Huber’s study while taking into account the Zhu remarks.

At present, let us briefly introduce Huber’s methodology. We will then expose our approach and objective.

1.1. Huber’s analytic approach

Let f be a density on ${\mathbb{R}}^d$. We define an instrumental density g with same mean and variance as f. Huber’s methodology requires us to start with performing the $K(f,g)=0$ test—with K being the Kullback–Leibler divergence. Should this test turn out to be positive, then $f=g$ and the algorithm stops. If the test were not to be verified, the first step of Huber’s algorithm amounts to defining a vector $a_1$ and a density $f^{(1)}$ by

$$a_1=arg\underset{a\in {\mathbb{R}}_{*}^d}{inf}K(f\frac{g_a}{f_a},g)\text{and}{f}^{(1)}=f\frac{g_{a_1}}{f_{a_1}}$$

(1.1)

where ${\mathbb{R}}_{*}^d$ is the set of non-null vectors of ${\mathbb{R}}^d$, where $f_a$ (resp. $g_a$) stands for the density of $a^{\top }X$ (resp. $a^{\top }Y$) when f (resp. g) is the density of X (resp. Y). More exactly, this results from the maximisation of $a\mapsto K(f_a,g_a)$ since $K(f,g)=K(f_a,g_a)+K(f\frac{g_a}{f_a},g)$ and it is assumed that $K(f,g)$ is finite. In a second step, Huber replaces f with $f^{(1)}$ and goes through the first step again.

By iterating this process, Huber thus obtains a sequence $(a_1,a_2,...)$ of vectors of ${\mathbb{R}}_{*}^d$ and a sequence of densities $f^{(i)}$.

$\mathcal{R}emark$ 1.1. Huber stops his algorithm when the Kullback–Leibler divergence equals zero or when his algorithm reaches the $d^{th}$ iteration, he then obtains an approximation of f from g:

When there exists an integer j such that $K(f^{(j)},g)=0$ with $j\le d$, he obtains $f^{(j)}=g$, i.e., $f=g{\Pi }_{i=1}^j\frac{f_{a_i}^{(i-1)}}{g_{a_i}}$ since by induction $f^{(j)}=f{\Pi }_{i=1}^j\frac{g_{a_i}}{f_{a_i}^{(i-1)}}$. Similarly, when, for all j, Huber gets $K(f^{(j)},g)>0$ with $j\le d$, he assumes $g=f^{(d)}$ in order to derive $f=g{\Pi }_{i=1}^d\frac{f_{a_i}^{(i-1)}}{g_{a_i}}$.

He can also stop his algorithm when the Kullback–Leibler divergence equals zero without the condition $j\le d$ is met. Therefore, since by induction we have $f^{(j)}=f{\Pi }_{i=1}^j\frac{g_{a_i}}{f_{a_i}^{(i-1)}}$ with $f^{(0)}=f$, we obtain $g=f{\Pi }_{i=1}^j\frac{g_{a_i}}{f_{a_i}^{(i-1)}}$. Consequently, we derive a representation of f as $f=g{\Pi }_{i=1}^j\frac{f_{a_i}^{(i-1)}}{g_{a_i}}.$

Finally, he obtains $K(f^{(0)},g)\ge K(f^{(1)},g)\ge .....\ge 0$ with $f^{(0)}=f$.

1.2. Huber’s synthetic approach

Keeping the notations of the above section, we start with performing the $K(f,g)=0$ test; should this test turn out to be positive, then $f=g$ and the algorithm stops, otherwise, the first step of his algorithm would consist in defining a vector $a_1$ and a density $g^{(1)}$ by

$$a_1=arg\underset{a\in {\mathbb{R}}_{*}^d}{inf}K(f,g\frac{f_a}{g_a})\text{and}{g}^{(1)}=g\frac{f_{a_1}}{g_{a_1}}$$

(1.2)

More exactly, this optimisation results from the maximisation of $a\mapsto K(f_a,g_a)$ since $K(f,g)=K(f_a,g_a)+K(f,g\frac{f_a}{g_a})$ and it is assumed that $K(f,g)$ is finite. In a second step, Huber replaces g with $g^{(1)}$ and goes through the first step again. By iterating this process, Huber thus obtains a sequence $(a_1,a_2,...)$ of vectors of ${\mathbb{R}}_{*}^d$ and a sequence of densities $g^{(i)}$.

$\mathcal{R}emark$ 1.2. First, in a similar manner to the analytic approach, this methodology enables us to approximate and even to represent f from g:

To obtain an approximation of f, Huber either stops his algorithm when the Kullback–Leibler divergence equals zero, i.e., $K(f,g^{(j)})=0$ implies $g^{(j)}=f$ with $j\le d$, or when his algorithm reaches the $d^{th}$ iteration, i.e., he approximates f with $g^{(d)}$.

To obtain a representation of f, Huber stops his algorithm when the Kullback–Leibler divergence equals zero, since $K(f,g^{(j)})=0$ implies $g^{(j)}=f$. Therefore, since by induction we have $g^{(j)}=g{\Pi }_{i=1}^j\frac{f_{a_i}}{g_{a_i}^{(i-1)}}$ with $g^{(0)}=g$, we then obtain $f=g{\Pi }_{i=1}^j\frac{f_{a_i}}{g_{a_i}^{(i-1)}}.$

Second, he gets $K(f,g^{(0)})\ge K(f,g^{(1)})\ge .....\ge 0$ with $g^{(0)}=g$.

1.3. Proposal

Let us first introduce the concept of $\varphi -$divergence.

Let ϕ be a strictly convex function defined by $\phi :\overline{{\mathbb{R}}^{+}}\to \overline{{\mathbb{R}}^{+}},$ and such that $\phi (1)=0$. We define a $\varphi -$divergence of P from Q—where P and Q are two probability distributions over a space Ω such that Q is absolutely continuous with respect to P—by

$$D_{\varphi }(Q,P)=\int \phi (\frac{dQ}{dP})dP$$

or $D_{\varphi }(q,p)=\int \phi (\frac{q(x)}{p(x)})p(x)dx$, if P and Q present p and q as density respectively.

Throughout this article, we will also assume that $\phi (0)<\infty$, that ${\phi }^{\prime }$ is continuous and that this divergence is greater than the $L^1$ distance—see also Appendix A.1 page 1604.

Now, let us introduce our algorithm.

We start with performing the $D_{\varphi }(g,f)=0$ test; should this test turn out to be positive, then $f=g$ and the algorithm stops, otherwise, the first step of our algorithm would consist in defining a vector $a_1$ and a density $g^{(1)}$ by

$$a_1=arg\underset{a\in {\mathbb{R}}_{*}^d}{inf}{D}_{\varphi }(g\frac{f_a}{g_a},f)\text{and}{g}^{(1)}=g\frac{f_{a_1}}{g_{a_1}}$$

(1.3)

Later on, we will prove that $a_1$ simultaneously optimises (1.1), (1.2) and (1.3).

In our second step, we will replace g with $g^{(1)}$, and we will repeat the first step.

And so on, by iterating this process, we will end up obtaining a sequence $(a_1,a_2,...)$ of vectors in ${\mathbb{R}}_{*}^d$ and a sequence of densities $g^{(i)}$.

We will thus prove that the underlying structures of f evidenced through this method are identical to the ones obtained through Huber’s method. We will also evidence the above structures, which will enable us to infer more information on f—see example below.

$\mathcal{R}emark$ 1.3. As in the previous algorithm, we first provide an approximate and even a representation of f from g: To obtain an approximation of f, we stop our algorithm when the divergence equals zero, i.e., $D_{\varphi }(g^{(j)},f)=0$ implies $g^{(j)}=f$ with $j\le d$, or when our algorithm reaches the $d^{th}$ iteration, i.e., we approximate f with $g^{(d)}$.

To obtain a representation of f, we stop our algorithm when the divergence equals zero. Therefore, since by induction we have $g^{(j)}=g{\Pi }_{i=1}^j\frac{f_{a_i}}{g_{a_i}^{(i-1)}}$ with $g^{(0)}=g$, we then obtain $f=g{\Pi }_{i=1}^j\frac{f_{a_i}}{g_{a_i}^{(i-1)}}.$

Second, we get $D_{\varphi }(g^{(0)},f)\ge D_{\varphi }(g^{(1)},f)\ge .....\ge 0$ with $g^{(0)}=g$.

Finally, the specific form of relationship (1.3) establishes that we deal with M-estimation. We can therefore state that our method is more robust than Huber’s—see Yohai [4], Toma [5] as well as Huber [6].

At present, let us study two examples:

$\mathcal{E}xample$ 1.1. Let f be a density defined on ${\mathbb{R}}^3$ by $f(x_1,x_2,x_3)=n(x_1,x_2)h(x_3)$, with n being a bi-dimensional Gaussian density, and h being a non-Gaussian density. Let us also consider g, a Gaussian density with same mean and variance as f.

Since $g(x_1,x_2/x_3)=n(x_1,x_2)$, we then have $D_{\varphi }(g\frac{f_3}{g_3},f)=D_{\varphi }(n.f_3,f)=D_{\varphi }(f,f)=0$ as $f_3=h$, i.e., the function $a\mapsto D_{\varphi }(g\frac{f_a}{g_a},f)$ reaches zero for $e_3={(0,0,1)}^{\prime }$—where $f_3$ and $g_3$ are the third marginal densities of f and g respectively.

We therefore obtain $g(x_1,x_2/x_3)=f(x_1,x_2/x_3)$.

$\mathcal{E}xample$ 1.2. Assuming that the φ-divergence is greater than the $L^2$ norm. Let us consider ${(X_n)}_{n\ge 0}$, the Markov chain with continuous state space E. Let f be the density of $(X_0,X_1)$ and let g be the normal density with same mean and variance as f.

Let us now assume that $D_{\varphi }(g^{(1)},f)=0$ with $g^{(1)}(x)=g(x)\frac{f_1}{g_1}$, i.e., let us assume that our algorithm stops for $a_1={(1,0)}^{\prime }$. Consequently, if $(Y_0,Y_1)$ is a random vector with g density, then the distribution law of $X_1$ given $X_0$ is Gaussian and is equal to the distribution law of $Y_1$ given $Y_0$.

And then, for any sequence $(A_i)$—where $A_i\subset E$—we have

$$\begin{array}{l}\mathbf{P}\left(X_{n+1}\in A_{n+1}\mid X_0\in A_0,X_1\in A_1,\dots ,X_{n-1}\in A_{n-1},X_n\in A_n\right)\\ =\mathbf{P}\left(X_{n+1}\in A_{n+1}\mid X_n\in A_n\right),basedontheverydefinitionofaMarkovchain,\\ =\mathbf{P}\left(X_1\in A_1\mid X_0\in A_0\right),throughtheMarkovproperty,\\ =\mathbf{P}\left(Y_1\in A_1\mid Y_0\in A_0\right),asaconsequenceoftheabovenullityofthe\varphi -divergence.\end{array}$$

To recapitulate our method, if $D_{\varphi }(g,f)=0$, we derive f from the relationship $f=g$; should a sequence ${(a_i)}_{i=1,...j}$, $j<d$, of vectors in ${\mathbb{R}}_{*}^d$ defining $g^{(j)}$ and such that $D_{\varphi }(g^{(j)},f)=0$ exist, then $f(./a_i^{\top }x,1\le i\le j)=g(./a_i^{\top }x,1\le i\le j)$, i.e., f coincides with g on the complement of the vector subspace generated by the family ${\{a_i\}}_{i=1,...,j}$—see also Section 2 for a more detailed explanation.

In this paper, after having clarified the choice of g, we will consider the statistical solution to the representation problem, assuming that f is unknown and $X_1$, $X_2$,... $X_m$ are i.i.d. with density f. We will provide asymptotic results pertaining to the family of optimizing vectors $a_{k,m}$—that we will define more precisely below—as m goes to infinity. Our results also prove that the empirical representation scheme converges towards the theoretical one. As an application, Section 3.4 permits a new test of fit pertaining to the copula of an unknown density f, Section 3.5 gives us an estimate of a density deconvoluted with a Gaussian component and Section 3.6 presents some applications to regression analysis. Finally, we will present simulations and an application to real datasets.

2. The Algorithm

2.1. The model

As explained by Friedman [1] and Diaconis [7], the choice of g depends on the family of distribution one wants to find in f. Until now, the choice has only been to use the class of Gaussian distributions. This can be extended to the class of elliptic distributions with almost all $\varphi -$divergences.

Elliptical laws 

The interest of this class lies in the fact that conditional densities with elliptical distributions are also elliptical—see Cambanis [8], Landsman [9]. This very property allows us to use this class in our algorithm.

Definition 2.1. 

X is said to abide by a multivariate elliptical distribution—noted $X\sim E_d(\mu ,\Sigma ,{\xi }_d)$—if X presents the following density, for any x in ${\mathbb{R}}^d$ :

$$f_X(x)=\frac{c_d}{{|\Sigma |}^{1/2}}{\xi }_d\left(\frac{1}{2}{(x-\mu )}^{\prime }{\Sigma }^{-1}(x-\mu )\right)$$

• with Σ, being a $d\times d$ positive-definite matrix and with μ, being a d-column vector,
• with ${\xi }_d$, being referred as the “density generator”,
• with $c_d$, being a normalisation constant, such that $c_d=\frac{\Gamma (d/2)}{{(2\pi )}^{d/2}}{\left({\int }_0^{\infty }{x}^{d/2-1}{\xi }_d(x)dx\right)}^{-1}$, with ${\int }_0^{\infty }{x}^{d/2-1}{\xi }_d(x)dx<\infty$.

Property 2.1. 

1/ For any $X\sim E_d(\mu ,\Sigma ,{\xi }_d)$, for any A, being an $m\times d$ matrix with rank $m\le d,$ and for any b, being an m-dimensional vector, we have $AX+b\sim E_m(A\mu +b,A\Sigma A^{\prime },{\xi }_m)$.

Therefore, any marginal density of multivariate elliptical distribution is elliptic, i.e., $X=(X_1,X_2,...,X_d)\sim E_d(\mu ,\Sigma ,{\xi }_d)\Rightarrow X_i\sim E_1({\mu }_i,{\sigma }_i^2,{\xi }_1),$ $f_{X_i}(x)=\frac{c_1}{{\sigma }_i}{\xi }_1\left(\frac{1}{2}{(\frac{x-{\mu }_i}{\sigma })}^2\right),$ $1\le i\le d$.

2/ Corollary 5 of Cambanis [8] states that conditional densities with elliptical distributions are also elliptic. Indeed, if $X={(X_1,X_2)}^{\prime }\sim E_d(\mu ,\Sigma ,{\xi }_d)$, with $X_1$ (resp. $X_2$) being a size $d_1<d$ (resp. $d_2<d$), then $X_1/(X_2=a)\sim E_{d_1}({\mu }^{\prime },{\Sigma }^{\prime },{\xi }_{d_1})$ with ${\mu }^{\prime }={\mu }_1+{\Sigma }_{12}{\Sigma }_{22}^{-1}(a-{\mu }_2)$ and ${\Sigma }^{\prime }={\Sigma }_{11}-{\Sigma }_{12}{\Sigma }_{22}^{-1}{\Sigma }_{21},$ with $\mu =({\mu }_1,{\mu }_2)$ and $\Sigma ={({\Sigma }_{ij})}_{1\le i,j\le 2}$.

$\mathcal{R}emark$ 2.1. 

Landsman [9] shows that multivariate Gaussian distributions derive from ${\xi }_d(x)=e^{-x}$. He also shows that if $X=(X_1,...,X_d)$ has an elliptical density such that its marginals verify $E(X_i)<\infty$ and $E(X_i^2)<\infty$ for $1\le i\le d,$ then μ is the mean of X and Σ is a multiple of the covariance matrix of X. Consequently, from now on, we will assume that we are in this case.

Definition 2.2. 

Let t be an elliptical density on ${\mathbb{R}}^k$ and let q be an elliptical density on ${\mathbb{R}}^{k^{\prime }}$. The elliptical densities t and q are said to belong to the same family—or class—of elliptical densities, if their generating densities are ${\xi }_k$ and ${\xi }_{k^{\prime }}$ respectively, which belong to a common given family of densities.

$\mathcal{E}xample$ 2.1. 

Consider two Gaussian densities $\mathcal{N}(0,1)$ and $\mathcal{N}((0,0),I{d}_2)$. They are said to belong to the same elliptical families as they both present $x\mapsto e^{-x}$ as generating density.

Choice of g 

Let us begin with studying the following case:

Let f be a density on ${\mathbb{R}}^d$. Let us assume there exists d non-null linearly independent vectors $a_j$, with $1\le j\le d,$ of ${\mathbb{R}}^d$, such that

$$f(x)=n(a_{j+1}^{\top }x,...,a_d^{\top }x)h(a_1^{\top }x,...,a_j^{\top }x)$$

(2.1)

with $j<d$, with n being an elliptical density on ${\mathbb{R}}^{d-j-1}$ and with h being a density on ${\mathbb{R}}^j$, which does not belong to the same family as n. Let $X=(X_1,...,X_d)$ be a vector presenting f as density.

Define g as an elliptical distribution with same mean and variance as f.

For simplicity, let us assume that the family ${\{a_j\}}_{1\le j\le d}$ is the canonical basis of ${\mathbb{R}}^d$:

The very definition of f implies that $(X_{j+1},...,X_d)$ is independent from $(X_1,...,X_j)$. Hence, the density of $(X_{j+1},...,X_d)$ given $(X_1,...,X_j)$ is n.

Let us assume that $D_{\varphi }(g^{(j)},f)=0,$ for some $j\le d$. We then get $\frac{f(x)}{f_{a_1}{f}_{a_2}...f_{a_j}}=\frac{g(x)}{g_{a_1}^{(1-1)}{g}_{a_2}^{(2-1)}...g_{a_j}^{(j-1)}}$, since, by induction, we have $g^{(j)}(x)=g(x)\frac{f_{a_1}}{g_{a_1}^{(1-1)}}\frac{f_{a_2}}{g_{a_2}^{(2-1)}}...\frac{f_{a_j}}{g_{a_j}^{(j-1)}}$.

Consequently, the fact that conditional densities with elliptical distributions are also elliptical enables us to infer that

$$n(a_{j+1}^{\top }x,.,a_d^{\top }x)=f(./a_i^{\top }x,1\le i\le j)=g(./a_i^{\top }x,1\le i\le j)$$

In other words, f coincides with g on the complement of the vector subspace generated by the family ${\{a_i\}}_{i=1,...,j}$.

Now, if the family ${\{a_j\}}_{1\le j\le d}$ is no longer the canonical basis of ${\mathbb{R}}^d$, then this family is again a basis of ${\mathbb{R}}^d$. Hence, Lemma D.1—page 1607—implies that

$$g(./a_1^{\top }x,...,a_j^{\top }x)=n(a_{j+1}^{\top }x,...,a_d^{\top }x)=f(./a_1^{\top }x,...,a_j^{\top }x)$$

(2.2)

which is equivalent to having $D_{\varphi }(g^{(j)},f)=0$—since by induction $g^{(j)}=g\frac{f_{a_1}}{g_{a_1}^{(1-1)}}\frac{f_{a_2}}{g_{a_2}^{(2-1)}}...\frac{f_{a_j}}{g_{a_j}^{(j-1)}}$.

The end of our algorithm implies that f coincides with g on the complement of the vector subspace generated by the family ${\{a_i\}}_{i=1,...,j}$. Therefore, the nullity of the $\varphi -$divergence provides us with information on the density structure.

In summary, the following proposition clarifies our choice of g which depends on the family of distribution one wants to find in f:

Proposition 2.1. 

With the above notations, $D_{\varphi }(g^{(j)},f)=0$ is equivalent to

$$g(./a_1^{\top }x,...,a_j^{\top }x)=f(./a_1^{\top }x,...,a_j^{\top }x)$$

More generally, the above proposition leads us to defining the co-support of f as the vector space generated from vectors $a_1,...,a_j$.

Definition 2.3. 

Let f be a density on ${\mathbb{R}}^d$. We define the co-vectors of f as the sequence of vectors $a_1,...,a_j$ which solves the problem $D_{\varphi }(g^{(j)},f)=0$ where g is an elliptical distribution with same mean and variance as f. We define the co-support of f as the vector space generated from vectors $a_1,...,a_j$.

$\mathcal{R}emark$ 2.2. 

Any $(a_i)$ family defining f as in (2.1), is an orthogonal basis of ${\mathbb{R}}^d$—see Lemma D.2

2.2. Stochastic outline of our algorithm

Let $X_1$, $X_2$,..,$X_m$ (resp. $Y_1$, $Y_2$,..,$Y_m$) be a sequence of m independent random vectors with same density f (resp. g). As customary in nonparametric $\varphi -$divergence optimizations, all estimates of f and $f_a$ as well as all uses of Monté Carlo’s methods are being performed using subsamples $X_1$, $X_2$,..,$X_n$ and $Y_1$, $Y_2$,..,$Y_n$—extracted respectively from $X_1$, $X_2$,..,$X_m$ and $Y_1$, $Y_2$,..,$Y_m$—since the estimates are bounded below by some positive deterministic sequence ${\theta }_m$—see Appendix B.

Let ${\mathbb{P}}_n$ be the empirical measure of the subsample $X_1$, $X_2$,.,$X_n$. Let $f_n$ (resp. $f_{a,n}$ for any a in ${\mathbb{R}}_{*}^d$) be the kernel estimate of f (resp. $f_a$), which is built from $X_1$, $X_2$,..,$X_n$ (resp. $a^{\top }{X}_1$, $a^{\top }{X}_2$,..,$a^{\top }{X}_n$).

As defined in Section 1.3, we introduce the following sequences ${(a_k)}_{k\ge 1}$ and ${(g^{(k)})}_{k\ge 1}$:

$$\begin{array}{ccc}& & •a_k\text{is}\text{a}\text{non}\text{null}\text{vector}\text{of}{\mathbb{R}}^d\text{such}\text{that}{a}_k=arg{min}_{a\in {\mathbb{R}}_{*}^d}{D}_{\varphi }(g^{(k-1)}\frac{f_a}{g_a^{(k-1)}},f)\hfill \end{array}$$

(2.3)

$$\begin{array}{ccc}& & •g^{(k)}\text{is}\text{the}\text{density}\text{such}\text{that}{g}^{(k)}=g^{(k-1)}\frac{f_{a_k}}{g_{a_k}^{(k-1)}}\text{with}{g}^{(0)}=g\hfill \end{array}$$

The stochastic setting up of the algorithm uses $f_n$ and $g_n^{(0)}=g$ instead of f and $g^{(0)}=g$—since g is known. Thus, at the first step, we build the vector ${\check{a}}_1$ which minimizes the $\varphi -$divergence between $f_n$ and $g\frac{f_{a,n}}{g_a}$ and which estimates $a_1$ :

Proposition B.1 page 1606 and Lemma D.3 page 1607 enable us to minimize the $\varphi -$divergence between $f_n$ and $g\frac{f_{a,n}}{g_a}$. Defining ${\check{a}}_1$ as the argument of this minimization, Proposition 3.3 page 1589 shows us that this vector tends to $a_1$.

Finally, we define the density ${\check{g}}_m^{(1)}$ as ${\check{g}}_m^{(1)}=g\frac{f_{{\check{a}}_1,m}}{g_{{\check{a}}_1}}$ which estimates $g^{(1)}$ through Theorem 3.1.

Now, from the second step and as defined in Section 1.3, the density $g^{(k-1)}$ is unknown. Consequently, once again, we have to truncate the samples:

All estimates of f and $f_a$ (resp. $g^{(1)}$ and $g_a^{(1)}$) are being performed using a subsample $X_1$, $X_2$,..,$X_n$ (resp. $Y_1^{(1)}$, $Y_2^{(1)}$,..,$Y_n^{(1)}$) extracted from $X_1$, $X_2$,..,$X_m$ (resp. $Y_1^{(1)}$, $Y_2^{(1)}$,..,$Y_m^{(1)}$—which is a sequence of m independent random vectors with same density $g^{(1)}$) such that the estimates are bounded below by some positive deterministic sequence ${\theta }_m$—see Appendix B.

Let ${\mathbb{P}}_n$ be the empirical measure of the subsample $X_1$, $X_2$,..,$X_n$. Let $f_n$ (resp. $g_n^{(1)}$, $f_{a,n}$, $g_{a,n}^{(1)}$ for any a in ${\mathbb{R}}_{*}^d$) be the kernel estimate of f (resp. $g^{(1)}$ and $f_a$ as well as $g_a^{(1)}$) which is built from $X_1$, $X_2$,..,$X_n$ (resp. $Y_1^{(1)}$, $Y_2^{(1)}$,..,$Y_n^{(1)}$ and $a^{\top }{X}_1$, $a^{\top }{X}_2$,..,$a^{\top }{X}_n$ as well as $a^{\top }{Y}_1^{(1)}$, $a^{\top }{Y}_2^{(1)}$,..,$a^{\top }{Y}_n^{(1)}$). The stochastic setting up of the algorithm uses $f_n$ and $g_n^{(1)}$ instead of f and $g^{(1)}$.

Thus, we build the vector ${\check{a}}_2$ which minimizes the $\varphi -$divergence between $f_n$ and $g_n^{(1)}\frac{f_{a,n}}{g_{a,n}^{(1)}}$—since $g^{(1)}$ and $g_a^{(1)}$ are unknown—and which estimates $a_2$.

Proposition B.1 page 1606 and Lemma D.3 page 1607 enable us to minimize the $\varphi -$divergence between $f_n$ and $g_n^{(1)}\frac{f_{a,n}}{g_{a,n}^{(1)}}$. Defining ${\check{a}}_2$ as the argument of this minimization, Proposition 3.3 page 1589 shows us that this vector tends to $a_2$ in n. Finally, we define the density ${\check{g}}_n^{(2)}$ as ${\check{g}}_n^{(2)}=g_n^{(1)}\frac{f_{{\check{a}}_2,n}}{g_{{\check{a}}_2,n}^{(1)}}$ which estimates $g^{(2)}$ through Theorem 3.1.

And so on, we will end up obtaining a sequence $({\check{a}}_1,{\check{a}}_2,...)$ of vectors in ${\mathbb{R}}_{*}^d$ estimating the co-vectors of f and a sequence of densities ${({\check{g}}_n^{(k)})}_k$ such that ${\check{g}}_n^{(k)}$ estimates $g^{(k)}$ through Theorem 3.1.

3. Results

3.1. Convergence results

3.1.1. Hypotheses on f

In this paragraph, we define the set of hypotheses on f which could possibly be of use in our work. Discussion on several of these hypotheses can be found in Appendix C.

In this section, to be more legible we replace g with $g^{(k-1)}$. Let

$$\Theta ={\mathbb{R}}^d,{\Theta }^{D_{\varphi }}=\{b\in \Theta |\int {\phi }^{*}({\phi }^{\prime }(\frac{g(x)}{f(x)}\frac{f_b(b^{\top }x)}{g_b(b^{\top }x)}))d\mathbf{P}<\infty \}$$

$$M(b,a,x)=\int {\phi }^{\prime }(\frac{g(x)}{f(x)}\frac{f_b(b^{\top }x)}{g_b(b^{\top }x)})g(x)\frac{f_a(a^{\top }x)}{g_a(a^{\top }x)}dx-{\phi }^{*}({\phi }^{\prime }(\frac{g(x)}{f(x)}\frac{f_b(b^{\top }x)}{g_b(b^{\top }x)}))$$

$${\mathbb{P}}_{n}M(b,a)=\int M(b,a,x)d{\mathbb{P}}_n,\mathbf{P}M(b,a)=\int M(b,a,x)d\mathbf{P}$$

where $\mathbf{P}$ is the probability measure presenting f as density.

Similarly as in chapter V of Van der Vaart [10], let us define :

(H1)

: $\text{For}\text{all}\epsilon >0,\text{there}\text{is}\eta >0,\text{such}\text{that}\text{for}\text{all}c\in {\Theta }^{D_{\varphi }}\text{verifying}\parallel c-a_k\parallel \ge \epsilon ,\text{we}\text{have}\mathbf{P}M(c,a)-\eta >\mathbf{P}M(a_k,a),\text{with}a\in \Theta .$

(H2)

: $\exists Z<0,n_0>0\text{such}\text{that}(n\ge n_0\Rightarrow {sup}_{a\in \Theta }{sup}_{c\in {\{{\Theta }^{D_{\varphi }}\}}^c}{\mathbb{P}}_{n}M(c,a)<Z)$

(H3)

: There is a neighbourhood V of a_k, and a positive function H, such that, for all $c\in V,\text{we}\text{have}|M(c,a_k,x)|\le H(x)(\mathbf{P}-a.s.)\text{with}\mathbf{P}H<\infty ,$

(H4)

: There is a neighbourhood V of a_k, such that for all ε, there is a η such that for all $c\in V\text{and}a\in \Theta ,\text{verifying}\parallel a-a_k\parallel \ge \epsilon ,\text{we}\text{have}\mathbf{P}M(c,a_k)<\mathbf{P}M(c,a)-\eta .$

Putting $I_{a_k}=\frac{{\partial }^2}{\partial a^2}{D}_{\varphi }(g\frac{f_{a_k}}{g_{a_k}},f),$ and $x\to \rho (b,a,x)={\phi }^{\prime }(\frac{g(x)f_b(b^{\top }x)}{f(x)g_b(b^{\top }x)})\frac{g(x)f_a(a^{\top }x)}{g_a(a^{\top }x)}$, putting:

(H5)

: The function φ is ${\mathcal{C}}^3$ in $(0,+\infty$) and there is a neighbourhood $V_k^{\prime }$ of $(a_k,a_k)$ such that, for all $(b,a)$ of $V_k^{\prime }$, the gradient $\nabla (\frac{g(x)f_a(a^{\top }x)}{g_a(a^{\top }x)})$ and the Hessian $\mathcal{H}(\frac{g(x)f_a(a^{\top }x)}{g_a(a^{\top }x)})$ exist ($\lambda \_a.s.$), and the first order partial derivatives $\frac{g(x)f_a(a^{\top }x)}{g_a(a^{\top }x)}$ and the first and second order derivatives of $(b,a)\mapsto \rho (b,a,x)$ are dominated ($\lambda \_$a.s.) by λ-integrable functions.

(H6)

: The function $(b,a)\mapsto M(b,a)$ is ${\mathcal{C}}^3$ in a neighbourhood $V_k$ of $(a_k,a_k)$ for all x; and the partial derivatives of $(b,a)\mapsto M(b,a)$ are all dominated in $V_k$ by a $\mathbf{P}\_$integrable function $H(x)$.

(H7)

: $\mathbf{P}\parallel \frac{\partial }{\partial b}M(a_k,a_k){\parallel }^2$ and $\mathbf{P}\parallel \frac{\partial }{\partial a}M(a_k,a_k){\parallel }^2$ are finite and the expressions $\mathbf{P}\frac{{\partial }^2}{\partial b_i\partial b_j}M(a_k,a_k)$ and $I_{a_k}$ exist and are invertible.

(H8)

: There exists k such that $\mathbf{P}M(a_k,a_k)=0$.

(H9)

: ${(Va{r}_{\mathbf{P}}(M(a_k,a_k)))}^{1/2}$ exists and is invertible.

(H0)

: f and g are assumed to be positive and bounded and such that $K(g,f)\ge \int |f(x)-g(x)|dx$.

3.1.2. Estimation of the first co-vector of f

Let $\mathcal{R}$ be the class of all positive functions r defined on $\mathbb{R}$ and such that $g(x)r(a^{\top }x)$ is a density on ${\mathbb{R}}^d$ for all a belonging to ${\mathbb{R}}_{*}^d$. The following proposition shows that there exists a vector a such that $\frac{f_a}{g_a}$ minimizes $D_{\varphi }(gr,f)$ in r:

Proposition 3.1. 

There exists a vector a belonging to ${\mathbb{R}}_{*}^d$ such that

$$arg\underset{r\in \mathcal{R}}{min}{D}_{\varphi }(gr,f)=\frac{f_a}{g_a}andr(a^{\top }x)=\frac{f_a(a^{\top }x)}{g_a(a^{\top }x)}$$

$\mathcal{R}emark$ 3.1. 

This proposition proves that $a_1$ simultaneously optimises (1.1), (1.2) and (1.3). In other words, it proves that the underlying structures of f evidenced through our method are identical to the ones obtained through Huber’s methods.

Following Broniatowski [11], let us introduce the estimate of $D_{\varphi }(g\frac{f_{a,n}}{g_a},f_n)$, through

$$\check{D_{\varphi }}(g\frac{f_{a,n}}{g_a},f_n)=\int M(a,a,x)d{\mathbb{P}}_n(x)$$

Proposition 3.2. 

Let $\check{a}$ be such that $\check{a}:=arg{inf}_{a\in {\mathbb{R}}_{*}^d}\check{D_{\varphi }}(g\frac{f_{a,n}}{g_a},f_n).$

Then, $\check{a}$ is a strongly convergent estimate of a, as defined in Proposition 3.1.

Let us also introduce the following sequences ${({\check{a}}_k)}_{k\ge 1}$ and ${({\check{g}}_n^{(k)})}_{k\ge 1}$, for any given n—see Section 2.2.:

• ${\check{a}}_k$ is an estimate of $a_k$ as defined in Proposition 3.2 with ${\check{g}}_n^{(k-1)}$ instead of g,
• ${\check{g}}_n^{(k)}$ is such that ${\check{g}}_n^{(0)}=g$, ${\check{g}}_n^{(k)}(x)={\check{g}}_n^{(k-1)}(x)\frac{f_{{\check{a}}_k,n}({\check{a}}_k^{\top }x)}{{[{\check{g}}^{(k-1)}]}_{{\check{a}}_k,n}({\check{a}}_k^{\top }x)}$, i.e., ${\check{g}}_n^{(k)}(x)=g(x){\Pi }_{j=1}^k\frac{f_{{\check{a}}_j,n}({\check{a}}_j^{\top }x)}{{[{\check{g}}^{(j-1)}]}_{{\check{a}}_j,n}({\check{a}}_j^{\top }x)}$.

We also note that ${\check{g}}_n^{(k)}$ is a density.

3.1.3. Convergence study at the $k^{\text{th}}$ step of the algorithm:

In this paragraph, we will show that the sequence ${({\check{a}}_k)}_n$ converges towards $a_k$ and that the sequence ${({\check{g}}_n^{(k)})}_n$ converges towards $g^{(k)}$.

Let ${\check{c}}_n(a)=arg{sup}_{c\in \Theta }{\mathbb{P}}_{n}M(c,a),$ with $a\in \Theta$, and ${\check{\gamma }}_n=arg{inf}_{a\in \Theta }{sup}_{c\in \Theta }{\mathbb{P}}_{n}M(c,a)$. We state

Proposition 3.3. 

Both ${sup}_{a\in \Theta }\parallel {\check{c}}_n(a)-a_k\parallel$ and ${\check{\gamma }}_n$ converge toward $a_k$ a.s.

Finally, the following theorem shows that ${\check{g}}_n^{(k)}$ converges almost everywhere towards $g^{(k)}$:

Theorem 3.1. 

It holds ${\check{g}}_n^{(k)}{\to }_n{g}^{(k)}a.s.$

3.2. Asymptotic Inference at the $k^{th}$ step of the algorithm

The following theorem shows that ${\check{g}}_n^{(k)}$ converges towards $g^{(k)}$ at the rate $O_{\mathbf{P}}(n^{-\frac{2}{2+d}})$ in three different cases, namely for any given x, with the $L^1$ distance and with the Kullback–Leibler divergence:

Theorem 3.2. 

It holds $|{\check{g}}_n^{(k)}(x)-g^{(k)}(x)|=O_{\mathbf{P}}(n^{-\frac{2}{2+d}}),$ $\int |{\check{g}}_n^{(k)}(x)-g^{(k)}(x)|dx=O_{\mathbf{P}}(n^{-\frac{2}{2+d}})$ and $|K({\check{g}}_n^{(k)},f)-K(g^{(k)},f)|=O_{\mathbf{P}}(n^{-\frac{2}{2+d}}).$

The following theorem shows that the laws of our estimators of $a_k$, namely ${\check{c}}_n(a_k)$ and ${\check{\gamma }}_n$, converge towards a linear combination of Gaussian variables.

Theorem 3.3. 

It holds $\sqrt{n}\mathcal{A}.({\check{c}}_n(a_k)-a_k)\stackrel{\mathcal{L}\mathit{aw}}{\to }\mathcal{B}.{\mathcal{N}}_d(0,\mathbf{P}\parallel \frac{\partial }{\partial b}M(a_k,a_k){\parallel }^2)+\mathcal{C}.{\mathcal{N}}_d(0,\mathbf{P}\parallel \frac{\partial }{\partial a}M(a_k,a_k){\parallel }^2)$ and $\sqrt{n}\mathcal{A}.({\check{\gamma }}_n-a_k)\stackrel{\mathcal{L}\mathit{aw}}{\to }\mathcal{C}.{\mathcal{N}}_d(0,\mathbf{P}\parallel \frac{\partial }{\partial b}M(a_k,a_k){\parallel }^2)+\mathcal{C}.{\mathcal{N}}_d(0,\mathbf{P}\parallel \frac{\partial }{\partial a}M(a_k,a_k){\parallel }^2)$ where $\mathcal{A}=\mathbf{P}\frac{{\partial }^2}{\partial b\partial b}M(a_k,a_k)(\mathbf{P}\frac{{\partial }^2}{\partial a_i\partial a_j}M(a_k,a_k)+\mathbf{P}\frac{{\partial }^2}{\partial a_i\partial b_j}M(a_k,a_k))$, $\mathcal{C}=\mathbf{P}\frac{{\partial }^2}{\partial b\partial b}M(a_k,a_k)$ and $\mathcal{B}=\mathbf{P}\frac{{\partial }^2}{\partial b\partial b}M(a_k,a_k)+\mathbf{P}\frac{{\partial }^2}{\partial a_i\partial a_j}M(a_k,a_k)+\mathbf{P}\frac{{\partial }^2}{\partial a_i\partial b_j}M(a_k,a_k).$

3.3. A stopping rule for the procedure

In this paragraph, we will call ${\check{g}}_n^{(k)}$ (resp. ${\check{g}}_{a,n}^{(k)}$) the kernel estimator of ${\check{g}}^{(k)}$ (resp. ${\check{g}}_a^{(k)}$). We will first show that $g_n^{(k)}$ converges towards f in k and n. Then, we will provide a stopping rule for this identification procedure.

3.3.1. Estimation of f

The following proposition provides us with an estimate of f:

Theorem 3.4. 

We have ${lim}_n{lim}_k{\check{g}}_n^{(k)}=f$ a.s.

Consequently, the following corollary shows that $D_{\varphi }(g_n^{(k-1)}\frac{f_{a_k,n}}{g_{a_k,n}^{(k-1)}},f_{a_k,n})$ converges towards zero as k and then as n go to infinity:

Corollary 3.1. 

We have ${lim}_n{lim}_k{D}_{\varphi }({\check{g}}_n^{(k)}\frac{f_{a_k,n}}{{[{\check{g}}^{(k)}]}_{a_k,n}},f_n)=0$ a.s.

3.3.2. Testing of the criteria

In this paragraph, through a test of our criteria, namely $a\mapsto D_{\varphi }({\check{g}}_n^{(k)}\frac{f_{a,n}}{{[{\check{g}}^{(k)}]}_{a,n}},f_n)$, we will build a stopping rule for this procedure. First, the next theorem enables us to derive the law of our criteria:

Theorem 3.5. 

For a fixed k, we have

$$\sqrt{n}{(Va{r}_{\mathbf{P}}(M({\check{c}}_n({\check{\gamma }}_n),{\check{\gamma }}_n)))}^{-1/2}({\mathbb{P}}_{n}M({\check{c}}_n({\check{\gamma }}_n),{\check{\gamma }}_n)-{\mathbb{P}}_{n}M(a_k,a_k))\stackrel{\mathcal{L}\mathit{aw}}{\to }\mathcal{N}(0,I),$$

where k represents the $k^{th}$ step of our algorithm and where I is the identity matrix in ${\mathbb{R}}^d$.

Note that k is fixed in Theorem 3.5 since ${\check{\gamma }}_n=arg{inf}_{a\in \Theta }{sup}_{c\in \Theta }{\mathbb{P}}_{n}M(c,a)$ where M is a known function of k—see Section 3.1. Thus, in the case when $D_{\varphi }(g^{(k-1)}\frac{f_{a_k}}{g_{a_k}^{(k-1)}},f)=0$, we obtain

Corollary 3.2. 

We have $\sqrt{n}{(Va{r}_{\mathbf{P}}(M({\check{c}}_n({\check{\gamma }}_n),{\check{\gamma }}_n)))}^{-1/2}{\mathbb{P}}_{n}M({\check{c}}_n({\check{\gamma }}_n),{\check{\gamma }}_n)\stackrel{\mathcal{L}\mathit{aw}}{\to }\mathcal{N}(0,I)$.

Hence, we propose the test of the null hypothesis

$(H_0)\text{:}{D}_{\varphi }(g^{(k-1)}\frac{f_{a_k}}{g_{a_k}^{(k-1)}},f)=0versus\text{the}\text{alternative}(H_1)\text{:}{D}_{\varphi }(g^{(k-1)}\frac{f_{a_k}}{g_{a_k}^{(k-1)}},f)\ne 0\text{.}$

Based on this result, we stop the algorithm, then, defining $a_k$ as the last vector generated, we derive from Corollary 3.2 a α-level confidence ellipsoid around $a_k$, namely

$${\mathcal{E}}_k=\{b\in {\mathbb{R}}^d;\sqrt{n}{(Va{r}_{\mathbf{P}}(M(b,b)))}^{-1/2}{\mathbb{P}}_{n}M(b,b)\le q_{\alpha }^{\mathcal{N}(0,1)}\}$$

where $q_{\alpha }^{\mathcal{N}(0,1)}$ is the quantile of a α-level reduced centered normal distribution and where ${\mathbb{P}}_n$ is the empirical measure arising from a realization of the sequences $(X_1,\dots ,X_n)$ and $(Y_1,\dots ,Y_n)$.

Consequently, the following corollary provides us with a confidence region for the above test:

Corollary 3.3. 

${\mathcal{E}}_k$ is a confidence region for the test of the null hypothesis $(H_0)$ versus $(H_1)$.

3.4. Goodness-of-fit test for copulas

Let us begin with studying the following case:

Let f be a density defined on ${\mathbb{R}}^2$ and let g be an elliptical distribution with same mean and variance as f. Assuming first that our algorithm leads us to having $D_{\varphi }(g^{(2)},f)=0$ where family $(a_i)$ is the canonical basis of ${\mathbb{R}}^2$. Hence, we have $g^{(2)}(x)=g(x)\frac{f_1}{g_1}\frac{f_2}{g_2^{(1)}}=g(x)\frac{f_1}{g_1}\frac{f_2}{g_2}$—through Lemma D.4 page 1608—and $g^{(2)}=f$. Therefore, $f=g(x)\frac{f_1}{g_1}\frac{f_2}{g_2},$ i.e., $\frac{f}{f_1{f}_2}=\frac{g}{g_1{g}_2}$, and then $\frac{{\partial }^2}{\partial x\partial y}{C}_f=\frac{{\partial }^2}{\partial x\partial y}{C}_g$ where $C_f$ (resp. $C_g$) is the copula of f (resp. g).

At present, let f be a density on ${\mathbb{R}}^d$ and let g be the density defined in Section 2.1.

Let us assume that our algorithm implies that $D_{\varphi }(g^{(d)},f)=0$.

Hence, we have, for any $x\in {\mathbb{R}}^d$, $g(x){\Pi }_{k=1}^d\frac{f_{a_k}(a_k^{\top }x)}{{[g^{(k-1)}]}_{a_k}(a_k^{\top }x)}=f(x)$, i.e., $\frac{g(x)}{{\Pi }_{k=1}^d{g}_{a_k}(a_k^{\top }x)}=\frac{f(x)}{{\Pi }_{k=1}^d{f}_{a_k}(a_k^{\top }x)}$, since Lemma D.4 page 1608 implies that $g_{a_k}^{(k-1)}=g_{a_k}$ if $k\le d$.

Moreover, the family ${(a_i)}_{i=1...d}$ is a basis of ${\mathbb{R}}^d$—see Lemma D.5 page 1608. Hence, putting $A=(a_1,...,a_d)$ and defining vector y (resp. density $\tilde{f}$, copula ${\tilde{C}}_f$ of $\tilde{f}$, density $\tilde{g}$, copula ${\tilde{C}}_g$ of $\tilde{g}$) as the expression of vector x (resp. density f, copula $C_f$ of f, density g, copula $C_g$ of g) in basis A, the above equality implies $\frac{{\partial }^d}{\partial y_1...\partial y_d}{\tilde{C}}_f=\frac{{\partial }^d}{\partial y_1...\partial y_d}{\tilde{C}}_g.$

Finally, we perform a statistical test of the null hypothesis $(H_0)$ : $\frac{{\partial }^d}{\partial y_1...\partial y_d}{\tilde{C}}_f=\frac{{\partial }^d}{\partial y_1...\partial y_d}{\tilde{C}}_g$ versus the alternative $(H_1)$ : $\frac{{\partial }^d}{\partial y_1...\partial y_d}{\tilde{C}}_f\ne \frac{{\partial }^d}{\partial y_1...\partial y_d}{\tilde{C}}_g$. Since, under $(H_0)$, we have $D_{\varphi }(g^{(d)},f)=0$, then, as explained in Section 3.3, Corollary 3.3 provides us with a confidence region for our test.

Theorem 3.6. 

Keeping the notations of Corollary 3.3, we infer that ${\mathcal{E}}_d$ is a confidence region for the test of the null hypothesis $(H_0)$ versus the alternative hypothesis $(H_1)$.

3.5. Rewriting of the convolution product

In the present paper, we first elaborated an algorithm aiming at isolating several known structures from initial data. Our objective was to verify if for a known density on ${\mathbb{R}}^d$, a known density n on ${\mathbb{R}}^{d-j-1}$ such that, for $d>1$,

$$f(x)=n(a_{j+1}^{\top }x,...,a_d^{\top }x)h(a_1^{\top }x,...,a_j^{\top }x)$$

(3.1)

did indeed exist, with $j<d$, with $(a_1,\dots ,a_d)$ being a basis of ${\mathbb{R}}^d$ and with h being a density on ${\mathbb{R}}^j$.

Secondly, our next step consisted in building an estimate (resp. a representation) of f without necessarily assuming that f meets relationship (3.1)—see Theorem 3.4.

Consequently, let us consider $Z_1$ and $Z_2$, two random vectors with respective densities $h_1$ and $h_2$—which is elliptical—on ${\mathbb{R}}^d$. Let us consider a random vector X such that $X=Z_1+Z_2$ and let f be its density. This density can then be written as $f(x)=h_1*h_2(x)={\int }_{{\mathbb{R}}^d}{h}_1(x)h_2(t-x)dt.$

Then, the following property enables us to represent f under the form of a product and without the integral sign.

Proposition 3.4. 

Let φ be a centered elliptical density with ${\sigma }^2.I_d$, ${\sigma }^2>0$, as covariance matrix, such that it is a product density in all orthogonal coordinate systems and such that its characteristic function $s\mapsto \Psi (\frac{1}{2}|s{|}^2{\sigma }^2)$ is integrable—see Landsman [9]. Let f be a density on ${\mathbb{R}}^d$ which can be deconvoluted with ϕ, i.e., $f=\overline{f}*\varphi ={\int }_{{\mathbb{R}}^d}\overline{f}(x)\varphi (t-x)dt,$ where $\overline{f}$ is some density on ${\mathbb{R}}^d$. Let $g^{(0)}$ be the elliptical density belonging to the same elliptical family as f and having same mean and variance as f.

Then, the sequence ${(g^{(k)})}_k$ converges uniformly a.s. and in $L^1$ towards f in k, i.e.,

$$\underset{k\to \infty }{lim}\underset{x\in {\mathbb{R}}^d}{sup}|g^{(k)}(x)-f(x)|=0,and\underset{k\to \infty }{lim}{\int }_{{\mathbb{R}}^d}|g^{(k)}(x)-f(x)|dx=0$$

Finally, with the notations of Section 3.3 and of Proposition 3.4, the following theorem enables us to estimate any convolution product of a multivariate elliptical density φ with a continuous density $\overline{f}$:

Theorem 3.7. 

It holds ${lim}_n{lim}_k{\check{g}}_n^{(k)}=\overline{f}*\varphi a.s.$

3.6. On the regression

In this section, we will study several applications of our algorithm pertaining to the regression analysis. We define $(X_1,...,X_d)$ (resp. $(Y_1,...,Y_d)$) as a vector with density f (resp. g—see Section 2.1).

$\mathcal{R}emark$ 3.2. 

In this paragraph, we will work in the $L^2$ space. Then, we will first only consider the $\varphi -$divergences which are greater than or equal to the $L^2$ distance—see Vajda [12]. Note also that the co-vectors of f can be obtained in the $L^2$ space—see Lemma D.3 and Proposition B.1.

3.6.1. The basic idea

In this paragraph, we will assume that $\Theta ={\mathbb{R}}_{*}^2$ and that our algorithm stops for $j=1$ and $a_1={(0,1)}^{\prime }$. The following theorem provides us with the regression of $X_1$ on $X_2$ :

Theorem 3.8. 

The probability measure of $X_1$ given $X_2$ is the same as the probability measure of $Y_1$ given $Y_2$. Moreover, the regression between $X_1$ and $X_2$ is $X_1=E(Y_1/Y_2)+\epsilon ,$ where ε is a centered random variable orthogonal to $E(X_1/X_2)$.

$\mathcal{R}emark$ 3.3 

This theorem implies that $E(X_1/X_2)=E(Y_1/Y_2)$. This equation can be used in many fields of research. The Markov chain theory has been used for instance in Example 1.2.

Moreover, if g is a Gaussian density with same mean and variance as f, then Saporta [14] implies that $E(Y_1/Y_2)=E(Y_1)+\frac{Cov(Y_1,Y_2)}{Var(Y_2)}(Y_2-E(Y_2))$ and then $X_1=E(Y_1)+\frac{Cov(Y_1,Y_2)}{Var(Y_2)}(Y_2-E(Y_2))+\epsilon .$

3.6.2. General case

In this paragraph, we will assume that $\Theta ={\mathbb{R}}_{*}^d$ and that our algorithm stops with j for $j<d$. Lemma D.6 implies the existence of an orthogonal and free family ${(b_i)}_{i=j+1,..,d}$ of ${\mathbb{R}}_{*}^d$ such that ${\mathbb{R}}^d=Vect\{a_i\}\stackrel{\perp }{\oplus }Vect\{b_k\}$ and such that

$$\begin{array}{c}\hfill g(b_{j+1}^{\top }x,...,b_d^{\top }x/a_1^{\top }x,...,a_j^{\top }x)=f(b_{j+1}^{\top }x,...,b_d^{\top }x/a_1^{\top }x,...,a_j^{\top }x)\end{array}$$

(3.2)

Hence, the following theorem provides us with the regression of $b_k^{\top }X$, $k=1,...,d$, on $(a_1^{\top }X,...,a_j^{\top }X)$:

Theorem 3.9. 

The probability measure of $(b_{j+1}^{\top }X,...,b_d^{\top }X)$ given $(a_1^{\top }X,...,a_j^{\top }X)$ is the same as the probability measure of $(b_{j+1}^{\top }Y,...,b_d^{\top }Y)$ given $(a_1^{\top }Y,...,a_j^{\top }Y)$. Moreover, the regression of $b_k^{\top }X$, $k=1,...,d$, on $(a_1^{\top }X,...,a_j^{\top }X)$ is $b_k^{\top }X=E(b_k^{\top }Y/a_1^{\top }{Y}_1,...,a_j^{\top }Y)+b_k^{\top }\epsilon$, where ε is a centered random vector such that $b_k^{\top }\epsilon$ is orthogonal to $E(b_k^{\top }X/a_1^{\top }X,...,a_j^{\top }X)$.

Corollary 3.4. 

If g is a Gaussian density with same mean and variance as f, and if $Cov(X_i,X_j)=0$ for any $i\ne j$, then, the regression of $b_k^{\top }X$, $k=1,...,d$, on $(a_1^{\top }X,...,a_j^{\top }X)$ is $b_k^{\top }X=E(b_k^{\top }Y)+b_k^{\top }\epsilon$, where ε is a centered random vector such that $b_k^{\top }\epsilon$ is orthogonal to $E(b_k^{\top }X/a_1^{\top }X,...,a_j^{\top }X)$.

4. Simulations

Let us study five simulations. The first involves a ${\chi }^2$-divergence, the second a Hellinger distance, the third and the fourth a Cressie–Read divergence (still with $\gamma =1.25$), and the fifth a Kullback–Leibler divergence.

In each example, our program will follow our algorithm and will aim at creating a sequence of densities $(g^{(j)})$, $j=1,..,k$, $k<d$, such that $g^{(0)}=g,$ $g^{(j)}=g^{(j-1)}{f}_{a_j}/{[g^{(j-1)}]}_{a_j}$ and $D_{\varphi }(g^{(k)},f)=0,$ with $D_{\varphi }$ being a divergence and $a_j=arg{inf}_b{D}_{\varphi }(g^{(j-1)}{f}_b/{[g^{(j-1)}]}_b,f),$ for all $j=1,...,k$. Moreover, in the second example, we will study the robustness of our method with two outliers. In the third and the fourth example, defining $(X_0,X_1)$ as a vector with f as density, we will study the regression of $X_1$ on $X_0$. And finally, in the fifth example, we will perform our goodness-of-fit test for copulas.

$\mathcal{S}imulation$ 4.1 

(With the ${\chi }^2$ divergence).

We are in dimension 3(=d), and we consider a sample of 50(=n) values of a random variable X with a density law f defined by

$$f(x)=Gaussian(x_1+x_2).Gaussian(x_0+x_2).Gumbel(x_0+x_1)$$

where the Normal law parameters are $(-5,2)$ and $(1,1)$ and where the Gumbel distribution parameters are $-3$ and 4. Let us generate then a Gaussian random variable Y with a density—that we will name g—presenting the same mean and variance as f.

We theoretically obtain $k=1$ and $a_1=(1,1,0)$. To get this result, we perform the following test:

$$H0:a_1=(1,1,0)versus(H_1):a_1\ne (1,1,0).$$

Then, Corollary 3.3 enables us to estimate $a_1$ by the following 0.9(=α) level confidence ellipsoid

$${\mathcal{E}}_1=\{b\in {\mathbb{R}}^3;{(Va{r}_{\mathbf{P}}(M(b,b)))}^{(-1/2)}{\mathbb{P}}_{n}M(b,b)\le q_{\alpha }^{\mathcal{N}(0,1)}/\sqrt{n}\simeq 0,2533/7.0710678=0.03582203\}$$

And, we obtain

Table 1. Simulation 1: Numerical results of the optimisation.

Our Algorithm
Projection Study 0 :	minimum : 0.0201741
	at point : (1.00912,1.09453,0.01893)
	P-Value : 0.81131
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
${\chi }^2$(Kernel Estimation of $g^{(1)}$, $g^{(1)}$)	6.1726

Table 1. Simulation 1: Numerical results of the optimisation.

Our Algorithm
Projection Study 0 :	minimum : 0.0201741
	at point : (1.00912,1.09453,0.01893)
	P-Value : 0.81131
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
${\chi }^2$(Kernel Estimation of $g^{(1)}$, $g^{(1)}$)	6.1726

Therefore, we conclude that f = g⁽¹⁾.

$\mathcal{S}imulation$ 4.2 

(With the Hellinger distance H).

We are in dimension 20(=d). We first generate a sample with 100(=n) observations, namely two outliers $x=(2,0,\dots ,0)$ and 98 values of a random variable X with a density f defined by

$$f(x)=Gumbel(x_0).Normal(x_1,\dots ,x_9)$$

where the Gumbel law parameters are -5 and 1 and where the normal distribution is reduced and centered. Our reasoning is the same as in Simulation 4.1.

In the first part of the program, we theoretically obtain $k=1$ and $a_1=(1,0,\dots ,0)$. To get this result, we perform the following test

$$(H_0):a_1=(1,0,\dots ,0)versus(H_1):a_1\ne (1,0,\dots ,0)$$

We estimate $a_1$ by the following 0.9(=α) level confidence ellipsoid

$${\mathcal{E}}_i=\{b\in {\mathbb{R}}^2;{(Va{r}_{\mathbf{P}}(M(b,b)))}^{-1/2}{\mathbb{P}}_{n}M(b,b)\le q_{\alpha }^{\mathcal{N}(0,1)}/\sqrt{n}\simeq 0.02533\}$$

And, we obtain

Table 2. Simulation 2: Numerical results of the optimisation.

Our Algorithm
Projection Study 0	minimum : 0.002692
	at point : (1.01326, 0.0657, 0.0628, 0.1011, 0.0509, 0.1083,
	0.1261, 0.0573, 0.0377, 0.0794, 0.0906, 0.0356, 0.0012,
	0.0292, 0.0737, 0.0934, 0.0286, 0.1057, 0.0697, 0.0771)
	P-Value : 0.80554
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
H(Est. of $g^{(1)}$, $g^{(1)}$)	3.042174

Table 2. Simulation 2: Numerical results of the optimisation.

Our Algorithm
Projection Study 0	minimum : 0.002692
	at point : (1.01326, 0.0657, 0.0628, 0.1011, 0.0509, 0.1083,
	0.1261, 0.0573, 0.0377, 0.0794, 0.0906, 0.0356, 0.0012,
	0.0292, 0.0737, 0.0934, 0.0286, 0.1057, 0.0697, 0.0771)
	P-Value : 0.80554
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
H(Est. of $g^{(1)}$, $g^{(1)}$)	3.042174

Therefore, we conclude that f = g⁽¹⁾.

$\mathcal{S}imulation$ 4.3 

(With the Cressie-Read divergence ($D_{\varphi }$)).

We are in dimension 2(=d), and we consider a sample of 50(=n) values of a random variable $X=(X_0,X_1)$ with a density law f defined by

$$f(x)=Gumbel(x_0).Normal(x_1)$$

where the Gumbel law parameters are -5 and 1 and where the normal distribution parameters are $(0,1)$. Let us generate then a Gaussian random variable Y with a density—that we will name g—presenting the same mean and variance as f.

We theoretically obtain $k=1$ and $a_1=(1,0)$. To get this result, we perform the following test

$$H0:a_1=(1,0)versus(H_1):a_1\ne (1,0)$$

Then, Corollary 3.3 enables us to estimate $a_1$ by the following 0.9(=α) level confidence ellipsoid

$${\mathcal{E}}_1=\{b\in {\mathbb{R}}^2;{(Va{r}_{\mathbf{P}}(M(b,b)))}^{(-1/2)}{\mathbb{P}}_{n}M(b,b)\le q_{\alpha }^{\mathcal{N}(0,1)}/\sqrt{n}\},with{q}_{\alpha }^{\mathcal{N}(0,1)}/\sqrt{n}\simeq 0.03582203.$$

And, we obtain

Table 3. Simulation 3: Numerical results of the optimisation.

Our Algorithm
Projection Study 0 :	minimum : 0.0210058
	at point : (1.001,0.0014)
	P-Value : 0.989552
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
$D_{\varphi }$(Kernel Estimation of $g^{(1)}$, $g^{(1)}$)	6.47617

Table 3. Simulation 3: Numerical results of the optimisation.

Our Algorithm
Projection Study 0 :	minimum : 0.0210058
	at point : (1.001,0.0014)
	P-Value : 0.989552
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
$D_{\varphi }$(Kernel Estimation of $g^{(1)}$, $g^{(1)}$)	6.47617

Therefore, we conclude that f = g⁽¹⁾.

Figure 1. Graph of the distribution to estimate (red) and of our own estimate (green).

Figure 2. Graph of the distribution to estimate (red) and of Huber’s estimate (green).

At present, keeping the notations of this simulation, let us study the regression of $X_1$ on $X_0$.

Our algorithm leads us to infer that the density of $X_1$ given $X_0$ is the same as the density of $Y_1$ given $Y_0$. Moreover, Property A.1 implies that the co-factors of f are the same for any divergence. Consequently, applying Theorem 3.8 implies that $X_1=E(Y_1/Y_0)+\epsilon ,$ where ε is a centered random variable orthogonal to $E(X_1/X_0)$.

Thus, since g is a Gaussian density, Remark 3.3 implies that

$$X_1=E(Y_1)+\frac{Cov(Y_1,Y_0)}{Var(Y_0)}(Y_0-E(Y_0))+\epsilon$$

Now, using the least squares method, we estimate ${\alpha }_1$ and ${\alpha }_2$ such that $X_1={\alpha }_1+{\alpha }_2.X_0+\epsilon .$

Thus, the following table presents the results of our regression and of the least squares method if we assume that ε is Gaussian.

Table 4. Simulation 3: Numerical results of the regression.

Our Regression	$E(Y_1)$	-4.545483
	$Cov(Y_1,Y_0)$	0.0380534
	$Var(Y_0)$	0.9190052
	$E(Y_0)$	0.3103752
	correlation $(Y_1,Y_0)$	0.02158213
Least squares method	${\alpha }_1$	-4.34159227
	Std Error of ${\alpha }_1$	0.19870
	${\alpha }_2$	0.06803317
	Std Error of ${\alpha }_2$	0.21154
	correlation $(X_1,X_0)$	0.04888484

Table 4. Simulation 3: Numerical results of the regression.

Our Regression	$E(Y_1)$	-4.545483
	$Cov(Y_1,Y_0)$	0.0380534
	$Var(Y_0)$	0.9190052
	$E(Y_0)$	0.3103752
	correlation $(Y_1,Y_0)$	0.02158213
Least squares method	${\alpha }_1$	-4.34159227
	Std Error of ${\alpha }_1$	0.19870
	${\alpha }_2$	0.06803317
	Std Error of ${\alpha }_2$	0.21154
	correlation $(X_1,X_0)$	0.04888484

Figure 3. Graph of the regression of $X_1$ on $X_0$ based on the least squares method (red) and based on our theory (green).

$\mathcal{S}imulation$ 4.4 

(With the Cressie-Read divergence ($D_{\varphi }$)).

We are in dimension 2(=d), and we consider a sample of 500(=n) values of a random variable $X=(X_0,X_1)$ with a density law f defined by

$$f(x)=Gumbel(x_1-x_0).Normal(x_1+x_0)$$

where the Gumbel law parameters are -5 and 1 and where the normal distribution parameters are $(0,1)$. Let us generate then a Gaussian random variable Y with a density—that we will name g—presenting the same mean and variance as f.

We theoretically obtain $k=1$ and $a_1=(1,0)$. To get this result, we perform the following test $H0:a_1=(1,-1)versus(H_1):a_1\ne (1,-1).$ Then, Corollary 3.3 enables us to estimate $a_1$ by the following 0.9(=α) level confidence ellipsoid

$${\mathcal{E}}_1=\{b\in {\mathbb{R}}^2;{(Va{r}_{\mathbf{P}}(M(b,b)))}^{(-1/2)}{\mathbb{P}}_{n}M(b,b)\le q_{\alpha }^{\mathcal{N}(0,1)}/\sqrt{n}\simeq 0,2533/\sqrt{500}=0.01132792\}$$

And, we obtain

Table 5. Simulation 4: Numerical results of the optimisation.

Our Algorithm
Projection Study 0 :	minimum : 0.010920
	at point : (1.09,-0.9701)
	P-Value : 0.889400
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
$D_{\varphi }$(Kernel Estimation of $g^{(1)}$, $g^{(1)}$)	5.25077

Table 5. Simulation 4: Numerical results of the optimisation.

Our Algorithm
Projection Study 0 :	minimum : 0.010920
	at point : (1.09,-0.9701)
	P-Value : 0.889400
Test :	$H_0$ : $a_1\in {\mathcal{E}}_1$ : True
$D_{\varphi }$(Kernel Estimation of $g^{(1)}$, $g^{(1)}$)	5.25077

Therefore, we conclude that f = g⁽¹⁾.

At present, keeping the notations of this simulation, let us study the regression of $X_1+X_0$ on $X_1-X_0$. Our algorithm leads us to infer that the density of $X_1+X_0$ given $X_1-X_0$ is the same as the density of $Y_1+Y_0$ given $Y_1-Y_0$. Moreover, Property A.1 implies that the co-factors of f are the same for any divergence. Consequently, putting $U=X_1+X_0$, $V=X_1-X_0$, $U^{\prime }=Y_1+Y_0$ and $V^{\prime }=Y_1-Y_0$, and since $\{{(1,1)}^{\prime },{(1,-1)}^{\prime }\}$ is an orthogonal basis, we can therefore infer from Theorem 3.8 that $U=E(U^{\prime }/V^{\prime })+\epsilon ,$ where ε is a centered random variable orthogonal to $E(U/V)$.

Thus, since g is a Gaussian density, Remark 3.3 implies that

$$U=E(U^{\prime })+\frac{Cov(U^{\prime },V^{\prime })}{Var(V^{\prime })}(V^{\prime }-E(V^{\prime }))+\epsilon$$

In other words, we apply the same reasoning as the one used in the regression studies in Simulation 4.3 to $(U,V)$ instead of $(X_1,X_0)$. This is possible since $\{{(1,1)}^{\prime },{(1,-1)}^{\prime }\}$ is an orthogonal basis of ${\mathbb{R}}^2$, i.e., we implement a change in basis from the canonical basis of ${\mathbb{R}}^2$ to $\{{(1,1)}^{\prime },{(1,-1)}^{\prime }\}$.

Thus, in the canonical basis $U=E(U^{\prime }/V^{\prime })+\epsilon$ becomes $X_1+X_0=E(Y_1+Y_0/Y_1-Y_0)+\epsilon$, i.e., we obtain that

$$X_1+X_0=E(Y_1+Y_0)+\frac{Cov(Y_1+Y_0,Y_1-Y_0)}{Var(Y_1-Y_0)}(Y_1-Y_0-E(Y_1-Y_0))+\epsilon$$

where ε is a centered random variable orthogonal to $E(X_1+X_0/X_1-X_0)$.

The following table presents the results of our regression.

We simulate 10 times the regression and we obtain a and b such that $X_1=a+b{X}_0+\epsilon$ :

Table 6. Simulation 4: Numerical results of the regression.

Simulation	a	Std Error of a	b	Std Error of b
1	-4.83739	0.11149	-0.95861	0.04677
2	-4.56895	0.09989	-0.88577	0.04225
3	-4.4926	0.1057	-1.2085	0.0452
4	-4.70619	0.10350	-1.04549	0.04235
5	-4.40331	0.10248	-1.00890	0.0438
6	-4.61757	0.09813	-1.20890	0.04649
7	-4.40572	0.09172	-1.16085	0.04091
8	-4.39581	0.10174	-1.38696	0.04487
9	-4.42780	0.10018	-0.93672	0.04066
10	-4.55394	0.09923	-0.98065	0.04382

Table 6. Simulation 4: Numerical results of the regression.

Simulation	a	Std Error of a	b	Std Error of b
1	-4.83739	0.11149	-0.95861	0.04677
2	-4.56895	0.09989	-0.88577	0.04225
3	-4.4926	0.1057	-1.2085	0.0452
4	-4.70619	0.10350	-1.04549	0.04235
5	-4.40331	0.10248	-1.00890	0.0438
6	-4.61757	0.09813	-1.20890	0.04649
7	-4.40572	0.09172	-1.16085	0.04091
8	-4.39581	0.10174	-1.38696	0.04487
9	-4.42780	0.10018	-0.93672	0.04066
10	-4.55394	0.09923	-0.98065	0.04382

Figure 4. Graph of the regression of $X_1$ on $X_0$ based on our theory (green).

$\mathcal{S}imulation$ 4.5 

(With the Kullback-Leibler divergence K).

We are in dimension 2(=d), and we use the Kullback–Leibler divergence to perform our optimisations. Let us consider a sample of 50(=n) values of a random variable X with a density law f defined by :

$$f(x)=c_{\rho }(F_{Gumbel}(x_0),F_{Exponential}(x_1)).Gumbel(x_0).Exponential(x_1)$$

where :

• c is the Gaussian copula with correlation coefficient $\rho =0.5$,
• the Gumbel distribution parameters are $-1$ and 1 and
• the Exponential density parameter is 2.

Let us generate then a Gaussian random variable Y with a density—that we will name g—presenting the same mean and variance as f. We theoretically obtain $k=2$ and $(a_1,a_2)=((1,0),(0,1))$. To get this result, we perform the following test

$$(H_0):(a_1,a_2)=((1,0),(0,1))versus(H_1):(a_1,a_2)\ne ((1,0),(0,1))$$

Then, Theorem 3.6 enables us to verify $(H_0)$ by the following 0.9(=α) level confidence ellipsoid

$${\mathcal{E}}_2=\{b\in {\mathbb{R}}^2;{(Va{r}_{\mathbf{P}}(M(b,b)))}^{(-1/2)}{\mathbb{P}}_{n}M(b,b)\le q_{\alpha }^{\mathcal{N}(0,1)}/\sqrt{n}\simeq 0,2533/7.0710678=0.0358220\}$$

And, we obtain

Table 7. Simulation 5: Numerical results of the optimisation.

Our Algorithm
Projection Study number 0 :	minimum : 0.445199
	at point : (1.0142,0.0026)
	P-Value : 0.94579
Test :	$H_1$ : $a_1\notin {\mathcal{E}}_1$ : True
Projection Study number 1 :	minimum : 0.0263
	at point : (0.0084,0.9006)
	P-Value : 0.97101
Test :	$H_0$ : $a_2\in {\mathcal{E}}_2$ : True
K(Kernel Estimation of $g^{(2)}$, $g^{(2)}$)	4.0680

Table 7. Simulation 5: Numerical results of the optimisation.

Our Algorithm
Projection Study number 0 :	minimum : 0.445199
	at point : (1.0142,0.0026)
	P-Value : 0.94579
Test :	$H_1$ : $a_1\notin {\mathcal{E}}_1$ : True
Projection Study number 1 :	minimum : 0.0263
	at point : (0.0084,0.9006)
	P-Value : 0.97101
Test :	$H_0$ : $a_2\in {\mathcal{E}}_2$ : True
K(Kernel Estimation of $g^{(2)}$, $g^{(2)}$)	4.0680

Therefore, we can conclude that H₀ is verified.

Figure 5. Graph of the estimate of $(x_0,x_1)\mapsto c_{\rho }(F_{Gumbel}(x_0),F_{Exponential}(x_1))$.

5. Critics of the Simulations

In the case where f is unknown, we will never be sure to have reached the minimum of the φ-divergence: we have indeed used the simulated annealing method to solve our optimisation problem, and therefore it is only when the number of random jumps tends in theory towards infinity that the probability to reach the minimum tends to 1. We also note that no theory on the optimal number of jumps to implement does exist, as this number depends on the specificities of each particular problem. Moreover, we choose the ${50}^{-\frac{4}{4+d}}$ (resp. ${500}^{-\frac{4}{4+d}}$ and ${100}^{-\frac{4}{4+d}}$) for the AMISE of Simulations 4.1, 4.2 and 4.3 (resp. Simulations 4.4 and 4.5). This choice leads us to simulate 50 (resp. 500 and 100) random variables—see Scott [15] page 151—none of which have been discarded to obtain the truncated sample. This has also been the case in our application to real datasets.

Finally, we remark that some of the key advantages of our method over Huber’s consist in the fact that—since there exist divergences smaller than the Kullback–Leibler divergence—our method requires a considerably shorter computation time and also in the superior robustness of our method.

6. Conclusions

Projection Pursuit is useful in evidencing characteristic structures as well as one-dimensional projections and their associated distributions in multivariate data. Huber [2] shows us how to achieve it through maximization of the Kullback–Leibler divergence.

The present article shows that our ϕ-divergence method constitutes a good alternative to Huber’s particularly in terms of regression and robustness as well as in terms of copula’s study. Indeed, the convergence results and simulations we carried out, convincingly fulfilled our expectations regarding our methodology.