Asymptotic Convergence of Soft-Constrained Neural Networks for Density Estimation

Abstract

A soft-constrained neural network for density estimation (SC-NN-4pdf) has recently been introduced to tackle the issues arising from the application of neural networks to density estimation problems (in particular, the satisfaction of the second Kolmogorov axiom). Although the SC-NN-4pdf has been shown to outperform parametric and non-parametric approaches (from both the machine learning and the statistics areas) over a variety of univariate and multivariate density estimation tasks, no clear rationale behind its performance has been put forward so far. Neither has there been any analysis of the fundamental theoretical properties of the SC-NN-4pdf. This paper narrows the gaps, delivering a formal statement of the class of density functions that can be modeled to any degree of precision by SC-NN-4pdfs, as well as a proof of asymptotic convergence in probability of the SC-NN-4pdf training algorithm under mild conditions for a popular class of neural architectures. These properties of the SC-NN-4pdf lay the groundwork for understanding the strong estimation capabilities that SC-NN-4pdfs have only exhibited empirically so far.

1. Introduction

Density estimation has long been a fundamental open issue in statistics and pattern classification. Implicitly or explicitly, it is at the core of statistical pattern recognition and unsupervised learning [1]. Applications embrace data compression and model selection [2], coding [3], and bioinformatics [4]. Density estimation was applied to the modeling of sequences [5,6] and structured data [7,8], as well. Finally, the task of estimating conditional probability distributions is fundamental to the broad area of probabilistic graphical models [9,10]. Statistical parametric and non-parametric techniques are available to practitioners [1], but they suffer from several significant shortcomings [11]. In fact, parametric techniques require a strong assumption on the form of the probability density function (pdf) at hand, while non-parametric approaches are memory-based (i.e., prone to overfitting), overly complex in time and space, and unreliable over small data samples. Consequently, density estimation via artificial neural networks (ANN) has been receiving increasing attention from researchers. Despite the ease of training ANNs for Bayes posterior probability estimation aimed at pattern classification [12,13], learning density functions raises problems entailed by the intrinsically unsupervised task and, above all, to the requirement of satisfying (at least numerically) the axioms of probability [14]. In particular, the integral of the function realized by the ANN shall be equal to one.

Traditional ANN-based attempts to tackle the pdf estimation problem could not overcome some major shortcomings [11]. First of all, when applied to pdf estimation, the so-called “probabilistic neural network” [15] reduces to a neural version of the Parzen window estimator [1], with a hidden neuron per each Parzen kernel, such that it inherits the aforementioned drawbacks of statistical non-parametric techniques. The maximum-likelihood framework [16] does not offer explicit solutions to the numeric computation of the integral of the neural estimate, and its application has been limited to univariate distributions. The extension of that framework handed out by [17] does not offer solutions to such issues, focusing only on the rate of convergence of the algorithm. A parametric approach is presented in [18], in the form of a self-organizing mixture network. The latter realizes a density estimator based on a self-organizing map (SOM) [19] lattice of units associated with a neuron-specific component density (e.g., a Gaussian distribution), resulting in a mixture of such component pdfs. Although the SOM-like learning procedure is original and effective, the model (being parametric) suffers from the aforementioned limitations of statistical parametric techniques. The approaches based on indirect pdf estimation by differentiation of an estimated cumulative distribution function [20,21] end up violating the axioms of probability (they do not ensure that a proper pdf is obtained) and can hardly be applied to multivariate data, as well. Given these drawbacks of traditional approaches, researchers have recently proposed more robust ANN-based techniques for multivariate density estimation [22,23,24]. In particular, soft-constrained neural networks for pdf estimation (SC-NN-4pdf) [25] realize a non-parametric neural density estimator that explicitly satisfies the second Kolmogorov axiom. The SC-NN-4pdf was shown to outperform both statistical estimation techniques (parametric and non-parametric) and its neural competitors over difficult multivariate, multimodal pdf estimation tasks [25]. No clear rationale behind such an empirical evidence has been put forward so far, neither has it any analysis of the fundamental theoretical properties of the SC-NN-4pdf. To this end, hereafter we deliver a mathematical study of the major theoretical properties of the SC-NN-4pdf. After introducing the pdf estimation setup (Section 2), followed by a concise review of the fundamentals of the model and its training algorithm (Section 2.1), the formal analysis is devised in Section 3. We first offer a formal statement of the class of pdfs that can be modeled by SC-NN-4pdf to any degree of precision (Section 3.1). Then, in Section 3.2 we hand out a proof of asymptotic convergence in probability under mild conditions of the SC-NN-4pdf training algorithm for a popular class of architectures, namely multilayer perceptrons (MLP) with logistic sigmoid activation functions. Conclusions are drawn in Section 4.

2. Materials and Methods

Let us write $\mathcal{T}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$ to represent the sample of n multivariate random vectors, (${\mathbf{x}}_k\in {\mathbb{R}}^d$ for $k=1,\dots ,n$) independently drawn from the unknown pdf $p\left(\mathbf{x}\right)$. A feed-forward neural network (FFNN) with d input units, one output unit, one or more hidden layers is used to learn $p\left(\mathbf{x}\right)$ from $\mathcal{T}$ in an unsupervised fashion. We write $f_i\left(a_i\right)$ to represent the activation function associated with the generic neuron i in the FFNN. The quantity $a_i$ is computed as $a_i={\sum }_j{w}_{ij}{f}_j\left(a_j\right)$ where $w_{ij}$ is the weight of the connection between neurons j and i. We write $\mathbf{w}$ to represent the weight vector embracing all the adaptive parameters of the FFNN (including the biases of sigmoids). When the FFNN is fed with a generic input $\mathbf{x}=(x_1,\dots ,x_d)$, the i-th input neuron realizes the identity function $f_i\left(a_i\right)=f_i\left(x_i\right)=x_i$. On the contrary, any nonlinearity can be associated with the hidden neurons, but for the intents and purposes of this paper we will assume that logistic sigmoids are used. Finally, the output neuron shall have an activation function whose counterdomain matches the notion of pdf, namely the range $[0,+\infty )$. There are different choices for $f_i\left(a_i\right)$ that end up satisfying this requirement, e.g., $f_i\left(a_i\right)=exp\left(a_i\right)$. In this paper, we resort to the ReLU activation function, that is $f_i\left(a_i\right)=max(a_i,0)$, which allows for both the respect of the axioms of probability and the nice modeling properties offered by linear combinations of sigmoids.

In summary, the FFNN computes a function $\phi (\mathbf{x},\mathbf{w})$ of its input $\mathbf{x}$. In [25], without loss of generality, it is assumed that the random samples of interest are limited to a compact $S\subset {\mathbb{R}}^d$ (therefore, henceforth S can be treated as the domain of $\phi (\mathbf{x},\mathbf{w})$). As explained in [25], any data normalization method may be applied to the aim of satisfying this requirement. The training algorithm presented in [25] revolves around a learning rule capable of adapting $\mathbf{w}$ given $\mathcal{T}$ such that a proper estimate of $p\left(\mathbf{x}\right)$ is achieved via $\phi (\mathbf{x},\mathbf{w})$. To this end, two purposes are pursued: (1) capitalizing on the information brought by $\mathcal{T}$ so as to approximate the unknown pdf; (2) holding the FFNN back from reaching degenerate solutions, such that the constraint ${\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}=1$ holds true. Section 2.1 reviews the fundamentals of the training algorithm presented in [25], which actually reaches both of these purposes.

2.1. Review of the Fundamentals of the Training Algorithm

The present Section summarizes the main points of the SC-NN-4pdf training algorithm, as handed out in [25]. In so doing, the notation used in the remainder of the paper is introduced, and a common ground is fixed that allows the reader to make sense of the theoretical analysis presented in Section 3. The estimate $\tilde{p}(·)$ of the unknown pdf $p(·)$ is defined as

$$\tilde{p}(\mathbf{x},\mathbf{w})=\frac{\phi (\mathbf{x},\mathbf{w})}{{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}}$$

(1)

for all $\mathbf{x}$ in S. The FFNN training algorithm is built on non-parametric statistics (see [1]). Given a generic pattern $\widehat{\mathbf{x}}\in \mathcal{T}$, let us consider a d-ball $B(\widehat{\mathbf{x}},\mathcal{T})$ centered on $\widehat{\mathbf{x}}$ and with the minimum radius $r(\widehat{\mathbf{x}},\mathcal{T})$ s.t. $\left|B(\widehat{\mathbf{x}},\mathcal{T})\cap \mathcal{T}\right|=k_n+1$ where $k_n=\lfloor k\sqrt{n}\rfloor$ and $k\in \mathbb{N}$ is a hyperparameter. Please note that $\widehat{\mathbf{x}}$ is in $B(\widehat{\mathbf{x}},\mathcal{T})\cap \mathcal{T}$ by construction, therefore a proper estimate of $p(·)$ over $\widehat{\mathbf{x}}$ shall not involve the latter in the computation: this is the rationale behind considering $k_n+1$ patterns within the ball instead of $k_n$.

We write P to represent the probability that a generic pattern drawn from $p\left(\mathbf{x}\right)$ is in $B(\widehat{\mathbf{x}},\mathcal{T})$. By definition of pdf, $P={\int }_{B(\widehat{\mathbf{x}},\mathcal{T})}p\left(\mathbf{x}\right)d\mathbf{x}$. It is seen that $P\simeq k_n/n$ and that ${\int }_{B(\widehat{\mathbf{x}},\mathcal{T})}p\left(\mathbf{x}\right)d\mathbf{x}\simeq p\left(\widehat{\mathbf{x}}\right)V\left(B(\widehat{\mathbf{x}},\mathcal{T})\right)$ where $V(.)$ denotes the volume of its argument. As a consequence, $p\left(\widehat{\mathbf{x}}\right)\simeq \frac{k_n/n}{V\left(B(\widehat{\mathbf{x}},\mathcal{T})\right)}$. Therefore, the SC-NN-4pdf training algorithm consists of a gradient-descent minimization of the unsupervised criterion function

$$\mathcal{L}(\mathcal{T},\mathbf{w})=\frac{1}{2}\sum _{{\mathbf{x}}_i\in \mathcal{T}}{\left(\frac{k_n/n}{V\left(B({\mathbf{x}}_i,\mathcal{T})\right)}-\tilde{p}({\mathbf{x}}_i,\mathbf{w})\right)}^2+\frac{\rho }{2}{\left(1-{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}\right)}^2$$

(2)

Regarding $\mathbf{w}$ (Equation (2) represents a batch criterion, although a stochastic approach can be taken as well by minimizing on-line a similar pattern-wise criterion upon presentation of individual training patterns one at a time [25]). The first term in the definition of $\mathcal{L}(\mathcal{T},\mathbf{w})$ aims at normalized FFNN outputs that approximate the aforementioned non-parametric estimate of $p(·)$. The second term results in a “soft” constraint enforcing a unit integral of $\tilde{p}(\mathbf{x},\mathbf{w})$ on S. The quantity $\rho \in {\mathbb{R}}^{+}$ weights the relative contribution from the constraint. In practice, $\rho$ is realized as a vanishing quantity, such that $\rho \to 0$ as long as training proceeds, i.e., as long as the function realized by the FFNN approaches the pdf sought (a similar asymptotic convergence of $\rho$ to zero is exploited shortly in the following theoretical analysis). The gradient-descent learning rule for the generic parameter w is devised in [25]. The learning rule prescribes a weight modification $\Delta w$ in the form $\Delta w=-\eta \frac{\partial \mathcal{L}(.)}{\partial w}$, where $\eta \in {\mathbb{R}}^{+}$ is the learning rate. To calculate the partial derivative of $\mathcal{L}(\mathcal{T},\mathbf{w})$ regarding w we need to take the derivatives of both terms on the right of Equation (2). As shown in [25], the derivative of the first term can be written as

$$\begin{array}{ll}& \frac{\partial }{\partial w}\left\{\frac{1}{2}{\left(\frac{k_n/n}{V\left(B(\widehat{\mathbf{x}},\mathcal{T})\right)}-\frac{\phi (\widehat{\mathbf{x}},\mathbf{w})}{{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}}\right)}^2\right\}=\hfill \\ =& -\frac{1}{{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}}\left(\frac{k_n/n}{V\left(B(\widehat{\mathbf{x}},\mathcal{T})\right)}-\frac{\phi (\widehat{\mathbf{x}},\mathbf{w})}{{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}}\right)·\hfill \\ & \left\{\frac{\partial \phi (\widehat{\mathbf{x}},\mathbf{w})}{\partial w}-\frac{\phi (\widehat{\mathbf{x}},\mathbf{w})}{{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}}{\int }_S\frac{\partial \phi (\mathbf{x},\mathbf{w})}{\partial w}d\mathbf{x}\right\}\hfill \end{array}$$

(3)

where Leibniz rule was applied. Similarly, the derivative of the second term can be written as [25]

$$\begin{array}{c}\hfill \frac{\partial }{\partial w}\left\{\frac{\rho }{2}{\left(1-{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}\right)}^2\right\}=\\ \hfill =-\rho \left(1-{\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}\right){\int }_S\frac{\partial \phi (\mathbf{x},\mathbf{w})}{\partial w}d\mathbf{x}.\end{array}$$

(4)

In the equations, the derivative $\frac{\partial \phi (\mathbf{x},\mathbf{w})}{\partial w}$ of the FFNN output regarding one of its parameters is obtained via backpropagation (BP), as usual. Let $w=w_{ij}$ represent the weight of the connection holding between the j-th neuron in layer $\ell -1$ and the i-th neuron in layer ℓ. We can write $\frac{\partial \phi (\mathbf{x},\mathbf{w})}{\partial w}={\delta }_i{f}_j\left(a_j\right)$ where ${\delta }_i=f_i^{\prime }\left(a_i\right)$ if ℓ represents the output layer, or ${\delta }_i=\left({\sum }_{u\in \ell +1}{w}_{ui}{\delta }_u\right)f_i^{\prime }\left(a_i\right)$ otherwise, provided that the deltas for the neurons in layer $\ell +1$ had already been determined as prescribed by plain BP. Extension of the algorithm to the other adaptive parameters of the FFNN (e.g., the bias values) is straightforward.

To compute the right-hand side of Equations (3) and (4), efficient algorithms for the computation of ${\int }_S\phi (\mathbf{x},\mathbf{w})d\mathbf{x}$ and ${\int }_S\frac{\partial }{\partial w}\phi (\mathbf{x},\mathbf{w})d\mathbf{x}$ are presented in [25] that scale properly with the dimensionality of the data and the size of the network. Eventually, the resulting overall algorithm takes the form of an iterative application of BP-based gradient-descent steps where the deltas at the output layer are properly defined, while the backpropagation of the deltas downward through the network takes place in a plain BP fashion. An illustrative example of the behavior of the SC-NN-4pdf is given in Figure 1. The figure represents the estimate obtained from a sample of $n=1000$ random observations drawn from a mixture of three generalized extreme value (GEV) pdfs with different priors and parameters. The SC-NN-4pdf architecture and the hyperparameters are the same we used in [25] for the experiments with univariate data.

3. Results

We hereafter present the results of the theoretical analysis of SC-NN-4pdfs. Section 3.1 investigates the modeling capabilities of the machine, that is the class of pdfs that can actually be modeled by the machine to any degree of precision (an investigation that turns out to be straightforward in the light of [26,27]). Section 3.2 proves a Theorem of asymptotic convergence in probability of the SC-NN-4pdf training algorithm to the pdf sought under mild conditions. Both Sections rely on established results on the approximation capabilities of ANNs [26,27,28], as well as on non-parametric statistics [1,29,30].

3.1. Modeling Capabilities

This Section shows that for any nonpaltry pdf $p(·)$ over X [11] and any $ϵ\in {\mathbb{R}}^{+}$ a SC-NN-4pdf exists which computes ${\phi }^{\left(ϵ\right)}(·)$ such that the corresponding density estimate ${\tilde{p}}^{\left(ϵ\right)}(·)=\frac{{\phi }^{\left(ϵ\right)}(·)}{{\int }_X{\phi }^{\left(ϵ\right)}(·)d\mathbf{x}}$ approximates $p(·)$ over X with precision $ϵ$. In qualitative terms, nonpaltry pdfs form the class of “interesting” pdfs $p:{\mathbb{R}}^d\to \mathbb{R}$ for which there is a compact subset X of $I_d$ (where $I_d={[0,1]}^d$) covering those finite regions of ${\mathbb{R}}^d$ s.t. the integral of any such $p(·)$ over ${\mathbb{R}}^d\backslash X$ is $<{ϵ}_p$ for an arbitrarily small ${ϵ}_p\in {\mathbb{R}}^{+}$. In practice, whenever the support of a pdf $p(·)$ of interest is not a subset of $I_d$, we expect that any suitable data normalization method can be applied so as to restrict the support of $p(·)$ such that ${\int }_{{\mathbb{R}}^d\backslash X}p\left(\mathbf{x}\right)d\mathbf{x}<{ϵ}_p$. On the other hand, if the support of $p(·)$ is a subset of $I_d$ then it can be extended to the whole $I_d$ by setting $p\left(\mathbf{x}\right)=0$ for all $\mathbf{x}\in I_d\backslash X$ (note that this does not affect satisfaction of the axioms of probability). These notions are readily formalized in terms of the following, mildly stricter definition [11]:

Definition 1.

Let $X\subseteq I_d$ be compact. A continuous pdf $p:{\mathbb{R}}^d\to \mathbb{R}$ such that ${\int }_{X}p\left(\mathbf{x}\right)d\mathbf{x}=1$ is said to be nonpaltry over X.

We write $\mathcal{P}\left(X\right)$ to represent the set of nonpaltry pdfs over X. As observed in [11], $\mathcal{P}\left(X\right)$ is not a linear space but a linear closure$\widehat{\mathcal{P}}\left(X\right)$ of $\mathcal{P}\left(X\right)$ can be defined as $\widehat{\mathcal{P}}\left(X\right)=\{ap(·)+bq(·)\mid a,b\in \mathbb{R},p(·),q(·)\in \mathcal{P}\left(X\right)\}$. The universe $\widehat{\mathcal{P}}\left(X\right)$ is actually a linear subspace of $\mathcal{C}\left(X\right)$, that is the set of all continuous, real-valued functions on X. Furthermore, if $f(·)\in \mathcal{C}\left(X\right)$, $\alpha ={\int }_{X}f\left(\mathbf{x}\right)d\mathbf{x}$ and $\alpha \ne 0$, then $p_f(·)=f(·)/\alpha$ is in $\widehat{\mathcal{P}}\left(X\right)$, i.e., any function in $\mathcal{C}\left(X\right)$ can be written as a linear combination of functions in $\widehat{\mathcal{P}}\left(X\right)$. Therefore, $\widehat{\mathcal{P}}\left(X\right)$ turns out to be a Banach space with Chebyshev norm, as well. Then, since the functions in $\mathcal{P}\left(X\right)$ are bounded (in fact, they are continuous over a closed domain), the formal analysis presented in [26] applies, proving that for any nonpaltry pdf $p(·)$ over X at least one SC-NN-4pdf exists that computes ${\phi }^{\left(ϵ\right)}(·)$ s.t. $\parallel {\tilde{p}}^{\left(ϵ\right)}{(·)-p(·)\parallel }_{\infty ,X}<ϵ$ for any $ϵ\in {\mathbb{R}}^{+}$ (from now on we write ${\parallel .\parallel }_{\infty ,X}$ to represent the Chebyshev norm over X). In point of fact, a SC-NN-4pdf computing such ${\phi }^{\left(ϵ\right)}(·)$ is found in the class of multilayer perceptrons (MLP) with only one hidden layer with logistic sigmoid activation functions.

While Cybenko’s analysis [26] revolves around the uniform norm, the arguments put forward in [27] entail likewise conclusions as pertains the $L^l$ norm on X for $l=1,2,\dots$. In the following, we make the dependence of this norm on X explicit by writing $L^l\left(X\right)$. Since the functions in $\mathcal{P}\left(X\right)$ are bounded, non-negative, and l-integrable on X, Corollary 2.2 in [27] proves the following theorem.

Theorem 1.

Given any nonpaltry pdf $p(·)$ over X, a SC-NN-4pdf with one hidden layer of sigmoid activation functions exists that computes ${\phi }^{\left(ϵ\right)}(·)$ s.t. $\parallel {\tilde{p}}^{\left(ϵ\right)}{(·)-p(·)\parallel }_{L^l\left(X\right)}<ϵ$ for any $ϵ\in {\mathbb{R}}^{+}$ and for any $l\in \mathbb{N}$.

Qualitatively speaking, these modeling capabilities of SC-NN-4pdfs (as well as the convergence Theorem presented in the next Section) can be seen as consequences of the fact that since $\parallel {\phi }^{\left(ϵ\right)}{(·)-p(·)\parallel }_{L^1\left(X\right)}$ is arbitrarily small [27] then ${\int }_X{\phi }^{\left(ϵ\right)}(·)d\mathbf{x}$ is arbitrarily close to 1 (see Equation (1)).

3.2. Asymptotic Convergence in Probability

Aside from the theoretical modeling capabilities presented in Section 3.1, the SC-NN-4pdf training process can be shown to actually converge in probability to the true, nonpaltry, and unknown pdf $\widehat{p}(·)\in \mathcal{P}\left(X\right)$ underlying the distribution of the data at hand, under mild condition, for the popular class of MLPs with a single hidden layer with logistic sigmoid activation functions. The formal proof is given hereafter. It is seen that convergence cannot be guaranteed in general, nonetheless convergence to a pdf that is arbitrarily close to $\widehat{p}(·)$ is proven under the mild conditions stated in [31,32].

Let $\phi (·,A,\mathbf{w})$ be the function on X computed by a SC-NN-4pdf with support X (i.e., $S=X$), architecture A, and parameters $\mathbf{w}$ to be learned from the data. Accordingly, $\varphi (·,A,{\mathbf{w}}_n)$ represents the estimate of $\widehat{p}(·)$ realize via the SC-NN-4pdf trained over a given training sample ${\mathcal{T}}_n=\{{\mathbf{x}}_1,\cdots ,{\mathbf{x}}_n\}$ of n independent random vectors drawn from $\widehat{p}(·)$, that is $\varphi (·,A,{\mathbf{w}}_n)=\frac{\phi (.,A,{\mathbf{w}}_n)}{{\int }_X\phi (.,A,{\mathbf{w}}_n)d\mathbf{x}}$. In fact, $\varphi (·,A,{\mathbf{w}}_n)$ is a random quantity that depends on ${\mathcal{T}}_n$, i.e., on the value ${\mathbf{w}}_n$ of the SC-NN-4pdf parameters upon completion of the training stage on ${\mathcal{T}}_n$. Hereafter, we analyze the behavior of SC-NN-4pdfs (that is, of the estimators $\varphi (·,A,{\mathbf{w}}_n)$ they learn to compute) as $n\to \infty$. First, note that $\frac{k_n/n}{V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)}\in \mathcal{C}\left(X\right)$ by construction. Moreover, since $k_n=\lfloor k\sqrt{n}\rfloor$ with $k\in \mathbb{N}$, it is seen that $V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)$ converges asymptotically to zero, that is ${lim}_{n\to \infty }V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)=0$, yet slower than $1/n$, that is ${lim}_{n\to \infty }nV\left(B(\mathbf{x},{\mathcal{T}}_n)\right)=\infty$.

Let $ϵ\in {\mathbb{R}}^{+}$ be any (small) positive real value, which is to say the “degree of precision” of the SC-NN-4pdf solution sought. Since ${\mathcal{T}}_n$ is a random quantity, we study the convergence of the SC-NN-4pdf estimate $\varphi (·,A,{\mathbf{w}}_n)$ to $\widehat{p}(·)$ in mean square (due to Markov’s inequality, this entails convergence in probability). Convergence in mean square prescribes [33] that an integer $n_0$ exists such that for all $n>n_0$, the following inequalities hold:

$$E\left[\parallel \varphi (·,A,{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\right]<ϵ$$

(5)

and

$${\sigma }^2\left[\parallel \varphi (·,A,{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\right]<ϵ$$

(6)

where $E[·]$ denotes the expectation of a random variable (such an expected value is ideally evaluated over all possible samples $\{{\mathbf{x}}_1,\cdots ,{\mathbf{x}}_n\}$ drawn from $\widehat{p}(·)$) and ${\sigma }^2[·]$ is the corresponding variance. Let ${ϵ}_1\in {\mathbb{R}}^{+}$ be s.t. ${ϵ}_1<ϵ$. Theorem 1 guarantees that at least one SC-NN-4pdf with a single hidden layer of logistic sigmoid activation functions exists that computes ${\phi }^{\left({ϵ}_1\right)}(·,A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)})$ s.t. the estimate ${\varphi }^{\left({ϵ}_1\right)}(·,A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)})={\phi }^{\left({ϵ}_1\right)}(·,A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)})/{\int }_X\phi (\mathbf{x},A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)})d\mathbf{x}$ satisfies $\parallel \widehat{p}(·)-{\varphi }^{\left({ϵ}_1\right)}(·,A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)}){\parallel }_{L^2\left(X\right)}<{ϵ}_1$. The actual convergence of a given SC-NN-4pdf depends on its architecture. Therefore (and, since $ϵ$ and ${ϵ}_1$ have been fixed), the present analysis focuses on the class of SC-NN-4pdfs with architecture $A^{\left({ϵ}_1\right)}$ and whose parameters $\mathbf{w}$ are to be learned from the training sample. Of course, these SC-NN-4pdfs compute $\mathbf{w}$-specific functions $\phi (·,A^{\left({ϵ}_1\right)},\mathbf{w})$. In the light of the theoretical results found in [32], let us assume that $ker\left[{\mathcal{X}}_n^{⊺}\right]\cap {\mathcal{D}}_1^{\mathcal{Y}}=\left\{0\right\}$ for all the values of n. Herein, the writing ${\mathcal{X}}_n^{⊺}$ denotes the transpose of the $n\times (d+1)$ matrix yielded by concatenation of the n training vectors ${\mathbf{x}}_1,\dots ,{\mathbf{x}}_n$ (each of them being d-dimensional), where the “$+1$” accounts for the presence of the bias terms in the sigmoid activation functions of the hidden layer. The quantity ${\mathcal{D}}_1^{\mathcal{Y}}$ represents the set of all delta layer traces (refer to [31]) ${\mathcal{Y}}_1=\left[\frac{\partial C_n(\mathbf{w}.)}{\partial a_j\left(m\right)}\right]$ for the hidden layer of $A^{\left({ϵ}_1\right)}$, generated by varying all the parameters $\mathbf{w}$ over their domains, where: $C_n\left(\mathbf{w}\right)=\frac{1}{2}{\sum }_{{\mathbf{x}}_i\in {\mathcal{T}}_n}{\left(\frac{k_n/n}{V\left(B({\mathbf{x}}_i,{\mathcal{T}}_n)\right)}-\phi ({\mathbf{x}}_i,A^{\left({ϵ}_1\right)},\mathbf{w})\right)}^2$ (compare the latter quantity with the first term in Equation (2)), j ranges over all the hidden neurons in $A^{\left({ϵ}_1\right)}$, m ranges over the training patterns ($m=1,\dots ,n$), and $a_j\left(m\right)$ is the input fed to j-th hidden neuron when the SC-NN-4pdf is fed with m-th input pattern. The following convergence Theorem can now be proven to hold true:

Theorem 2.

Let $ϵ\in {\mathbb{R}}^{+}$, ${ϵ}_1\in {\mathbb{R}}^{+}$, and ${ϵ}_1<ϵ$. Let $\phi (·,A^{\left({ϵ}_1\right)},\mathbf{w})$ be the function computed by a SC-NN-4pdf with architecture $A^{\left({ϵ}_1\right)}$ and parameters $\mathbf{w}$. Let us assume that for all the values of n, BP training of the SC-NN-4pdf is applied to the minimization of criterion $\mathcal{L}({\mathcal{T}}_n,\mathbf{w})$ with no early stopping, and that $ker\left[{\mathcal{X}}_n^{⊺}\right]\cap {\mathcal{D}}_1^{\mathcal{Y}}=\left\{0\right\}$ holds true. Let ${\lambda }_{max}$ be the largest eigenvalue of the Hessian of $\phi (·,A^{\left({ϵ}_1\right)},\mathbf{w})$ with respect to $\mathbf{w}$. If the learning rate η satisfies $|1-\eta {\lambda }_{max}|<1$ at each step of BP, then an integer $n_0$ exists s.t., for all $n>n_0$, training the SC-NN-4pdf over ${\mathcal{T}}_n$ converges to parameters ${\mathbf{w}}_n$ with the result that $\varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)$ satisfies Equations (5) and (6).

Proof. 

Under the hypotheses, we prove the existence of an integer $n_0$ s.t. both Equations (5) and (6) hold true for $n>n_0$. First, let us work out the expectation, that is Equation (5). Let ${ϵ}_2\in {\mathbb{R}}^{+}$ s.t. $ϵ={ϵ}_1+{ϵ}_2$. Independently from the specific architecture A of the SC-NN-4pdf, we have:

$$\parallel \varphi (·,A,{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\le {∥\varphi (·,A,{\mathbf{w}}_n)-\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}∥}_{L^2\left(X\right)}+{∥\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}-\widehat{p}(·)∥}_{L^2\left(X\right)}$$

(7)

From Equation (7) we can write:

$$\begin{array}{lll}E\left[\parallel \varphi (·,A,{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\right]& \le & E\left[{∥\varphi (·,A,{\mathbf{w}}_n)-\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}∥}_{L^2\left(X\right)}\right]+\hfill \\ & & E\left[{∥\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}-\widehat{p}(·)∥}_{L^2\left(X\right)}\right]\hfill \end{array}$$

(8)

Given Equation (8), the existence of an integer $n_0$ s.t. Equation (5) holds true for $n>n_0$ is guaranteed if the following conditions hold:

$$E\left[{∥\varphi (·,A,{\mathbf{w}}_n)-\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}∥}_{L^2\left(X\right)}\right]<{ϵ}_1$$

(9)

and

$$E\left[{∥\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}-\widehat{p}(·)∥}_{L^2\left(X\right)}\right]<{ϵ}_2$$

(10)

for all $n>n_0$. It is convenient to focus on Equation (10) first. Bearing in mind that $r(\mathbf{x},{\mathcal{T}}_n)$ is the radius of $B(\mathbf{x},{\mathcal{T}}_n)$, let us define the function $\psi (.)$ of $\mathbf{u}\in X$, $\mathbf{v}\in X$, and ${\mathcal{T}}_n$ as follows:

$$\psi (\mathbf{u},\mathbf{v},{\mathcal{T}}_n)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathbf{v}\in B(\mathbf{u},{\mathcal{T}}_n)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(11)

s.t. ${\int }_X\psi (\mathbf{u},\mathbf{v},{\mathcal{T}}_n)d\mathbf{v}=V\left(B(\mathbf{u},{\mathcal{T}}_n)\right)$. Accordingly, we have

$$\begin{array}{lll}E\left[\frac{k_n/n}{V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)}\right]& =& \frac{1}{n}\sum _{{\mathbf{x}}_j\in {\mathcal{T}}_n}E\left[\frac{\psi (\mathbf{x},{\mathbf{x}}_j,{\mathcal{T}}_n)}{V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)}\right]\hfill \\ & =& {\int }_X\frac{\psi (\mathbf{x},\mathbf{v},{\mathcal{T}}_n)}{V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)}\widehat{p}\left(\mathbf{v}\right)d\mathbf{v}\hfill \end{array}$$

(12)

which is a convolution of the unknown density and of the normalized $\psi (.)$ function. Since $\psi (·)$ is well-behaved by construction and since $V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)\to 0$ for $n\to \infty$ s.t. ${\int }_X\psi (\mathbf{x},\mathbf{v},{\mathcal{T}}_n)d\mathbf{v}\to 0$ too, then in the infinite sample case $\frac{\psi (\mathbf{x},·,{\mathcal{T}}_n)}{V\left(B(\mathbf{x},{\mathcal{T}}_n)\right)}$ converges to a Dirac’s delta centered at $\mathbf{x}$. Therefore, since $\widehat{p}(·)$ is continuous over its support (being nonpaltry over X), then ${lim}_{n\to \infty }E\left[\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}\right]=\widehat{p}(·)$. Consequently, there is an integer $n_p$ such that Equation (10) holds true for all $n>n_p$.

Then, let us consider Equation (9). At any time during training, regardless of n and ${\mathcal{T}}_n$, the SC-NN-4pdf computing $\phi (·,A^{\left({ϵ}_1\right)},\mathbf{w})$ can be set and kept in canonical form (Theorem 3 in [32]) without affecting $\phi (·,A^{\left({ϵ}_1\right)},\mathbf{w})$. In so doing, Theorem 4 in [32] ensures that the loss function $C_n\left(\mathbf{w}\right)$ presents no local minima with respect to $\mathbf{w}$ (since by hypotheses $ker\left[{\mathcal{X}}_n^{⊺}\right]\cap {\mathcal{D}}_1^{\mathcal{Y}}=\left\{0\right\}$), therefore neither $\mathcal{L}({\mathcal{T}}_n,\mathbf{w})$ does (as we let $\rho \to 0$). Consequently, for any value of n, BP training (without early stopping) applied to this SC-NN-4pdf converges to the global minimum ${\mathbf{w}}_n^{*}$ of the criterion $\mathcal{L}({\mathcal{T}}_n,\mathbf{w})$, i.e., ${\mathbf{w}}_n={\mathbf{w}}_n^{*}$, given the fact that the learning rate $\eta$ was chosen (by hypotheses) s.t. $|1-\eta {\lambda }_{max}|<1$ at each step of BP (see [12], Sec. 7.5.1, p. 266).

A paramount consequence of this argument is that if we let ${\overline{\mathcal{L}}}_n\left({\mathbf{w}}^{\left({ϵ}_1\right)}\right)=\frac{1}{2}{\sum }_{{\mathbf{x}}_i\in {\mathcal{T}}_n}{\left({\varphi }^{\left({ϵ}_1\right)}({\mathbf{x}}_i,A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)})-\frac{k_n/n}{V\left(B({\mathbf{x}}_i,{\mathcal{T}}_n)\right)}\right)}^2+\frac{\rho }{2}{\left(1-{\int }_X{\phi }^{\left({ϵ}_1\right)}(\mathbf{x},A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)})d\mathbf{x}\right)}^2$ then $\mathcal{L}({\mathcal{T}}_n,{\mathbf{w}}_n)\le {\overline{\mathcal{L}}}_n\left({\mathbf{w}}^{\left({ϵ}_1\right)}\right)$ for all the values of n. Ergo, bearing in mind that $\widehat{p}(·)={lim}_{n\to \infty }E\left[\frac{k_n/n}{V\left(B(·,{\mathcal{T}}_n)\right)}\right]$ as well as that $A^{\left({ϵ}_1\right)}$ and ${\mathbf{w}}^{\left({ϵ}_1\right)}$ were chosen in such a way that $\parallel \widehat{p}(·)-{\varphi }^{\left({ϵ}_1\right)}(·,A^{\left({ϵ}_1\right)},{\mathbf{w}}^{\left({ϵ}_1\right)}){\parallel }_{L^2\left(X\right)}<{ϵ}_1$, it is seen that

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}E\left[\parallel \mathcal{L}({\mathcal{T}}_n,{\mathbf{w}}_n){\parallel }_{L^2\left(X\right)}\right]& \le & \underset{n\to \infty }{lim}E\left[\parallel {\overline{\mathcal{L}}}_n\left({\mathbf{w}}^{\left({ϵ}_1\right)}\right){\parallel }_{L^2\left(X\right)}\right]\hfill \\ \hfill & <& {ϵ}_1^2\hfill \\ \hfill & <& {ϵ}_1\hfill \end{array}$$

(13)

where we exploited the definition of $L^2$ norm, as well as the fact that the integral of the function realized by the SC-NN-4pdf converges to 1 as $\phi (.)$ approaches a pdf. Equation (13) entails the existence of an integer $n_{\varphi }$ s.t. Equation (9) holds true for all $n>n_{\varphi }$. Since Equation (10) was already proven for all $n>n_p$, and since we fixed ${ϵ}_1$ and ${ϵ}_2$ s.t. ${ϵ}_1+{ϵ}_2=ϵ$, if we let $n_0=max(n_p,n_{\varphi })$ then Equation (8) yields $E\left[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\right]<ϵ$ for all $n>n_0$, as sought.

Then, let us work out the convergence of the variance, aiming at satisfying the asymptotic condition (6). Given the fact that the variance is non-negative, and resorting to one of its computational formulae, it is possible to write

$$\begin{array}{ccc}\hfill 0& \le & {\sigma }^2\left[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\right]\hfill \\ \hfill & =& E\left[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}^2\right]-{\left\{E\left[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}\right]\right\}}^2\hfill \\ \hfill & \le & E\left[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}^2\right]\hfill \end{array}$$

(14)

where we relied on the fact that $\left\{E\right[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}{\left]\right\}}^2$ is non-negative. Following in the footsteps of the analysis we applied earlier to the convergence of the mean, it is seen that $E[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}^2]<ϵ$ for all $n>n_0$ which, as a consequence of Equation (14), proves that also ${\sigma }^2[\parallel \varphi (·,A^{\left({ϵ}_1\right)},{\mathbf{w}}_n)-\widehat{p}(·){\parallel }_{L^2\left(X\right)}]<ϵ$ for all $n>n_0$, as desired. This completes the proof. □

4. Conclusions

Density estimation is at the core of many practical applications rooted in pattern recognition, unsupervised learning, statistical analysis. and coding. Despite its having long been investigated, it is still an open problem. On the one hand, statistical techniques suffer from severe drawbacks. On the other hand, ANN-based pdf estimation algorithms struggle to break-through due to the difficulties posed by the very nature of the unsupervised estimation task and the requirement of satisfying Kolmogorov’s axioms of probability. The topic has recently been receiving an increasing attention from the Community, and some algorithmic attempts to tackle the issues were presented in the literature. In particular, the SC-NN-4pdf was proposed and successfully applied over several univariate and multivariate density estimation tasks, yielding improvements over the established approaches. Despite the empirical evidence stressing its effectiveness, no theoretical analysis of its properties had been carried out so far. This paper contributed filling such a gap, along two major directions: (1) formal identification of the class of pdfs that can be modeled by SC-NN-4pdfs to any degree $ϵ$ of precision, and (2) proof of the asymptotic convergence in probability of the SC-NN-4pdf training algorithm. In particular, it was shown that under the assumption $ker\left[{\mathcal{X}}_n^{⊺}\right]\cap {\mathcal{D}}_1^{\mathcal{Y}}=\left\{0\right\}$, 2-layer SC-NN-4pdfs with proper architecture converge in probability to an expected solution that is close to the true pdf to the desired degree $ϵ$ of precision. These theoretical properties of the SC-NN-4pdf lay the groundwork for understanding the strong estimation capabilities of the present family of ANN-based density estimators. The generality of the class of estimated pdfs (the nonpaltry pdfs) and the asymptotic convergence to the true pdf underlying a data distribution make the SC-NN-4pdf a promising practical tool especially over large data samples. In particular, in classification tasks involving c classes ${\omega }_1,\dots ,{\omega }_c$, the corresponding class-conditional pdfs $p\left(\mathbf{x}\right|{\omega }_1),\dots ,p\left(\mathbf{x}\right|{\omega }_c)$ can be estimated via c class-specific SC-NN-4pdfs that, as n increases, tend to the true pdfs such that the estimates of the class-posterior probabilities $P\left({\omega }_1\right|\mathbf{x}),\dots ,P\left({\omega }_c\right|\mathbf{x})$ computed relying on these pdfs estimates will, in turn, converge to the true class-posteriors, and the maximum-a-posteriori decision rule will end up realizing the ideal Bayes decision rule, i.e., the classifier with minimum probability of error. An even more intriguing application can be found in semi-supervised learning, where a limited subsample of the overall training set is actually supervised, while a huge fraction of the training data is unlabeled. Again, SC-NN-4pdf models of the class-conditional pdfs can be estimated from the supervised subsample first, and then applied to the unsupervised data, where a maximum-likelihood criterion based on the SC-NN-4pdf outputs (due to their being proper pdfs) can be applied in order to label a fraction of the unlabeled patterns, enlarging the supervised subset, and so forth. Finally, since the SC-NN-4pdf training algorithm offers a technique for sampling from the neural network (a step required in the computation of the integrals involved), once trained the SC-NN-4pdf can be applied as a generative model in order to draw new data from the pdf learned. Nevertheless, like most statistical and neural density estimators, SC-NN-4pdfs still suffer from the curse of dimensionality [1]. Although the paper did not offer a general solution to such a problem, the asymptotic convergence for $n\to \infty$ has the obvious practical implication that the curse is less and less dramatic (in probability) as the cardinality of the training set increases. Yet, Theorem 2 proves convergence, but it does not offer insights on the convergence rate, i.e., convergence may be quite slow, especially over high-dimensional feature spaces.