An Upper Bound of the Bias of Nadaraya-Watson Kernel Regression under Lipschitz Assumptions

Abstract

The Nadaraya-Watson kernel estimator is among the most popular nonparameteric regression technique thanks to its simplicity. Its asymptotic bias has been studied by Rosenblatt in 1969 and has been reported in several related literature. However, given its asymptotic nature, it gives no access to a hard bound. The increasing popularity of predictive tools for automated decision-making surges the need for hard (non-probabilistic) guarantees. To alleviate this issue, we propose an upper bound of the bias which holds for finite bandwidths using Lipschitz assumptions and mitigating some of the prerequisites of Rosenblatt’s analysis. Our bound has potential applications in fields like surgical robots or self-driving cars, where some hard guarantees on the prediction-error are needed.

1. Introduction

Nonparametric regression and density estimation have been used in a wide spectrum of applications, ranging from economics [1], system dynamics identification [2,3], and reinforcement learning [4,5,6,7]. In recent years, nonparametric density estimation and regression have been dominated by parametric methods such as those based on deep neural networks. These parametric methods have demonstrated an extraordinary capacity in dealing with both high-dimensional data—such as images, sounds, or videos—and large datasets. However, it is difficult to obtain strong guarantees on such complex models, which have been shown easy to fool [8]. Nonparametric techniques have the advantage of being easier to understand, and recent work overcame some of their limitations by, e.g., allowing linear-memory and sub-linear query time for density kernel estimation [9,10]. These methods allowed nonparametric kernel density estimation to be performed on datasets of ${10}^6$ samples and up to 784 input dimension. As such, nonparametric methods are a suitable choice when one is willing to trade performance for statistical guarantees; and the contribution of this paper is to advance the state-of-the-art on such guarantees.

Studying the error of a statistical estimator is important. It can be used, for example, to tune the hyper-parameters by minimizing the estimated error [11,12,13,14]. To this end, the estimation error is usually decomposed into an estimation bias and variance. When it is not possible to derive these quantities, one performs an asymptotic behavior analysis or a convergence to a probabilistic distribution of the error. While all aforementioned analyses give interesting insights on the error and allow for hyper-parameter optimization, they do not provide any strong guarantee on the error, i.e., we cannot upper bound it with absolute certainty.

Beyond hyper-parameter optimization, we argue that another critical aspect of error analysis is to provide hard (non-probabilistic) bounds of the error for critical data-driven algorithms. We believe that learning agents taking autonomous, data-driven, decisions will be increasingly present in the near future. These agents will, for example, be autonomous surgeons, self-driving cars or autonomous manipulators. In many critical applications involving these agents, it is of primary importance to bound the prediction error in order to provide some technical guarantees on the agent’s behavior. In this paper we derive a hard upper bound of the estimation bias in non-parametric regression with minimal assumptions on the problem. The bound can be readily applied to a wide range of applications.

Specifically, we consider the Nadaraya-Watson kernel regression [15,16], which can be seen as a conditional kernel density estimate. We derive an upper bound of the estimation bias under weak local Lipschitz assumptions. The reason for our choice of estimator falls in its inherent simplicity compared to more sophisticated techniques. The bias of the Nadaraya-Watson kernel regression has been previously studied by [17], and has been reported in a number of related work [18,19,20,21]. The analysis of the bias conducted by Rosenblatt (1969) [17] still remains the main reference for this regression technique. The main assumptions of Rosenblatt’s analysis are $h_n\to 0$ (where n is the number of samples) and $n{h}_n\to \infty$ where $h_n$ is the kernel’s bandwidth. Rosenblatt’s analysis suffers from an asymptotic error $o\left(h_n^2\right)$, which means that for large bandwidths it is not accurate; making it inapplicable to derive a hard upper bound. To the best of our knowledge, the only proposed bound on the bias requires the restrictive assumption that the samples must be placed evenly on a closed interval [22]. In contrast, we derive an upper bound of the bias of the Nadaraya-Watson kernel regression that is valid for a large class of design and for any choice of bandwidth.

We build our analysis on weak Lipschitz assumptions [23], which are milder than the (global) Lipschitz, as we require only $\left|f\right(x)-f(y\left)\right|\le L|x-y|$$\forall y\in \mathcal{C}$ given a fixed x, instead of the classic $\left|f\right(x)-f(y\left)\right|\le L|x-y|$$\forall y,x\in \mathcal{C}$—where $\mathcal{C}$ is the data domain. Lipschitz assumptions are common in different fields, and usually allow a wide family of admissible functions. This is particularly true when the Lipschitz is require for only a subset of the function’s domain (like in our case). Moreover, notice that the classical analysis requires the knowledge of the second derivative of the regression function m, and therefore the continuity of $m^{\prime }$. Our Lipschitz assumption is less restrictive, allowing us to obtain a bias upper bound even for functions like $m\left(x\right)=\left|x\right|$, at points (like $x=0$) where $m^{″}$ is undefined. The Rosenblatt analysis builds on a Taylor expansion of the estimator and therefore when the bandwidth $h_n$ is large, Rosenblatt’s bias analysis tends to provide wrong estimates of the bias, as observed in the experimental section. We consider multidimensional input space, and we apply the bound to a realistic regression problem.

2. Preliminaries

Consider the problem of estimating $\mathbb{E}\left[Y\right|X=\mathbf{x}]$ where $X\sim f_X$ and $Y=m\left(X\right)+\epsilon$, with noise $\epsilon$, i.e., $\mathbb{E}\left[\epsilon \right]=0$. The noise can depend on $\mathbf{x}$, but since our analysis is conducted point-wise for a given $\mathbf{x}$, ${\epsilon }_{\mathbf{x}}$ will be simply denoted by $\epsilon$. Let $m:{\mathbb{R}}^d\to \mathbb{R}$ be the regression function and $f_X$ a probability distribution on X called design. In our analysis we consider $X\in {\mathbb{R}}^d$ and $Y\in \mathbb{R}$. The Nadaraya-Watson kernel estimate of $\mathbb{E}\left[Y\right|X=\mathbf{x}]$ is

$$\begin{array}{c}\hfill \widehat{m}\left(\mathbf{x}\right)=\frac{{\sum }_{i=1}^n{K}_{\mathbf{h}}(\mathbf{x}-{\mathbf{x}}_i)y_i}{{\sum }_{j=1}^n{K}_{\mathbf{h}}(\mathbf{x}-{\mathbf{x}}_j)},\end{array}$$

(1)

where $K_{\mathbf{h}}$ is a kernel function with bandwidth-vector $\mathbf{h}$, the ${\mathbf{x}}_i$ are drawn from the design $f_X$ and $y_i$ from $m\left({\mathbf{x}}_i\right)+\epsilon$. Note that both the numerator and the denominator are proportional to Parzen-Rosenblatt density kernel estimates [24,25]. We are interested in the point-wise bias of such estimate $\mathbb{E}\left[\widehat{m}\left(\mathbf{x}\right)\right]-m\left(\mathbf{x}\right)$. In the prior analysis of [17], knowledge of $m^{\prime },m^{″},f_X,f_X^{\prime }$ is required and $f,m^{\prime }$ must be continuous in a neighborhood of x. In addition, and as discussed in the introduction, the analysis is limited to a one-dimensional design and an infinitesimal bandwidth. We briefly present the classical bias analysis of [17] before introducing our results for clarity of exposition.

Theorem 1.

Classic Bias Estimation [17]. Let $m:\mathbb{R}\to \mathbb{R}$ be twice differentiable. Assume a set ${\{x_i,y_i\}}_{i=1}^n$, where ${\mathbf{x}}_i$ are i.i.d. samples from a distribution with non-zero differentiable density $f_X$. Assume $y_i=m\left({\mathbf{x}}_i\right)+{\epsilon }_i$, with noise ${\epsilon }_i\sim \epsilon \left({\mathbf{x}}_i\right)$. The bias of the Nadaraya-Watson kernel in the limit of infinite samples and for $h\to 0$ and $n{h}_n\to \infty$ is

$$\begin{array}{cc}\hfill \mathbb{E}\left[\underset{n\to \infty }{lim}{\widehat{m}}_n\left(x\right)\right]-m\left(x\right)& =h_n^2\left(\frac{1}{2}{m}^{″}\left(x\right)+\frac{m^{\prime }\left(x\right)f_X^{\prime }\left(x\right)}{f_X\left(x\right)}\right)\int u^{2}K\left(u\right)\mathrm{d}u+o_P\left(h_n^2\right)\hfill \\ \hfill & \approx h_n^2\left(\frac{1}{2}{m}^{″}\left(x\right)+\frac{m^{\prime }\left(x\right)f_X^{\prime }\left(x\right)}{f_X\left(x\right)}\right)\int u^{2}K\left(u\right)\mathrm{d}u.\hfill \end{array}$$

(2)

Note that Equation (2) must be normalized with ${\int }_{-\infty }^{\infty }k\left(u\right)\mathrm{d}u$ when the kernel function does not integrate to one. The $o_P$ term denotes the asymptotic behavior w.r.t. the bandwidth. Therefore, for a larger value of the bandwidth, the bias estimation becomes worse, as is illustrated in Figure 1.

3. Main Result

In this section, we present two bounds on the bias of the Nadaraya-Watson estimator. The first one considers a bounded regression function m (i.e., $\left|m\right(x\left)\right|\le M$), and allows for weak Lipschitz conditions on a subset of the design’s support. Instead, the second bound does not require the regression function to be bounded but only the weak Lipschitz continuity to hold on all of its support. The definition of “weak” Lipschitz continuity will be given below.

To develop our bound on the bias for multidimensional inputs is essential to define some subset of the ${\mathbb{R}}^d$ space. In more detail, we consider an open n-dimensional interval in ${\mathbb{R}}^d$ which is defined as $\Omega ({\mathbf{\tau }}^{-},{\mathbf{\tau }}^{+})\equiv ({\tau }_1^{-},{\tau }_1^{+})\times \cdots \times ({\tau }_d^{-},{\tau }_d^{+})$ where ${\tau }^{-},{\tau }^{+}\in {\overline{\mathbb{R}}}^d$. We now formalize what is meant by weak (log-)Lipschitz continuity. This will prove useful as we need knowledge of the weak-Lipschitz constants of m and $log{f}_X$ in our analysis.

Definition 1.

Weak Lipschitz continuity at$\mathbf{x}$on the set$\mathcal{C}$under the$L_1$-norm.

Let$\mathcal{C}\subseteq {\mathbb{R}}^d$and$f:\mathcal{C}\to \mathbb{R}$. We call f weak Lipschitz continuous at$\mathbf{x}\in \mathcal{C}$if and only if

$$\begin{array}{c}\hfill \left|f\right(\mathbf{x})-f(\mathbf{y}\left)\right|\le L|\mathbf{x}-\mathbf{y}|\forall \mathbf{y}\in \mathcal{C},\end{array}$$

where$|·|$denotes the$L_1$-norm.

Definition 2.

Weak log-Lipschitz continuity at$\mathbf{x}$on the set$\mathcal{C}$under the$L_1$-norm.

Let$\mathcal{C}\subseteq {\mathbb{R}}^d$. We call f weak log-Lipschitz continuous at$\mathbf{x}$on the set$\mathcal{C}$if and only if

$$\begin{array}{c}\hfill |logf(\mathbf{x})-logf(\mathbf{y}\left)\right|\le L|\mathbf{x}-\mathbf{y}|\forall \mathbf{y}\in \mathcal{C}.\end{array}$$

Note that the set$\mathcal{C}$can be a subset of the function’s domain.

It is important to note that, in contrast to the global Lipschitz continuity, which requires $\left|f\right(\mathbf{y})-f(\mathbf{z}\left)\right|\le L|\mathbf{y}-\mathbf{z}|$$\forall \mathbf{y},\mathbf{z}\in \mathcal{C}$, the weak Lipschitz continuity is defined at a specific point $\mathbf{x}$ and therefore allows the function to be discontinuous elsewhere. The Lipschitz assumptions are not very restrictive, and in practice require a bounded gradient. They have been widely used in various fields. Note that when the Lipschitz constants are not known, they can be estimated from the dataset [26]. In the following we list the set of assumptions that we use in our theorems.

A1. 

$f_X$ and m are defined on $\mathrm{Y}\equiv \Omega (\mathbf{x}-{\upsilon }^{-},\mathbf{x}+{\upsilon }^{+})$ and ${\upsilon }^{-},{\upsilon }^{+}\in {\overline{\mathbb{R}}}_{+}^d$;

A2. 

$f_X$ is log weak Lipschitz with constant $L_f$ at $\mathbf{x}$ on the set $\mathcal{D}\equiv \Omega (\mathbf{x}-{\mathit{\delta }}^{-},\mathbf{x}+{\mathit{\delta }}^{-})\subseteq \mathrm{Y}$ and $f_X\left(\mathbf{x}\right)\ge f_X\left(\mathbf{z}\right)$$\forall \mathbf{z}\in Y\backslash \mathcal{D}$ with positive defined ${\mathit{\delta }}^{-},{\mathit{\delta }}^{+}\in {\overline{\mathbb{R}}}_{+}^d$ (note that this implies $f_X\left(\mathbf{y}\right)>0$$\forall \mathbf{y}\in \mathcal{D}$);

A3. 

m is weak Lipschitz with constant $L_m$ at $\mathbf{x}$ on a the set $\mathcal{G}\equiv \Omega (\mathbf{x}-{\mathit{\gamma }}^{-},\mathbf{x}+{\mathit{\gamma }}^{+})\subseteq \mathcal{D}$ with positive defined ${\mathit{\gamma }}^{-},{\mathit{\gamma }}^{+}\in {\overline{\mathbb{R}}}_{+}^d$.

To work out a bound on the bias valid for a wide class of kernels, we must enumerate some assumption and quantify some integrals with respect to the kernel.

A4. 

The multidimensional kernel $K_{\mathbf{h}}:{\mathbb{R}}^d\to \mathbb{R}$ can be decomposed in a product of independent uni-dimensional kernels, i.e., $K_{\mathbf{h}}\left(\mathbf{x}\right)={\prod }_{i=1}^d{k}_i(x/h_i)$ with $k_i:\mathbb{R}\to \mathbb{R}$;

A5. 

the kernels are non-negative $k_i\left(x\right)\ge 0$ and symmetric $k_i\left(x\right)=k_i(-x)$;

A6. 

for every $a,x\in \mathbb{R}$ and $h\ne 0$, the integrals ${\int }_0^a{k}_i\left(x\right)\mathrm{d}x={\Phi }_i\left(a\right)$, ${\int }_0^a{k}_i\left(x\right)e^{-x{L}_f}\mathrm{d}x=B_i(x,L_f)$, ${\int }_0^a{k}_i\left(x\right)x{e}^{-x{L}_f}\mathrm{d}x=C_i(x,L_f)$ are finite (i.e., $<+\infty$).

Assumptions A4–A6 are not really restrictive, and includes any kernel with both finite domain and co-domain, or not heavy-tailed (e.g., Gaussian-like). Furthermore, Axiom A4 allows any independent composition of different kernel functions. In Appendix B we detail the integrals of Axiom A6 for different kernels. Note that when the integrals listed in A6 exist in closed form, the computation of the bound is straightforward, and requires negligible computational effort.

In the following, we propose two different bounds of the bias. The first version considers a bounded regression function ($M<+\infty$), this allows both the regression function and the design to be weak Lipschitz on a subset of their domain. In the second version instead, we consider the case of an unbounded regression function ($M=+\infty$) or an unknown bound M. In this case both the regression function and the design must be weak Lipschitz on the entire domain $\mathrm{Y}$.

Theorem 2.

Bound on the Bias with Bounded Regression Function.

Assuming A1–A3,$\mathbf{h}\in {\mathbb{R}}_{+}^d$a positive defined vector of bandwidths$\mathbf{h}={[h_1,h_2,\cdots ,h_n]}^{⊺}$, $K_h$the multivariate kernel defined in A4–A6, ${\widehat{m}}_n\left(\mathbf{x}\right)$the Nadaraya-Watson kernel estimate using n observations${\{{\mathbf{x}}_i,y_i\}}_{i=1}^n$with$x_i\sim f_X$, $y_i=m\left({\mathbf{x}}_i\right)+{\epsilon }_i$and with noise${\epsilon }_i\sim \epsilon \left({\mathbf{x}}_i\right)$centered in zero ($\mathbb{E}\left[\epsilon \left({\mathbf{x}}_i\right)\right]=0$), $n\to \infty$, and furthermore assuming there is a constant$0\le M<+\infty$such that$\left|m\right(\mathbf{y})-m(\mathbf{z}\left)\right|\le M$$\forall \mathbf{y},\mathbf{z}\in \mathrm{Y}$, the considered Nadaraya-Watson kernel regression bias is bounded by

$$\begin{array}{cc}\hfill & |\mathbb{E}\left[\underset{n\to \infty }{lim}{\widehat{m}}_n\left(\mathbf{x}\right)\right]-m\left(\mathbf{x}\right)|\le \frac{L_m{\displaystyle \sum _{k=1}^d}{\xi }_k({\varphi }_k^{-},{\varphi }_k^{+}){\displaystyle \prod _{i\ne k}^d}\zeta ({\varphi }_i^{-},{\varphi }_i^{+})+M\left({\displaystyle \prod _{i=1}^d}\zeta ({\mathsf{\gamma }}_i^{-},{\mathsf{\gamma }}_i^{+})-{\displaystyle \prod _{i=1}^d}\zeta ({\varphi }_i^{-},{\varphi }_i^{+})+D\right)}{{\prod }_{i=1}^d{\psi }_i({\delta }_i^{-},{\delta }_i^{+})}\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill {\psi }_i(a,b)& =h_i\left(B_i(b/h_i,L_f{h}_i)-B_i(-a/h_i,-L_f{h}_i)\right),{\zeta }_i(a,b)=h_i\left(B_i(b/h_i,-L_f{h}_i)-B_i(-a/h_i,L_f{h}_i)\right),\hfill \\ \hfill {\xi }_i(a,b)& =h_i^2\left(C_i(b/h_i,-L_f{h}_i)+C_i(-a{h}_i,L_f{h}_i)\right),D=\underset{\omega \to +\infty }{lim}{\displaystyle \prod _{i=1}^d}{h}_i{\mathsf{\Phi }}_i\left(\frac{\omega }{h_i}\right)-{\displaystyle \prod _{i=1}^d}{h}_i{\phi }_i,\hfill \end{array}$$

with${\phi }_i={\Phi }_i({\mathsf{\gamma }}_i^{+}/h_i)+{\Phi }_i({\mathsf{\gamma }}_i^{-}/h_i)$, and$0<{\varphi }_i^{-}\le {\mathsf{\gamma }}_i^{-},0<{\varphi }_i^{+}\le {\mathsf{\gamma }}_i^{+}$can be freely chosen to obtain a tighter bound. We suggest${\varphi }_i^{+}=min({\mathsf{\gamma }}_i^{+},M/L_m),{\varphi }_i^{-}=min({\mathsf{\gamma }}_i^{-},M/L_m)$.

In the case where M is unknown or infinite, we propose the following bound.

Theorem 3.

Bound on the Bias with Unbounded Regression Function.

Assuming A1–A3,$\mathbf{h}\in {\overline{\mathbb{R}}}_{+}^d$a positive defined vector of bandwidths$\mathbf{h}={[h_1,h_2,\cdots ,h_n]}^{⊺}$, $K_h$the multivariate kernel defined in A4–A6,${\widehat{m}}_n\left(\mathbf{x}\right)$the Nadaraya-Watson kernel estimate using n observations${\{{\mathbf{x}}_i,y_i\}}_{i=1}^n$with$x_i\sim f_X$, $y_i=m\left({\mathbf{x}}_i\right)+{\epsilon }_i$and with noise${\epsilon }_i\sim \epsilon \left({\mathbf{x}}_i\right)$centered in zero ($\mathbb{E}\left[\epsilon \left({\mathbf{x}}_i\right)\right]=0$),$n\to \infty$, and furthermore assuming that$\mathrm{Y}\equiv \mathcal{D}\equiv \mathcal{G}$, the considered Nadaraya-Watson kernel regression bias is bounded by

$$\begin{array}{cc}\hfill & |\mathbb{E}\left[\underset{n\to \infty }{lim}{\widehat{m}}_n\left(\mathbf{x}\right)\right]-m\left(\mathbf{x}\right)|\le \frac{L_m{\displaystyle \sum _{k=1}^d}{\xi }_k({\upsilon }_k^{-},{\upsilon }_k^{+}){\displaystyle \prod _{i\ne k}^d}{\zeta }_i({\upsilon }_i^{-},{\upsilon }_i^{+})}{{\prod }_{i=1}^d{\psi }_i({\upsilon }_i^{-},{\upsilon }_i^{+})},\hfill \end{array}$$

where${\xi }_k,{\zeta }_i,{\psi }_i$are defined as in Theorem 2.

We detail the proof of both theorems in Appendix A. Note that the conditions required by our theorems are mild, and they allow a wide range of random designs, including and not limited to Gaussian, Cauchy, Pareto, Uniform, and Laplace distributions. In general, every continuously differentiable density distribution is also weak log-Lipschitz in some closed subset of its domain. For example, the Gaussian distribution does not have a finite Lipschitz constant on its entire domain, but there is a finite weak Lipschitz constant on any closed interval. Examples of distributions that are weak log-Lipschitz are presented in

Table 1. Examples of parameters to use for different univariate random design.

Distribution	Density	$\mathbf{Y}$	$\mathcal{D}$	${\mathit{L}}_{\mathit{f}}$
$\mathrm{Laplace}(\mu ,\lambda )$	$\frac{1}{2\lambda }exp\left(-\frac{|x-\mu |}{\lambda }\right)$	$(-\infty ,+\infty )$	$(-\infty ,+\infty )$	${\lambda }^{-1}$
$\mathrm{Cauchy}(\mu ;\mathsf{\gamma })$	${\left(\pi \mathsf{\gamma }+\pi \frac{{(x-\mu )}^2}{\mathsf{\gamma }}\right)}^{-1}$	$(-\infty ,+\infty )$	$(-\infty ,+\infty )$	$\frac{2(z-\mu )}{{\mathsf{\gamma }}^2+{(z-\mu )}^2}$1
$\mathrm{Uniform}(a,b)$	$\left\{\begin{array}{c}\frac{1}{b-a}\mathrm{if}a\le x\le b\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.$	$(a,b)$	$(a,b)$	0
$\mathrm{Pareto}\left(\alpha \right)$	$\left\{\begin{array}{c}\frac{\alpha }{x^{\alpha +1}}\mathrm{if}x\ge 1\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.$	$(1,+\infty )$	$(1,+\infty )$	$1+\alpha$
$\mathrm{Normal}(\mu ,\sigma )$	$\frac{1}{\sqrt{2\pi {\sigma }^2}}exp-\frac{{(x-\mu )}^2}{2{\sigma }^2}$	$(-\infty ,+\infty )$	$(a,b)$	$f_{\mu ,\sigma }(a,b)$

.

4. Analysis

Although the bound applies to different kernel functions, in the following we analyze the most common Gaussian kernel. It worth noting that for $k\left(x\right)=e^{-x^2}$,

$$\begin{array}{cc}\hfill \varphi \left(x\right)& =\frac{\sqrt{\pi }}{2}erf\left(a\right),B(a,c)=\frac{\sqrt{\pi }}{2}{e}^{\frac{c^2}{4}}\left(erf\left(a+\frac{c}{2}\right)-erf\left(\frac{c}{2}\right)\right),\hfill \\ \hfill C(a,c)& =\frac{1}{2}\left(1-e^{-a(a+c)}-cB(a,c)\right).\hfill \end{array}$$

Note that we removed the subscripts from the functions $\psi$, B, and C, as we consider only the Gaussian kernel. To provide a tight bound, we consider many quantities that describe the design’s domain, the Lipschitz constants of the design and of the regression function, the bound of the image of the regression function, and the different bandwidths for each dimension of the space. This complexity results in an effective but poorly readable bound. In this section, we try to simplify the problem and to analyze the behavior of the bound in the limit.

Asymptotic Analysis: Let us consider, for the moment, the case of one-dimension ($d=1$) and infinite domains and co-domains (M unknown and ${\upsilon }^{-}={\upsilon }^{+}={\delta }^{-}={\delta }^{+}={\mathsf{\gamma }}^{-}={\mathsf{\gamma }}^{+}=\infty$). In this particular case, the bound becomes

$$\begin{array}{cc}\hfill & |\mathbb{E}\left[\underset{n\to \infty }{lim}{\widehat{m}}_n\left(\mathbf{x}\right)\right]-m\left(\mathbf{x}\right)|\le L_{m}h\left(\frac{1}{2\sqrt{\pi }exp\frac{L_f^2{h}^2}{4}}+h{L}_f\left(erf\left(\frac{h{L}_f}{2}\right)+1\right)\right)=A_1.\hfill \end{array}$$

As expected, for $h\to 0$ or for $L_m=0$, $B_1=0$. This result is in line with (2) (since $L_m=0\Rightarrow m^{\prime }=0, m^{″}=0$). A completely flat design, e.g., uniformly distributed, does not imply a zero bias. This can be seen either in Rosenblatt’s analysis or by just considering the fact that, notoriously, the Nadaraya-Watson estimator suffers from the boundary bias. When we analyze our bound, we find in fact that ${lim}_{L_f\to 0}{B}_1\propto L_{m}h$. It is also interesting to analyze the asymptotic behavior when these quantities tend to infinity. Similarly to (2), we observe that $A_1$ grows quadratically w.r.t. the kernel’s bandwidth h and it scales linearly w.r.t. the Lipschitz constant of the regression function (which is linked to $m^{\prime }$). A further analysis brings us to the consideration that Rosenbatt’s analysis is linear w.r.t. $\mathrm{d}/\mathrm{d}xlog{f}_X\left(x\right)$ (since $f_X^{\prime }\left(x\right)/f_X\left(x\right)=\mathrm{d}/\mathrm{d}xlog{f}_X\left(x\right)$). Our bound has a similar implication, as the Log-Lipschitz constant is also related to the derivative of the logarithm of the design function, and $A_1=\mathcal{O}\left(L_f\right)$.

Boundary Bias: The Nadaraya-Watson kernel estimator is affected by the boundary bias. The boundary bias is an additive bias term affecting the estimation in the region close to the boundaries of the design’s domains. Since in our framework, we can consider a closed domain of the design, we can also see what is happening close to the border. Let us consider still a one-dimensional regression, but this time ${\upsilon }^{-}\to 0$, ${\upsilon }^{-}={\delta }^{-}={\mathsf{\gamma }}^{-}$ and ${\upsilon }^{+}={\delta }^{+}={\mathsf{\gamma }}^{+}=\infty$. In this case, we obtain

$$\begin{array}{cc}\hfill & |\mathbb{E}\left[\underset{n\to \infty }{lim}{\widehat{m}}_n\left(\mathbf{x}\right)\right]-m\left(\mathbf{x}\right)|\le \frac{L_{m}h}{\sqrt{\pi }{e}^{\frac{L_f^2{h}^2}{4}}\left(1-erf\left(\frac{h{L}_f}{2}\right)\right)}+\frac{L_m{L}_f{h}^2\left(1+erf\left(\frac{h{L}_f}{2}\right)\right)}{2-2erf\left(\frac{h{L}_f}{2}\right)}=A_2.\hfill \end{array}$$

Keeping in account that $\mathrm{d}/\mathrm{d}xerf\left(x\right)\propto e^{-x}$ and using L’Hôpital’s rule, we can observe that the bound is now exponential w.r.t. h and $L_f$, i.e., $A_2=\mathcal{O}\left(e^{h{L}_f}\right)$, which implies that it is more “sensible” to higher bandwidths or less smooth design. Interestingly, instead, the bounds maintains its linear relation w.r.t. $L_m$.

Dimensionality: Let us now study the multidimensional case, supposing that each dimension has same bandwidth and same values for the boundaries. In this case,

$$\begin{array}{cc}\hfill & |\mathbb{E}\left[\underset{n\to \infty }{lim}{\widehat{m}}_n\left(\mathbf{x}\right)\right]-m\left(\mathbf{x}\right)|\le \frac{d\xi ({\upsilon }^{-},{\upsilon }^{+})\zeta {({\upsilon }^{-},{\upsilon }^{+})}^{d-1}}{\psi {({\upsilon }^{-},{\upsilon }^{+})}^d}\propto d{\left(\frac{\zeta ({\upsilon }^{-},{\upsilon }^{+})}{\psi ({\upsilon }^{-},{\upsilon }^{+})}\right)}^{d-1}.\hfill \end{array}$$

Therefore, the bound scales exponentially w.r.t. the dimension. We observe an exponential behavior when $\mathbf{x}$ is close to the boundary of the design’s domain. In these regions, in fact the ratio $\zeta ({\upsilon }^{-},{\upsilon }^{+})/\psi ({\upsilon }^{-},{\upsilon }^{+})$ is particularly high. Of course, when the aforementioned ratio tends to one, the linearity w.r.t. d is predominant.

We can conclude the analysis by noticing that our bound has similar limiting behavior with the Rosenblatt’s analysis, but it provides a hard bound on the bias.

5. Numerical Simulation

In this section, we provide three numerical analyses of our bounds on the bias (the code of our numerical simulations can be found at http://github.com/SamuelePolimi/UpperboundNWBias). The first analysis of our method is conducted on uni-dimensional input spaces for display purposes and aims to show the properties of our bounds in different scenarios. The second analysis aims instead at testing the behavior of our method on a multidimensional input space. The third analysis emulates a realistic scenario where our bound can be applied.

Uni-dimensional Analysis: We select a set of regression functions with different Lipschitz constants and different bounds,

• $y=sin\left(5x\right)$; $L_m=5$ and $M=1$,
• $y=logx$ which for $\mathcal{G}\equiv \Omega (-1,+\infty )$ has $L_m=1$ and $M=+\infty$,
• $y={60}^{-1}logcosh60x$ which has $L_m=1$, is unbuounded, and has a particularly high second derivative in $x=0$, with $m^{″}\left(0\right)=60$,
• $y=\sqrt{x^2+1}$ which has $L_m=1$ and is unbounded.

A zero-mean Gaussian noise with standard deviation $\sigma =0.05$ has been added to the output y. Our theory applies to a wide family of kernels. In this analysis we consider a Gaussian kernel, with $k\left(x\right)=e^{-x^2}$, a box kernel, with $k\left(x\right)=I\left(x\right)$, and a triangle kernel, with $k\left(x\right)=I\left(x\right)(1-|x\left|\right)$ with

$$\begin{array}{c}\hfill I\left(x\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\left|x\right|\le 1,\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

We further analyze the aforementioned kernels in Appendix B. In order to provide as many different scenarios as possible we also used the distributions from Table 1, using therefore both infinite domain distributions, such as Cauchy and Laplace, and finite domain such as Uniform. In order to numerically estimate the bias, we approximate $E\left[{\widehat{m}}_n\left(x\right)\right]$ with an ensemble of estimates $N^{-1}{\sum }_{j=1}^N{\widehat{m}}_{n,j}\left(x\right)$ where each estimate ${\widehat{m}}_{n,j}$ is built on a different dataset (drawn from the same distribution $f_X$). In order to “simulate” $n\to \infty$ we used $n={10}^5$ samples, and to obtain high confidence of the bias’ estimate, we used $N=100$ models.

In this section we provide some simulations of our bound presented in Theorems 2 and 3, and for the Rosenblatt’s case we use

$$\left|h_n^2\left(\frac{1}{2}{m}^{″}\left(x\right)+\frac{m^{\prime }\left(x\right)f_X^{\prime }\left(x\right)}{f_X\left(x\right)}\right)\int u^{2}K\left(u\right)\mathrm{d}u\right|.$$

Since the Rosenblatt’s bias estimate is not an upper bound, it can happen that the true bias is higher (as well as lower) than this estimate, as it is possible to see in Figure 1. We presented different scenarios, both with bounded and unbounded functions, infinite and finite design domains, and a larger or smaller bandwidth choice. It is possible to observe that, thanks to the knowledge of $f,f^{\prime },m^{\prime },m^{″}$ the Rosenblatt’s estimation of the bias tends to be more accurate than our bound. However, it can happen that it largely overestimates the bias, like in the case of $m\left(x\right)={60}^{-1}logcosh\left(60x\right)$ in $x=0$ or to underestimate it, most often in boundary regions. In contrast, our bound always overestimates the true bias, and despite its lack of knowledge of $f,f^{\prime },m^{\prime },m^{″}$, it is most often tight. Moreover, when the bandwidth is small, both our method and Rosenblatt’s deliver an accurate estimation of the bias. In general, Rosenblatt tends to deliver a better estimate of the bias, but it does not behave as a bound, and in some situations, it also can deliver larger mispredictions. In detail, the plot (a) in Figure 1 shows that with a tight bandwidth, both our method and Rosenblatt’s method achieve good approximations of the bias, but only our method correctly upper bounds the bias. When increasing the bandwidth, we obtain both a larger bias and subsequent larger estimates of the bias. Our method consistently upper bounds the bias, while in many cases, Rosenblatt’s method underestimates it, especially in proximity of boundaries (subplots b, d, e). An interesting case can be observed in subplot (c), where we test the function $m\left(x\right)={60}^{-1}logcosh\left(60x\right)$, which has a high second-order derivative in $x=0$: in this case, Rosenblatt’s method largely overestimates the bias.

The figure shows that our bound can deal with different functions and random designs, being reasonably tight, if compared to Rosenblatt’s estimation, which requires the knowledge of the regression function and the design, and respective derivatives.

Multidimensional Analysis: We want to study if our bounds work in a multidimensional case and how much it overestimates the true bias (therefore, how tight it is). For this purpose, we took a linear function $m\left(\mathbf{x}\right)={\mathbf{1}}^{⊺}\mathbf{x}$ where $\mathbf{1}$ is a column-vector of d ones. This function, for any dimension d, has a Lipschitz constant $L_m=1$ and is unbounded $(M=\infty )$. We set a Gaussian design with zero mean and unit diagonal covariance. Since in higher dimensions, the estimation’s variance grows exponentially [22], we used a large number of samples ($n={10}^6$), and we averaged over $N={10}^5$ independent estimations. In Figure 2 we show how the “true” bias (estimated numerically averaging over a thousand Nadaraya-Watson regressions) and our bound evolve with a growing number of dimensions d. Far from the low-density region $\mathbf{x}=\mathbf{0}$ we notice that the bias tends to have a linear behavior, while close to the boundary the bias tends to be exponential. We can observe that our bound correctly bounds the bias in all the cases.

Realistic Scenario: Let us consider the regression problem of the dynamics of an under-actuated pendulum of length l and mass m. In particular, the state of the pendulum can be described by its angle $\alpha$ and its angular rotation $\dot{\alpha }$. Furthermore, a force u can be applied to the pendulum. The full system is described by,

$$\begin{array}{c}\hfill \ddot{\alpha }=\frac{3}{m{l}^2}u-\frac{3g}{2l}sin(\alpha +\pi ),\end{array}$$

where g is the gravitational acceleration. In practice, when this model is discretized in time, the next state is estimated via numerical integration. Fort this reason, the Lipschitz constant $L_m$ is unknown. Notice that also $m^{\prime }$ and $m^{″}$ required by the Rosenblatt’s analysis are unknown. We estimated the Lipschitz constant $L_m$ by selecting the highest ratio $|y_i-y_j|/|{\mathbf{x}}_i-{\mathbf{x}}_i|$ in the dataset. In our analysis, we want to predict all the states with fixed $\dot{\alpha }=0$, $u=0$, but variable $\alpha \in [-\pi ,\pi ]$. In order to generate the dataset, we use the simulator provided by gym [27]. To train our models, we generate tuples of $\alpha ,\dot{\alpha },u$ by sampling independently each variable from a uniform distribution i.e., $\alpha \sim \mathrm{Uniform}(-\pi ,\pi )$, $\dot{\alpha }\sim \mathrm{Uniform}(-8,8)$ and $u\sim \mathrm{Uniform}(-2,2)$ (hence, $L_f=0$). We fit 100 different models with 50,000 samples. We choose a Gaussian kernel with bandwidth $\mathbf{h}=[0.2,0.2,0.2]$. Figure 3 depicts our bound and the estimated bias. We notice the bias is low and increases close to the boundaries. Our upper bound is tight, but, as expected, it becomes overly pessimistic in the boundary region.

6. Applications

Our bound can be applied alongside any use of the Nadaraya-Watson regression estimate. In this section, we want to point out an interesting application in reinforcement learning. In reinforcement learning, we aim to approximate, from samples, an optimal behavior of an agent acting in a given environment. To do so, one can find a so-called “value-function” that determines how well the agent behaves. This function can be approximated using dynamic programming combined with functional approximators such as neural networks [28], but using also regression-trees [29], Gaussian processes [4], etc. In our prior work [30], we used the presented bound to subsequently bound the bias of the value function estimated via Nadaraya-Watson regression.

7. Conclusions

The Nadaraya-Watson kernel regression is one of the most well-known nonparametric estimator, used in a wide range of applications. Its asymptotic bias and variance are well-known in the literature. However, to the best of our knowledge, such an estimator’s bias has never been bounded before. In this paper, we proposed a hard bound of the bias that requires mild assumptions. Our proposed bound is numerically tight and accurate for a large class of regression functions, kernels, and random designs. We believe that providing hard, non-probabilistic guarantees on a regression error is an essential step in adopting data-driven algorithms in real-world applications. Our future research will focus on extending the bias analysis to a broader class of kernel functions and with a finite-samples analysis.