Multivariate Universal Local Linear Kernel Estimators in Nonparametric Regression: Uniform Consistency

Abstract

In this paper, for a wide class of nonparametric regression models, new local linear kernel estimators are proposed that are uniformly consistent under close-to-minimal and visual conditions on design points. These estimators are universal in the sense that their designs can be either fixed and not necessarily satisfying the traditional regularity conditions, or random, while not necessarily consisting of independent or weakly dependent random variables. With regard to the design elements, only dense filling of the regression function domain with the design points without any specification of their correlation is assumed. This study extends the dense data methodology and main results of the authors’ previous work for the case of regression functions of several variables.

1. Introduction

We consider the following regression model:

$$X_i=f\left({\mathbf{z}}_i\right)+{\epsilon }_i,i=1,\dots ,n,$$

(1)

where $f\left(\mathbf{t}\right)$, $\mathbf{t}={(t_1,\dots ,t_k)}^{\top }\in \mathcal{P}\subset {\mathbb{R}}_{+}^k$, $k\ge 1$, is an unknown real-valued random function (random field) that is continuous on a compact set $\mathcal{P}$ with probability 1. The design $\{{\mathbf{z}}_i;i=1,\dots ,n\}$ consists of a set of observable random k-dimensional vectors taking values in $\mathcal{P}$ with possibly unknown distributions, and it is not necessarily independent or identically distributed. We consider the design as an array of random vectors $\{{\mathbf{z}}_i,i=1,\dots ,n\}$ that may depend on n. In particular, this scheme includes models with a fixed design. It is not assumed that the random function $f\left(\mathbf{t}\right)$ does not depend on the design. Some conditions on the random error $\{{\epsilon }_i;i=1,\dots ,n\}$ are given below.

This paper is devoted to the construction of uniformly consistent (in the sense of convergence in probability) kernel-type estimators for the regression function $f\left(\mathbf{t}\right)$ under minimal assumptions on the dependence of the design points.

Let us review publications related to kernel estimators in the problem under consideration. Here, we do not aspire to present a comprehensive overview of this actively developing area of nonparametric estimation and we will only indicate publications representing the main trends in this direction. The most popular procedures for kernel estimation in the classical case of a nonrandom regression function are apparently related to the Nadaraya–Watson estimators, local polynomial estimators (in particular, local linear estimators), Priestley–Chao estimators, and Gasser–Müller, as well as their modifications (see, for example, books [1,2,3,4,5]).

We are primarily interested in the conditions on the design elements, and in this regard, we note that traditionally in regression problems, it is customary to consider deterministic and random designs separately. Part of this division seems to be due to different approaches to the study of estimators in one or another case. Moreover, initially, there was some specification by design type between kernel-type estimators: the Nadaraya–Watson estimators were studied, for example, only in the case of random design, while the Priestley–Chao and Gasser-Müller estimators are for a one-dimensional nonrandom case. Further in the direction mentioned, many various generalizations were obtained and the above-mentioned bounds were blurred (see, for example, [6,7,8,9]). The Nadaraya–Watson estimators in the case of a nonrandom regular multidimensional design were studied, for example, in [10].

In the case of deterministic design in the vast majority of works, one or another conditions for the design regularity are assumed (for example, see [10,11,12,13,14,15,16,17,18,19]). In papers dealing with random design, independent identically distributed design elements are often considered [1,11,20,21,22,23,24,25,26,27,28,29,30]. But over the past decades, many forms of dependent random variables have been proposed and the corresponding limit theorems for sequences with such properties (as well as probabilistic inequalities) were proved. Development in this direction of probability theory has also fully affected the problems of nonparametric regression so that as design elements, samples from a stationary sequence of random variables are often considered that satisfy one or another known form of dependence. In particular, to construct the design elements, the authors used various mixing conditions, moving average schemes, associated random variables, Markov or martingale properties, etc. In this regard, we note, for example, the papers [1,18,21,31,32,33,34,35,36,37,38,39,40,41]. In the recent papers [42,43,44,45,46,47,48], there are considered nonstationary sequences of design elements with certain special forms of dependence (Markov chains, autoregression, partial sums of moving averages, etc.). The uniform consistency of the kernel-type estimators of the regression function, both in the case of deterministic and random design, was studied by many authors (see, for example, [11,16,18,23,24,25,26,36,38,39,42,43,44,49,50] and the references therein).

It is worth noting that the nature of dependence of real sample data in statistics is difficult to determine if the dependence of observations takes place by the nature of the stochastic experiment. In this regard, the creation and development of new approaches to the statistical analysis of dependent observations of large size that do not satisfy the classical mixing conditions and other known forms of correlation, as well as the study of new forms of dependence, which would be statistically more clear and justified, is of interest not only from a theoretical point of view but is also relevant and especially important for applications.

In the present paper, we continue to develop the concept of dense data proposed in [51,52,53]. In these papers, it is established that to restore the regression function, it is enough to know the noisy values of this function on some dense (in one sense or another) set of points from the regression function domain. We succeeded in constructing new kernel-type estimators by using special sums of weighted observations with the structure of Riemann integral sums that are close to the corresponding integrals in the case of dense design. In this case, the stochastic nature of the design points does not play any role. In the papers of predecessors, in the case of random design, the fulfillment of the dense filling with design points of the regression function domain was satisfied due to various concrete forms of the weak dependence of design points, and the asymptotic properties of the estimators were studied using corresponding probabilistic limit theorems. It is important to emphasize that the new estimators are uniformly consistent not only in the cases of weak dependence mentioned above but also for significantly different correlations of observations when the conditions of ergodicity or stationarity are not satisfied, as well as the classical mixing conditions and other known conditions of dependence (see Example 2 in Section 2 below). In addition, the new estimators have the property of universality regarding the nature of the design: it can be either fixed and not necessarily regular or random, while not necessarily satisfying the traditional correlation conditions.

Note that the proposed estimators belong to the class of local linear kernel estimators, but with some weights other than those used in the classical version. These weights are given by the Lebesgue measure of the elements of some finite random partition of the design sample space, where each element of the partition corresponds to one design point. In this paper, explicit upper bounds on the rate of uniform convergence in the probability of the new estimators to the regression function are obtained simultaneously for both fixed and random designs, while, in contrast to the previously known results, the rate of convergence of our estimators are insensitive to the correlation structure of the design points. The only design characteristic explicitly included in the resulting upper bounds is the minimal radius of the epsilon net formed by the design elements in the regression function domain. This minimal radius is a qualitatively different characteristic compared with previously known ones, in terms of which it is possible to describe sufficient conditions for the uniform consistency of kernel-type estimators. Its advantage over classical weak dependence characteristics is that it is insensitive to the correlation of the design observations. The main thing is that as the size of the observations indefinitely increases, this radius tends to zero in probability. Such a requirement, as we have already noted above, is essentially necessary since only when the design is densely filled with the regression function domain, it is possible to restore it with one or another accuracy.

Previously, similar ideas were implemented in [51,52] for local constant estimators and in [53] for local linear estimators with a univariate design. The estimators from [53] are a particular case (for $k=1$) of the estimators proposed here. Note that the construction of estimators from [53], when the weights of the weighted least squares method are the spacing statistics constructed by the variational series of design points, has no direct generalization to the case of functions of many variables. Similar conditions on design elements were also used in [54,55] for nonparametric regression, and in [56,57,58] for nonlinear regression. In particular, in [54,55], similar conditions were proposed for the Nadaraya–Watson estimators, but these conditions guarantee only pointwise consistency. In [54,55,59], conditions for the uniform consistency of Nadaraya–Watson estimators and classical local linear estimators are obtained in terms of dense data, but the conditions are not as simple as in the present paper and require a more uniform dense filling with design points of the regression function domain than is required in this paper.

Note also that the model (1) assumes that the unknown function $f\left(\mathbf{t}\right)$ is a random process with almost surely continuous trajectories (random field). This more general statement (in comparison with the classical one), is considered partly in order to use the obtained results as an application for estimating the mean and covariance functions of a continuous random field. In connection with the random regression function, we note, for example, recent works [60,61,62,63,64,65,66,67], in which the mean and covariance functions of the random regression function f are estimated in the case of N independent realizations $f_1,\dots ,f_N$ of the function f, there are noisy values of each of these random curves in some set of the design elements (the design can be either common to all trajectories or different from series to series). We consider one of the variants of this problem as an application of the main result. Previously, we considered some statements of this problem in the case of a univariate design (see [51,53,68,69,70]).

To conclude the Introduction, it is worth noting that all kernel methods have the so-called “curse of dimensionality” property: as the dimension of the design points increases, the convergence rates of the kernel methods decrease. Therefore, in this case, kernel methods require relatively large samples to achieve the required accuracy. In this regard, reducing the dimension of the design by selecting relevant factors play an important role. As a guide in this direction, we point out the recent paper by [71], where one can find a detailed bibliography.

This paper has the following structure. Section 2 contains the main results. In Section 3, as a corollary, the problem of estimating the mean function in the dense design case is considered. The proofs of the results are found in Section 4.

Currently, the authors are preparing a continuation of this work for publication that will provide both a computer simulation of the statistical procedures proposed in this work and examples of processing real data.

2. Main Results

2.1. Notation and Main Assumptions

We agree that vectors are column vectors. Vectors are denoted by bold letters, and matrices are in straight capital letters. Denote by $\mathrm{diag}\{d_1,\dots ,d_m\}$ the diagonal matrix of dimension $m\times m$ with relevant elements on the main diagonal. The symbol ^⊤ denotes the transposition of a vector or matrix, and the determinant of some matrix $\mathrm{A}$ is denoted by $det\mathrm{A}$. Denote by ${\Lambda }_k(·)$ the Lebesgue measure in ${\mathbb{R}}^k$. For any vector $\mathbf{x}={(x_1,\dots ,x_k)}^{\top }$, the symbol $\parallel \mathbf{x}\parallel$ means the supnorm in ${\mathbb{R}}^k$, i.e., $\parallel \mathbf{x}\parallel ={max}_{j=1,\dots k}\left|x_j\right|$. For an arbitrary matrix $\mathrm{X}$, as a matrix norm, we consider the norm subordinated to the vector supnorm, i.e., $\parallel \mathrm{X}\parallel ={max}_{l\le k}{\sum }_{m\le k}\left|{\mathrm{X}}_{lm}\right|$, where the symbol ${\mathrm{X}}_{lm}$ here and below denotes the matrix entry at the intersection of the l-th row and m-th column. Notice that we may consider any vector column $\mathbf{x}$ as a matrix of dimension $k\times 1$ and its supnorm $\parallel \mathbf{x}\parallel$ coincides with the matrix norm above. Therefore, we use one symbol $\parallel ·\parallel$ to denote these two norms.

By $d\left(A\right)$, we denote the diameter of a set A, i.e., $d\left(A\right)={sup}_{\mathbf{x},\mathbf{y}\in A}\parallel \mathbf{x}-\mathbf{y}\parallel$. We also need a notation for the modulus of continuity of a function f defined on a unit k-dimensional cube ${[0,1]}^k$:

$${\omega }_f\left(h\right)=\underset{\mathbf{x},\mathbf{y}:\parallel \mathbf{x}-\mathbf{y}\parallel \le h}{sup}|f\left(\mathbf{x}\right)-f\left(\mathbf{y}\right)|.$$

Next, by $O_p\left({\eta }_n\right)$, we denote a random variable ${\zeta }_n$ such that for all $M>0$, one has

$$\underset{n\to \infty }{lim\; sup}\mathbb{P}\left(\right|{\zeta }_n|/{\eta }_n>M)\le \beta \left(M\right),$$

where ${lim}_{M\to \infty }\beta \left(M\right)=0$ and $\left\{{\eta }_n\right\}$ are positive (maybe random or not) variables and the function $\beta \left(M\right)$ does not depend on the parameters of the model under consideration. We agree that throughout what follows, all limits, unless otherwise stated, are taken for $n\to \infty$. In what follows, without loss of generality, we assume that $\mathcal{P}={[0,1]}^k$.

We now formulate the following four main assumptions.

$\left(A_1\right)$ The observations $\{({\mathbf{z}}_i,X_i),i=1,\dots ,n\}$ have the structure (1), where $f\left(\mathbf{t}\right)$, $\mathbf{t}\in {[0,1]}^k$ is an unknown real-valued random function on ${[0,1]}^k$ that is continuous with probability 1. The design points $\{{\mathbf{z}}_i;i=1,\dots ,n\}$ are a set of observable k-dimensional random variables with, generally speaking, unknown distributions and values in ${[0,1]}^k$, not necessarily independent or identically distributed; in this case, the random variables $\left\{{\mathbf{z}}_i\right\}$ may depend on n.

$\left(A_2\right)$ For all $n\ge 1$, the unobservable random errors $\{{\epsilon }_i;i=1,\dots ,n\}$ form a sequence of martingale differences under the condition

$$M_p=\underset{i\le n}{sup}\mathbb{E}{\left|{\epsilon }_i\right|}^p<\infty \mathrm{when}\mathrm{some}p>k\mathrm{and}p\ge 2,$$

where $M_p$ does not depend on n. It is also assumed that $\left\{{\epsilon }_i\right\}$ is independent of $\left\{{\mathbf{z}}_i\right\}$ and the random function $f(·)$; moreover, the random variables $\left\{{\epsilon }_i\right\}$ may depend on n.

$\left(A_3\right)$ The kernel function $K\left(\mathbf{t}\right)$, $\mathbf{t}\in {\mathbb{R}}^k$, vanishes outside the cube ${[-1,1]}^k$ and can be represented as $K\left(\mathbf{t}\right)={\prod }_{j=1}^k{K}_o\left(t_j\right)$ for $\mathbf{t}={(t_1,\dots ,t_k)}^{\top }$, where $K_o(·)$ is a symmetric distribution density with the support $[-1,1]$. We assume that the function $K_o\left(t\right)$ satisfies the Lipschitz condition with constant $1\le L<\infty$ and $K_o(\pm 1)=0$.

In what follows, we need the notation $K_h\left(\mathbf{t}\right)=h^{-k}K\left(h^{-1}\mathbf{t}\right)$. It is clear that $K_h\left(\mathbf{t}\right)$ is a distribution density on ${[-h,h]}^k$.

The following condition is the only limitation on the design.

$\left(A_4\right)$ For each n, there exists a random partition of the set ${[0,1]}^k$ into n Jordan-measurable subsets $\{{\mathcal{P}}_i;i=1,\dots ,n\}$ such that every element of this partition contains exactly one point from the set $\{{\mathbf{z}}_i;i=1,\dots ,n\}$ (the numbering of elements partition is such that ${\mathbf{z}}_i\in {\mathcal{P}}_i$), and ${\delta }_n={max}_{i\le n}d\left({\mathcal{P}}_i\right)\stackrel{p}{\to }0$. Here, it is assumed that the diameters are random variables, i.e., measurable mappings of the probability space.

Remark 1.

Traditionally, a family of nonempty sets ${\mathcal{P}}_1,\dots ,{\mathcal{P}}_n$ forms a partition of the set $\mathcal{P}$ if the elements of the family $\left\{{\mathcal{P}}_i\right\}$ are pairwise disjoint and ${\displaystyle \bigcup _{i=1}^n}{\mathcal{P}}_i=\mathcal{P}$. Let us agree that in $\left(A_4\right)$, it is allowed that the elements of the set $\left\{{\mathcal{P}}_i\right\}$ intersect along sets of zero Lebesgue measure (for example, along boundaries). Such a reservation allows us not to exclude the situation of multiple points in the design. In the case of pairwise different design points, such a reservation is not required. Note also that without the above convention, condition $\left(A_4\right)$ can be formulated as follows: for each n, there exists a random partition of the set ${[0,1]}^k$ into n Jordan-measurable subsets $\{{\mathcal{P}}_i;i=1,\dots ,n\}$ such that ${\delta }_n:={max}_{i\le n}d({\mathcal{P}}_i\cup {\mathbf{z}}_i)\stackrel{p}{\to }0.$

Remark 2.

Condition $\left(A_4\right)$ means that for any n, the set of design points $\{{\mathbf{z}}_i;i=1,\dots ,n\}$ forms an ε-net of ${[0,1]}^k$ for $\epsilon ={\delta }_n$ provided that ${\delta }_n\stackrel{p}{\to }0$.

Remark 3.

Note that condition $\left(A_4\right)$ is satisfied for any nonrandom regular design. If $\left\{{\mathbf{z}}_i\right\}$ are independent and identically distributed and the set ${[0,1]}^k$ is the support of the distribution ${\mathbf{z}}_1$, then condition $\left(A_1\right)$ is also fulfilled. If, additionally, the distribution density $X_1$ is separated from zero on ${[0,1]}^k$, then ${\delta }_n=O\left(\frac{logn}{n^{1/k}}\right)$ with probability 1. If $\{{\mathbf{z}}_i;i\ge 1\}$ is a stationary sequence with an α-mixing condition and the marginal distribution support is ${[0,1]}^k$, then condition $\left(A_4\right)$ is also satisfied. Note that all kinds of dependence of design points $\left\{{\mathbf{z}}_i\right\}$ known to the authors satisfy condition $\left(A_4\right)$, but the fulfillment of this condition is also quite possible for other types of dependence not yet described in the modern literature on nonparametric regression (see Example 2 below).

Let

$$\begin{array}{c}\mathbf{x}={(X_1,\dots ,X_n)}^{\top },{\mathrm{W}}_{\mathbf{t}}=\mathrm{diag}\{K_h(\mathbf{t}-{\mathbf{z}}_1){\Lambda }_k\left({\mathcal{P}}_1\right),\dots ,K_h(\mathbf{t}-{\mathbf{z}}_n){\Lambda }_k\left({\mathcal{P}}_n\right)\},\end{array}$$

(2)

$$\begin{array}{c}{\mathrm{Z}}_{\mathbf{t}}=\left[\begin{array}{cc}1& {(\mathbf{t}-{\mathbf{z}}_1)}^{\top }\\ ⋮& ⋮\\ 1& {(\mathbf{t}-{\mathbf{z}}_n)}^{\top }\end{array}\right].\end{array}$$

(3)

We introduce the following class of estimators for the regression function f:

$$\begin{array}{c}{\widehat{f}}_{n,h}\left(\mathbf{t}\right):={\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathbf{x},\end{array}$$

(4)

where ${\mathbf{e}}_1={(1,0,\dots ,0)}^{\top }$ is the $(k+1)$-dimensional vector such that the first coordinate equals 1 and the other ones vanish.

Remark 4.

It is easy to check that the kernel estimator (4) is the first coordinate of the $(k+1)$-dimensional estimator of the following variant of the weighted least squares method:

$$\begin{array}{c}\underset{a,\mathbf{b}}{min}\sum _{i=1}^n{\left(X_i-\left(a+{\mathbf{b}}^{\top }(\mathbf{t}-{\mathbf{z}}_i)\right)\right)}^2{K}_h(\mathbf{t}-{\mathbf{z}}_i){\Lambda }_k\left({\mathcal{P}}_i\right).\end{array}$$

(5)

Thus, the proposed class of estimators in a certain sense (in fact, by construction) is close to the classical multidimensional local linear kernel estimators, but in weighted least squares (5), we use slightly different weights.

2.2. The Main Theorem, Corollaries, and Examples

The following theorem is the main result of this paper. It allows us to construct a confidence region (tube) for an unknown regression function.

Theorem 1.

Let conditions $\left(A_1\right)$–$\left(A_4\right)$ be satisfied. Then, for any fixed $h\in (0,1/2]$ with probability 1, the following relation is valid:

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\widehat{f}}_{n,h}\left(\mathbf{t}\right)-f\left(\mathbf{t}\right)|\le C_1^{*}{\omega }_f\left(h\right)+{\zeta }_n\left(h\right),\end{array}$$

(6)

with a nonnegative random variable ${\zeta }_n\left(h\right)$ such that

$$\begin{array}{c}\mathbb{P}\left({\zeta }_n\left(h\right)>y\right)\le C_2^{*}{y}^{-p}{h}^{-k(p/2+1)}\mathbb{E}\left({\delta }_n^{kp/2}\right)+\mathbb{P}({\delta }_n>c^{*}h),\end{array}$$

(7)

where the constants $c^{*}<1$, $C_1^{*}$, and $C_2^{*}$ defined in (36), Lemmas 6 and 8, depend on k and the kernel K; furthermore, the constant $C_2^{*}$ additionally depends on p and $M_p$.

Remark 5.

Let f be a nonrandom regression function. Substitute the following intp (7):

$$y={\left(h^{-k(p/2+1)}\mathbb{E}\left({\delta }_n^{kp/2}\right)\right)}^{1/p}.$$

Applying Markov’s power inequality with exponent $kp/2$ for the second term in (7), it is easy to see that under the conditions of the above theorem,

$${\zeta }_n\left(h\right)=O_p\left(C^{*}{\left(h^{-k(p/2+1)}\mathbb{E}\left({\delta }_n^{kp/2}\right)\right)}^{1/p}\right),$$

(here, $C^{*}$ is a positive constant that depends on k, p, $M_p$, and K) and there exists a solution $h\equiv h_n$ to the equation

$$\begin{array}{c}\mathbb{E}\left({\delta }_n^{kp/2}\right)=h^{k(p/2+1)}{\omega }_f^p\left(h\right).\end{array}$$

(8)

It is clear that this solution tends to zero as n grows. In fact, the quantity $h_n$ minimizes in h the order of smallness of the right-hand side of relation (6). Notice that in virtue of (8), the limit relations ${\left(h_n\right)}^{-k(p/2+1)}\mathbb{E}({\delta }_n^{kp/2})\to 0$ and ${\delta }_n/h_n\stackrel{p}{\to }0$ are valid.

Theorem 1 implies the following two assertions.

Corollary 1.

Let conditions $\left(A_1\right)$–$\left(A_4\right)$ be fulfilled and let $\mathcal{C}$ be a set of equicontinuous nonrandom functions from the space $C\left({[0,1]}^k\right)$. Then,

$${\gamma }_n\left(\mathcal{C}\right)=\underset{f\in \mathcal{C}}{sup}\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\widehat{f}}_{n,{\tilde{h}}_n}\left(\mathbf{t}\right)-f\left(\mathbf{t}\right)|\stackrel{p}{\to }0,$$

where ${\tilde{h}}_n$ is a solution to Equation (8), where the modulus of continuity ${\omega }_f\left(h\right)$ is replaced with the universal modulus ${\omega }_f^{\mathcal{C}}\left(h\right)=\underset{f\in \mathcal{C}}{sup}{\omega }_f\left(h\right)$. Moreover, there is the valid relation

$${\gamma }_n\left(\mathcal{C}\right)=O_p\left({\omega }_f^{\mathcal{C}}\left({\tilde{h}}_n\right)\right).$$

Corollary 2.

If conditions $\left(A_1\right)$–$\left(A_4\right)$ are fulfilled and the modulus of continuity of the random regression function $f\left(\mathbf{t}\right)$, $\mathbf{t}\in {[0,1]}^k$ satisfies with probability 1 the condition ${\omega }_f\left(h\right)\le \zeta \phi \left(h\right)$ for some proper random variable $\zeta >0$ and a positive nonrandom function $\phi \left(h\right)$ with the condition $\phi \left(h\right)\to 0$ as $h\to 0$, then there is the valid limit relation

$$\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\widehat{f}}_{n,{\widehat{h}}_n}\left(\mathbf{t}\right)-f\left(\mathbf{t}\right)|\stackrel{p}{\to }0,$$

where ${\widehat{h}}_n$ is a solution to Equation (8), in which the modulus of continuity ${\omega }_f\left(h\right)$ is replaced with $\phi \left(h\right)$.

Example 1.

${\delta }_n\le \nu n^{-1/k}$, with $\mathbb{E}{\nu }^{kp/2}<\infty$, and ${\omega }_f\left(h\right)\le \zeta h^{\alpha }$ for $\alpha \in (0,1]$ and some proper random variable ζ. Then,

$$h_n=O\left(n^{-\frac{1}{2k(1/p+1/2)+\alpha }}\right)and\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\widehat{f}}_{n,h}\left(\mathbf{t}\right)-f\left(\mathbf{t}\right)|=O_p\left(n^{-\frac{\alpha }{2k(1/p+1/2)+2\alpha }}\right).$$

In particular, if $f(·)=W(·)$ is a Wiener process on $[0,1]$ and the independent identically distributed random variables ${\epsilon }_i$ have a normal distribution with zero mean, then for all sufficiently small $\nu >0$,

$$\underset{t\in [0,1]}{sup}|{\widehat{f}}_{n,h}\left(t\right)-f\left(t\right)|=O_p\left(n^{-1/3+\nu }\right).$$

Here, $k=1$, $\alpha =1/2-{\nu }_1$, and $1/p<{\nu }_2$ for arbitrarily small positive ${\nu }_1$ and ${\nu }_2$.

Example 2.

Let a sequence of bivariate random variables $\{{\mathbf{z}}_i;i\ge 1\}$ be defined by the relation

$${\mathbf{z}}_i={\nu }_i{\mathbf{u}}_i^l+(1-{\nu }_i){\mathbf{u}}_i^r,$$

(9)

where $\left\{{\mathbf{u}}_i^l\right\}$ and $\left\{{\mathbf{u}}_i^r\right\}$ are independent and uniformly distributed on the respective rectangles $[0,1/2]\times [0,1]$ and $[1/2,1]\times [0,1]$, and the sequence $\left\{{\nu }_i\right\}$ does not depend on $\left\{{\mathbf{u}}_i^l\right\}$ and $\left\{{\mathbf{u}}_i^r\right\}$ and consists of Bernoulli random variables with the success probability $1/2$, i.e., the distribution of random variables ${\mathbf{z}}_i$ is an equilibrium mixture of two uniform distributions in the corresponding rectangles. The dependence between the random variables ${\nu }_i$ for any natural i is defined by the equalities ${\nu }_{2i-1}={\nu }_1$ and ${\nu }_{2i}=1-{\nu }_1$. In this case, the random variables $\{{\mathbf{z}}_i;i\ge 1\}$ in (9) form a stationary sequence of random variables uniformly distributed on the square $[0,1]\times [0,1]$, but, say, all known mixing conditions are not satisfied here since for all natural m and n,

$$\begin{array}{c}\mathbb{P}\left({\mathbf{z}}_{2m}\in [0,1/2]\times [0,1],{\mathbf{z}}_{2n-1}\in [0,1/2]\times [0,1]\right)=0.\end{array}$$

On the other hand, it is easy to check that the stationary sequence $\left\{{\mathbf{z}}_i\right\}$ satisfies the Glivenko–Cantelli theorem. This means that for any fixed $h>0$,

$$c_1{h}^k\le lim\underset{\mathbf{t}\in {[0,1]}^k}{sup}\#\{i:\parallel \mathbf{t}-{\mathbf{z}}_i\parallel \le h,1\le i\le n\}\le c_2{h}^k$$

is almost surely uniform in t; here, # is the counting measure and the constants $c_1$ and $c_2$ do not depend on t and h. In other words, the sequence $\left\{{\mathbf{z}}_i\right\}$ satisfies condition $\left(A_4\right)$.

In the general case, according to the scheme of this example, one can construct various sequences of dependent random variables uniformly distributed over $[0,1]\times [0,1]$ based on the choice of various sequences of Bernoulli switches with the conditions ${\nu }_{j_k}=1$ and ${\nu }_{l_k}=0$ for unlimited numbers of indices $\left\{j_k\right\}$ and $\left\{l_k\right\}$, respectively. In this case, condition $\left(A_4\right)$ is also satisfied. But the corresponding sequence $\left\{{\mathbf{z}}_i\right\}$ (not necessarily stationary) may not satisfy the strong law of large numbers. For example, this situation occurs when ${\nu }_j=1-{\nu }_1$ for $j=2^{2k-1},\dots ,2^{2k}-1$, and ${\nu }_j={\nu }_1$ for $j=2^{2k},\dots ,2^{2k+1}-1$, where $k=1,2,\dots$; i.e., we play in which of the rectangles we throw the first point at random, and then alternate the number of throws for each of the two specified rectangles as follows: 2, 4, 8, 16, and so on. Indeed, we introduce the notation $n_k=2^{2k}-1$, ${\tilde{n}}_k=2^{2k+1}-1$, and $S_m={\sum }_{i=1}^m{z}_{1i}$ for ${\mathbf{z}}_i=(z_{1i},z_{2i})$, and we note that for all outcomes that make up the event $\{{\nu }_1=1\}$, the following is fulfilled:

$$\frac{S_{n_k}}{n_k}=\frac{1}{n_k}\sum _{i\in N_{1,k}}{u}_{1i}^l+\frac{1}{n_k}\sum _{i\in N_{2,k}}{u}_{1i}^r,$$

where ${\mathbf{u}}_i^l=(u_{1i}^l,u_{2i}^l)$ and ${\mathbf{u}}_i^r=(u_{1i}^r,u_{2i}^r)$, and $N_{1,k}$ and $N_{2,k}$ are the collections of indices such that the observations $\{{\mathbf{z}}_i,i\le n_k\}$ lie in the rectangles $[0,1/2]\times [0,1]$ or $[1/2,1]\times [0,1]$, respectively. It is easy to see that $\#\left(N_{1,k}\right)=n_k/3$ and $\#\left(N_{2,k}\right)=2\#\left(N_{1,k}\right)$. Hence, $S_{n_k}/n_k\to 7/12$ almost surely as $k\to \infty$ due to the strong law of large numbers for the sequences $\left\{u_{1i}^l\right\}$ and $\left\{u_{1i}^r\right\}$. On the other hand, for all elementary outcomes belonging to the event $\{{\nu }_1=1\}$, as $k\to \infty$,

$$\frac{S_{{\tilde{n}}_k}}{{\tilde{n}}_k}=\frac{1}{{\tilde{n}}_k}\sum _{i\in {\tilde{N}}_{1,k}}{u}_{1i}^l+\frac{1}{{\tilde{n}}_k}\sum _{i\in {\tilde{N}}_{2,k}}{u}_{1i}^r\to \frac{5}{12}$$

(10)

with probability 1, where ${\tilde{N}}_{1,k}$ and ${\tilde{N}}_{2,k}$ are the collections of indices such that the observations $\{{\mathbf{z}}_i,i\le {\tilde{n}}_k\}$, lie in the respective rectangles $[0,1/2]\times [0,1]$ or $[1/2,1]\times [0,1]$. When justifying the convergence in (10), we took into account the fact that $\#\left({\tilde{N}}_{1,k}\right)=(2^{2k+2}-1)/3$ and $\#\left({\tilde{N}}_{2,k}\right)=2n_k/3$, i.e., $\#\left({\tilde{N}}_{1,k}\right)=2\#\left({\tilde{N}}_{2,k}\right)+1$.

Similar conclusions are valid for all outcomes that make up the event $\{{\nu }_1=0\}$.

3. Estimating the Mean and Covariance of a Random Regression Function with a Dense Design

In this section, as an application of Theorem 1, we consider one of the variants of the problem of estimating the mean function of an almost surely continuous random process. The estimation of the mean and covariance functions plays an important role and many recent works were devoted to solving this problem (see, for example, [60,67,72,73] and the references below). As in the classical formulation of the problem of nonparametric regression of the form (1), the design in the problem of estimating the mean and covariance functions is considered to be either random or deterministic. For a random design, as a rule, it is assumed that the design points are independent identically distributed random variables (see, for example, [60,61,62,63,64,65,66,74,75]). In the case of deterministic design, one or another regularity condition is usually used. For example, the nonrandom equidistant design was discussed in [61]. In addition, in the problem of estimating the mean function, it is customary to subdivide the design into certain types depending on the number of design points for a particular trajectory.

The literature focuses on two opposing types of data: the design is in one sense or another “sparse” (for example, the number of design points for each of the realizations of the regression process is uniformly bounded; see [60,61,62,74]), or the design is somewhat “dense” (the number of design elements in each series grows simultaneously with the number of series; see [60,62,65,74]). In Theorem 2 given below, the second of the mentioned types of design is considered, provided only condition $\left(A_4\right)$ is satisfied in each of the independent series. Note that our formulation of the problem of estimating the mean function also includes the situation of a general deterministic design. Mean function estimation approaches that are used for dense or sparse data are often different (see, for example, [73,76]). In the case of a growing number of observations in each series, it is natural to first estimate the random regression function in each series, and then average their estimators over all series (see, for example, [61,65]). This is exactly what we do next (see Formula (12) below) by following this generally accepted approach. The uniform consistency of the mean function estimators was studied, for example, in [60,62,64,66,72].

Therefore, consider the following statement of the problem under consideration. We have N independent copies of model (1) from condition $\left(A_1\right)$:

$$X_{i,j}=f_j\left({\mathbf{z}}_{i,j}\right)+{\epsilon }_{i,j},i=1,\dots ,n,j=1,\dots ,N,$$

(11)

where $f_1\left(\mathbf{t}\right),\dots ,f_N\left(\mathbf{t}\right)$, $\mathbf{t}\in {[0,1]}^k$, are independent identically distributed unobservable random processes with a.s. continuous trajectories, and the collections $\{{\epsilon }_{i,j};i=1,\dots ,n\}$ and $\{{\mathbf{z}}_{i,j};i=1,\dots ,n\}$ for all j satisfy conditions $\left(A_1\right),\left(A_2\right)$, and $\left(A_4\right)$. Here and below in this section, the index j means the copy number of model (1). We define an estimator for the mean function by the equality

$${\overline{f^{*}}}_{N,n,h}\left(\mathbf{t}\right)=\frac{1}{N}\sum _{j=1}^N{\widehat{f}}_{n,h,j}\left(\mathbf{t}\right),$$

(12)

where the estimators ${\widehat{f}}_{n,h,j}\left(\mathbf{t}\right)$ are determined by Formula (4), where the values from (1) are replaced with the corresponding characteristics from (11).

A corollary of Theorem 1 is the following assertion.

Theorem 2.

For model (11), let conditions $\left(A_1\right)$–$\left(A_4\right)$ be fulfilled and

$$\mathbb{E}\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|f_1\left(\mathbf{t}\right)\right|<\infty .$$

(13)

Moreover, let a sequence $h\equiv h_n\to 0$ and a subsequence of naturals $N\equiv N_n\to \infty$ satisfy the conditions

$$h^{-k(p/2+1)}\mathbb{E}\left({\delta }_n^{kp/2}\right)\to 0andN\mathbb{P}({\delta }_n>c^{*}h\})\to 0.$$

(14)

Then,

$$\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\overline{f^{*}}}_{N,n,h}\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(\mathbf{t}\right)|\stackrel{p}{\to }0.$$

Remark 6.

If condition (13) is replaced with a slightly stronger constraint

$$\mathbb{E}\underset{\mathbf{t}\in {[0,1]}^k}{sup}{f}_1^2\left(\mathbf{t}\right)<\infty ,$$

then by (14), one can show the uniform consistency of the estimator

$$M_{N,n,h}^{*}(\mathbf{t},\mathbf{s})=\frac{1}{N}\sum _{j=1}^N{\widehat{f}}_{n,h,j}\left(\mathbf{t}\right){\widehat{f}}_{n,h,j}\left(\mathbf{s}\right),\mathbf{t},\mathbf{s}\in {[0,1]}^k,$$

for the mixed second moment $\mathbb{E}f\left(\mathbf{t}\right)f\left(\mathbf{s}\right)$, where $h\equiv h_n$ and $N\equiv N_n$ are defined in (14). The proof of this fact is similar to the proof of Theorem 2, and thus, we omit the detailed arguments. In other words, under the above conditions, the estimator

$${\mathrm{Cov}}_{N,n,h}^{*}(\mathbf{t},\mathbf{s})=M_{N,n,h}^{*}(\mathbf{t},\mathbf{s})-{\overline{f^{*}}}_{N,n,h}\left(\mathbf{t}\right){\overline{f^{*}}}_{N,n,h}\left(\mathbf{s}\right)$$

is uniformly consistent for the covariance function of the random regression field $f\left(\mathbf{t}\right)$.

4. Proofs

Throughout this section, we assume that conditions $\left(A_1\right)$–$\left(A_3\right)$ are satisfied. To prove Theorem 1, we need a number of auxiliary assertions. Let

$$\begin{array}{c}{\kappa }_j={\int }_{-1}^1{\left|u\right|}^j{K}_o\left(u\right)du,j=1,2,\\ {\lambda }_i\left(\mathbf{t}\right)=K_h(\mathbf{t}-{\mathbf{z}}_i){\Lambda }_k\left({\mathcal{P}}_i\right),w_{n0}\left(\mathbf{t}\right)=\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right),\end{array}$$

(15)

$$\begin{array}{c}{w}_{nuv}\left(\mathbf{t}\right)=\sum _{i=1}^n(t_u-z_{ui})(t_v-z_{vi}){\lambda }_i\left(\mathbf{t}\right),w_{nu}\left(\mathbf{t}\right)=\sum _{i=1}^n(t_u-z_{ui}){\lambda }_i\left(\mathbf{t}\right),u,v=1,\dots ,k,\end{array}$$

(16)

$$\begin{array}{c}{w}_0\left(\mathbf{t}\right)=\int K_h(\mathbf{t}-\mathbf{z})d\mathbf{z},w_u\left(\mathbf{t}\right)=\int (t_u-z_u)K_h(\mathbf{t}-\mathbf{z})d\mathbf{z},u=1,\dots ,k,\end{array}$$

(17)

$$\begin{array}{c}{w}_{uv}\left(\mathbf{t}\right)=\int (t_u-z_u)(t_v-z_v)K_h(\mathbf{t}-\mathbf{z})d\mathbf{z},u,v=1,\dots ,k,\end{array}$$

(18)

where $\mathbf{t}={(t_1,\dots ,t_k)}^{\top }$, ${\mathbf{z}}_i={(z_{i1},\dots ,z_{ik})}^{\top }$, and $\mathbf{z}={(z_1,\dots ,z_k)}^{\top }$.

Remark 7.

We emphasize that due to the density properties of $K_h(·)$, the summation domain in all sums in (15) and (16), as well as in all sums in the formulas given below in (41), (42), (44), and (51) coincide with the set

$$N_h\left(\mathbf{t}\right)=\{i:\parallel \mathbf{t}-{\mathbf{z}}_i\parallel \le h,1\le i\le n\},$$

(19)

and the domain of integration in (17) and (18), respectively, coincides with the set

$$A_h{\left(\mathbf{t}\right)=\{\mathbf{z}\in [0,1]}^k:\parallel \mathbf{t}-\mathbf{z}\parallel \le h\}.$$

These facts are fundamental for the further analysis.

Lemma 1.

For $h<1/2$, the following relations are valid:

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{inf}\left(w_0\left(\mathbf{t}\right)w_{uu}\left(\mathbf{t}\right)-w_u^2\left(\mathbf{t}\right)\right)\ge 2^{-k-1}\left({\kappa }_2-{\kappa }_1^2\right)h^2,u=1,\dots ,k,\end{array}$$

(20)

$$\begin{array}{c}\forall \mathbf{t}\in {[0,1]}^k{w}_0\left(\mathbf{t}\right)w_{uv}\left(\mathbf{t}\right)-w_u\left(\mathbf{t}\right)w_v\left(\mathbf{t}\right)=0,u\ne v,u,v=1,\dots ,k,\end{array}$$

(21)

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{inf}{w}_0\left(\mathbf{t}\right)=2^{-k},\underset{\mathbf{t}\in {[0,1]}^k}{sup}{w}_0\left(\mathbf{t}\right)=1,\end{array}$$

(22)

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}{w}_u\left(\mathbf{t}\right)=2^{-1}{\kappa }_{1}h,\underset{\mathbf{t}\in {[0,1]}^k}{sup}{w}_{uv}\left(\mathbf{t}\right)=2^{-2}{\kappa }_1^2{h}^2,u,v=1,\dots ,k.\end{array}$$

(23)

Proof. 

The relation (21) obviously follows from the representation $K\left(\mathbf{t}\right)={\prod }_{j=1}^k{K}_o\left(t_j\right)$ introduced in condition $\left(A_3\right)$. The statements (20), (22), and (23) follow from Lemma 1 in [53] and the abovementioned representation for the kernel function. □

Lemma 2.

For $h<1/2$, on the subset of elementary events defined by the relation ${\delta }_n\le h$, the following inequalities hold:

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}{w}_{n0}\left(\mathbf{t}\right)\le 2^{2k}{L}^k,\underset{\mathbf{t}\in {[0,1]}^k}{sup}|w_{n0}\left(\mathbf{t}\right)-w_0\left(\mathbf{t}\right)|\le k{2}^{2k}{L}^k{h}^{-1}{\delta }_n,\end{array}$$

(24)

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}|w_{nu}\left(\mathbf{t}\right)|\le 2^{2k}{L}^{k}h,\underset{\mathbf{t}\in {[0,1]}^k}{sup}|w_{nu}\left(\mathbf{t}\right)-w_u\left(\mathbf{t}\right)|\le (k+1)2^{2k}{L}^k{\delta }_n,u=1,\dots ,k,\end{array}$$

(25)

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}|w_{nuv}\left(\mathbf{t}\right)|\le 2^{2k}{L}^k{h}^2,\underset{\mathbf{t}\in {[0,1]}^k}{sup}|w_{nuv}\left(\mathbf{t}\right)-w_{uv}\left(\mathbf{t}\right)|\le (k+2)2^{2k}{L}^{k}h{\delta }_n,u,v=1,\dots ,k;\end{array}$$

(26)

and on the subset of elementary events ${\delta }_n\le {\left(k{2}^{3k+1}{L}^k\right)}^{-1}h$,

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{inf}{w}_{n0}\left(\mathbf{t}\right)\ge 2^{-k-1},\end{array}$$

(27)

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|w_{nuv}\left(\mathbf{t}\right)-\frac{w_{nu}\left(\mathbf{t}\right)w_{nv}\left(\mathbf{t}\right)}{w_{n0}\left(\mathbf{t}\right)}-w_{uv}\left(\mathbf{t}\right)+\frac{w_u\left(\mathbf{t}\right)w_v\left(\mathbf{t}\right)}{w_0\left(\mathbf{t}\right)}\right|\le (k+2)2^{6k+3}{L}^{k+k\wedge 2}h{\delta }_n\end{array}$$

(28)

for any $u,v=1,\dots ,k$.

Proof. 

To display the first estimates in (24)–(26), one should take into account Remark 7 and the following relations:

$$\underset{\mathbf{s}\in {[-1,1]}^k}{sup}K\left(\mathbf{s}\right)\le L^k,\sum _{i=1}^n{K}_h(\mathbf{t}-{\mathbf{z}}_i){\Lambda }_k\left({\mathcal{P}}_i\right)\le h^{-k}{L}^k{(2h+2{\delta }_n)}^k\le 2^{2k}{L}^k.$$

(29)

The second estimates in (24)–(26) follow from the well-known error estimate for the approximation by Riemann integral sums of the corresponding integrals of functions satisfying the Lipschitz condition:

$$\left|\sum _{i\in N_h\left(\mathbf{t}\right)}{g}_{\mathbf{t},\alpha ,\beta }^{u,v}\left({\mathbf{z}}_i\right){\Lambda }_k\left({\mathcal{P}}_i\right)-\underset{\mathbf{z}\in A_h\left(\mathbf{t}\right)}{\int }{g}_{\mathbf{t},\alpha ,\beta }^{u,v}\left(\mathbf{z}\right)d\mathbf{z}\right|\le {\delta }_n{L}_{g_{\mathbf{t},\alpha ,\beta }^{u,v}}\sum _{i=1}^n{\Lambda }_k\left({\mathcal{P}}_i\right),$$

(30)

where the functions $g_{\mathbf{t},\alpha ,\beta }^{u,v}\left(\mathbf{z}\right)={(t_u-z_u)}^{\alpha }{(t_v-z_v)}^{\beta }{K}_h(\mathbf{t}-\mathbf{z})$, $\alpha ,\beta =0,1$, are defined for $z_j\in [0\vee t_j-h,1\wedge t_j+h]$, $j=1,\dots ,k$, and $L_{g_{\mathbf{t},\alpha ,\beta }^{u,v}}$ is a Lipschitz constant of the function $g_{\mathbf{t},\alpha ,\beta }^{u,v}\left(\mathbf{z}\right)$. It is easy to verify that under the conditions of the lemma,

$$\underset{t\in {[0,1]}^k}{sup}{L}_{g_{\mathbf{t},0,0}^{u,v}}\le k{L}^k{h}^{-k-1},\underset{t\in {[0,1]}^k}{sup}{L}_{g_{\mathbf{t},1,0}^{u,v}}\le (k+1)L^k{h}^{-k},\underset{t\in {[0,1]}^k}{sup}{L}_{g_{\mathbf{t},1,1}^{u,v}}\le (k+2)L^k{h}^{-k+1}.$$

To complete deriving (24)–(26), we need to take into account the definitions (15)–(18) and the estimates (29) and (30).

Finally, (27) follows from the first relation in (22) and the second relation in (24). When deriving the last assertion (28) of the lemma, we use the assertions of Lemmas 1 and 2. □

In what follows, we also need to establish the Lipschitz property of the introduced functions $w_{n0}\left(\mathbf{t}\right)$, $w_{nu}\left(\mathbf{t}\right)$, and $w_{nuv}\left(\mathbf{t}\right)$.

Lemma 3.

For $h<1/2$, on the set of elementary outcomes defined by the relation ${\delta }_n\le h$, for any $\mathbf{t}$, ${\mathbf{t}}^{\prime }\in {[0,1]}^k$, the following inequalities hold:

$$\begin{array}{c}|w_{n0}\left({\mathbf{t}}^{\prime }\right)-w_{n0}\left(\mathbf{t}\right)|\le k{2}^{2k+1}{L}^k{h}^{-1}\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel ,\end{array}$$

(31)

$$\begin{array}{c}|w_{nu}\left({\mathbf{t}}^{\prime }\right)-w_{nu}\left(\mathbf{t}\right)|\le (k+1)2^{2k+1}{L}^k\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel ,u=1,\dots ,k,\end{array}$$

(32)

$$\begin{array}{c}|w_{nuv}\left({\mathbf{t}}^{\prime }\right)-w_{nuv}\left(\mathbf{t}\right)|\le (k+2)2^{2k+1}{L}^{k}h\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel ,u,v=1,\dots ,k.\end{array}$$

(33)

On the set of elementary outcomes with the condition ${\delta }_n\le {\left(k{2}^{3k+1}{L}^k\right)}^{-1}h$,

$$\left|w_{nuv}\left({\mathbf{t}}^{\prime }\right)-\frac{w_{nu}\left({\mathbf{t}}^{\prime }\right)w_{nv}\left({\mathbf{t}}^{\prime }\right)}{w_{n0}\left({\mathbf{t}}^{\prime }\right)}-w_{nuv}\left(\mathbf{t}\right)+\frac{w_{nu}\left(\mathbf{t}\right)w_{nv}\left(\mathbf{t}\right)}{w_{n0}\left(\mathbf{t}\right)}\right|\le 5(k+1)2^{8k+2}{L}^{3k}h\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel .$$

(34)

Proof. 

First of all, we note that the normalized kernel $K_h$ satisfies the Lipschitz condition with constant $k{2}^{2k+1}{L}^k{h}^{-k-1}$, and for the sets $N_h\left(\mathbf{t}\right)$ defined in (19), under the conditions of the lemma, we have the estimate

$$\sum _{i\in N_h\left(\mathbf{t}\right)}{\Lambda }_k\left({\mathcal{P}}_i\right)\le 2^k{(h+{\delta }_n)}^k\le 2^{2k}{h}^k.$$

From here, we easily obtain (31):

$$|w_{n0}\left({\mathbf{t}}^{\prime }\right)-w_{n0}\left(\mathbf{t}\right)|\le \sum _{i\in N_h\left({\mathbf{t}}^{\prime }\right)\cup N_h\left(\mathbf{t}\right)}|{\lambda }_i\left({\mathbf{t}}^{\prime }\right)-{\lambda }_i\left(\mathbf{t}\right)|$$

$$\le k{L}^k{h}^{-k-1}\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel \left(\sum _{i\in N_h\left({\mathbf{t}}^{\prime }\right)}{\Lambda }_k\left({\mathcal{P}}_i\right)+\sum _{i\in N_h\left(\mathbf{t}\right)}{\Lambda }_k\left({\mathcal{P}}_i\right)\right)\le k{2}^{2k+1}{L}^k{h}^{-1}\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel .$$

The proof of estimate (32) differs slightly from the one just shown:

$$|w_{nu}\left({\mathbf{t}}^{\prime }\right)-w_{nu}\left(\mathbf{t}\right)|\le \sum _{i\in N_h\left({\mathbf{t}}^{\prime }\right)\cup N_h\left(\mathbf{t}\right)}{\lambda }_i\left(\mathbf{t}\right)|t_u^{\prime }-t_u|+\sum _{i\in N_h\left({\mathbf{t}}^{\prime }\right)\cup N_h\left(\mathbf{t}\right)}|{\lambda }_i\left({\mathbf{t}}^{\prime }\right)-{\lambda }_i\left(\mathbf{t}\right)\left|\right|t_u-z_{ui}|$$

$$\le 2^{2k+1}{L}^k\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel +k{2}^{2k+1}{L}^k\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel .$$

Similarly, estimate (33) is proved after dividing the original sum into three subsums. Inequality (33) follows directly from the upper bounds (24) and (25) for the values under consideration, as well as from (31) and (32). We only deduce the main estimate:

$$|w_{nu}\left({\mathbf{t}}^{\prime }\right)w_{nv}\left({\mathbf{t}}^{\prime }\right)w_{n0}\left(\mathbf{t}\right)-w_{nu}\left(\mathbf{t}\right)w_{nv}\left(\mathbf{t}\right)w_{n0}\left({\mathbf{t}}^{\prime }\right)|\le |w_{nu}\left({\mathbf{t}}^{\prime }\right)-w_{nu}\left(\mathbf{t}\right)\left|\right|w_{nv}\left({\mathbf{t}}^{\prime }\right)|w_{n0}\left(\mathbf{t}\right)$$

$$+|w_{nv}\left({\mathbf{t}}^{\prime }\right)-w_{nv}\left(\mathbf{t}\right)\left|\right|w_{nu}\left(\mathbf{t}\right)|w_{n0}\left(\mathbf{t}\right)+|w_{n0}\left({\mathbf{t}}^{\prime }\right)-w_{n0}\left(\mathbf{t}\right)\left|\right|w_{nu}\left(\mathbf{t}\right)\left|\right|w_{nv}\left(\mathbf{t}\right)|$$

$$\le (5k+4)2^{6k}{L}^{3k}h\parallel {\mathbf{t}}^{\prime }-\mathbf{t}\parallel .$$

To prove (34), we only need to use estimate (33), as well as the uniform lower bound $\underset{\mathbf{t}\in {[0,1]}^k}{inf}{w}_{n0}\left(\mathbf{t}\right)\ge 2^{-k-1}$ of the set of outcomes defined by the inequality ${\delta }_n\le {\left(k{2}^{3k+1}{L}^k\right)}^{-1}h$ (see (27)). After this, you just need to add up the obtained estimates and carry out elementary calculations, which we omit. □

We define the entries of a square matrix $\mathrm{H}$ of size k as follows:

$${\mathrm{H}}_{uv}=w_{nuv}\left(\mathbf{t}\right)-w_{n0}^{-1}\left(\mathbf{t}\right)w_{nu}\left(\mathbf{t}\right)w_{nv}\left(\mathbf{t}\right),u,v=1,\dots ,k.$$

(35)

Note that by virtue of the Cauchy–Bunyakowsky inequality, one has ${\mathrm{H}}_{uv}\ge 0$ for all $u,v$. We also need the following notation:

$$c^{*}=\frac{{\left({\kappa }_2-{\kappa }_1^2\right)}^k}{(k+2)!2^{6k^2+4k+2}{L}^{2k^2-k+k\wedge 2}}.$$

(36)

It is easy to see that the difference ${\kappa }_2-{\kappa }_1^2$ is the variance of a non-degenerate distribution, and thus, it is strictly positive. Also, $c^{*}<1.$

The following assertion holds.

Lemma 4.

For $h<1/2$, on the set of outcomes defined by the relation ${\delta }_n\le c^{*}h$,

$$\underset{t\in {[0,1]}^k}{sup}\parallel {\mathrm{H}}^{-1}\parallel \le k!2^{6k^2-2k-1}{L}^{2k(k-1)}{({\kappa }_2-{\kappa }_1^2)}^{-k}{h}^{-2}.$$

Proof. 

Denote by ${\mathrm{H}}^0$ the square matrix of size k with entries

$${\mathrm{H}}_{uv}^0=w_{uv}\left(\mathbf{t}\right)-w_0^{-1}\left(\mathbf{t}\right)w_u\left(\mathbf{t}\right)w_v\left(\mathbf{t}\right),u,v=1,\dots ,k.$$

By Lemma 1,

$${\mathrm{H}}^0=\mathrm{diag}\left\{w_{11}\left(\mathbf{t}\right)-w_0\left(\mathbf{t}\right)w_1^2\left(\mathbf{t}\right),\dots ,w_{kk}\left(\mathbf{t}\right)-w_0\left(\mathbf{t}\right)w_k^2\left(\mathbf{t}\right)\right\}$$

and the following estimate holds for the determinant of this diagonal matrix with positive entries:

$$\underset{\mathbf{t}\in {[0,1]}^k}{inf}det\left({\mathrm{H}}^0\right)\ge 2^{-k(k+1)}{\left({\kappa }_2-{\kappa }_1^2\right)}^k{h}^{2k}.$$

(37)

Next,

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|det\mathrm{H}-det{\mathrm{H}}^0\right|\le k!k(k+2)2^{6k+3}{L}^{k+k\wedge 2}{\left(L^{2k}{2}^{5k+2}\right)}^{k-1}{\delta }_{n}h{h}^{2(k-1)}\le \\ (k+2)!2^{5k^2+3k+1}{L}^{2k^2-k+k\wedge 2}{\delta }_n{h}^{2k-1}.\end{array}$$

(38)

When deriving (4), we use the definition of the matrix determinant and estimate (28), and took into account the fact that due to Lemmas 1 and 2, for all $u,v$, one has

$$\underset{\mathbf{t}\in {[0,1]}^k}{sup}{\mathrm{H}}_{uv}\le 2^{5k+2}{L}^{2k}{h}^2,\underset{\mathbf{t}\in {[0,1]}^k}{sup}{\mathrm{H}}_{uv}^0\le 2^{k-1}{L}^2{h}^2.$$

Hence, on the set of elementary outcomes defined by the relation ${\delta }_n\le c^{*}h$,

$$\underset{\mathbf{t}\in {[0,1]}^k}{inf}det\mathrm{H}\ge 2^{-k(k+1)-1}{\left({\kappa }_2-{\kappa }_1^2\right)}^k{h}^{2k}.$$

(39)

The assertion of the lemma can now be easily derived directly from the definition of the inverse matrix and relation (39). □

We need the following notation of n-dimensional vectors:

$$\begin{array}{c}\mathbf{1}={(1,\dots ,1)}^{\top },\mathit{ϵ}={({\epsilon }_1,\dots ,{\epsilon }_n)}^{\top },\\ {\mathit{\delta }}_f={\left((f\left({\mathbf{z}}_1\right)-f\left(\mathbf{t}\right)),\dots ,(f\left({\mathbf{z}}_n\right)-f\left(\mathbf{t}\right))\right)}^{\top }\end{array}$$

In this case, $\mathbf{x}=f\left(\mathbf{t}\right)\mathbf{1}+{\mathit{\delta }}_f+\mathit{ϵ}$, and for the estimate ${\widehat{f}}_{n,h}\left(\mathbf{t}\right)$ in (4), there is the valid representation

$$\begin{array}{c}\hfill {\widehat{f}}_{n,h}\left(\mathbf{t}\right)=f\left(\mathbf{t}\right){\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathbf{1}+{\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathit{\delta }}_f\\ \hfill +{\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathit{ϵ}.\end{array}$$

(40)

Using definitions (2) and (3) and the first notation in (15), one can rewrite the matrix ${\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}$ of dimension $(k+1)\times (k+1)$ and $(k+1)$-dimensional vectors ${\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathbf{1}$, ${\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathit{\delta }}_f$, and ${\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathit{ϵ}$ as follows:

$$\begin{array}{c}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}=\left[\begin{array}{cc}\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right)& \sum _{i=1}^n{(\mathbf{t}-{\mathbf{z}}_i)}^{\top }{\lambda }_i\left(\mathbf{t}\right)\\ \sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right)& \sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){(\mathbf{t}-{\mathbf{z}}_i)}^{\top }{\lambda }_i\left(\mathbf{t}\right)\end{array}\right],{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathbf{1}=\left[\begin{array}{c}\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right)\\ \sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right)\end{array}\right],\end{array}$$

(41)

$$\begin{array}{c}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathit{\delta }}_f=\left[\begin{array}{c}\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right)\left(f\left({\mathbf{z}}_i\right)-f\left(\mathbf{t}\right)\right)\\ \sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right)\left(f\left({\mathbf{z}}_i\right)-f\left(\mathbf{t}\right)\right)\end{array}\right],{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathit{ϵ}=\left[\begin{array}{c}\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right){\epsilon }_i\\ \sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right){\epsilon }_i\end{array}\right].\end{array}$$

(42)

The following assertion holds.

Lemma 5.

The following identity is valid:

$${\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathbf{1}=1.$$

Proof. 

We use the Frobenius formula to invert a nondegenerate block matrix:

$$\begin{array}{c}{\left[\begin{array}{cc}\mathrm{A}& \mathrm{B}\\ \mathrm{C}& \mathrm{D}\end{array}\right]}^{-1}=\left[\begin{array}{cc}{\mathrm{A}}^{-1}+{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}& -{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\\ -{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}& {\mathrm{H}}^{-1}\end{array}\right],for\mathrm{H}=\mathrm{D}-\mathrm{C}{\mathrm{A}}^{-1}\mathrm{B},\end{array}$$

where $\mathrm{A}$ is a square nondegenerate matrix of size $m_1$ and $\mathrm{D}$ is a square matrix of size $m_2$. In this formula, we let $m_1=1$, $m_2=k$,

$$\begin{array}{c}\begin{array}{c}\hfill \mathrm{A}=\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right)\equiv w_{n0}\left(\mathbf{t}\right),\mathrm{B}=\sum _{i=1}^n{(\mathbf{t}-{\mathbf{z}}_i)}^{\top }{\lambda }_i\left(\mathbf{t}\right),\\ \hfill \mathrm{C}=\sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right),\mathrm{D}=\sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){(\mathbf{t}-{\mathbf{z}}_i)}^{\top }{\lambda }_i\left(\mathbf{t}\right).\end{array}\end{array}$$

(43)

Then, in view of (41) and the Frobenius formula, the first component of the vector of interest to us is

$$\left({\mathrm{A}}^{-1}+{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}\right)\mathrm{A}-{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}\equiv 1.$$

□

Lemma 6.

For $h<1/2$, on the set of elementary outcomes defined by the relation ${\delta }_n\le c^{*}h$,

$$\underset{t\in {[0,1]}^k}{sup}|{\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathit{\delta }}_f|\le C_1^{*}{\omega }_f\left(h\right),with{C}_1^{*}=1+k·k!2^{6k^2}{L}^{2k^2-k}{({\kappa }_2-{\kappa }_1^2)}^{-k}.$$

Proof. 

We use definition (43). Next, using the Frobenius formula (see the proof of Lemma 5), we obtain the identity

$$\begin{array}{c}{\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathit{\delta }}_f={\mathrm{A}}^{-1}{\mathrm{A}}_f+{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}{\mathrm{A}}_f-{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}{\mathrm{C}}_f,\end{array}$$

(44)

where

$${\mathrm{A}}_f=\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right)\left(f\left({\mathbf{z}}_i\right)-f\left(\mathbf{t}\right)\right),{\mathrm{C}}_f=\sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right)\left(f\left({\mathbf{z}}_i\right)-f\left(\mathbf{t}\right)\right).$$

Consider each of the terms on the right side of this relation. Notice that for the standard inner product $(\mathbf{x},\mathbf{y})$ in $R^k$ and the supnorm $\parallel ·\parallel$ under consideration, we have the inequality $\left|\right(\mathbf{x},\mathbf{y}\left)\right|\le k\parallel \mathbf{x}\parallel \parallel \mathbf{y}\parallel$. Therefore,

$$\begin{array}{c}|{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}{\mathrm{A}}_f|\le k{\mathrm{A}}^{-1}|{\mathrm{A}}^{-1}{\mathrm{A}}_f|\parallel {\mathrm{B}}^{\top }\parallel \parallel {\mathrm{H}}^{-1}\parallel \parallel \mathrm{C}\parallel ,\\ |{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}{\mathrm{C}}_f|\le k{\mathrm{A}}^{-1}\parallel {\mathrm{B}}^{\top }\parallel \parallel {\mathrm{H}}^{-1}\parallel \parallel {\mathrm{C}}_f\parallel .\end{array}$$

Further, taking into account Remark 7, we obtain

$$|{\mathrm{A}}^{-1}{\mathrm{A}}_f|\le {\omega }_f\left(h\right),\parallel {\mathrm{C}}_f\parallel \le {\omega }_f\left(h\right)\mathrm{A}h,\parallel \mathrm{C}\parallel \le \mathrm{A}h,\underset{t\in {[0,1]}^k}{sup}\parallel {\mathrm{B}}^{\top }\parallel \le 2^{2k}{L}^{k}h.$$

(45)

The above relations, together with the identity (44) and the estimates (45), complete the proof of the lemma. □

Lemma 7.

For any $h<1/2$, $\mathbf{t}\in {[0,1]}^k$, and $m\in N_h\left(\mathbf{t}\right)$, on the set of outcomes defined by the relation ${\delta }_n\le c^{*}h$, the following estimates are valid:

$$\begin{array}{c}|{\alpha }_{n,m}\left(\mathbf{t}\right)|\le 1+(k+1)!2^{6k^2-1}{L}^{2k^2-k}{({\kappa }_2-{\kappa }_1^2)}^{-k}\equiv \alpha ,\\ |{\alpha }_{n,m}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\alpha }_{n,m}\left(\mathbf{t}\right)|\le \overline{\alpha }{2}^{-l}{h}^{-1}if{2}^{-l}\le h,\\ with\overline{\alpha }=30{(k!)}^2{k}^{3-2k}{L}^{10k-2}{2}^{8k^2+21k+5}{\left({\kappa }_2-{\kappa }_1^2\right)}^{-2k},\end{array}$$

where ${\alpha }_{n,m}\left(\mathbf{t}\right)=1+\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}-\mathrm{B}{\mathrm{H}}^{-1}(\mathbf{t}-{\mathbf{z}}_m)$, and ${\mathbf{e}}_r$ is the k-dimensional vector with unit r-th component and zeros as the other ones.

Proof. 

Since $\parallel \mathbf{t}-{\mathbf{z}}_m\parallel \le h$ for $m\in N_h\left(\mathbf{t}\right)$, then the first assertion of the lemma follows from Lemma 4, relations (45), and the following estimate:

$$\begin{array}{c}|{\alpha }_{n,m}\left(\mathbf{t}\right)|\le 1+k{\mathrm{A}}^{-1}\parallel {\mathrm{B}}^{\top }\parallel \parallel {\mathrm{H}}^{-1}\parallel \parallel \mathrm{C}\parallel +\parallel {\mathrm{B}}^{\top }\parallel \parallel {\mathrm{H}}^{-1}\parallel \parallel \mathbf{t}-{\mathbf{z}}_m\parallel .\end{array}$$

To prove the second assertion, first of all, we note

$${\mathrm{H}}^{-1}=\frac{\mathrm{adj}\mathrm{H}}{det\mathrm{H}},$$

where the elements of the adjoint matrix $\mathrm{adj}\mathrm{H}$ are, up to the signs ±, the complementary minors of the original matrix $\mathrm{H}$. As is known, the determinant of any square matrix $\mathrm{M}$ of size k with entries ${\mathrm{M}}_{ij}$, $i,j=1,\dots ,k$, can be represented as the multilinear form

$$det\mathrm{M}=\sum _{{\alpha }_1,\dots ,{\alpha }_k}{(-1)}^{I\left(\overline{\alpha }\right)}{\mathrm{M}}_{1{\alpha }_1}\dots {\mathrm{M}}_{k{\alpha }_k},$$

(46)

where summation is over all permutations $\overline{\alpha }\equiv ({\alpha }_1\dots {\alpha }_k)$ of natural numbers $1,\dots ,k$, and $I\left(\overline{\alpha }\right)$ is the number of inversions in a particular permutation $\overline{\alpha }$. We need Formula (46) to represent the determinant $det\mathrm{H}$ and the complementary minors of the matrix $\mathrm{H}$ in the inverse matrix ${\mathrm{H}}^{-1}$, respectively, as k- and $(k-1)$-linear forms of type (46) constructed by entries (35).

To prove this, in the definition of the function ${\alpha }_{n,m}\left(\mathbf{t}\right)$, the term $\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}$ is Lipschitz and the summand $\mathrm{B}{\mathrm{H}}^{-1}(\mathbf{t}-{\mathbf{z}}_m)$ is local Lipschitz, and the method is carried out the same as that in Lemma 3 (see the proof of (34)). First, we note that the following representation holds:

$$\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}=\frac{{\displaystyle \sum _{i,j=1}^k}{w}_{ni}\left(\mathbf{t}\right)w_{nj}\left(\mathbf{t}\right){(-1)}^{i+j}det{\mathrm{H}}^{\left(ij\right)}}{Adet\mathrm{H}},$$

(47)

where $det{\mathrm{H}}^{\left(ij\right)}$ is the complementary minor to the entry ${\mathrm{H}}_{ij}=w_{nij}\left(\mathbf{t}\right)+\frac{w_{ni}\left(\mathbf{t}\right)w_{nj}\left(\mathbf{t}\right)}{w_{n0}\left(\mathbf{t}\right)}$ of the matrix $\mathrm{H}$. Note that in virtue of notation (43), relation (27) can be rewritten as follows:

$$\underset{t\in {[0,1]}^k}{inf}\mathrm{A}\ge 2^{-k-1}.$$

(48)

Further, to calculate the Lipschitz constant for the ratio in (47), we need the lower bounds (39) and (48) for the denominator of the fractions on the right-hand side of (47) that are valid on the set of elementary outcomes ${\delta }_n\le c_{*}h$, taking into account the fact that the lower bound in (39) has the order of smallness $h^{2k}$. Moreover, Lemmas 2 and 3, together with the representation (46) (with the elements ${\mathrm{H}}_{ij}$ instead of $M_{ij}$), allow us to assert that the upper bound for $det\mathrm{H}$ has the same order of smallness in h, and the Lipschitz constant for $det\mathrm{H}$ is estimated from above as $C_3^{*}{h}^{2k-1}$. At the same time, the complementary minors $det{\mathrm{H}}^{\left(ij\right)}$ are estimated from above as $C_4^{*}{h}^{2k-2}$, with a Lipschitz constant of order $h^{2k-3}$ uniformly over all $i,j=1,\dots ,k$. In fact, we only need to repeat the calculations when deriving estimate (34) to calculate the Lipschitz constant of the fraction on the right side of (47), from which it follows that the mentioned constant has the form $C_5^{*}{h}^{-1}$, where as a constant $C_5^{*}$, we can take the value

$$C_5^{*}=15{(k!)}^2{k}^{3-2k}{L}^{10k-2}{2}^{8k^2+21k+5}{\left({\kappa }_2-{\kappa }_1^2\right)}^{-2k}.$$

(49)

For the term $\mathrm{B}{\mathrm{H}}^{-1}(\mathbf{t}-{\mathbf{z}}_m))$, we establish that in the h-neighborhood of the point $\mathbf{t}$, it has the property of being locally Lipschitz with a constant of the form $C_6^{*}{h}^{-1}$. By analogy with (47), we represent the specified summand as

$$\mathrm{B}{\mathrm{H}}^{-1}(\mathbf{t}-{\mathbf{z}}_m)=\frac{{\displaystyle \sum _{i,j=1}^k}{w}_{ni}\left(\mathbf{t}\right)(t_j-z_{mj}){(-1)}^{i+j}det{\mathrm{H}}^{\left(ij\right)}}{det\mathrm{H}}.$$

Repeating almost verbatim the previous arguments in proving this lemma, we come to the conclusion that in the h-neighbourhood of any point $\mathbf{t}\in {[0,1]}^k$ (under the condition $\parallel \mathbf{t}-{\mathbf{z}}_m\parallel \le h$), the function $\mathrm{B}{\mathrm{H}}^{-1}(\mathbf{t}-{\mathbf{z}}_m)$ satisfies the Lipschitz condition with constant $C_6^{*}$, which is significantly less than that in (49). Therefore, the final constant in the lemma can be set equal to $2C_5^{*}$. □

Lemma 8.

For any $y>0$ and $h<1/2$, on the subset of elementary events defined by the relation ${\delta }_n\le c^{*}h$, the following estimate holds:

$$\begin{array}{c}{\mathbb{P}}_{{\mathcal{F}}_n}\left(\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|{\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathit{ϵ}\right|>y\right)\le C_2^{*}{y}^{-p}{\delta }_n^{kp/2}{h}^{-k(1+(p/2\left)\right)},\end{array}$$

where the symbol ${\mathbb{P}}_{{\mathcal{F}}_n}$ denotes the conditional probability given the σ-algebra ${\mathcal{F}}_n$ generated by the design $\{{\mathbf{z}}_i;i\le n\}$, and the constant $C_2^{*}$ is defined in (61).

Proof. 

We use the notation (43). Then, from (41), (42), and the Frobenius formula, we obtain the identity

$${\nu }_{n,h}\left(\mathbf{t}\right):={\mathbf{e}}_1^{\top }{\left({\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}{\mathrm{Z}}_{\mathbf{t}}\right)}^{-1}{\mathrm{Z}}_{\mathbf{t}}^{\top }{\mathrm{W}}_{\mathbf{t}}\mathit{ϵ}=\left({\mathrm{A}}^{-1}+{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}\mathrm{C}{\mathrm{A}}^{-1}\right){\mathrm{A}}_{\epsilon }-{\mathrm{A}}^{-1}\mathrm{B}{\mathrm{H}}^{-1}{\mathrm{C}}_{\epsilon },$$

(50)

where

$$\begin{array}{c}{\mathrm{A}}_{\epsilon }=\sum _{i=1}^n{\lambda }_i\left(\mathbf{t}\right){\epsilon }_i,{\mathrm{C}}_{\epsilon }=\sum _{i=1}^n(\mathbf{t}-{\mathbf{z}}_i){\lambda }_i\left(\mathbf{t}\right){\epsilon }_i.\end{array}$$

(51)

Let

$${\mu }_{n,h}\left(\mathbf{t}\right)=\sum _{i:\parallel \mathbf{t}-{\mathbf{z}}_i\parallel \le h}{\alpha }_{n,i}\left(\mathbf{t}\right){\lambda }_i\left(\mathbf{t}\right){\epsilon }_i.$$

In virtue of estimate (27), Lemma 2, and Remark 7, one has

$$|{\nu }_{n,h}\left(\mathbf{t}\right)|\le 2^{k+1}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|.$$

(52)

The distribution tail $\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|$ is estimated by Kolmogorov’s dyadic chaining (see, for example, [77]). First of all, we note that the set ${[0,1]}^k$ under the supremum sign can be replaced with the set of dyadic rational points $\mathcal{R}={\cup }_{l\ge 1}{\mathcal{R}}_l$, where

$${\mathcal{R}}_l=\{(j_1/2^l,\dots ,j_k/2^l):j_1=1,\dots ,2^l-1;\dots ;j_k=1,\dots ,2^l-1\}.$$

Therefore,

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\mu }_{n,h}\left(\mathbf{t}\right)|=\underset{\mathbf{t}\in \mathcal{R}}{sup}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|\le \\ \le \underset{\mathbf{t}\in {\mathcal{R}}_m}{max}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|+\sum _{l=m+1}^{\infty }\sum _{r=1}^k\underset{\mathbf{t}\in {\mathcal{R}}_l}{max}|{\mu }_{n,h}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\mu }_{n,h}\left(\mathbf{t}\right)|,\end{array}$$

where $m=\lceil -{log}_{2}h\rceil$ (here, $\lceil a\rceil$ is the smallest integer greater than or equal to a). Hence,

$$\begin{array}{c}{\mathbb{P}}_{{\mathcal{F}}_n}\left(\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|>y\right)\le {\mathbb{P}}_{{\mathcal{F}}_n}\left(\underset{\mathbf{t}\in {\mathcal{R}}_m}{max}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|>a_{m}y\right)+\\ +\sum _{l=m+1}^{\infty }\sum _{r=1}^k{\mathbb{P}}_{{\mathcal{F}}_n}\left(\underset{\mathbf{t}\in {\mathcal{R}}_l}{max}|{\mu }_{n,h}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{l}y/k\right)\le \\ \le \sum _{\mathbf{t}\in {\mathcal{R}}_m}{\mathbb{P}}_{{\mathcal{F}}_n}\left(\right|{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{m}y)\\ +\sum _{l=m+1}^{\infty }\sum _{r=1}^k\sum _{\mathbf{t}\in {\mathcal{R}}_l}{\mathbb{P}}_{{\mathcal{F}}_n}\left(|{\mu }_{n,h}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{l}y/k\right),\end{array}$$

(53)

where $a_m,a_{m+1},\dots$ is a sequence of positive numbers, with $a_m+a_{m+1}+\cdots =1$.

To evaluate the probabilities on the right side of (53), we need the following martingale inequality (see [78], Theorem 2.1):

$$\begin{array}{c}\mathbb{E}|\sum _{i=1}^n{\eta }_i{|}^p\le {\left((p-1)\sum _{i=1}^n{\left(\mathbb{E}|{\eta }_i{|}^p\right)}^{2/p}\right)}^{p/2},\end{array}$$

(54)

where $\left\{{\eta }_i\right\}$ is a martingale difference with finite moments of order $p\ge 2$.

To obtain an upper bound for ${\mathbb{P}}_{{\mathcal{F}}_n}\left(\right|{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{m}y)$, we use (54) for

$${\eta }_i={\alpha }_{n,i}\left(\mathbf{t}\right)K_h(\mathbf{t}-{\mathbf{z}}_i){\Lambda }_k\left({\mathcal{P}}_i\right){\epsilon }_i.$$

From (29), Lemma 7, and the elementary estimate $K_h(\mathbf{t}-{\mathbf{z}}_i){\Lambda }_k\left({\mathcal{P}}_i\right)\le L^k{({\delta }_n/h)}^k$ we obtain that with probability 1,

$$\sum _{i=1}^n{\left({\mathbb{E}}_{{\mathcal{F}}_n}{\left|{\eta }_i\right|}^p\right)}^{2/p}\le M_p^{2/p}{\alpha }^2{2}^{2k}{L}^{2k}{({\delta }_n/h)}^k.$$

Next, taking into account the estimate ${\delta }_n\le h\le 1$, the above inequality and (54) imply the relation

$$\begin{array}{c}{\mathbb{P}}_{{\mathcal{F}}_n}\left(\right|{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{m}y)\le \frac{{\mathbb{E}}_{{\mathcal{F}}_n}\left(\right|{\mu }_{n,h}\left(\mathbf{t}\right){|}^p)}{{\left(a_{m}y\right)}^p}\le G_1\frac{{({\delta }_n/h)}^{kp/2}}{{\left(a_{m}y\right)}^p}\mathrm{a}.\mathrm{s}.,\end{array}$$

(55)

where

$$G_1={(p-1)}^{p/2}{\alpha }^p{2}^{kp}{L}^{kp}{M}_p.$$

(56)

To estimate the probability ${\mathbb{P}}_{{\mathcal{F}}_n}\left(|{\mu }_{n,h}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{l}y/k\right)$, we use inequality (54), with

$${\eta }_i=\left({\alpha }_{n,i}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)K_h(\mathbf{t}-{\mathbf{z}}_i+2^{-l}{\mathbf{e}}_r)-{\alpha }_{n,i}\left(\mathbf{t}\right)K_h(\mathbf{t}-{\mathbf{z}}_i)\right){\Lambda }_k\left({\mathcal{P}}_i\right){\epsilon }_i.$$

We need the following relations:

$$\begin{array}{c}|{\alpha }_{n,i}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)K_h(\mathbf{t}-{\mathbf{z}}_i+2^{-l}{\mathbf{e}}_r)-{\alpha }_{n,i}\left(\mathbf{t}\right)K_h(\mathbf{t}-{\mathbf{z}}_i)|{\Lambda }_k\left({\mathcal{P}}_i\right)\le \\ \le \left|{\alpha }_{n,i}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)\right||K_h(\mathbf{t}-{\mathbf{z}}_i+2^{-l}{\mathbf{e}}_r)-K_h(\mathbf{t}-{\mathbf{z}}_i)|{\Lambda }_k\left({\mathcal{P}}_i\right)+\\ +K_h(\mathbf{t}-{\mathbf{z}}_i)|{\alpha }_{n,i}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\alpha }_{n,i}\left(\mathbf{t}\right)|{\Lambda }_k\left({\mathcal{P}}_i\right)\le \\ \le \alpha L^k{h}^{-k-1}{2}^{-l}{\delta }_n^k+\overline{\alpha }{h}^{-1}{2}^{-l}{L}^k{h}^{-k}{\delta }_n^k=(\alpha +\overline{\alpha })L^k{h}^{-k-1}{2}^{-l}{\delta }_n^k,\end{array}$$

(57)

$$\begin{array}{c}\sum _{i=1}^n\left({\alpha }_{n,i}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)K_h(\mathbf{t}-{\mathbf{z}}_i+2^{-l}{\mathbf{e}}_r)-{\alpha }_{n,i}\left(\mathbf{t}\right)K_h(\mathbf{t}-{\mathbf{z}}_i)\right){\Lambda }_k\left({\mathcal{P}}_i\right)\le \\ \le (\alpha +\overline{\alpha })L^k{h}^{-k-1}{2}^{-l+1}{(2h+2{\delta }_n)}^k\le (\alpha +\overline{\alpha })L^k{h}^{-1}{2}^{-l+1}{2}^{2k}.\end{array}$$

(58)

Note that $|\mathbf{t}-{\mathbf{z}}_i+2^{-l}{\mathbf{e}}_r-\mathbf{t}+{\mathbf{z}}_i|=2^{-l}\le h$ for $l>m$. Moreover, the summation domain in (4) coincides with the set

$$N_h(\mathbf{t}+2^{-l}{\mathbf{e}}_r)\cup N_h\left(\mathbf{t}\right)for\mathbf{t}\in {\mathcal{R}}_l.$$

These facts, Lemma 7, and the relation ${\delta }_n\le h\le 1$ were taken into account when deriving (4). To prove (57), we used the second assertion of Lemma 7.

Thus,

$$\sum _{i=1}^n{\left({\mathbb{E}}_{{\mathcal{F}}_n}{\left|{\eta }_i\right|}^p\right)}^{2/p}\le M_p^{2/p}{(\alpha +\overline{\alpha })}^2{2}^{2k}{L}^{2k}{2}^{-2l+1}{h}^{-2}{({\delta }_n/h)}^k,$$

and in virtue of inequality (54), we obtain

$$\begin{array}{c}{\mathbb{P}}_{{\mathcal{F}}_n}\left(|{\mu }_{n,h}(\mathbf{t}+2^{-l}{\mathbf{e}}_r)-{\mu }_{n,h}\left(\mathbf{t}\right)|>a_{l}y/k\right)\le \frac{G_2}{k}\frac{{({\delta }_n/h)}^{kp/2}{h}^{-p}{2}^{-lp}}{{\left(a_{l}y\right)}^p},\end{array}$$

(59)

where

$$G_2=2^{p/2}{(\alpha +\overline{\alpha })}^p{k}^{p+1}{(p-1)}^{p/2}{2}^{kp}{L}^{kp}{M}_p=2^{p/2}{k}^{p+1}{\alpha }^{-p}{(\alpha +\overline{\alpha })}^p{G}_1.$$

(60)

Now, using relations (53), (55), and (59), we conclude that

$${\mathbb{P}}_{{\mathcal{F}}_n}\left(\underset{\mathbf{t}\in \mathcal{P}}{sup}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|>y\right)<y^{-p}{({\delta }_n/h)}^{kp/2}\left(G_1{2}^{km}{a}_m^{-p}+G_2{h}^{-p}\sum _{l=m+1}^{\infty }{2}^{-(p-k)l}{a}_l^{-p}\right).$$

The optimal sequence $a_l$ that minimizes the right-hand side of this inequality is

$$a_m=c{\left(G_1{2}^{km}\right)}^{1/(p+1)},a_l=c{\left(G_2{h}^{-p}{2}^{-(p-k)l}\right)}^{1/(p+1)}$$

for $l=m+1,m+2,\dots$, where the coefficient c is given by the equality $a_m+a_{m+1}+\cdots =1$. For the specified sequence, we obtain

$$\begin{array}{c}{\mathbb{P}}_{{\mathcal{F}}_n}\left(\underset{\mathbf{t}\in \mathcal{P}}{sup}\left|{\mu }_{n,h}\left(\mathbf{t}\right)\right|>y\right)\le \\ \le y^{-p}{({\delta }_n/h)}^{kp/2}{\left({\left(G_1{2}^{km}\right)}^{1/(p+1)}+\sum _{l=m+1}^{\infty }{\left(G_2{h}^{-p}{2}^{-(p-k)l}\right)}^{1/(p+1)}\right)}^{p+1}\le \\ \le y^{-p}{\delta }_n^{kp/2}{h}^{-k(1+(p/2\left)\right)}{\left({\left(2G_1\right)}^{1/(p+1)}+{\left(G_2{2}^{-(p-k)}\right)}^{1/(p+1)}\sum _{l=0}^{\infty }{2}^{-(p-k)l/(p+1)}\right)}^{p+1}\le \\ \le y^{-p}{\delta }_n^{kp/2}{h}^{-k(1+(p/2\left)\right)}{\left({\left(2G_1\right)}^{1/(p+1)}+{\left(G_2{2}^{-(p-k)}\right)}^{1/(p+1)}{\left(1-2^{-(p-k)/(p+1)}\right)}^{-1}\right)}^{p+1}.\end{array}$$

This relation, together with the estimate (52), completes the derivation of Lemma 8 if we let

$$C_2^{*}=2^{p(k+1)}{\left({\left(2G_1\right)}^{1/(p+1)}+{\left(G_2{2}^{-(p-k)}\right)}^{1/(p+1)}{\left(1-2^{-(p-k)/(p+1)}\right)}^{-1}\right)}^{p+1},$$

(61)

where the constants $G_1$ and $G_2$ are defined in (56) and (60), respectively. □

We complete the proof of Theorem 1. We set ${\zeta }_n\left(h\right)={sup}_{\mathbf{t}\in {[0,1]}^k}\left|{\nu }_{n,h}\left(\mathbf{t}\right)\right|$, where ${\nu }_{n,h}\left(\mathbf{t}\right)$ is defined in (50), and together with Lemma 8, we take into account the relation

$$\mathbb{P}\left({\zeta }_n\left(h\right)>y,{\delta }_n\le c^{*}h\right)=\mathbb{E}I\left({\delta }_n\le c^{*}h\right){\mathbb{P}}_{{\mathcal{F}}_n}\left({\zeta }_n\left(h\right)>y\right).$$

It remains for us to use the identity (40) and the assertions of the Lemmas 5 and 6. Theorem 1 is proved.

To prove Theorem 2, we need the following auxiliary assertion.

Lemma 9.

If condition (13) is satisfied, then ${lim}_{\epsilon \to 0}\mathbb{E}{\omega }_{f_1}\left(\epsilon \right)=0$, and for the independent copies $f_1\left(\mathbf{t}\right),\dots ,f_n\left(\mathbf{t}\right)$ of an almost sure continuous random process, the following uniform law of large numbers holds:

$$\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\overline{f}}_N\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(\mathbf{t}\right)|\stackrel{p}{\to }0,where{\overline{f}}_N\left(\mathbf{t}\right)=N^{-1}\sum _{i=1}^N{f}_i\left(\mathbf{t}\right).$$

(62)

Proof. 

The first assertion of the lemma follows from (13) and Lebesgue’s dominated convergence theorem. Let

$${\omega }_{{\overline{f}}_N}\left(\epsilon \right)=\underset{\mathbf{t},\mathbf{s}:\parallel \mathbf{t}-\mathbf{s}\parallel \le \epsilon }{sup}|{\overline{f}}_N\left(\mathbf{t}\right)-{\overline{f}}_N\left(\mathbf{s}\right)|,{\omega }_{\mathbb{E}{f}_1}\left(\epsilon \right)=\underset{\mathbf{t},\mathbf{s}:\parallel \mathbf{t}-\mathbf{s}\parallel \le \epsilon }{sup}|\mathbb{E}{f}_1\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(\mathbf{s}\right)|.$$

For simplicity, let $k=2$. For an arbitrary fixed $r>0$ and $u,v=0,\dots ,r$, the following relations hold:

$$\begin{array}{c}\underset{\mathbf{t}\in {[0,1]}^2}{sup}|{\overline{f}}_N\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(\mathbf{t}\right)|\le \underset{0\le u,v\le r}{max}\left|{\overline{f}}_N\left(u/r,v/r\right)-\mathbb{E}{f}_1\left(u/r,v/r\right)\right|+\\ +\underset{1\le u,v\le r}{max}\underset{\frac{u-1}{r}\le t_1\le \frac{u}{r},\frac{v-1}{r}\le t_2\le \frac{v}{r}}{sup}\left|{\overline{f}}_N\left(\mathbf{t}\right)-{\overline{f}}_N\left(u/r,v/r\right)\right|+\\ \underset{1\le u,v\le r}{max}\underset{\frac{u-1}{r}\le t_1\le \frac{u}{r},\frac{v-1}{r}\le t_2\le \frac{v}{r}}{sup}\left|\mathbb{E}{f}_1\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(u/r,v/r\right)\right|\le \\ \le \underset{0\le u,v\le r}{max}\left|{\overline{f}}_N\left(u/r,v/r\right)-\mathbb{E}{f}_1\left(u/r,v/r\right)\right|+{\omega }_{{\overline{f}}_N}\left(1/r\right)+{\omega }_{\mathbb{E}{f}_1}\left(1/r\right).\end{array}$$

(63)

Now, notice that ${\omega }_{\mathbb{E}{f}_1}\left(\epsilon \right)\le \mathbb{E}{\omega }_{f_1}\left(\epsilon \right)$ and

$${\overline{f}}_N\left(u/r,v/r\right)\stackrel{p}{\to }\mathbb{E}{f}_1\left(u/r,v/r\right),{\omega }_{{\overline{f}}_N}\left(\epsilon \right)\le \frac{1}{N}\sum _{i=1}^N{\omega }_{f_i}\left(\epsilon \right)\stackrel{p}{\to }\mathbb{E}{\omega }_{f_1}\left(\epsilon \right).$$

Therefore, the right side in (4) does not exceed $2\mathbb{E}{\omega }_{f_1}\left(1/r\right)+o_p\left(1\right)$, and due to the arbitrariness of r and the first assertion of the lemma, the relation (62) for $k=2$ is proved. For arbitrary k, the derivation of the statement of the lemma is similar. □

Proof of Theorem 2.

We let

$${\Delta }_{n,h,j}=\underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\widehat{f}}_{n,h,j}\left(\mathbf{t}\right)-f_j\left(\mathbf{t}\right)|$$

and prove the following version of the law of large numbers for the values $\left\{{\Delta }_{n,h,j}\right\}$:

$$\frac{1}{N}\sum _{j=1}^N{\Delta }_{n,h,j}\stackrel{p}{\to }0asN\to \infty ,$$

(64)

where the sequences $h=h_n$ and $N=N_n$ are defined in (14). We introduce the events

$$A_{n,h,j}=\{{\delta }_{n,j}\le c^{*}h\},j=1,\dots ,N$$

and notice that for any positive $\nu$, one has

$$\mathbb{P}\left\{\frac{1}{N}\sum _{j=1}^N{\Delta }_{n,h,j}>\nu \right\}\le \mathbb{P}\left\{\frac{1}{N}\sum _{j=1}^N{\Delta }_{n,h,j}I\left(A_{n,h,j}\right)>\nu \right\}+N\mathbb{P}\left(\overline{A_{n,h,1}}\right).$$

(65)

Next, from Theorem 1, it follows that

$$\begin{array}{c}\mathbb{E}{\Delta }_{n,h,j}I\left(A_{n,h,j}\right)\le C_1^{*}\mathbb{E}{\omega }_f\left(h\right)+\underset{0}{\overset{\infty }{\int }}\mathbb{P}\left({\zeta }_n\left(h\right)>y,{\delta }_n\le c^{*}h\right)dy\le \\ \le C_1^{*}\mathbb{E}{\omega }_f\left(h\right)+{\gamma }_n+\underset{{\gamma }_n}{\overset{\infty }{\int }}\mathbb{P}\left({\zeta }_n\left(h\right)>y,{\delta }_n\le c^{*}h\right)dy\le \\ \le C_1^{*}\mathbb{E}{\omega }_f\left(h\right)+(1+C_2^{*}){\gamma }_n,\end{array}$$

where ${\gamma }_n={\left(h^{-k(p/2+1)}\mathbb{E}{\delta }_n^{kp/2}\right)}^{1/p}$. To complete the proof of (64), it remains to estimate the first term on the right-hand side of (65) using Markov’s inequality, taking into account the limit relations from (14) and the last estimate.

The assertion of Theorem 2 follows from Lemma 9, limit relation (64), and the simple estimate

$$\underset{\mathbf{t}\in {[0,1]}^k}{sup}\left|N^{-1}\sum _{j=1}^N{\widehat{f}}_{n,h,j}\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(\mathbf{t}\right)\right|\le \underset{\mathbf{t}\in {[0,1]}^k}{sup}|{\overline{f}}_N\left(\mathbf{t}\right)-\mathbb{E}{f}_1\left(\mathbf{t}\right)|+N^{-1}\sum _{j=1}^N{\Delta }_{n,h,j}.$$

Thus, Theorem 2 is proved. □

5. Conclusions

This paper considers the classic problem of nonparametric regression: estimating a regression function from observations of its noisy values in some known set of points from the domain of its definition (design points). In order to estimate the regression function, you need to impose certain restrictions on the design. In numerous works by predecessors, it is usually assumed that the design points are either fixed and, in a certain sense, regularly fill the domain of definition of a regression function or random process, or are random and consist of independent or weakly dependent random variables. Our idea is as follows: in order to restore the function with respect to the design, it is enough to only require the condition for the design points to fill asymptotically densely the domain of the regression function as the sample size increases. In contrast to previously known results, the new condition is insensitive to the nature of the dependence of the design points and is essentially necessary to restore the function with varying the accuracy, and at the same time, includes both the situation of fixed design without requirements of regularity and random design, which may not satisfy the conditions of weak dependence.

We propose to implement this very clear idea by applying the construction of Riemann integral sums to the structure of new kernel estimators, which open up the opportunity to study asymptotic properties of the estimators due to the proximity of integral sums and corresponding integrals without using probabilistic limit theorems. We emphasize that this approach allows us to reconstruct a regression function without any specification regarding the structure of dependence of the design points and allows us to significantly weaken the previously known conditions for these characteristics.

In [51,52], we use local constant estimators, and in [53], we use local linear ones, but in the case of a one-dimensional design. In the present paper, new universal local linear kernel-type estimators of an unknown regression function are constructed in the case of a multivariate design.