Estimating the Conditional Density in Scalar-On-Function Regression Structure: k-N-N Local Linear Approach

Abstract

In this study, the problem of conditional density estimation of a scalar response variable, given a functional covariable, is considered. A new estimator is proposed by combining the k-nearest neighbors (k-N-N) procedure with the local linear approach. Then, the uniform consistency in the number of neighbors (UNN) of the proposed estimator is established. Such result is useful in the study of some data-driven rules. As a direct application and consequence of the conditional density estimation, we derive the UNN consistency of the conditional mode function estimator. Finally, to highlight the efficiency and superiority of the obtained results, we applied our new estimator to real data and compare it to its existing competitive estimator.

1. Introduction

Functional data analysis (FDA) has acquired a great deal of consideration in the last few years. We cite, for instance, some precursor contributions developed by [1,2,3], as well as some recent advances published in the special issues [4,5]. In this data analysis area, the nonparametric fitting is actually in intensive development (see, for examples, [6,7], among others). While many of the works in this field consider nonparametric estimation of the functional models by employing the kernel estimation method, we are interested in estimating the conditional density using the method of local linear (L-L) estimation smoothed by the k-nearest neighbors procedure (k-N-N).

The subject of this paper is an intersection between three emergent fields in statistical mathematics: functional statistics, local linear modeling, and the nearest number neighborhood. The first axis regarding the functional statistics analysis is in a colossal development. Indeed, to cite a few recent advances in this year, we refer to [8], who proposed to estimate the mean and the covariance function using the full quasi-likelihood and the kernel method. Ref. [9] used the functional Horvitz–Thompson estimator to approximate the confidence bands for the mean of curves. As a real-life application, they used their approach to fit the tourists’ daily expenditure in the Majella National Park in Italy. We refer to [10] for the quantile estimation in multivariate functional data. They constructed a new estimator obtained by combining nonparametric techniques, the time-varying coefficient model, and basic functions. Such an estimator is employed in air pollution forecasting and is used for estimation and prediction. For practical purposes, [11] provided a new clustering algorithm, allowing them to proceed without the normality assumption of the random function. Concerning the local linear analysis, a recent advance can be found in [12]. They studied an estimation of the volume under an ROC surface using a semiparametric regression model. In this cited work, the local linear approach is used to estimate the nonparametric competence of the location-scale regression models. Ref. [13] considered the local linear estimator of the derivative of the quantile regression. Estimation of semi-varying coefficient models for longitudinal data was studied by [14]. They established the asymptotic property of the estimator whose nonparametric component is obtained by the local linear approach. The third axis of this paper concerns the estimation of the nearest number neighborhood. The latter has been developed as a flexible tool for many practical issues in applied statistics. In particular, it was used in clustering analysis in [15], in classification by [16], and in learning machine by [17], among others.

1.1. Related Works

On one hand, among the wide existing literature in nonparametric functional statistics, we cite, for instance, [18], that considers the nonparametric estimation of the generalized regression function when the regressors are sampled from a continuous time process valued in semimetric space. They established the suboptimal rate of the $L_2$ consistency of the constructed estimator. Ref. [19] investigated another nonparametric functional model. They studied the recursive version of the kernel estimator of the density function without condition. They stated the asymptotic properties of the constructed estimator when the functional observations are spatially correlated. We return to [20] for an exhaustive overview in spatio-functional data analysis and its feasibility as an attractive approach in spatial big data modeling. Considering the ergodicity structure, [21] show the uniform consistency of the nonparametric estimator of functional modal regression. For recent results, Ref. [22] have established the asymptotic properties of the conditional density for continuous time processes valued in functional space. As application, they derived the convergence of the conditional mode and the conditional expectation. An alternative estimation of the conditional density was studied by [23]. They used Galerkin algorithm to provide an efficient cross-validation procedure for the bandwidth parameter choice. Their estimator was constructed using B-spline basis. We refer to [24] for new ideas on the nonparametric estimation in functional statistics. In particular, he combined the Bayesian procedure with nonparametric approach to estimate the regression function when the error is correlated. For the same regression function, [25] investigated another nonparametric technique based on recursive procedure with two timescales. He states the stochastic property of the constructed estimator and he compares it to the single timescale recursive estimator through simulated data. For practical issues, the functional statistics ideas have been used to model some clinical data by [26]. More precisely, they used the functional smoothing to analyze the dynamics of COVID-19 in different French departments. To this end, let us refer the reader interested by the recent advances in nonparametric functional data analysis to some survey papers or some special issues [7,27,28], among others.

On the other hand, motivated by its small bias property, the local linear estimation method has earned great attention in the FDA literature. In fact, this interest was started by [29] who established the mean quadratic consistency of the L-L estimator (L-L-E) of the regression operator, assuming that the regressor is valued in a Hilbert space. Next, the almost complete (a.co.) consistency of an alternative version of this estimator, based on covariate values in a Banach space, was proven by [30]. After that, Ref. [31] studied the strong consistency of a functional L-L-E of the conditional density and then gave its leading term in the mean square error in [32]. Furthermore, the k-N-N smoothing method has been introduced, in the field of FDA, by [33]. Since this pioneering work, many authors have become concerned with the nonparametric estimation method of the functional models based on the k-N-N approach (see, for example, the recent works [34,35] and references therein). It is well known that this smoothing method is well adapted to functional data because this approach has a sophisticated bandwidth selection method. In fact, the smoothing parameters are locally chosen concerning the vicinity of the functional data, allowing their local structure exploration.

1.2. Contribution

In this paper we wish to take the benefit of both methods in order to develop a sophisticated and comprehensive estimator of the conditional density. The idea is, therefore, to combine the two approaches, starting from the advantages of the two techniques. Thus, the novelty of this study is to proposed a new estimator of the functional conditional density by combining both the L-L approach and the k-N-N smoothing method. Such a combination allows us to obtain an attractive estimator with a small bias and overcome the bandwidth selection problem. However, there are two big challenges which are related to this new estimator: (i) in contrast to the kernel approach, where the smoothing parameter is a fixed scalar, the chosen bandwidth parameter is a random variable, making the study of the asymptotic properties of the k-N-N method more challenging than in the kernel approach, and (ii) the number of neighbors that define the bandwidth parameter is deterministic, whereas, in practice, the optimal one varies with respect to the data. Thus, traditional asymptotic properties obtained with a fixed number of neighbors are insufficient to cover the practical case. Therefore, the main objective of this study is to establish the a.co. consistency of the built estimator uniformly, concerning the number of neighbors, which allows us to incorporate the practical case.

1.3. Organization

In the remainder of this paper, we present our built estimator in Section 2, and our main results are presented in Section 3. Then, in Section 4, we emphasize the direct impact of our results in nonparametric functional data analysis. Finally, a practical implementation of the constructed estimator is studied in Section 5. The proof of the intermediate results is given in the Appendix.

2. Methodology: Notations and Necessary Background Knowledge

2.1. Model and Its Estimator

Assume n couple of sequence ${\left({\mathcal{X}}_i,Y_i\right)}_{i=1,2,\dots ,n}$ taken from the random variables $(\mathcal{X},Y)$. The latter pair is valued in the functional space $\mathfrak{F}\times \Re$, where $\mathfrak{F}$ is assumed to be a semi-metric space equipped with a semi-metric d. In the sequel, denote by $\mathfrak{X}$ a fixed curve in $\mathfrak{F}$, ${\aleph }_{\mathfrak{X}}$ a neighborhood of $\mathfrak{X}$, and set by $\mathbf{B}(\mathfrak{X},r)$ the closed ball in the semi-metric space $(\mathfrak{F},d)$. The $\mathfrak{X}$ and r are, respectively, the center and the radius of the ball $\mathbf{B}(\mathfrak{X},r)$. Further, we assume that the distribution of Y given $\mathfrak{X}$ is absolutely continuous and has a conditional probability density (CPD) $f(·|\mathfrak{X})$ such that:

There exist $b_1 b_2 \in (0,1]$ such that, for all $({\mathfrak{X}}_1,{\mathfrak{X}}_2)\in {\aleph }_{\mathfrak{X}}\times {\aleph }_{\mathfrak{X}}$ and $(y_1,y_2)\in {\Re }^2$,

$$|f\left(y_1\right|{\mathfrak{X}}_1)-f\left(y_2\right|{\mathfrak{X}}_2)|\le C\left(d^{b_1}({\mathfrak{X}}_1,{\mathfrak{X}}_2)+{|y_1-y_2|}^{b_2}\right).$$

(1)

The present paper aims to estimate the CPD using the L-L algorithm. The idea of this algorithm is based on linear approximation of the function $f(·|\mathfrak{X})$. The L-L algorithm is performed in vectorial case under the Taylor expansion of $f(·|\mathfrak{X})$. Such expansion is not available in FDA. Instead, we suppose that:

For all ${\mathfrak{X}}_0\in {\aleph }_{\mathfrak{X}}$, and for all $y\in \Re$,

$$\forall y\in \Re , f\left(y\right|{\mathfrak{X}}_0)=a\left(y\right|\mathfrak{X})+b\left(y\right|\mathfrak{X})d({\mathfrak{X}}_0,\mathfrak{X})+o\left(d({\mathfrak{X}}_0,\mathfrak{X})\right).$$

(2)

Observe that both regularity Assumptions (1) and (2) are standard condition in this context of functional local linear analysis. They are verified for many class of conditional density function. For example, it is verified when the conditional density of Y given $\mathfrak{X}$ is exponential distribution with parameter $\lambda =R\left(\mathfrak{X}\right)$, when $R(·)$ is Gateaux differentiable. Now, the L-L-k-N-N estimators of the functionals $a\left(y\right|\mathfrak{X})$ and $b\left(y\right|\mathfrak{X})$ are minimizers of

$$\underset{(a,b)\in {\Re }^2}{min}\sum _{i=1}^n{\left({♭}_l^{-1}\mathbf{K}\left({♭}_l^{-1}(y-Y_i)\right)-a-bd({\mathcal{X}}_i,\mathfrak{X})\right)}^2\mathbf{K}\left({\hslash }_k^{-1}d(\mathfrak{X},{\mathcal{X}}_i)\right),$$

(3)

where K is a real function of class $C^1$ supported within the interval (0, 1/2) and $({\hslash }_k,{♭}_l)$ are the k-N-N smoothing parameters defined by

$$\begin{array}{cc}& \hfill {\hslash }_k=min\{\hslash \in {\Re }^{+}, \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{h} \mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t} \sum _{i=1}^n {\mathbf{𝟙}}_{\mathbf{B}(\mathfrak{X},\hslash )}\left(X_i\right)=k\},\\ \hfill \mathrm{a}\mathrm{n}\mathrm{d}& {♭}_l=min\{♭\in {\Re }^{+}, \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{h} \mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t} \sum _{i=1}^n {\mathbf{𝟙}}_{(y-♭, y+♭)}\left(Y_i\right)=l\}.\end{array}$$

Taking ${\mathfrak{X}}_0\in {\aleph }_{\mathfrak{X}}$ one obtains $f(·|\mathfrak{X})=a(y\left|\mathfrak{X}\right)$. Then $b\left(y\right|\mathfrak{X})$ is a kind of derivative of $f(·|\mathfrak{X})$ with respect to $\mathfrak{X}$. Thus, (3) estimates $f(·|\mathfrak{X})$ and $b\left(y\right|\mathfrak{X})$. Both estimators can be explicitly expressed. Indeed, we have

$$\begin{array}{cc}\hfill {}^t\mathfrak{L}=\left(\begin{array}{c}1,1,\dots ,1\\ d({\mathcal{X}}_1,\mathfrak{X})\dots ,d({\mathcal{X}}_n,\mathfrak{X})\end{array}\right),& \mathbb{K}=\left(\begin{array}{c}\mathbf{K}\left({\hslash }_k^{-1}d(\mathfrak{X},{\mathcal{X}}_1)\right),0,\dots ,0\\ 0,\dots ,0,\mathbf{K}\left({\hslash }_k^{-1}d(\mathfrak{X},{\mathcal{X}}_n)\right))\end{array}\right),\\ \hfill \mathrm{a}\mathrm{n}\mathrm{d}& {\rm Y}=\left(\begin{array}{c}{♭}_l^{-1}\mathbf{K}\left({♭}_l^{-1}(y-Y_1)\right)\\ ⋮\\ {♭}_l^{-1}\mathbf{K}\left({♭}_l^{-1}(y-Y_n)\right)\end{array}\right).\end{array}$$

Using the partial derivative to show that the minimizers of (3) are zeros of

$${}^t\mathfrak{L}\left(\mathbb{K}\mathrm{Y}-\mathbb{K}\mathfrak{L}\left(\begin{array}{c}\widehat{a}\left(y\right|\mathfrak{X})\\ \widehat{b}\left(y\right|\mathfrak{X})\end{array}\right)\right)=0.$$

It follows that $\left(\begin{array}{c}\widehat{a}\left(y\right|\mathfrak{X})\\ \widehat{b}\left(y\right|\mathfrak{X})\end{array}\right)={{(}^t\mathfrak{L}\mathbb{K}\mathfrak{L})}^{-1}{(}^t\mathfrak{L}\mathbb{K}\mathrm{Y}).$ Hence, $\widehat{a}\left(y\right|\mathfrak{X})=(1,0){{(}^t\mathfrak{L}\mathbb{K}\mathfrak{L})}^{-1}{(}^t\mathfrak{L}\mathbb{K}\mathrm{Y}).$ Therefore, the L-L-k-N-N estimator of $f\left(y\right|\mathfrak{X})$ is

$$\tilde{f}\left(y\right|\mathfrak{X})=\frac{{\sum }_{i,j=1}^n{\mathrm{W}}_{ij}\left(\mathfrak{X}\right)\mathbf{K}\left({♭}_l^{-1}(y-Y_j)\right)}{{♭}_l{\sum }_{i,j=1}^n{\mathrm{W}}_{ij}\left(\mathfrak{X}\right)},$$

(4)

where

$$\begin{array}{cc}{\mathrm{W}}_{ij}\left(\mathfrak{X}\right)=\hfill & d({\mathcal{X}}_i,\mathfrak{X})\left(d({\mathcal{X}}_i,\mathfrak{X})-d({\mathcal{X}}_j,\mathfrak{X})\right)\hfill \\ & \times \mathbf{K}\left({\hslash }_k^{-1}d(\mathfrak{X},{\mathcal{X}}_i)\right)\mathbf{K}\left({\hslash }_k^{-1}d(\mathfrak{X},{\mathcal{X}}_j)\right),\hfill \end{array}$$

2.2. Functional Statistics Framework: General Assumptions

Our primary goal in this study is to establish the a.co. consistency of $\tilde{f}\left(y\right|\mathfrak{X})$ uniformly with respect to the numbers of neighbors $(k,l)\in (k_{1,n},k_{2,n})\times (l_{1,n},l_{2,n})$. To achieve this, we use C and C′ to denote some strictly positive generic constants, S a compact subset of ℜ, and we introduce the following class of functions, for $\iota =0,1,2 \mathrm{a}\mathrm{n}\mathrm{d} \kappa =0,1$,

$$\begin{array}{cc}{\mathcal{K}}^{\iota ,\kappa }=\left\{\left({·}_1,{·}_2\right)\mapsto \right.\hfill & {\gamma }^{-\iota }\mathbf{K}\left({\gamma }^{-1}d\right(\mathfrak{X},{·}_1\left)\right)d^{\iota }\left(·,\mathfrak{X}\right)\hfill \\ & \left.\times {\lambda }^{-\kappa }{\mathbf{K}}^{\kappa }\left({\lambda }^{-1}\right(y-{·}_2\left)\right), \gamma >0,\lambda >0\right\}\hfill \end{array}$$

Throughout this paper, we denote by $\mathcal{N}(ϵ,\mathcal{C},\parallel .{\parallel }_{L_2\left(Q\right)})$ the minimal number of open balls (with radius $ϵ$) which is essential to cover the set of functions $\mathcal{C}$ with respect to the $L_2\left(Q\right)$ metric. Moreover, for any application $\psi :\mathcal{C}\mapsto \mathrm{I}\mathrm{R}$, we set ${\parallel \psi \parallel }_{\mathcal{C}}={sup}_{g\in \mathcal{C}}\left|\psi \left(g\right)\right|.$ The basic assumptions which are needed to establish the results are commented on in the rest of this section.

Assumption 1.

Functional Regressor Condition

$$\begin{array}{cc}& {\displaystyle \forall r>0,\mathbb{P}\left(\mathcal{X}\in \mathbf{B}(\mathfrak{X},r)\right):={\zeta }_{\mathfrak{X}}\left(r\right)>0 and \forall s\in (0,1),\underset{r\to 0}{lim}\frac{{\zeta }_{\mathfrak{X}}\left(sr\right)}{{\zeta }_{\mathfrak{X}}\left(r\right)}={\tau }_{\mathfrak{X}}\left(s\right)}\\ & {\displaystyle such that \mathbf{K}(1/2){\tau }_{\mathfrak{X}}(1/2)-{\int }_0^{1/2}{\mathbf{K}}^{\prime }\left(s\right){\tau }_{\mathfrak{X}}\left(s\right)ds>0}\\ \hfill ,& {\displaystyle and (1/4)\mathbf{K}(1/2){\tau }_{\mathfrak{X}}(1/2)-{\int }_0^{1/2}{\left(s^2\mathbf{K}\left(s\right)\right)}^{\prime }{\tau }_{\mathfrak{X}}\left(s\right)ds>0,}\end{array}$$

Such condition is classic in the asymptotic theory of nonparametric functional statistics. Indeed, the first part of this condition is a fundamental tool in this setting (cf. [2] for a deeper discussion and more comments on this hypothesis). Recall that all the asymptotic results, in this topic, are expressed together with this quantity. The second part is also considered in many papers treating the nonparametric functional statistics (cf., for instance, [36]). The reader will find in [36] several examples on the small ball probability function, for which Assumption 1 is verified.

Assumption 2.

Measurability Condition

For ι = 0, 1, 2 and κ = 0, 1, the class of functions ${\mathcal{K}}^{\iota ,\kappa }$ are pointwise measurable classes. Notice that this weak condition of measurability is very important in our context. In fact, it allows us to express our claimed results in their usual form under the classical definition of the probability notion and then to avoid the complicated notion of outer probability or outer expectation (cf. [37]).

Assumption 3.

Entropy Condition

$$\underset{Q}{sup}{\int }_0^1\sqrt{1+log\mathcal{N}(ϵ\parallel F^{\iota ,\kappa }{\parallel }_{L_2\left(Q\right)},{\mathcal{K}}^{\iota ,\kappa },\parallel .{\parallel }_{L_2\left(Q\right)})}dϵ<\infty$$

where $F^{\iota ,\kappa }$ is the envelope function of the class ${\mathcal{K}}^{\iota ,\kappa }$ and ${\parallel .\parallel }_{L_2\left(Q\right)}$ is the $L_2\left(Q\right)$ norm. We point out that the supremum takes all probability measures Q in the functional space $\mathcal{F}$ with $Q\left(F^2\right)<\infty$. Elsewhere, it is well known that the asymptotic uniformity is closely linked to the notions of entropy and of compactness. The UNN consistency will then be obtained under a uniform entropy integral condition. Indeed, the uniform entropy integral condition is used to characterize the Donsker classes of functions (cf. [37]) in order to derive the uniform low limit or to evaluate the moment of the empirical processes. Notice that this consideration is frequently used in some similar contexts where one can find several versions. Moreover, it is more general than the VC-class condition (cf. [38]).

Assumption 4.

The Smoothing Parameter Condition

The sequences$\left(k_{1,n}\right), \left(k_{2,n}\right), \left(l_{1,n}\right), and \left(l_{2,n}\right)$verify:$\frac{l_{2,n}}{n}\to 0, {\zeta }_{\mathfrak{X}}^{-1}\left(\frac{k_{2,n}}{n}\right)\to 0$and

$$\frac{nlogn}{l_{1,n}min\left(n{\zeta }_{\mathfrak{X}}^{-1}\left(\frac{k_{1,n}}{n}\right),k_{1,n}\right)},n^{\vartheta -1}{l}_{1,n}\to \infty ,$$

for some ϑ > 0.

All these general assumptions are sufficiently weak relative to the different objects involved in the statement of our uniform consistency results. They cover and exploit the principal axes of this contribution, which are the topological structure of the functional variables, the probability measure in this functional space, the measurability concept on the class of function, and the uniformity controlled by the entropy propriety

3. Main Result and Proof

Then, the UNN consistency of $\tilde{f}(·|\mathfrak{X})$ is stated with its proofs in Theorem 1, below.

Theorem 1.

Based on Assumptions (1–4), we obtain ${sup}_{k_{1,n}\le k\le k_{2,n}}{sup}_{l_{1,n}\le l\le l_{2,n}}{sup}_{y\in S}|\tilde{f}\left(y\right|\mathfrak{X})-f\left(y\right|\mathfrak{X})|=O\left({\zeta }_{\mathfrak{X}}^{-1}{\left(\frac{k_{2,n}}{n}\right)}^{b_1}\right)+O{\left(\frac{l_{2,n}}{n}\right)}^{b_2}+O_{a.co.}\left(\sqrt{\frac{nlogn}{l_{1,n}{k}_{1,n}}}\right)$.

Proof of Theorem 1.

Set, for a fixed $\alpha \in ]0,1[$, that

$$z_n={\zeta }_{\mathfrak{X}}^{-1}{\left(\frac{k_{2,n}}{n}\right)}^{b_1}+{\left(\frac{l_{2,n}}{n}\right)}^{b_2}+\left(\sqrt{\frac{nlogn}{l_{1,n}{k}_{1,n}}}\right),$$

where

$${\displaystyle 4a_n={\zeta }_{\mathfrak{X}}^{-1}\left(\frac{\alpha k_{1,n}}{n}\right),b_n={\zeta }_{\mathfrak{X}}^{-1}\left(\frac{k_{2,n}}{n\alpha }\right),c_n=\frac{\alpha l_{1,n}}{n}\mathrm{a}\mathrm{n}\mathrm{d}{d}_n=\frac{l_{2,n}}{n\alpha }.}$$

Thus, for all ϵ > 0, we obtain

$$\begin{array}{c}\mathbb{P}\left\{\underset{k_{1,n}\le k\le k_{2,n}}{sup}\underset{l_{1,n}\le l\le l_{2,n}}{sup}\underset{y\in S}{sup}|\tilde{f}\left(y\right|\mathfrak{X})-f\left(y\right|\mathfrak{X})|\ge ϵz_n\right\}\hfill \\ \le \mathbb{P}\left\{{\displaystyle \underset{k_{1,n}\le k\le k_{2,n}}{sup}\underset{l_{1,n}\le l\le l_{2,n}}{sup}\underset{y\in S}{sup}|\tilde{f}\left(y\right|\mathfrak{X})-f\left(y\right|\mathfrak{X})|}\right.\hfill \\ \left.\times \mathbb{1}{\mathbf{𝟙}}_{\left\{a_n\le {\hslash }_k\le {\zeta }_{\mathfrak{X}}^{-1}\left(\frac{k_{2,n}}{n\alpha }\right),\frac{\alpha l_{1,n}}{n}\le {♭}_l\le \frac{l_{2,n}}{n\alpha }\right\}}\ge \frac{ϵz_n}{2}\right\}\hfill \\ +\mathbb{P}\left\{{\displaystyle {\hslash }_k\notin \left({\zeta }_{\mathfrak{X}}^{-1}\left(\frac{\alpha k_{1,n}}{n}\right),{\zeta }_{\mathfrak{X}}^{-1}\left(\frac{k_{2,n}}{n\alpha }\right)\right)}\right\}\hfill \\ +\mathbb{P}\left\{{\displaystyle {♭}_l\notin \left(\frac{\alpha l_{1,n}}{n},\frac{l_{2,n}}{n\alpha }\right)}\right\}.\hfill \end{array}$$

The last two probabilities have been evaluated in [35]. Therefore, it suffices to examine the first one. This latter is mainly a consequence of the two lemmas below. □

Lemma 1.

Based on the assumptions of Theorem 1, we obtain

$$\underset{c_n\le ♭\le d_n}{sup}\underset{a_n\le \hslash \le b_n}{sup}\left|\frac{\mathbb{E}\left[{\tilde{f}}_N\left(y\right|\mathfrak{X})\right]-f\left(y\right|\mathfrak{X})\mathbb{E}\left[{\tilde{f}}_D\left(\mathfrak{X}\right)\right]}{\mathbb{E}\left[{\tilde{f}}_D\left(\mathfrak{X}\right)\right]}\right|=O\left(b_n^{b_1}\right)+O\left(d_n^{b_2}\right),$$

where

$${\tilde{f}}_N\left(y\right|\mathfrak{X})=\frac{1}{n(n-1))♭{\hslash }^2{\zeta }_{\mathfrak{X}}^2(\hslash )}\sum _{i\ne j}{W}_{ij}(\mathfrak{X},\hslash )K\left({♭}^{-1}(y-Y_j)\right),$$

and

$${\tilde{f}}_D\left(\mathfrak{X}\right)=\frac{1}{n(n-1)){\hslash }^2{\zeta }_{\mathfrak{X}}^2(\hslash )}\sum _{i\ne j}{W}_{ij}(\mathfrak{X},\hslash ).$$

with$W_{ij}(\mathfrak{X},\hslash )=d({\mathcal{X}}_i,\mathfrak{X})\left(d({\mathcal{X}}_i,\mathfrak{X})-d({\mathcal{X}}_j,\mathfrak{X})\right)\mathbf{K}\left({\hslash }^{-1}d(\mathfrak{X},{\mathcal{X}}_i)\right)\mathbf{K}\left({\hslash }^{-1}d(\mathfrak{X},{\mathcal{X}}_j)\right)$,

Lemma 2.

Based on the Theorem 1 assumptions, we have

$$\begin{array}{c}\underset{y\in S}{sup}\underset{c_n\le ♭\le d_n}{sup}\underset{a_n\le h\le b_n}{sup}\frac{1}{{\tilde{f}}_D\left(\mathfrak{X}\right)}|{\tilde{f}}_N\left(y\right|\mathfrak{X})-\mathbb{E}\left[{\tilde{f}}_N\left(y\right|\mathfrak{X})\right]|\hfill \\ =O_{a.co.}\left(\sqrt{\frac{logn}{n{c}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right).\end{array}$$

4. Discussion on the Impact of the Main Result and Its Usefulness

• UNN consistency vs. fixed number neighborhood consistency. As discussed in the introduction, kNN smoothing is an important estimation algorithm in nonparametric statistics. Concerning the functional L-L-E of the conditional density, the literature is limited; only the authors of [34] have proposed an estimator based on the Hilbertian version of the local linear approach introduced by [33]. They derived the almost -consistency for the fixed number neighborhood of the proposed estimator. Alternatively, we use here the fast version, which can be adapted for semi-metric space and applied even if the functional variables are discontinuous, unlike the Hilbertian version that requires oversmoothing curves to use the Fourier decomposition. Moreover, we state here the uniform consistency with respect to the number of the neighborhood, which allows us to derive the asymptotic property of the estimator even if this number is random. This last statement has a significant impact in practice. In particular, it covers the realistic situation where the best estimator (that corresponds to the best smoothing parameters, according to the criterion (5)), has a random number neighborhood. Indeed, usually, the best estimator of the conditional density is obtained by using the following cross-validation criterion,

  $${\displaystyle ({\tilde{k}}_{CV},{\tilde{l}}_{CV})=arg\underset{k\in (k_{1,n},k_{2,n}),l\in (l_{1,n},l_{2,n})}{min}CV(k,l),}$$

  (5)

  where

  $$\begin{array}{cc}CV(k,l)=\hfill & \frac{1}{n}\sum _{i=1}^n{W}_1\left(X_i\right)\int {\tilde{f}}_{(-i)}^2\left(y\right|{\mathcal{X}}_i)W_2\left(y\right)dy\hfill \\ & -\frac{2}{n}\sum _{i=1}^n{\tilde{f}}_{(-i)}\left(Y_i\right|{\mathcal{X}}_i)W_1\left({\mathcal{X}}_i\right)W_2\left(Y_i\right)\hfill \end{array}$$

  (6)

  with ${\displaystyle {\tilde{f}}_{(-k)}\left(y\right|{\mathcal{X}}_k)=\frac{{\displaystyle \sum _{\genfrac{}{}{0pt}{}{i,j=1}{i\ne k,j\ne k}}^n{\mathrm{W}}_{ij}\left({\mathcal{X}}_k\right)K({♭}^{-1}-l(y-Y_i))}}{{\displaystyle {♭}_l\sum _{i,j=1}^n{\mathrm{W}}_{ij}\left({\mathcal{X}}_k\right)}}}$ is the leave-out-one estimator of the conditional density. We point out that this criterion is inspired by the local constant estimation method (see [39] for the vectorial case or [40] for the functional case). Moreover, this rule was also used for the local linear estimator, in the nonfunctional case, by [41]. However, still now there is no mathematical support to justify the optimality of selected parameters by this rule. In fact, all the previous studies fail to guarantee the convergence of the best estimator which is associated with the values of $\tilde{k}$ and $\tilde{l}$ because they only treat the simple case when k and l are deterministic, unlike the optimal case $\tilde{k}$ and $\tilde{l}$, which are random variables. Then, the determined uniform consistency result offers the opportunity to derive asymptotic properties of the built estimator even if the smoothing parameter is a random variable. Precisely, we obtain the following corollary.

Corollary 1.

If (${\tilde{k}}_{CV},{\tilde{l}}_{CV})\in (k_{1,n},k_{2,n})\times (l_{1,n},l_{2,n})$ and if the assumptions of Theorem 1 hold, then we obtain

$$\begin{array}{cc}|\widehat{f_{opt}}\left(y\right|\mathfrak{X})-f\left(y\right|\mathfrak{X})|=\hfill & O\left({\zeta }_{\mathfrak{X}}^{-1}{\left(\frac{k_{2,n}}{n}\right)}^{b_1}\right)+O{\left(\frac{l_{2,n}}{n}\right)}^{b_2}\hfill \\ & +O_{a.co.}\left(\sqrt{\frac{nlogn}{l_{1,n}{k}_{1,n}}}\right),\hfill \end{array}$$

where

$${\displaystyle \widehat{f_{opt}}\left(y\right|\mathfrak{X})=\frac{{\sum }_{i\ne j}{W}_{ij}(\mathfrak{X},{\hslash }_{{\tilde{k}}_{CV}})K\left({♭}_{{\tilde{l}}_{CV}}^{-1}(y-Y_j)\right)}{{♭}_{{\tilde{l}}_{CV}}{\sum }_{i\ne j}{W}_{ij}(\mathfrak{X},{\hslash }_{{\tilde{k}}_{CV}})}}$$

Notice that this result has at least one practical interest since it gives a mathematical justification to the previous cross-validation selection rule.

• Local linear estimator vs. classical kernel estimator. The second advantage of our proposed estimator is the fact that the estimator for the conditional density is constructed by combining two excellent algorithms: the k-N-N smoothing technique and the L-L approach. Such consideration permits accumulation of the advantages of the two approaches. In particular, the k-N-N method allows obtaining flexibility in the bandwidth selection, which is adapted to the local structure of the functional data. Precisely, the smoothing parameter is locally selected with respect to the vicinity at the conditioning point, which allows to take into account the local behavior of the data. Despite the practical interest of this approach in functional statistics, it is not fully explored in this field. The first result on the functional conditional mode estimation was determined by [42]. They study the asymptotic properties of the kNN version of the functional Nadraya–Watson estimator. As an alternative, we use in this paper the L-L estimation method. Note that the bias that produced by the L-L method is smaller that that for the standard kernel method. This merit is also produced in the UNN consistency. Indeed, if we replace the regularity Assumption (2), for all $\mathfrak{Z}\in {\aleph }_{\mathfrak{X}}$, by $f\left(y\right|\mathfrak{Z})=f\left(y\right|\mathfrak{X})+bd(\mathfrak{Z},\mathfrak{X})+c{d}^2(\mathfrak{X},\mathfrak{Z})+o\left(d^2(\mathfrak{X},\mathfrak{Z})\right),$ and then use ${\sum }_{i,j=1}^{n}d({\mathcal{X}}_j,\mathfrak{x}){\mathrm{W}}_{ij}\left(\mathfrak{X}\right)=0$, to reformulate the estimator ${\tilde{f}}^x$ bias term as follows:

  $$\begin{array}{c}\mathbb{E}\left[\tilde{f}\left(y\right|\mathfrak{X})\right]-f\left(y\right|\mathfrak{X})\hfill \\ =\mathbb{E}\left[\frac{{\sum }_{i,j=1}^n{W}_{ij}\left(\mathfrak{X}\right)\left({♭}^{-1}K\left({♭}^{-1}(y-Y_j)\right)-f\left(y\right|{\mathcal{X}}_j\right)}{{\sum }_{i,j=1}^n{\mathrm{W}}_{ij}\left(\mathfrak{X}\right)}\right]\hfill \\ +\mathbb{E}\left[\frac{{\sum }_{i,j=1}^n{W}_{ij}\left(\mathfrak{X}\right)\left(f^{X_j}y)-f\left(y\right|\mathfrak{X})+bd({\mathcal{X}}_j,d(\mathfrak{X},\mathfrak{Z}))\right)}{{\sum }_{i,j=1}^n{\mathrm{W}}_{ij}\left(\mathfrak{X}\right)}\right].\hfill \end{array}$$

Thus, using the same ideas that we used to prove Theorem 1, we obtain

$$\begin{array}{c}\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\mathbb{E}\left[\tilde{f}\left(y\right|\mathfrak{X})\right]-f\left(y\right|\mathfrak{X})\hfill \\ =O\left({\zeta }_{\mathfrak{X}}^{-1}{\left(\frac{k_{2,n}}{n}\right)}^2\right)+O{\left(\frac{l_{2,n}}{n}\right)}^2.\end{array}$$

Significantly, this bias term is better than the standard kernel estimator when one supposes the same regularity assumption.

4.1. Empirical Analysis and Simulation Study

This section is devoted to emphasizing the ability to compute the estimator $\tilde{f}(·|·)$ in practice. As mentioned in Section 4, the number of neighborhood (k, l) is the principal element in the estimator $\tilde{f}(·|·)$. Thus, we focus in this section on evaluation of the impact of this element in the behavior of this estimator. To do that we conduct a Monte Carlo study using artificial data drawn by the following regression model

$$Y={\int }_{-1}^{1}exp\left(-\frac{1}{1+X^2\left(t\right)}\right)dt+ϵ,$$

where ϵ is generated from standard normal distribution N (0, 1). Such consideration permits to explicitly express the conditional density of Y given X by

$$f\left(y\right|\mathfrak{X})=\frac{1}{\sqrt{2\pi }}exp\left(-\frac{{\left(y-{\int }_{-1}^{1}exp\left(-\frac{1}{1+{\mathfrak{X}}^2\left(t\right)}\right)dt\right)}^2}{2}\right).$$

Concerning the functional covariate $\mathfrak{X}$, we sampled from the following random curves

$$\mathcal{X}\left(t\right)=-0.5\ast log\left(W{t}^2\right)+0.4\ast Wt. W⇝Uniform(0,0.5).$$

All the random curves ${\mathcal{X}}_i^{’}$ are discretized 100 equispaced measurements in (−1, 1). The functional curves are plotted in Figure 1.

As discussed in Section 4, the smoothing parameter (k, l) choice is one of the principal issues resolved by the present contribution. Thus, to highlight the importance of the selector Procedure (5) we compare the optimal estimator $\widehat{f_{opt}}\left(y\right|\mathfrak{X})$ to an arbitrary one obtained by adding an arbitrary number (ı, j) to $\left({\tilde{k}}_{CV},{\tilde{l}}_{CV}\right)$. This optimal estimator $\widehat{f_{opt}}\left(y\right|\mathfrak{X})$ is associated with the exact optimal number $\left({\tilde{k}}_{CV},{\tilde{l}}_{CV}\right)$ selected from {5, 10, 15, 20, 25, ⋯ n/2}. Furthermore, the estimator $\tilde{f}(·|·)$ is computed by a quadratic kernel supported within (0, 1) and by considering the L₂ metric. The efficiency of the estimator is examined by the Mean Absolute Error (MAE) expressed by ${\displaystyle MAE=\frac{1}{n}\sum _{i=1}^n\left|f\left(Y_i\right|{\mathcal{X}}_i)-\tilde{f}\left(Y_i\right|{\mathcal{X}}_i)\right|}$. In

Table 1. MAE of the conditional density estimation with respect to the number neighborhood.

Sample Size	The Optimal Case	(ı, j) = (−3, −4)	(ı, j) = (−3, 4)	(ı, j) = (3, 4)
n = 50	0.18	0.54	0.34	0.29
n = 100	0.11	0.42	0.38	0.24
n = 150	0.08	0.33	0.22	0.20
n = 200	0.06	0.19	0.15	0.17

, we compare the MAE of the optimal estimator $\widehat{f_{opt}}\left(y\right|\mathfrak{X})$ to a different arbitrary one obtained by replacing $\left({\tilde{k}}_{CV},{\tilde{l}}_{CV}\right)$ by $\left({\tilde{k}}_{CV}+ı,{\tilde{l}}_{CV}+j\right)$. The additional number (ı, j) is arbitrary chosen.

Foe more readability of the impact of the selection of the optimal neighborhood number (k, l) we plot the conditional density function $f(·|·)$ versus their estimators for the three cases (small values (k, l), optimal values ${\tilde{k}}_{CV},{\tilde{l}}_{CV})$, and large values (k, l) and for fixed n = 100.. All these functions are plotted at arbitrary conditioning curves ${\mathfrak{X}}_0$ (randomly choosing).

The MAE presented in Table 1 as well as Figure 2 shows that the choice of the optimal number of neighborhood significantly impacts the efficiency of the estimator $\tilde{f}(·|·)$. Small value underestimates the function $f(·|·)$ giving a rough estimator, while large value overestimates the true conditional density, giving an oversmooth estimator. Finally, and unsurprisingly, even if the efficiency of the optimal estimator decreases with respect to the sample-size n, it has satisfactory finite sample performance for a moderate sample size, n = 500, as well as a large one, n = 200.

4.2. Real Data Application

In this part, we conduct an empirical study based on real data example to compare the behavior of the local linear procedure to the Nadaraya–Watson using the k-N-N smoothing approach. This example studies the sugar quality using near-infrared spectroscopy data. More precisely, we compare the estimator $\widehat{\theta }$ to its competitive one proposed by [42], which is defined by $\overline{\theta }\left(x\right)=argmax\frac{{\sum }_{i=1}^{n}K\left(h_k^{-1}d(\mathfrak{X},X_i)\right)K\left({\ell }_l^{-1}(y-Y_i)\right)}{{\ell }_l{\sum }_{i=1}^{n}K\left(h_k^{-1}d(\mathfrak{X},X_i)\right)}.$ Of course, our main goal is to the show the easy implantation of our estimator and to compare the performance of the L-L approach to the Nadaraya–Watson in the prediction setting. For this reason, we use the near-infrared dataset; the data link is provided in the “Data Availability Statement” section. The spectrometric data of this data sample is obtained by PELS50B fluorescence spectrometer using spectra emission from 275–560 nm measured in 0.5 nm intervals at seven excitation wavelengths. For sake of shortness, we use only the last excitation wavelengths. Formally, for this application, we consider as functional regressor X (·) the spectrometry data associated with the 571 emission measurements for the last excitation wavelength (230 nm). The data curves are plotted in Figure 3.

Concerning the choice of the real response variable Y, we point out that there exist lots of parameters to control the sugar quality, such as the sucrose content, ash content, moisture content, or its color in solution. The ash content is the most commonly used. It is determined by conductivity and gives the percentage of the inorganic impurities in refined sugar. Thus, for this prediction issue, we take Y to be the ash content.

Now, to make an equitable comparison between the estimators $\overline{\theta }$ and ${\displaystyle \widehat{\theta }=argmax\tilde{f}\left(y\right|\mathfrak{X})}$, we must deal with both estimators on the same conditions, in the sense that we use the same kernels and the same bandwidth selection rule. In this application study, we used the same rules discussed in [2] and available at http://www.math.univ-toulouse.fr/~ferraty/online-resources.html (accessed on 20 December 2021). Specifically, we used the quadratic kernel on (0, 1), and took d equal to the $L_2$ metric between the curves. We are certain that the numbers k and I were selected from the subset {5, 15, 25 ⋯, 75, 85, 90}. They were selected by splitting the data into subsets (input sample and output samples (200 + 65). The 200 observations of the input sample (learning sample) were used to select the best k and l by the same cross-validation rule used in [2]. The prediction results are shown in Figure 4, where we display the predicted values versus the actual values for the two estimators.

The two estimators $\widehat{\theta }$ and $\overline{\theta }$ behavior is evaluated by using the Mean square Error (MSE) defined by $MSE\left(\theta \right)=\frac{1}{65}{\sum }_{k=1}^{65}{(Y_i-\dot{\theta }\left(X_i\right))}^2$, where $(X_i,Y_i)$ represents the output observations (testing sample) and $\dot{\theta }$ refers to either $\widehat{\theta }$ and $\overline{\theta }$. The prediction result shows that the L-L approach has significant superiority over the Nadaraya–Watson method. Such superiority is confirmed by the MSE 3.78 of $\widehat{\theta }$ against 5.14 for $\overline{\theta }$. This statement is, of course, not surprising because the main aim of the L-L method is to reduce the bias of the Nadaraya–Watson.

5. Conclusions and Perspectives

This new contribution deals with the L-L-k-N-N estimation of the CPD function of a real random variable in scalar-on-function regression structure. The new considered fitting approach is achieved by combining the L-L approach with those employed by the k-N-N weighting procedure. Of course, this idea allows improving the estimator’s efficiency in practice by reducing the bias term of the estimator and overcoming the problem of the selection of the smoothing parameter. Furthermore, the asymptotic results proved in this contribution greatly impact practice. In particular, it gives mathematical support to justify the use of the different cross-validation selectors rules (see Corollary 1).

It should be noted that the estimation of the conditional density is also very useful in practice. In particular, it offers more precious information on the link between the input and the output variables. Indeed, unlike the classical regression that analyzes the central tendency of the data, the conditional density describes the behavior of the center as well as the behavior extremes of the data. Moreover, it has a pivotal role in statistical modelings such as prediction by conditional mode function, interval predictive by shortest conditional modal interval, risk analysis by shortfall expectation model, or association testing. Finally, let us point out that the significance of this contribution is also highlighted by the numerous open questions offered by the present work. We list, for instance, the treatment of the functional time-series case and/or the surface time-series case. Furthermore, the asymptotic distribution of the L-L-k-N-N estimation of the CPD is also an essential prospect of the present contribution.

In addition to the listed prospects, we add some open questions related to the correlation of the functional data. In particular, for the functional time series analysis, it will be very interesting to study the asymptotic properties of this kind of estimator when the data exhibits some correlation assumptions such as the strong mixing structure, the quasi-association dependency, and the ergodicity correlation. Moreover, the L-L-k-N-N estimation of the conditional density in the spatial structure is also an essential prospect of the present contribution. It would be useful in surface time series via the conditional mode or the predictive region. Finally, let us point out that using the L-L-k-N-N estimation to approximate other alternative models is also very beneficial to open tracks for the future. These alternative models include expectile regression, relative error regression, robust regression, median expected shortfall, and regression structure, such as single index model, semi-palatial linear regression, additive regression model, etc.

6. Proofs of the Intermediate Results

Proof of Lemma 1.

By the linearity of the expectation, we have, for all $\hslash \in (a_n,b_n)$ and $♭\in (c_n,d_n)$,

$$\mathbb{E}\left[{\tilde{f}}_N\left(y\right|\mathfrak{X})\right]-f\left(y\right|\mathfrak{X})\mathbb{E}\left[{\tilde{f}}_D\left(\mathfrak{X}\right)\right]=\frac{1}{{\hslash }^2♭{\zeta }_{\mathfrak{X}}^2(\hslash )}\mathbb{E}\left[W_{12}(\mathfrak{X},\hslash )\left[\mathbb{E}\left[H_2\left(y\right)|{\mathcal{X}}_2\right]-f\left(y\right|\mathfrak{X})\right]\right],$$

where $H_2\left(y\right)={♭}^{-1}\mathbf{K}\left({♭}^{-1}(y-Y_2)\right)$. Next, we use (1), uniformly on $y\in S$, $\hslash \in (a_n,b_n)$ and $♭\in (c_n,d_n)$, to get

$$\begin{array}{c}{\mathbf{𝟙}}_{\mathbf{B}(\mathfrak{X},h)}\left({\mathcal{X}}_2\right)|\mathbb{E}\left[H_2\left(y\right)\right|{\mathcal{X}}_2]-f\left(y\right|\mathfrak{X})|\\ \le {\int }_{\mathbb{R}}K\left(t\right)\left({\hslash }^{b_1}+\left|t\right|{}^{b_2}♭^{b_2}\right)dt.\hfill \end{array}$$

.

Thus, the claimed result of the above lemma is a consequence of the stationarity of the sample. □

Proof of Lemma 2.

First, start with the compactness property of the subset S that permits to write a sequence ${\left(t_k\right)}_k$ in S. Hence, we have $S\subset {\bigcup }_{k=1}^{s_n}(t_k-l_n, t_k+l_n),$ with $l_n=n^{-\frac{3\vartheta }{2}-\frac{1}{2}}\mathrm{a}\mathrm{n}\mathrm{d}{s}_n=O\left(l_n^{-1}\right)$.

Next, for all $y\in S$, we place ${\displaystyle t_y=arg\underset{t\in \{t_1,\dots ,t_{s_n}\}}{min}|y-t|}$, and we use the Lipschitz’s condition on the kernel K to obtain

$$\begin{array}{c}\hfill \underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\underset{y\in S}{sup}\frac{\left|{\tilde{f}}_N\left(y\right|\mathfrak{X})-{\tilde{f}}_N\left(t_y\right|\mathfrak{X})\right|}{{\tilde{f}}_D\left(\mathfrak{X}\right)}=O\left(\sqrt{\frac{logn}{n{d}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right).\end{array}$$

(7)

and

$$\begin{array}{c}\hfill \underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\underset{y\in S}{sup}\frac{\left|\mathbb{E}\left[{\tilde{f}}_N\left(y\right|\mathfrak{X})\right]-\mathbb{E}\left[{\tilde{f}}_N\left(t_y\right|\mathfrak{X})\right]\right|}{{\tilde{f}}_D\left(\mathfrak{X}\right)}=\left(\sqrt{\frac{logn}{n{d}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right).\end{array}$$

(8)

Thus, to complete the proof, we have to evaluate the quantity

$$s_n\underset{t_y\in \{t_1,\dots ,t_{s_n}\}}{max}\mathbb{P}\left(\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\left|{\tilde{f}}_N\left(t_y\right|\mathfrak{X})-\mathbb{E}\left[{\tilde{f}}_N\left(t_y\right|\mathfrak{X})\right]\right|>\eta \sqrt{\frac{logn}{n{d}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right),$$

for some η > 0.

To do that, we place for all $t_y\in \{t_1,\dots ,t_{s_n}\}, \iota =0,1,2, \kappa =0,1,$, ${\Delta }_i^{\iota ,\kappa }=\frac{1}{{\hslash }^{\iota }{♭}^{\kappa }}\left(\mathbf{K}\left({\hslash }^{-1}d(\mathfrak{X},{\mathcal{X}}_1)\right)d^{\iota }({\mathcal{X}}_1,\mathfrak{X}){\mathbf{K}}^{\kappa }\left({♭}^{-1}(t_y-Y_1)\right)\right.$, $-\mathbb{E}\left[\mathbf{K}\left({\hslash }^{-1}d\left(\mathfrak{X},{\mathcal{X}}_1\right)\right)d^{\iota }\left({\mathcal{X}}_1,\mathfrak{X}\right){\mathbf{K}}^{\kappa }\left({♭}^{-1}\left(t_y-Y_1\right)\right)\right])$ and we define $S^{\iota ,\kappa }=\frac{1}{n{\zeta }_{\mathfrak{X}}(\hslash )}{\sum }_{i=1}^n{\Delta }_i^{\iota ,\kappa }$. Thus, using similar ideas to those used by [31] to prove that the claimed result is a direct consequence of the below three intermediate results:

$$\sum _n{s}_n\mathbb{P}\left\{\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\sqrt{\frac{n{d}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}{logn}}\left|S^{\iota ,\kappa }-\mathbb{E}\left[S^{\iota ,\kappa }\right]\right|>\eta \right\}<\infty ,$$

(9)

$$\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}Cov(S^{0,1},S^{2,0})=o\left(\sqrt{\frac{logn}{n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right),$$

(10)

$$\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}Cov(S^{1,1},S^{1,0})=o\left(\sqrt{\frac{logn}{n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right).$$

(11)

□

Proof of (9).

We follow similar ideas to those in [38] to evaluate this quantity. In particular, it is based on the Bernstein’s inequality applied to the empirical process, where ${\alpha }_n^{\iota ,\kappa }\left(\mathbf{K}\right)=\frac{1}{\sqrt{n}}{\sum }_{i=1}^n{\Delta }_i^{\iota ,\kappa }$ for ι = 0, 1, 2 and κ = 0, 1. To achieve that, we place $h_j=2^j{a}_n$, ${♭}_{j\prime }=2^{j\prime }{c}_n$ and we define the following class of functions: ${\mathcal{G}}_{\mathbf{K},j,j^{\prime }}^{\iota ,\kappa }=\left\{(z,t)\to \mathbf{K}\left({\hslash }^{-1}d(\mathfrak{X},{\mathcal{X}}_1)\right)d^{\iota }({\mathcal{X}}_1,\mathfrak{X})\right.$, ${\mathbf{K}}^{\kappa }\left({♭}^{-1}(t_y-Y_1)\right),$, $\left.h_{j-1}\le \gamma \le h_j,{♭}_{j^{\prime }-1}\le \delta \le {♭}_{j^{\prime }}\right\}$. Therefore,

$$\begin{array}{c}\hfill \mathbb{P}\left\{\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\sqrt{\frac{n{c}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}{logn}}\left|S^{\iota ,\kappa }-\mathbb{E}\left[S^{\iota ,\kappa }\right]\right|\ge \eta \right\}\\ \hfill \le L\left(n\right)L^{\prime }\left(n\right)\mathbb{P}\left\{\underset{1\le k\le n}{max}{\parallel \sqrt{k}{\alpha }_k^{\iota ,\kappa }\left(\mathbf{K}\right)\parallel }_{{\mathcal{G}}_{\mathbf{K},j,j^{\prime }}}>\eta \sqrt{n{♭}_{j^{\prime }}{\zeta }_{\mathfrak{X}}(h_j/2)logn}\right\},\end{array}$$

where $L\left(n\right)=max\left\{j:h_j\le 2b_0\right\}$ and $L\prime =max\left\{j\prime :{♭}_{j\prime }\le 2d_0\right\}$. Applying the Bernstein’s inequality to the empirical process ${\alpha }_n^{\iota ,\kappa }\left(\mathbf{K}\right)$, with the envelop function of the $G_{\mathbf{K},j,j^{\prime }}$, such that $G(z,t)\le C{\mathbf{𝟙}}_{\mathbf{B}\left(\mathfrak{X},h_j/2\right)}\left(z\right){\mathbf{𝟙}}_{\left[t_y-{♭}_{j\prime }/2,t_y\right]}\left(t\right)$. It follows that ${\sigma }^2:=\mathbb{E}\left[G_{\mathbf{K},j,j\prime }^2\left(X,Y\right)\right]=O\left({♭}_{j\prime }{\xi }_{\mathfrak{X}}(h_j/2)\right)$ and $\mathbb{E}\left[G_{\mathbf{K},j,j\prime }^p\left(X,Y\right)\le C^p{\sigma }^2\right]$, ${\beta }_n=\mathbb{E}\left[\parallel \sqrt{n{\alpha }_n}\left(\mathbf{K}\right){\parallel }_{G_{K,H,J}}\right]=O\left(\sqrt{n{h}_{H,j}{\zeta }_{\mathfrak{X}}\left(h_j/2\right)}\right)$. Furthermore, for $t=\eta \sqrt{n{♭}_{j\prime },{\zeta }_{\mathfrak{X}}\left(h_j/2\right)logn}-\beta$, we obtain

$$\begin{array}{c}\hfill \mathbb{P}\left\{\underset{1\le k\le n}{max}{\parallel \sqrt{k}{\alpha }_k^{\iota ,\kappa }\left(\mathbf{K}\right)\parallel }_{{\mathcal{G}}_{\mathbf{K},j,j^{\prime }}}>\beta +t\right\}\le exp\left\{-{\eta }^2\frac{logn}{8+C\sqrt{\frac{logn}{nn{♭}_{j^{\prime }}{\zeta }_{\mathfrak{X}}(h_j/2)}}}\right\}\le n^{-C^{\prime }{\eta }^2}.\end{array}$$

Since L (n) ≤ 2 log n and L’ (n) ≤ 2 log n we obtain

$$\begin{array}{c}\hfill \mathbb{P}\left\{\underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}\sqrt{\frac{n{c}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}{logn}}\left|S^{\iota ,\kappa }-\mathbb{E}\left[S^{\iota ,\kappa }\right]\right|\ge \eta \right\}\le {log}^{2}n{n}^{-C^{\prime }{\eta }^2}.\end{array}$$

Hence, combining this last result with the definition of s_n to conclude (9). □

Proof of (10) and (11)).

We write that

$$\begin{array}{c}\hfill {\displaystyle \underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}Cov(S^{0,1},S^{2,0})=\frac{1}{n^2{\zeta }_{\mathfrak{X}}^2(\hslash )}\sum _{i}Cov\left({\Delta }_i^{0,1},{\Delta }_i^{2,0}\right).}\end{array}$$

and

$$\begin{array}{c}\hfill {\displaystyle \underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}Cov(S^{1,1},S^{1,0})=\frac{1}{n^2{\zeta }_{\mathfrak{X}}^2(\hslash )}\sum _{i}Cov\left({\Delta }_i^{1,1},{\Delta }_i^{1,0}\right).}\end{array}$$

By simple algebra, we prove that, for k′ = 0, 1 and l′ = 1, 2, $\left|Cov\left({\hslash }^{-k^{\prime }}\mathbf{K}\left({\hslash }^{-1}d(\mathfrak{X},{\mathcal{X}}_1)\right)d^{k^{\prime }}({\mathcal{X}}_1,\mathfrak{X})\mathbf{K}\left({♭}^{-1}(t_y-Y_1)\right),{\hslash }^{-l^{\prime }}\mathbf{K}\left({♭}^{-1}(t_y-Y_1)\right)\right)\right.\le C♭{\zeta }_{\mathfrak{X}}(\hslash /2)$. Consequently,

$$\begin{array}{c}\hfill \underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}Cov(S^{0,1},S^{2,0})=O\left(\frac{1}{n{\zeta }_{\mathfrak{X}}\left(a_n\right)}\right)=o\left(\sqrt{\frac{logn}{n{c}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right),\end{array}$$

and

$$\begin{array}{c}\hfill \underset{a_n\le \hslash \le b_n}{sup}\underset{c_n\le ♭\le d_n}{sup}Cov(S^{1,1},S^{1,0})=O\left(\frac{1}{n{\zeta }_{\mathfrak{X}}\left(a_n\right)}\right)=o\left(\sqrt{\frac{logn}{n{c}_n{\zeta }_{\mathfrak{X}}\left(a_n\right)}}\right).\end{array}$$

□