Forward Selection of Relevant Factors by Means of MDR-EFE Method

Abstract

The suboptimal procedure under consideration, based on the MDR-EFE algorithm, provides sequential selection of relevant (in a sense) factors affecting the studied, in general, non-binary random response. The model is not assumed linear, the joint distribution of the factors vector and response is unknown. A set of relevant factors has specified cardinality. It is proved that under certain conditions the mentioned forward selection procedure gives a random set of factors that asymptotically (with probability tending to one as the number of observations grows to infinity) coincides with the “oracle” one. The latter means that the random set, obtained with this algorithm, approximates the features collection that would be identified, if the joint distribution of the features vector and response were known. For this purpose the statistical estimators of the prediction error functional of the studied response are proposed. They involve a new version of regularization. This permits to guarantee not only the central limit theorem for normalized estimators, but also to find the convergence rate of their first two moments to the corresponding moments of the limiting Gaussian variable.

1. Introduction

This paper is dedicated to the eminent scientist Professor A.S. Holevo, academician of the Russian Academy of Sciences, on occasion of his remarkable birthday.

The classical problem of regression analysis consists in the search for deterministic function f, which, in a certain sense, “well” approximates the observed random variable (response) Y by the value $f\left(X\right)$, where $X=(X_1,\dots ,X_p)$ is a vector of factors influencing the behavior of Y. This approach was initiated by the works of A.-M. Legendre and K. Gauss. At that time it found application in the processing of astronomical observations. Nowadays one widely uses the methods involving the appropriate choice of unknown real coefficients ${\beta }_1,\dots ,{\beta }_p$ for a linear model of the form $Y={\sum }_{i=1}^p{\beta }_i{X}_i+\epsilon$, where $\epsilon$ describes a random error. Clearly, $X_0=1$ can be included in the collection of factors, then $Y={\beta }_0+{\sum }_{i=1}^p{\beta }_i{X}_i+\epsilon$. For example, books [1,2] are devoted to regression. The close tasks also arise in observations classification, see, e.g., [3].

Since the end of the 20th century, stochastic models have been studied where the random response Y depended only on some subset of the factors in the set of $X_1,\dots ,X_p$. So, in article [4], the LASSO method (Least Absolute Shrinkage and Selection Operator) was introduced, using the idea of regularization (going back to A.N.Tikhonov), which allowed to find factors included with non-zero coefficients in a “sparse” linear model. Somewhat earlier, this approach was used by several authors for the treatment of geophysical data. Generalizations of the mentioned method are considered in monograph [5]. We emphasize that the idea of identifying some of the factors having a principle (in a certain sense) impact on a response is also intensely developing within the framework of nonlinear models. Such direction of modern mathematical statistics is called Feature Selection (FS), i.e., the choice of features (variables, factors). In this regard, we refer, e.g., to monographs [6,7,8,9] and also to reviews [10,11,12,13,14]. In [10] the authors consider filter, wrapper and embedded methods of FS. They concentrate on feature elimination and also demonstrate the application of FS technique on standard datasets. In [11] the modern mainstream dimensionality reduction methods are analyzed including ones for small samples and those based on deep learning. In [12] FS machinery is considered based on filtering methods for detecting the cyber attacks. Survey [13] is devoted to FS methods in machine learning (the structured information is contained in 20 tables). The authors of [14] concentrate on applications of FS to stock market prediction and applications of FS in the analysis of credit risks are considered, e.g., in [15]. Beyond financial mathematics the choice of relevant factors is very important in medicine and biology. For instance, in the field of genetic data analysis there is an extensive research area called GWAS (Genome-Wide Association Studies) aimed at studying the relationships between phenotypes and genotypes, see, e.g., [16,17]. The authors of [18] provide the survey of starting methods used by genetic algorithms. Review [19] is devoted to the FS methods for predicting the risk of diseases. Thus, research in the field of FS is not only of theoretical interest, but also admits various applications.

Note that there are a number of complementary methods for identifying relevant factors. Much attention is paid to those employing the basic concepts of information theory such as entropy, mutual information, conditional mutual information, interaction information, various divergences, etc. Here statistical estimation of information characteristics plays an important role. One can mention, e.g., works [20,21]. In this article, the accent is made on identifying a set of relevant factors in the framework of a certain stochastic model, when the quality of the response approximation is evaluated by means of some metric.

Recall that J.B. Herrick in 1910 described the Sickle cell anemia (HbS). Later it was discovered that all clinical manifestations of the presence of HbS are the consequences of the single change in the B-globin gene. This famous example shows that even the search of a single feature having impact on a disease is reasonable. Nowadays the researchers concentrate on complex diseases provoked by several disorders of the human genome. Even identification of two SNPs (single nucleotide polymorphisms) having impact on a certain disease is of interest, see, e.g., [22].

Now we turn to the description of the studied mathematical model. All the considered random variables are defined on a probability space $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$. Let a random variable Y map $\mathrm{\Omega }$ to some finite set $\mathbb{Y}$. We assume that, for $k\in T:=\{1,\dots ,p\}$, a random variable $X_k:\mathrm{\Omega }\to M_k$, where $M_k$ is an arbitrary finite set. Then the vector $X=(X_1,\dots ,X_p)$ takes the values in $\mathbb{X}=M_1\times \dots \times M_p$. For a set $S=\{i_1,\dots ,i_r\}$, where $1\le i_1<\dots <i_r\le p$, we put $X_S:=(X_{i_1},\dots ,X_{i_r})$. Similarly, for $x\in \mathbb{X}$, $x_S$ denotes a vector $(x_{i_1},\dots ,x_{i_r})$. A collection of indices $S\subset T$ (the symbol ⊂ is everywhere understood as a non-strict inclusion) is called relevant if the following relation holds for any $x\in \mathbb{X}$ and $y\in \mathbb{Y}$:

$$\mathsf{P}(Y=y|X=x)=\mathsf{P}(Y=y|X_S=x_S),$$

(1)

whenever $\mathsf{P}(Y=y|X=x)\ne 0$. In this case, the set of factors $X_S$ is called relevant as well. If (1) takes place for some $S=S_0$ then it will be obviously valid for any S containing $S_0$. Therefore, the natural desire is to identify a set S that satisfies (1) and has cardinality $r<p$ (if such a set other than T exists). Note that there are different definitions of the relevant factors collection, see, e.g., [23,24] and the references therein.

It is assumed that a collection of relevant factors has r elements $(1\le r<p$), however, the set S itself, which appears in (1), is unknown and should be identified. We label this assumption as (A). There is no restriction that S satisfying (1) and containing r elements is unique. Usually the joint distribution of $(X,Y)$ is also unknown. Therefore, a statistical estimator of S is constructed based on the first N observations ${\xi }_N:=({\xi }^{\left(1\right)},\dots ,{\xi }^{\left(N\right)})$ of a sequence ${\xi }^{\left(1\right)},{\xi }^{\left(2\right)},\dots$, consisting of i.i.d. random vectors, where, for $k\in \mathbb{N}$, ${\xi }^{\left(k\right)}:=(X^{\left(k\right)},Y^{\left(k\right)})$ has the same distribution as the vector $(X,Y)$.

In 2001, the authors of [25] proposed a method for identifying relevant factors, called MDR (Multifactor Dimensionality Reduction). According to article [26], more than 800 publications were devoted to the development of this method and its applications in the period from 2001 to 2014. Research in this direction has continued over the last decade, see, e.g., [27,28,29]. In [30], for the binary response Y, a modification of the MDR method was introduced, namely, MDR-EFE (Error Function Estimation), based on statistical estimates of the error functional of the response prediction using the K-fold cross-validation procedure, see also [31]. Later this method was extended in [32] to study the non-binary response.

Recall how the MDR-EFE method is employed. Let a non-random function $f:\mathbb{X}\to \mathbb{Y}$ be used to predict the response Y by the values of the factors vector X. Further we exclude considering the trivial case when $Y=y_0$ with probability one for some $y_0\in \mathbb{Y}$ (hence, X and Y are independent). The prediction quality is determined by applying the following error functional

$$\mathsf{Err}\left(f\right):=\mathsf{E}|Y-f(X\left)\right|\psi \left(Y\right),$$

(2)

where a penalty function $\psi :\mathbb{Y}\to {\mathbb{R}}_{+}$. The functional $\mathsf{Err}$ takes finite values for the discrete X and Y under consideration. The function $\psi$ allows to take into account the importance of approximating a particular value of Y using $f\left(X\right)$.

In biomedical research, one often considers the binary response Y characterizing the patient’s state of health, say, the value $Y=1$ corresponds to illness, and $Y=-1$ means that the patient is healthy. In many situations it is more important to consider the disease detection, so the value of 1 is attributed more weight. Of interest is the situation when $\mathbb{Y}=\{-1,0,1\}$. Then the value 0 describes some intermediate state of uncertainty (“gray zone”). Following [32], we will consider a more general scheme when the set $\mathbb{Y}:=\{-m,\dots ,0,\dots ,m\}$ for some $m\in \mathbb{N}$. Lemma 1 in [32] describes for such model all optimal functions $f_{opt}$ that deliver a minimum to the error functional (2). Note that we can suppose that the set of values of Y is strictly contained in $\{-m,\dots ,m\}$, i.e., some values are accepted with zero probability. For such y, we assume that $\psi \left(y\right)=0$. Thus, it is possible to study Y taking values in an arbitrary finite subset of $\mathbb{Z}$. In order to simplify the notation, we further consider $\mathsf{P}(Y=y)>0$ for all $y\in \mathbb{Y}=\{-m,\dots ,m\}$.

It is proved that in the framework of model (1) the relation $f_{opt}=f^S$ is valid, where, for $x\in \mathbb{X}$ and $U\subset T$, $f^U\left(x\right)=f\left(x_U\right)$ and a function f is constructed in a due way. At the same time, for any $U\subset T$ such that $♯U=♯S$ (♯ denotes the cardinality of a finite set) and S appearing in (1), the following inequality is true:

$$\mathsf{Err}\left(f^S\right)\le \mathsf{Err}\left(f^U\right).$$

(3)

For $U\subset T$, the function $f^U$ is introduced further. It depends on the joint distribution of $(X,Y)$ which is usually unknown. Thus we use observations ${\xi }_N=\{(X^{\left(j\right)},Y^{\left(j\right)}),$$j=1,\dots ,N\}$ for statistical estimates of the functional $\mathsf{Err}\left(f^U\right)$, where $U\subset T$, and then select as an estimator of S the set U on which the minimum of the corresponding statistical estimate is attained. This approach is described in the next section of the article.

We underline that consideration of all subsets (of the set T) having the cardinality r in the mentioned comparison procedure (involving regularized estimators, as explained in Section 2) for statistical estimates of the error functional is practically unfeasible, when p is large and r is moderately large. Therefore, a number of suboptimal methods of sequential feature selection have emerged. Such methods are used in various approaches to identify sets of relevant factors.

Mainly, one aims either to sequentially add indexes at each step of the algorithm for constructing a statistical estimator of a set S appearing in (1), or to sequentially exclude features from the general set T. In [33], algorithms of forward selection, i.e., sequential addition of indexes to the initial set, based on information theory, are considered. The authors of [33] show that the various algorithms employed can be interpreted as procedures based on proper approximations of the certain objective function. In [34] the principle attention is paid to simple models describing the phenomenon of epistasis observed in genetics, when individual factors do not affect the response, and some combinations of them lead to essential effects (in statistics one says “synergy interaction” of factors). Besides we also demonstrated that a number of well-known algorithms, for instance, mRMR (Minimum Redundancy Maximum Relevance) using mutual information and/or interaction information with a sequential procedure for selecting relevant factors can lead to the identification of the desired set with probability which is negligibly small. In [35] a variant is proposed for sequential (forward) application of the MDR-EFE method within the binary response model involving the naive Bayesian classifier scheme. The latter means that, for any $y\in \{-1,1\}$ and all $x\in \mathbb{X}$, the following relation holds:

$$\mathsf{P}(X=x|Y=y)=\prod _{k=1}^p\mathsf{P}(X_k=x_k|Y=y).$$

(4)

In other words, the factors $X_1,\dots ,X_p$ are conditionally independent for a given response Y. In [35] the joint distribution of X and Y was assumed known.

The principle goal of our work is to derive, for a non-binary, in general, random response, the probability that a sequential selection of features based on the (forward) application of the MDR-EFE method, without assuming the validity of (4), leads to identifying a suboptimal set that would be constructed by means of the same method from observations with a known joint distribution of the response and the vector of factors.

This result builds on the central limit theorem (CLT) for statistical estimates of the prediction error functional for a possibly non-binary response, proved in [32], which extends the CLT for the binary response model studied by the author previously. In addition, for the purposes of this work, we found the convergence rate of the first two moments of the considered statistics to the corresponding moments of the limiting Gaussian variable as the number of observations tends to infinity.

The article has the following structure. Section 2 describes statistical estimates of the error functional (for a response prediction) based on the MDR-EFE method. We also introduce the regularized versions of these estimators. In Section 3, the convergence rate of the first two moments of the regularized estimators of the error functional to the corresponding moments of the limiting Gaussian variable is established. Section 4 contains the main result related to the forward selection of relevant factors. The concluding remarks are given in Section 5. The proof of elementary Lemma 2 is provided in Appendix A for completeness of exposition.

2. Error Functional Estimators

Consider, in general, a non-binary response, i.e., let $\mathbb{Y}:=\{-m,\dots ,0,\dots ,m\}$ for some $m\in \mathbb{N}$. In the framework of the introduced discrete model, Lemma 1 of [32] gives a complete description of the class of optimal functions $f_{opt}$ providing the minimum error $\mathsf{Err}\left(f\right)$, determined by (2), in the class of all functions $f:\mathbb{X}\to \mathbb{Y}$. To define such a function (included in the optimal class) for $x\in \mathbb{X}$, we deal with a vector $w\left(x\right)$ having components

$$w_y\left(x\right):=\psi \left(y\right)\mathsf{P}(Y=y,X=x),y\in \mathbb{Y}.$$

It can be easily seen that

$$\mathsf{Err}\left(f\right)=\sum _{y,z\in \mathbb{Y}}|y-z|\psi \left(y\right)\mathsf{P}(Y=y,f\left(X\right)=z)=\sum _{z\in \mathbb{Y}}\sum _{x\in A_z}{w}^{\top }\left(x\right)q\left(z\right),$$

(5)

where $A_z:=\{x\in \mathbb{X}:f\left(x\right)=z\}$, $q\left(z\right)$ is a column of $(2m+1)\times (2m+1)$ matrix Q having elements $q_{y,z}:=|y-z|$ (the element $q_{-m,-m}$ is located in the upper left corner of the matrix Q), ⊤ stands for the transposition of column vectors. In other words, one employs in (5) the scalar product of the vectors $w\left(x\right)$ and $q\left(z\right)$. Thus, search for an optimal function $f_{opt}$ means finding the partition of $\mathbb{X}$ into such sets $A_z$, $z\in \mathbb{Y}$, that provide the minimum value of the right-hand side of (5). Note also that, according to Formula (13) of [32], the error of response prediction can be written as follows:

$$\mathsf{Err}\left(f\right)=\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\psi \left(y\right)\mathsf{P}(Y=y,|f\left(X\right)-y|>i).$$

(6)

Let, for $y\in \mathbb{Y}$, the vector $\Delta \left(y\right)$ have the first $m+y$ components equal to 1, and the remaining $m-y+1$ components equal to $(-1)$. For any $x\in \mathbb{X}$, we introduce a vector $L\left(x\right)$ with $2m$ components having the form

$$L_y\left(x\right):=w^{\top }\left(x\right)\Delta \left(y\right),y\in \mathbb{Y},y>-m.$$

(7)

According to formula (11) of [32] one infers that

$$f_{opt}\left(x\right)=y⟺\left\{\begin{array}{cc}{L}_{-m+1}\left(x\right)\ge 0,\hfill & y=-m,\hfill \\ L_{y+1}\left(x\right)\ge 0,L_y\left(x\right)<0,\hfill & y\ne \pm m,\hfill \\ L_m\left(x\right)<0,\hfill & y=m.\hfill \end{array}\right.$$

(8)

The joint distribution of $(X,Y)$ is, in general, unknown. Therefore, the optimal function $f_{opt}$ cannot be found in practice, so an algorithm is used to predict it, i.e., to approximate by means of specified statistical estimators. The response prediction algorithm is defined as a function ${\widehat{f}}_{PA}={\widehat{f}}_{PA}(x,\xi \left(W\right))$ given for $x\in \mathbb{X}$ and a set of observations

$$\xi \left(W\right):=\{{\xi }^{\left(j\right)}=(X^{\left(j\right)},Y^{\left(j\right)}),j\in W\},W\subset \mathbb{N},♯W<\infty .$$

(9)

The function ${\widehat{f}}_{PA}$ takes values in the set $\mathbb{Y}$. It is assumed that the value of ${\widehat{f}}_{PA}(x,\xi \left(W\right))$ becomes close, in a certain sense, to $f\left(x\right)$ for x in a specified subset of the set $\mathbb{X}$ when W is sufficiently “massive”. More precisely, we consider a family of functions ${\widehat{f}}_{PA}$ that depend on sets $\xi \left(W\right)$ of different cardinalities, but we will not complicate the notation. Consider $M=\{x\in \mathbb{X}:\mathsf{P}(X=x)>0\}$. For $x\in \mathbb{X}$, $U\subset T$ and $y\in \mathbb{Y}$, introduce a vector $w^U\left(x\right)$ with components

$$w_y^U\left(x\right):=\left\{\begin{array}{cc}\psi \left(y\right)\mathsf{P}(Y=y,X_U=x_U),\hfill & x\in M,\hfill \\ 0,\hfill & x\notin M.\hfill \end{array}\right.$$

Set

$$L_y^U\left(x\right):={\left(w^U\left(x\right)\right)}^{\top }\Delta \left(y\right),y\in \mathbb{Y},y>-m.$$

(10)

For $U\subset T$, let $f^U$ be defined by means of a counterpart of formula (8), where $L_y^U\left(x\right)$ is now written instead of $L_y\left(x\right)$. Then, according to Section 5 of [32] (the notation $\alpha$ is used there instead of U), in the framework of model (1), the optimal function $f_{opt}=f^S$, where S appears in (1) and $♯S=r$. Therefore relation (3) is valid for $f^U$ corresponding to any $U\subset T$ with $♯U=r$ (the assumption (A) holds).

To introduce an algorithm for predicting the function $f^U$, we employ statistical estimators of the penalty function $\psi$, as well as the values $L_y^U\left(x\right)$, where $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $y>-m$. Consider

$$\psi \left(y\right):=1/\mathsf{P}(Y=y),where\mathsf{P}(Y=y)>0,y\in \mathbb{Y}.$$

(11)

In the case of a binary response, such a choice of the penalty function was proposed in [36], the justification for this choice is given in [31], see also Section 4 in [32]. For the specified function $\psi \left(y\right)$ and observations $\xi \left(W\right)$, where the finite set $W\subset \mathbb{N}$, we use

$$\widehat{\psi }(y,\xi \left(W\right)):=\left\{\begin{array}{cc}\frac{1}{\widehat{P}(y,\xi \left(W\right))},\hfill & \widehat{P}(y,\xi \left(W\right))\ne 0,\hfill \\ 0,\hfill & \widehat{P}(y,\xi \left(W\right))=0,\hfill \end{array}\right.$$

(12)

where the frequency estimator of a probability $\mathsf{P}(Y=y)$ has the form

$$\widehat{P}(y,\xi \left(W\right)):=\frac{1}{♯W}\sum _{j\in W}\mathbb{I}\{Y^{\left(j\right)}=y\},y\in \mathbb{N}.$$

(13)

It is not difficult to see that the strong law of large numbers for arrays of random variables (see, e.g., [37]) entails for finite sets $W_N\subset \mathbb{N}$, such that $♯W_N\to \infty$, the relation

$$\widehat{\psi }(y,\xi \left(W_N\right))\to \psi \left(y\right)a.s.,N\to \infty .$$

(14)

Let the prediction algorithm ${\widehat{f}}_{PA}^U(x,\xi \left(W_N\right))$ of a function $f^U\left(x\right)$ be constructed by means of formula (8) analogue, where, for $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $y>-m$, and $W_N\subset \{1,\dots ,N\}$, one uses now statistical estimators ${\widehat{L}}_y^{U,W_N}\left(x\right)$ of functions $L_y^U\left(x\right)$ introduced in (10). Namely, let us define the following random variables:

$${\widehat{w}}_y^{U,W_N}\left(x\right):=\widehat{\psi }(y,\xi \left(W_N\right))\frac{1}{♯W_N}\sum _{j\in W_N}\mathbb{I}\{Y^{\left(j\right)}=y,X_U^{\left(j\right)}=x_U\},y\in \mathbb{Y},$$

where $\widehat{\psi }(y,\xi \left(W_N\right))$ is an estimator of $\psi \left(y\right)$ appearing in (12). For $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $y>-m$, set

$${\widehat{L}}_y^{U,W_N}\left(x\right):={\widehat{w}}_y^{U,W_N}{\left(x\right)}^{\top }\Delta \left(y\right).$$

Replace the value $L_y\left(x\right)$ in (8) by ${\widehat{L}}_y^{U,W_N}\left(x\right)$. Then one can claim that

$${\widehat{f}}_{PA}^U(x,\xi \left(W_N\right))=y⟺\left\{\begin{array}{cc}{\widehat{L}}_y^{U,W_N}\left(x\right)\ge 0,\hfill & y=-m,\hfill \\ {\widehat{L}}_{y+1}^{U,W_N}\left(x\right)\ge 0,{\widehat{L}}_y^{U,W_N}\left(x\right)<0,\hfill & y\ne \pm m,\hfill \\ {\widehat{L}}_y^{U,W_N}\left(x\right)<0,\hfill & y=m.\hfill \end{array}\right.$$

(15)

For $K\in \mathbb{N}$, $K>1$, we take a partition of a set $\{1,\dots ,N\}$ into subsets

$$D_k\left(N\right):=\{(k-1)[N/K]+1,\dots ,k[N/K]\mathbb{I}\{k<K\}+N\mathbb{I}\{K=N\}\},$$

(16)

here $k=1,\dots ,K$, $\left[a\right]$ is an integer part of a number $a\in \mathbb{R}$, $\mathbb{I}\left\{A\right\}$ is an indicator of a set A. These sets are applied in the K-fold cross-validation procedure increasing the stability of statistical inference (cross-validation procedure is studied, e.g., in [38]). Following [32], the estimator of the functional $\mathsf{Err}\left(f^U\right)$, i.e., a statistical estimator of the prediction error functional for a function $f^U$ and observations ${\xi }_N:=\xi \left(\{1,\dots ,N\}\right)$, involving the K-fold cross-validation procedure, is given by the formula:

$${\widehat{Err}}_{K,N}\left(f^U\right):=\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{1}{K}\sum _{k=1}^K\widehat{\psi }(y,\xi \left(D_k\left(N\right)\right))$$

$$\times \frac{1}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y,|{\widehat{f}}_{PA}^U(X^{\left(j\right)},\xi \left({\overline{D}}_k\left(N\right)\right))-y|>i\},$$

(17)

where ${\overline{D}}_k\left(N\right):=\{1,\dots ,N\}\setminus D_k\left(N\right)$ and $\widehat{\psi }(y,\xi \left(D_k\left(N\right)\right))$ are evaluated according to (12) for $W_N=D_k\left(N\right)$, $k=1,\dots ,K$. The estimator (17) is a natural statistical analogue of the error functional (2) written in the form (6) when one employs the K-cross-validation procedure. Namely, instead of $\psi \left(y\right)$ we apply its statistical estimator of the type (12) and instead of f we use its approximation by means of prediction algorithm based on the part $D_k\left(N\right)$ of observations. To obtain the statistical estimators of the probability appearing in Formula (6) we write the corresponding average of indicator functions. One employs also the averaging over different parts of observations.

By Theorem 2 of [32], if $S=\{i_1,\dots ,i_r\}$ is a set of relevant factors, i.e., (1) holds, then, for each $\epsilon >0$ and any set $U=\{m_1,\dots ,m_r\}\subset T$, the following inequality takes place almost sure for all N large enough:

$${\widehat{Err}}_{K,N}\left(f^S\right)\le {\widehat{Err}}_{K,N}\left(f^U\right)+\epsilon .$$

(18)

Thus, it is natural to consider all subsets $U=\{m_1,\dots ,m_r\}\subset T$ and choose as a statistical estimator of a relevant collection of indices $(i_1,\dots ,i_r)$ a set U on which the minimum of ${\widehat{Err}}_{K,N}\left(f^U\right)$ is attained. Here we also note that, for the study of asymptotic properties of the error functional, the regularization of the prediction algorithm by means of a sequence of positive numbers ${\left({\epsilon }_N\right)}_{N\in \mathbb{N}}$ such that ${\epsilon }_N\to 0$, as $N\to \infty$, plays an important role. Namely, for $W_N\subset \{1,\dots ,N\}$, we define

$${\widehat{f}}_{PA,{\epsilon }_N}^U(x,\xi \left(W_N\right))=y⟺\left\{\begin{array}{cc}{\widehat{L}}_y^{U,W_N}\left(x\right)+{\epsilon }_N\ge 0,\hfill & y=-m,\hfill \\ {\widehat{L}}_{y+1}^{U,W_N}\left(x\right)+{\epsilon }_N\ge 0,{\widehat{L}}_y^{U,W_N}\left(x\right)+{\epsilon }_N<0,\hfill & y\ne \pm m,\hfill \\ {\widehat{L}}_y^{U,W_N}\left(x\right)+{\epsilon }_N<0,\hfill & y=m.\hfill \end{array}\right.$$

(19)

As in article [32], we assume that

$${\epsilon }_N\to 0+,\sqrt{N}{\epsilon }_N\to \infty ,N\to \infty .$$

(20)

Now we introduce a statistical estimator ${\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)$ using an analogue of Formula (17), where one employs ${\widehat{f}}_{PA,{\epsilon }_N}^U$ instead of ${\widehat{f}}_{PA}^U$. For the regularized statistical estimators, as mentioned in [32], the analogue of Formula (18) holds. In [32], for estimators ${\widehat{f}}_{PA,{\epsilon }_N}^U$ constructed when condition (20) is met, the CLT is established. In the next section we apply a slightly different regularization for the error functional estimates, which will permit us to specify the convergence rate of the first two moments of these estimators to corresponding moments of the limiting Gaussian variable. This result is not only of independent interest, but is also applied in Section 4.

3. Asymptotic Behavior of the First Two Moments of Statistical Estimators of the Error Functional

As noted in Section 2, we will use the penalty function (11). Therefore, for $W_N=D_k\left(N\right)$, as a strongly consistent estimator $\widehat{\psi }(y,D_k\left(N\right))$ of $\psi \left(y\right)$ we will employ the variable appearing in (12), denoted below as ${\widehat{\psi }}_{N,k}\left(y\right)$, where $y\in \mathbb{Y}$, $k=1,\dots ,K$, $N\in \mathbb{N}$. Recall that the estimator ${\widehat{\mathsf{Err}}}_{K,N}\left(f^U\right)$ is defined by formula (2). If the regularized version ${\widehat{f}}_{PA,{\epsilon }_N}$ is substituted into this estimator instead of ${\widehat{f}}_{PA}^U$, where $x\in \mathbb{X}$ and $N\in \mathbb{N}$, then the notation ${\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)$ is used. We will apply the following Corollary 3 of [32] established in the framework of a model satisfying (1).

Theorem 1 ([32]).

Let U be an arbitrary subset of T having the cardinality r, the function $f^U$ be defined after formula (10), ${\widehat{f}}_{PA,{\epsilon }_N}^U$ appear in (19) for observations ${\xi }_N$, and the sequence ${\left({\epsilon }_N\right)}_{N\in \mathbb{N}}$ satisfy condition (20). Then

$$\sqrt{N}\left({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{Err}\left(f^U\right)\right)\stackrel{\mathcal{D}}{\to }Z\sim N(0,{\sigma }^2\left(U\right)),N\to \infty ,$$

(21)

and in this case ${\sigma }^2\left(U\right)$ is the variance of a random variable

$$V\left(U\right):=\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{\mathbb{I}\{Y=y\}}{\mathsf{P}(Y=y)}\left(\mathbb{I}\right\{|f^U\left(X\right)-y|>i\}-\mathsf{P}\left(\right|f^U\left(X\right)-y|>i|Y=y\left)\right).$$

(22)

It is known that the convergence in distribution of random variables, in general, does not ensure the convergence of their moments even when the moments exist. We will manage to establish the convergence rate of the first two moments of the error functional statistical estimators to the corresponding moments of the limit random variable. For this purpose we slightly strength the condition of estimates regularization. We require that a sequence ${\left({\epsilon }_N\right)}_{N\in \mathbb{N}}$ satisfies the following condition:

$${\epsilon }_N\to 0+,\frac{{\epsilon }_N\sqrt{N}}{\sqrt{log\frac{1}{{\epsilon }_N}}}\to \infty ,N\to \infty .$$

(23)

Clearly, (23) implies the validity of (20). Relation (23) holds if one takes ${\epsilon }_N=N^{-\delta }$, $N\in \mathbb{N}$, where $\delta \in (0,1/2)$.

Lemma 1.

Let condition (23) be met. Then, for every $K\in \mathbb{N}$, $K>1$, and any $U\subset T$, the statistical estimators ${\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)$ satisfy the following relation:

$$N\mathsf{E}{({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{Err}\left(f^U\right))}^2\to {\sigma }^2\left(U\right),N\to \infty ,$$

(24)

where ${\sigma }^2\left(U\right)=\mathsf{var}V\left(U\right)$ and $V\left(U\right)$ is introduced in formula (22).

Proof of Lemma 1.

Let us fix an arbitrary set $U\subset T$. For each $N\in \mathbb{N}$ one has

$$\begin{array}{ccc}\hfill {\mathbb{Z}}_N& :=& \sqrt{N}\left({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{Err}\left(f^U\right)\right)=\sqrt{N}({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-{\widehat{T}}_N\left(f^U\right))\hfill \\ & +& \sqrt{N}({\widehat{T}}_N\left(f^U\right)-T_N\left(f^U\right))+\sqrt{N}(T_N\left(f^U\right)-\mathsf{Err}\left(f^U\right)),\hfill \end{array}$$

(25)

where

$$T_N\left(f^U\right):=\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{1}{K}\sum _{k=1}^K\frac{\psi \left(y\right)}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y,|f^U\left(X^{\left(j\right)}\right)-y|>i\},$$

(26)

$${\widehat{T}}_N\left(f^U\right):=\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{1}{K}\sum _{k=1}^K\frac{{\widehat{\psi }}_{N,k}\left(y\right)}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y,|f^U\left(X^{\left(j\right)}\right)-y|>i\},$$

(27)

${\widehat{\psi }}_{N,k}\left(y\right)$ are defined by means of (12) for $W_N=D_k\left(N\right)$, $k=1,\dots ,K$, $N\in \mathbb{N}$. The proof is divided into several steps.

Step 1. At first we consider

$$R_N:=\sqrt{N}({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-{\widehat{T}}_N\left(f^U\right)),N\in \mathbb{N}.$$

To simplify the notation, we do not write that $R_N$ also depends on K, ${\xi }_N$ and ${\epsilon }_N$. Our aim is to show that if (23) holds then

$$\mathsf{E}{R}_N^2\to 0asN\to \infty .$$

(28)

In the light of formula (71) of [32], under condition (20) the following relation is valid:

$$R_N\stackrel{\mathsf{P}}{\to }0,N\to \infty .$$

(29)

Taking into account (29), by Theorem 5.4 of [39], relation (28) holds if (and only if) the sequence ${\left(R_N^2\right)}_{N\in \mathbb{N}}$ is uniformly integrable. Due to theorem by De La Vallé - Poussin (see, e.g., Theorem 1.3.4 of [40]) it is sufficient to verify that

$$\underset{N\in \mathbb{N}}{sup}\mathsf{E}\left(R_N^4\right)<\infty .$$

For $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $i\in {\mathbb{Z}}_{+}$, $k=1,\dots ,K$ and $N\in \mathbb{N}$ we introduce the following random variables:

$$F_{N,k}^{\left(i\right)}(x,y)=\mathbb{I}\left\{\right|{\widehat{f}}_{PA,{\epsilon }_N}^U(x,\xi \left({\overline{D}}_{N,k}\right))-y|>i\}-\mathbb{I}\left\{\right|f^U\left(x\right)-y|>i\},$$

(30)

$${\mathbb{S}}_k(i,y):=\frac{1}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y\}{F}_{N,k}^{\left(i\right)}(X^{\left(j\right)},y),$$

(31)

where, for $W\subset \mathbb{N}$, $\xi \left(W\right)$ is defined by Formula (9). Write $R_N=U_{N,1}+U_{N,2}$, here

$$\begin{array}{ccc}\hfill U_{N,1}& :=& \sqrt{N}\left(\frac{1}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\psi \left(y\right){\mathbb{S}}_k(i,y)\right),\hfill \\ \hfill U_{N,2}& :=& \sqrt{N}\left(\frac{1}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right)){\mathbb{S}}_k(i,y)\right).\hfill \end{array}$$

Now note that, for any real numbers $a_1,\dots ,a_v$, every $v\in \mathbb{N}$ and an arbitrary $\gamma >1$, the Hölder inequality implies that

$${\left(\sum _{r=1}^v\left|a_r\right|\right)}^{\gamma }\le v^{\gamma -1}\sum _{r=1}^v{\left|a_r\right|}^{\gamma }.$$

(32)

Evidently, (32) is true for $\gamma =1$ as well. Consequently, we get

$$R_N^4\le 8(U_{N,1}^4+U_{N,2}^4),N\in \mathbb{N}.$$

(33)

Clearly, for all $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $W_N\subset \{1,\dots ,N\}$ and $N\in \mathbb{N}$, one has

$${\widehat{L}}_{y,{\epsilon }_N}^{U,W_N}\left(x\right):={\widehat{L}}_y^{U,W_N}\left(x\right)+{\epsilon }_N=L_y^U\left(x\right)+{({\widehat{w}}_y^{U,W_N}\left(x\right)-w_y\left(x\right))}^{\top }\Delta \left(y\right)+{\epsilon }_N,$$

(34)

where the functions appearing in (34) were introduced in Section 2. For any $x\in \mathbb{X}$ and $y\in \mathbb{Y}$, the inequalities $L_y^U\left(x\right)\ge 0$, $L_{y+1}^U\left(x\right)<0$ are satisfied if and only if, for arbitrary ${\delta }_N(x,y;U)>0$ such that ${\delta }_N(x,y;U)\to 0$, as $N\to \infty$, and all sufficiently large $N\in \mathbb{N}$, the following inequalities are valid: $L_y^U\left(x\right)+{\delta }_N(x,y;U)>0$, $L_{y+1}^U\left(x\right)+{\delta }_N(x,y;U)<0$ (the analogous statement is true for inequalities corresponding to coordinates $y=m$ and $y=-m$ in Formula (19)). Obviously,

$$|{({\widehat{w}}_y^{U,W_N}\left(x\right)-w_y\left(x\right))}^{\top }\Delta \left(y\right)|$$

$$\le |\widehat{\psi }(y,\xi \left(W_N\right))-\psi \left(y\right)|+\psi \left(y\right)\left|\frac{1}{♯W_N}\sum _{q\in W_N}\mathbb{I}\{X_U^{\left(q\right)}=x_U,Y^{\left(q\right)}=y\}-\mathsf{P}(X_U=x_U,Y=y)\right|,$$

where $\widehat{\psi }(y,\xi \left(W_N\right))$ is defined in (12). One has

$$\begin{array}{ccc}& & \sum _{x_U}\left(\frac{1}{♯W_N}\sum _{q\in W_N}\mathbb{I}\{X_U^{\left(q\right)}=x_U,Y^{\left(q\right)}=y\}-\mathsf{P}(X_U=x_U,Y=y)\right)\hfill \\ & & =\frac{1}{♯W_N}\sum _{q\in W_N}\mathbb{I}\{Y^{\left(q\right)}=y\}-\mathsf{P}(Y=y)\hfill \\ & & =\widehat{P}(y,\xi \left(W_N\right))-\mathsf{P}(Y=y).\hfill \end{array}$$

(35)

For $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $W_N\subset \{1,\dots ,N\}$ and $N\in \mathbb{N}$, consider the following event

$$A_{W_N}(x,y)=\left\{\left|\frac{1}{♯W_N}\sum _{q\in W_N}\mathbb{I}\{X_U^{\left(q\right)}=x_U,Y^{\left(q\right)}=y\}-\mathsf{P}(X_U=x_U,Y=y)\right|\le \frac{p_0^2{\epsilon }_N}{8♯\mathbb{X}}\right\},$$

(36)

where $p_0={min}_{y\in \mathbb{Y}}\mathsf{P}(Y=y)$ (we assumed that $\mathsf{P}(Y=y)>0$ for $y\in \mathbb{Y}$). More precisely one can write $A_{W_N}(x,y)=A_{W_N}(x,y,U;\{(X^{\left(q\right)},Y^{\left(q\right)}),q\in W_N\})$. We will not include a set U in the list of arguments since this set is fixed. Then, for $\omega \in A_{W_N}(x,y)$, in view of (35), we get

$$\left|\widehat{P}(y,\xi \left(W_N\right))-\mathsf{P}(Y=y)\right|\le \frac{p_0^2{\epsilon }_N}{8}.$$

(37)

Then by virtue of (37), for any $y\in \mathbb{Y}$ and all N large enough, i.e., for $N\ge N_0(\mathbb{Y},{\left({\epsilon }_N\right)}_{N\in \mathbb{N}})$, one has

$$\widehat{P}(y,\xi \left(W_N\right))\ge \mathsf{P}(Y=y)-\frac{p_0^2{\epsilon }_N}{8}\ge \mathsf{P}(Y=y)-\frac{{\epsilon }_N}{8}>\frac{\mathsf{P}(Y=y)}{2}>0,$$

and hence the following relation holds

$$|\widehat{\psi }(y,\xi \left(W_N\right))-\psi \left(y\right)|=\frac{|\widehat{P}(y,\xi \left(W_N\right))-\mathsf{P}(Y=y)|}{\widehat{P}(y,\xi \left(W_N\right))\mathsf{P}(Y=y)}\le \frac{\frac{p_0^2{\epsilon }_N}{8}}{\frac{\mathsf{P}{(Y=y)}^2}{2}}\le \frac{{\epsilon }_N}{4}.$$

(38)

Thus if $\omega \in A_{W_N}(x,y)$, where $x\in \mathbb{X}$ and $y\in \mathbb{Y}$, then according to (36) and (38), for all N large enough, we can write

$$|{({\widehat{w}}_y^{U,W_N}\left(x\right)-w_y\left(x\right))}^{\top }\Delta \left(y\right)|\le \frac{{\epsilon }_N}{4}+\left(\frac{1}{p_0}\right)\frac{p_0^2{\epsilon }_N}{8♯\mathbb{X}}\le \frac{{\epsilon }_N}{2}.$$

Taking into account that the sets $\mathbb{X}$ and $\mathbb{Y}$ have finite cardinalities, we ascertain that, for any $x\in \mathbb{X}$, $y\in \mathbb{Y}$ and all N large enough, for $\omega \in A_{W_N}(x,y)$, one has

$${\widehat{f}}_{PA,{\epsilon }_N}^{U,W_N}\left(x\right)=f^U\left(x\right).$$

(39)

Consequently, for any $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $i=0,1,\dots ,2m-1$, $\omega \in A_{W_N}(x,y)$, where $W_N={\overline{D}}_k\left(N\right)$, $k=1,\dots ,K$, for all N large enough (i.e., $N\ge N_1$), the following inequality holds:

$$F_{N,k}^{\left(i\right)}(x,y)\mathbb{I}\left\{A_{{\overline{D}}_k\left(N\right)}(x,y)\right\}=0.$$

(40)

Applying (32) we come to the inequality

$$|U_{N,1}{|}^4\le N^2\frac{{\left(2m\right)}^6}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\psi {\left(y\right)}^4{\left(\frac{1}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y\}{F}_{N,k}^{\left(i\right)}(X^{\left(j\right)},y)\right)}^4.$$

Let $\tilde{\Sigma }$ denote the summation over all $x_j\in \mathbb{X}$ for $j\in D_k\left(N\right)$. For $N\ge N_1$ one has

$$\begin{array}{ccc}& & \mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y\}{F}_{N,k}^{\left(i\right)}(X^{\left(j\right)},y)\right)}^4\hfill \\ & =& \mathsf{E}\left(\tilde{\Sigma }{\left(\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y\}{F}_{N,k}^{\left(i\right)}(x_j,y)\right)}^4\mathbb{I}\left\{\bigcap _{j\in D_k\left(N\right)}\{X^{\left(j\right)}=x_j\}\right\}\right)\hfill \end{array}$$

$$\begin{array}{ccc}& =& \mathsf{E}\left(\tilde{\Sigma }{\left(\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y\}{F}_{N,k}^{\left(i\right)}(x_j,y)\mathbb{I}\left\{{\overline{A}}_{{\overline{D}}_k\left(N\right)}(x_j,y)\right\}\right)}^4\mathbb{I}\left\{\bigcap _{j\in D_k\left(N\right)}\{X^{\left(j\right)}=x_j\}\right\}\right)\hfill \\ & =& \mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}\mathbb{I}\{Y^{\left(j\right)}=y\}{F}_{N,k}^{\left(i\right)}(X^{\left(j\right)},y)\mathbb{I}\left\{{\overline{A}}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y)\right\}\right)}^4\hfill \\ & \le & \mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}\mathbb{I}\left\{{\overline{A}}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y)\right\}\right)}^4,\hfill \end{array}$$

here we employ (40) and take into account that $|F_{N,k}^{\left(i\right)}(x,y)|\le 1$. We see that

$$|U_{N,1}{|}^4\le N^2\frac{{\left(2m\right)}^6}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{\psi {\left(y\right)}^4}{{(♯D_k\left(N\right))}^4}{\left(\sum _{j\in D_k\left(N\right)}\mathbb{I}\left\{{\overline{A}}_{{\overline{D}}_N\left(k\right)}(X^{\left(j\right)},y)\right\}\right)}^4.$$

(41)

For $W_N\subset \{1,\dots ,N\}$, $y\in \mathbb{Y}$ and $j=1,\dots ,N$, introduce the functions

$$g_{W_N}(X^{\left(j\right)},y)=\mathbb{I}\left\{{\overline{A}}_{W_N}(X^{\left(j\right)},y)\right\}=\mathbb{I}\left\{{\overline{A}}_{W_N}(X^{\left(j\right)},y;\{(X^{\left(q\right)},Y^{\left(q\right)}),q\in W_N\})\right\}.$$

It is known (see, e.g., formula (15) in Chap. VI of [41]) that if a bounded Borel function $g:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$, $\xi$ and $\zeta$ are independent random vectors taking values in ${\mathbb{R}}^n$ and ${\mathbb{R}}^m$, respectively, then

$$\mathsf{E}\left(g(\xi ,\zeta )\right|\zeta =z)=\mathsf{E}g(\xi ,z),z\in {\mathbb{R}}^n.$$

Due to independence of $(X^{\left(j\right)},Y^{\left(j\right)})$, $j\in \mathbb{N}$, we can apply the lemma on grouping random vectors (see, e.g., [42], p. 28) to get the relation

$$\mathsf{E}\left(\right(\sum _{j\in D_k(N)}{g}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;(X^{\left(q\right)},Y^{\left(q\right)}),q\in {\overline{D}}_k\left(N\right))){)}^4\left|(X^{\left(q\right)},Y^{\left(q\right)})=(x_q,y_q),q\in {\overline{D}}_k\left(N\right))\right)$$

$$=\mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}{g}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;(x_q,y_q)),q\in {\overline{D}}_N\left(k\right)\left)\right)\right)}^4.$$

By the Rosenthal inequality (see, e.g., Theorem 2.9 of [43]), for independent centered random variables $Z_1,\dots ,Z_v$, having $\mathsf{E}|Z_j{|}^t<\infty$ for some $t\in [2,\infty )$ and each $j=1,\dots ,v$, one has

$$\mathsf{E}{\left|\sum _{j=1}^v{Z}_j\right|}^t\le C\left(t\right)\left(\sum _{j=1}^v\mathsf{E}{\left|Z_j\right|}^t+{\left(\sum _{j=1}^v\mathsf{E}{Z}_j^2\right)}^{\frac{t}{2}}\right),$$

(42)

where $C\left(t\right)>0$ depends on t but does not depend on v and distributions of variables $Z_j$, $j=1,\dots ,v$.

Set ${\eta }_{N,k}^{\left(j\right)}:=g_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;\left\{(x_q,y_q)\right),q\in {\overline{D}}_N\left(k\right)\})$, $j\in \mathbb{N}$. Note that $0\le {\eta }_{N,k}^{\left(j\right)}\le 1$ for all $j\in D_N\left(k\right)$. Then according to (42) we come to the inequality

$$\mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}({\eta }_{N,k}^{\left(j\right)}-\mathsf{E}{\eta }_{N,k}^{\left(j\right)})\right)}^4\le C{(♯D_k\left(N\right))}^2,$$

where $k=1,\dots ,K$ and $C=2C\left(4\right)$. Hence, applying (32) for $\gamma =4$ and $v=2$, one has

$$\mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}{\eta }_{N,k}^{\left(j\right)}\right)}^4\le 8\left(C{(♯D_k\left(N\right))}^2+{\left(\sum _{j\in D_k\left(N\right)}\mathsf{E}{\eta }_{N,k}^{\left(j\right)}\right)}^4\right)$$

$$\le 8C{(♯D_k\left(N\right))}^2+8{(♯D_k\left(N\right))}^4\underset{j\in D_k\left(N\right)}{max}{\left(\mathsf{E}{\eta }_{N,k}^{\left(j\right)}\right)}^4.$$

Evidently, we can write

$$\mathsf{E}\left({\eta }_{N,k}^{\left(j\right)}\right)=\mathsf{P}\left({\overline{A}}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;\left\{(x_q,y_q)\right),q\in {\overline{D}}_k\left(N\right)\}\right).$$

Let $M_k=♯{\overline{D}}_k\left(N\right)$, where $M_k=M_k\left(N\right)$, $k=1,\dots ,K$. Set ${\zeta }_q=\mathbb{I}\{X_U^{\left(q\right)}=x_U,Y^{\left(q\right)}=y\}$, where $q\in {\overline{D}}_k\left(N\right)$, ${\sigma }_0^2=\mathsf{var}{\zeta }_q$. Clearly, ${\zeta }_q$ depends on $x_U$, y and U. Random variables ${\zeta }_q$ are identically distributed for $q\in \mathbb{N}$. Therefore ${\sigma }_0^2={\sigma }_0^2(U,x,y)$, but does not depend on q. If ${\sigma }_0^2=0$, then the variables ${\zeta }_q$ are a.s. equal to some constant. According to (36), an event ${\overline{A}}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;\left\{(x_q,y_q)\right),q\in {\overline{D}}_k\left(N\right)\})$ occurrence means that the variable which is equal to zero a.s. turns greater than $\left(p_0^2{\epsilon }_N\right)/(8♯\mathbb{X})$. Therefore, in the degenerate case one has

$$\mathsf{P}({\overline{A}}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;(x_q,y_q)),q\in {\overline{D}}_k\left(N\right)))=0$$

and $\mathsf{E}{\eta }_{N,k}^{\left(j\right)}=0$ for all $j=1,\dots ,N$. Consider now the case when ${\sigma }_0^2>0$. Then we get

$$\mathsf{P}({\overline{A}}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;\{(x_q,y_q),q\in {\overline{D}}_k\left(N\right)\})=\mathsf{P}\left(\frac{{\sum }_{q\in {\overline{D}}_k\left(N\right)}({\zeta }_q-\mathsf{E}{\zeta }_q)}{{\sigma }_0\sqrt{M_k}}>\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{8♯\mathbb{X}{\sigma }_0}\right),$$

where $p_0$ appeared in (36).

Now we employ the Berry-Esseen estimate of the convergence rate in CLT for i.i.d. random variables. Let $Z_1,\dots ,Z_v$ be i.i.d. random variables such that $\mathsf{E}{Z}_1=0$, $\mathsf{var}{Z}_1={\sigma }^2\in (0,\infty )$, $\mathsf{E}|Z_1{|}^3=\rho <\infty$. We write F for the distribution function of $Z_1$ and $F_v$ stands for the distribution function of $(Z_1+\dots +Z_v)/\left(\sigma \sqrt{v}\right)$. Then (see, e.g., Theorem 5.4 of [43]), for any $v\in \mathbb{N}$,

$$\underset{u\in \mathbb{R}}{sup}|F_v\left(u\right)-\mathrm{\Phi }\left(u\right)|\le \frac{C_0\rho }{{\sigma }^3\sqrt{v}},$$

where $\mathrm{\Phi }\left(u\right)$ is the distribution function of a standard normal random variable, $C_0$ is a positive constant ($C_0$ does not depend on distribution of $Z_1$ and v). According to [44] one has $C_0$ 0.4693. Consequently, taking $Z\sim N(0,1)$, we have

$$\mathsf{P}\left(\left|\frac{{\sum }_{q\in {\overline{D}}_k\left(N\right)}({\zeta }_q-\mathsf{E}{\zeta }_q)}{{\sigma }_0\sqrt{M_k}}\right|>\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{8♯\mathbb{X}{\sigma }_0}\right)\le \mathsf{P}\left(\left|Z\right|>\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{8♯\mathbb{X}{\sigma }_0}\right)+\frac{2C_0}{{\sigma }_0^3\sqrt{M_k}}$$

(43)

since $\mathsf{E}|{\zeta }_q-\mathsf{E}{\zeta }_q{|}^3\le 1$ for $q\in {\overline{D}}_k\left(N\right)$, where ${\zeta }_q=\mathbb{I}\{X_U^{\left(q\right)}=x_U,Y^{\left(q\right)}=y\}$.

It is well-known (see, e.g., formula (29) of Chap. II of [41]), that, for $u>0$, the following inequality is true:

$$\mathsf{P}\left(\right|Z|\ge u)\le \frac{\sqrt{2/\pi }}{u}exp\left\{-\frac{u^2}{2}\right\}.$$

Therefore, by virtue of an inequality ${\sigma }_0^2\le 1/4$ (which is valid for the indicator variance) and as

$$(K-1)[N/K]\le M_k\le N,$$

(44)

we can write under condition (23) that

$$\mathsf{P}\left(\left|Z\right|>\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{8♯\mathbb{X}{\sigma }_0}\right)\le \frac{8♯\mathbb{X}\sqrt{2}{\sigma }_0}{p_0^2\sqrt{\pi M_k}{\epsilon }_N}exp\left\{-\frac{1}{2}{\left(\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{8♯\mathbb{X}{\sigma }_0}\right)}^2\right\}$$

$$\begin{array}{ccc}& \le & \frac{4\sqrt{2}♯\mathbb{X}}{p_0^2\sqrt{\pi M_k}{\epsilon }_N}exp\left\{-\frac{1}{32}{\left(\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{♯\mathbb{X}}\right)}^2\right\}\hfill \\ & =& \frac{4\sqrt{2}♯\mathbb{X}}{p_0^2\sqrt{\pi M_k}}exp\left\{-\frac{1}{32}{\left(\frac{p_0^2\sqrt{M_k}{\epsilon }_N}{♯\mathbb{X}}\right)}^2+log\left(\frac{1}{{\epsilon }_N}\right)\right\}\le \frac{C_1}{\sqrt{N}},N\in \mathbb{N},\hfill \end{array}$$

and $C_1$ does not depend on N.

Introduce

$${\tilde{\sigma }}^2:=\underset{U\subset T,x\in \mathbb{X},y\in \mathbb{Y}}{min}{\sigma }_0^2(U,x,y),$$

where one considers only strictly positive ${\sigma }_0^2(U,x,y)$. Then obviously ${\tilde{\sigma }}^2>0$, as there exists only a finite collection of different variants. Thus in view of (44), for all x, y and U under consideration, one has

$$\frac{2C_0}{{\tilde{\sigma }}^3\sqrt{M_k}}\le \frac{C_2}{\sqrt{N}},N\in \mathbb{N},$$

where $C_0$ appeared in (43) and $C_2$ does not depend on N.

Therefore, if condition (23) is satisfied then, for all $x\in \mathbb{X}$, $y\in \mathbb{Y}$, $k=1,\dots ,K$ and $j\in D_k\left(N\right)$, the following inequality holds:

$$\mathsf{E}{\eta }_{N,k}^{\left(j\right)}\le \frac{C_3}{\sqrt{N}},N\in \mathbb{N},$$

(45)

where $C_3$ does not depend on $x,y$, k and N. Hence, in view of (44) we come to the relation

$$\mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}{g}_{{\overline{D}}_k\left(N\right)}(X^{\left(j\right)},y;\{(X^{\left(q\right)},Y^{\left(q\right)}),q\in {\overline{D}}_k\left(N\right)\})\right)}^4$$

$$\le \left(8C{(♯D_k\left(N\right))}^2+8{(♯D_k\left(N\right))}^4\frac{C_3^4}{N^2}\right)\sum _{(x_q,y_q)),q\in {\overline{D}}_N\left(k\right)}\mathsf{P}((X^{\left(q\right)},Y^{\left(q\right)})=(x_q,y_q))\le C_4{N}^2,$$

where $C_4$ does not depend on x, y, k and N. Thus according to (41), for all N large enough, we have proved the inequality

$$\mathsf{E}{U}_{N,1}^4\le C_5,$$

(46)

where $C_5$ does not depend on N.

In a similar way (taking into account (42) and (45)), for $i=0,\dots ,2m-1$, $y\in \mathbb{Y}$, $k=1,\dots ,K$, and all N large enough, we get

$$\mathsf{E}{\mathbb{S}}_k{(i,y)}^8\le C_6{(♯D_N\left(k\right))}^{-4},$$

(47)

where ${\mathbb{S}}_k(i,y)$ is introduced in (31), and $C_6$ does not depend on N.

We will employ an elementary result for the Bernoulli scheme. Let $U_1,U_2,\dots ,$ be a sequence of i.i.d. random variables such that $\mathsf{P}(U_1=1)=\mathsf{p}$ and $\mathsf{P}(U_1=0)=1-\mathsf{p}$, where $\mathsf{p}\in (0,1)$. Consider the following frequency estimator of a probability $\mathsf{p}$:

$${\widehat{\mathsf{p}}}_N:=\frac{1}{N}\sum _{j=1}^N\mathbb{I}\{U_j=1\},N\in \mathbb{N}.$$

Define

$${\widehat{\psi }}_N:=\left\{\begin{array}{cc}\frac{1}{{\widehat{\mathsf{p}}}_N},\hfill & {\widehat{\mathsf{p}}}_N\ne 0,\hfill \\ 0,\hfill & {\widehat{\mathsf{p}}}_N=0.\hfill \end{array}\right.$$

(48)

Lemma 2.

For the Bernoulli scheme introduced above and the estimators ${\widehat{\psi }}_N$ provided by formula (48), for each $t\in \mathbb{N}$, the following relation holds:

$$\mathsf{E}{\left({\widehat{\psi }}_N-\frac{1}{\mathsf{p}}\right)}^t=O\left(\frac{1}{N}\right),N\to \infty .$$

(49)

More precisely, the absolute value of the function in the left-hand side of (49), for all $N\in \mathbb{N}$, admits a bound $c/N$ where $c=c(\mathsf{p},t)$ for $\mathsf{p}\in (0,1)$ and $t\in \mathbb{N}$.

For the sake of completeness the proof of this result is given in Appendix A.

Now we continue the proof corresponding to Step 1. For all considered k, i, y and any $N\in \mathbb{N}$, the Cauchy - Bunyakovsky - Schwarz inequality yields

$$\mathsf{E}{\left(({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right)){\mathbb{S}}_k(i,y)\right)}^4\le {\left(\mathsf{E}{({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))}^8\mathsf{E}{\mathbb{S}}_k{(i,y))}^8\right)}^{\frac{1}{2}}.$$

Due to Lemma 2 one has $\mathsf{E}{({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))}^8=O\left(\frac{1}{N}\right)$, $N\to \infty$. Employing the Minkowski inequality (to take into account the summation over i, y, k), for all $N\in \mathbb{N}$, we come to the bound

$$\mathsf{E}{U}_{N,2}^4\le N^2{C}_7{\left(\left(\frac{1}{N}\right)\left(\frac{1}{N^4}\right)\right)}^{\frac{1}{2}}=\frac{C_7}{\sqrt{N}},$$

(50)

where $C_7$ does not depend on N.

Consequently, by virtue of (33), (46) and (50) the uniform integrability of a sequence ${\left(R_N^2\right)}_{N\in \mathbb{N}}$ is established. Thus (28) is verified.

Step 2. Now we study the asymptotic behavior of the variables $\sqrt{N}({\widehat{T}}_N\left(f^U\right)-T_N\left(f^U\right))$, as $N\to \infty$, where ${\widehat{T}}_N\left(f^U\right)$ and $T_N\left(f^U\right)$ are given by Formulas (26) and (27), respectively. For $j\in \mathbb{N}$, $i=0,\dots ,2m-1$, $y\in \mathbb{Y}$, we set $Z_i^{\left(j\right)}\left(y\right)=\mathbb{I}\{Y^{\left(j\right)}=y,|f\left(X^{\left(j\right)}\right)-y|>i\}$. One has

$$\sqrt{N}({\widehat{T}}_N\left(f^U\right)-T_N\left(f^U\right))={\mathbb{W}}_{N,1}+{\mathbb{W}}_{N,2},$$

where

$$\begin{array}{ccc}\hfill {\mathbb{W}}_{N,1}& =& \frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}(Z_i^{\left(j\right)}\left(y\right)-\mathsf{E}{Z}_i^{\left(j\right)}\left(y\right)),\hfill \\ \hfill {\mathbb{W}}_{N,2}& =& \frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}\mathsf{P}(Y^{\left(j\right)}=y,|f^U\left(X^{\left(j\right)}\right)-y|>i)\hfill \\ \hfill & =& \frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i).\hfill \end{array}$$

(51)

The purpose of the second step is to prove that

$$\mathsf{E}{\mathbb{W}}_{N,1}^2\to 0,N\to \infty .$$

(52)

For $k=1,\dots ,K$, $i=0,\dots ,2m-1$ and $y\in \mathbb{Y}$ introduce

$${\mathbb{G}}_k(i,y)=\frac{1}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}(Z_i^{\left(j\right)}\left(y\right)-\mathsf{E}{Z}_i^{\left(j\right)}\left(y\right)).$$

The Cauchy-Bunyakovsky-Schwarz inequality yields

$$\mathsf{E}{\left(({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right)){\mathbb{G}}_k(i,y)\right)}^2\le {\left(\mathsf{E}{({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))}^4\right)}^{\frac{1}{2}}{\left(\mathsf{E}{\left({\mathbb{G}}_k(i,y)\right)}^4\right)}^{\frac{1}{2}}.$$

For each considered N, y, i and k, the variables $\{Z_i^{\left(j\right)}\left(y\right),j\in D_k\left(N\right)\}$ are independent and $|Z_i^{\left(j\right)}\left(y\right)-\mathsf{E}{Z}_i^{\left(j\right)}\left(y\right)|\le 1$, so by virtue of the Rosenthal inequality (42) we obtain

$$\mathsf{E}{\left(\sum _{j\in D_k\left(N\right)}(Z_i^{\left(j\right)}\left(y\right)-\mathsf{E}{Z}_i^{\left(j\right)}\left(y\right))\right)}^4=O(♯D_k{\left(N\right)}^2).$$

Taking into account Lemma 2 for $t=4$ and in view of (44), for each $k=1,\dots ,K$, we get the relation

$$\mathsf{E}{\mathbb{W}}_{N,1}^2=O\left(N^{-\frac{1}{2}}\right),N\to \infty .$$

Therefore, the goal of the second step has been achieved.

Step 3. The implementation of steps 1 and 2 permits to reduce the study of the asymptotic behavior (as $N\to \infty$) of ${\mathbb{Z}}_N$ given by Formula (25) to the study of variables

$${\eta }_N:=\sqrt{N}(T_N\left(f^U\right)-\mathsf{Err}\left(f^U\right))+{\mathbb{W}}_{N,2},N\in \mathbb{N},$$

where ${\mathbb{W}}_{N,2}$ is defined by Formula (51).

The aim of the third step is to prove that $\mathsf{E}{\left({\eta }_N\right)}^2\to {\sigma }^2\left(U\right)$, as $N\to \infty$, where ${\sigma }^2\left(U\right)$ is the variance of the random variable $V\left(U\right)$ appearing in Formula (22).

On this way, we will show that the sum of certain part of the terms in a specified representation of the variables ${\eta }_N$ does not affect (in the sense of $L^2(\mathrm{\Omega },\mathcal{F},\mathsf{P})$) the limit behavior of these variables for growing N. For $y\in \mathbb{Y}$ and $W_N\subset \{1,\dots ,N\}$, where $N\in \mathbb{N}$, we introduce the event

$$B_{W_N}\left(y\right):=\{\omega :\widehat{P}(y,\xi \left(W_N\right))\ne 0\},$$

(53)

where $\widehat{P}(y,\xi \left(W_N\right))$ is defined according to (13). Then, in view of the independence of observations ${\xi }^{\left(1\right)},{\xi }^{\left(2\right)},\dots$ we have

$$\mathsf{P}\left({\overline{B}}_{W_N}\left(y\right)\right)=\mathsf{P}\left(\bigcap _{j\in W_N}\{Y^{\left(j\right)}\ne y\}\right)={(1-\mathsf{P}(Y=y))}^{♯W_N}.$$

If $\omega \in {\overline{B}}_{W_N}\left(y\right)$ then $|\widehat{\psi }(y,\xi \left(W_N\right))-\psi \left(y\right)|=\psi \left(y\right)$. Set

$$H_N:=\frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}({\widehat{\psi }}_{N,k}\left(y\right)-\psi \left(y\right))\mathbb{I}\left\{B_{N,k}\left(y\right)\right\}\mathsf{P}(Y=y,|f\left(X\right)-y|>i),$$

where $B_{N,k}\left(y\right):=B_{D_k\left(N\right)}\left(y\right)$ and an event $B_{W_N}\left(y\right)$ is introduced by Formula (53). Then

$$\mathsf{E}{({\mathbb{W}}_{N,2}-H_N)}^2=\mathsf{E}{\left(\frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{\mathbb{I}\left\{{\overline{B}}_{N,k}\left(y\right)\right\}}{\mathsf{P}(Y=y)}\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i)\right)}^2$$

$$\le \frac{N{\left(2m\right)}^4}{p_0^2}\underset{y\in \mathbb{Y}}{max}{(1-\mathsf{P}(Y=y))}^{[N/K]}\to 0,N\to \infty ,$$

since $♯D_k\left(N\right)\ge [N/K]$ for $N\in \mathbb{N}$, $k=1,\dots ,K$ and because all $\mathsf{P}(Y=y)>0$ for each $y\in \mathbb{Y}$, $[·]$ stands for an integer part of a number.

We verify that $H_N$ for large N is approximated in the space $L^2(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ by the random variable

$${\tilde{H}}_N:=\frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\mathbb{I}\left\{B_{N,k}\left(y\right)\right\}\left(\frac{\mathsf{P}(Y=y)-{\widehat{p}}_{N,k}\left(y\right)}{\mathsf{P}{(Y=y)}^2}\right)\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i),$$

where ${\widehat{p}}_{N,k}\left(y\right):=\widehat{P}(y,\xi \left(D_k\left(N\right)\right))$ and $\widehat{P}(y,\xi \left(W_N\right))$ was introduced by (13) for $y\in \mathbb{Y}$ and $W_N\subset \{1,\dots ,N\}$. Evidently, $0\le \mathsf{P}(Y=y,|f^U\left(X\right)-y|>i)\le 1$ for all k, i, y and N under consideration. Consequently, it follows that

$$\begin{array}{ccc}\hfill {\Delta }_{N,k}(i,y)& :=& |\sqrt{N}\mathbb{I}\left\{B_{N,k}\left(y\right)\right\}\left(\frac{1}{{\widehat{p}}_{N,k}\left(y\right)}-\frac{1}{\mathsf{P}(Y=y)}\right)\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i)\hfill \\ & -& \sqrt{N}\mathbb{I}\left\{B_{N,k}\left(y\right)\right\}\left(\frac{\mathsf{P}(Y=y)-{\widehat{p}}_{N,k}\left(y\right)}{\mathsf{P}{(Y=y)}^2}\right)\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i)|\hfill \\ & \le & \sqrt{N}\left|\frac{\mathsf{P}(Y=y)-{\widehat{p}}_{N,k}\left(y\right)}{\mathsf{P}(Y=y)}\right|\left|{\widehat{\psi }}_{N,k}\left(y\right)-\frac{1}{\mathsf{P}(Y=y)}\right|\hfill \\ & =& \frac{\sqrt{N}}{\mathsf{P}(Y=y)\sqrt{♯D_k\left(N\right)}}\left|{\widehat{\psi }}_{N,k}\left(y\right)-\frac{1}{\mathsf{P}(Y=y)}\right|{\mathbb{J}}_N,\hfill \end{array}$$

where

$${\mathbb{J}}_N:=\frac{1}{\sqrt{♯D_k\left(N\right)}}\sum _{j\in D_k\left(N\right)}(\mathbb{I}\{Y^{\left(j\right)}=y\}-\mathsf{P}(Y^{\left(j\right)}=y)).$$

For any considered k, i, y and N the Cauchy - Bunyakovsky - Schwarz inequality implies that

$$\mathsf{E}{\left({\Delta }_{N,k}(i,y)\right)}^2\le \frac{N}{(\mathsf{P}{(Y=y)}^2♯D_k\left(N\right)}{\left(\mathsf{E}{\mathbb{J}}_N^4\mathsf{E}{\left({\widehat{\psi }}_{N,k}\left(y\right)-\frac{1}{\mathsf{P}(Y=y)}\right)}^4\right)}^{\frac{1}{2}}.$$

The Rosenthal inequality (42) yields that $\mathsf{E}{\mathbb{J}}_N^4\le 2C\left(4\right)$. By means of Lemma 2 (for $t=4$ and multipliers $c(\mathsf{p},t)$ with $\mathsf{p}=\mathsf{P}(Y=y)$), for all considered i, y, k and any $N\in \mathbb{N}$ we come to the bound

$$\mathsf{E}{\left({\Delta }_{N,k}(i,y)\right)}^2\le \frac{N}{(\mathsf{P}{(Y=y)}^2♯D_k\left(N\right)}\frac{{\left(2C\left(4\right)c(\mathsf{P}(Y=y),4)\right)}^{\frac{1}{2}}}{\sqrt{N}}.$$

Therefore, $\mathsf{E}{(H_N-{\tilde{H}}_N)}^2\to 0$ as $N\to \infty$.

Let us define the variable $G_N$ by formula similar to ${\tilde{H}}_N$ but without the multiplier $\mathbb{I}\left\{B_{N,k}\left(y\right)\right\}$. In view of (44) it is easily seen that

$$\mathsf{E}{({\tilde{H}}_N-G_N)}^2\le \frac{N{\left(2m\right)}^4}{p_0^4}\underset{y\in \mathbb{Y}}{max}{(1-\mathsf{P}(Y=y))}^{[N/K]}\left(\frac{1}{4}\right)\underset{k=1,\dots ,K}{max}\frac{1}{♯D_k\left(N\right)}\to 0,N\to \infty .$$

Thus $\mathsf{E}{({\eta }_N-Q_N)}^2\to 0$ as $N\to \infty$, where

$$Q_N:=\sqrt{N}(T_N\left(f^U\right)-\mathsf{Err}\left(f^U\right))+G_N,N\in \mathbb{N}.$$

Taking into account Formula (6) for the function $f=f^U$, we come to the relation

$$\begin{array}{ccc}\hfill Q_N& =& \frac{\sqrt{N}}{K}\sum _{k=1}^K\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{1}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}(\frac{\mathbb{I}\{Y^{\left(j\right)}=y,|f^U\left(X^{\left(j\right)}\right)-y|>i\}}{\mathsf{P}(Y=y)}\hfill \\ \hfill & -& \frac{\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i)}{\mathsf{P}(Y=y)}+\frac{(\mathsf{P}(Y=y)-\mathbb{I}\{Y^{\left(j\right)}=y\})\mathsf{P}(Y=y,|f^U\left(X\right)-y|>i)}{\mathsf{P}{(Y=y)}^2})\hfill \\ \hfill & =& \frac{\sqrt{N}}{K}\sum _{k=1}^K\frac{1}{♯D_k\left(N\right)}\sum _{j\in D_k\left(N\right)}{V}^{\left(j\right)},\hfill \end{array}$$

(54)

where, for $j\in \mathbb{N}$,

$$V^{\left(j\right)}:=\sum _{i=0}^{2m-1}\sum _{i-m<\left|y\right|\le m}\frac{\mathbb{I}\{Y^{\left(j\right)}=y\}}{\mathsf{P}(Y=y)}\left(\mathbb{I}\left\{\right|f^U\left(X^{\left(j\right)}\right)-y|>i)-\mathsf{P}\left(\right|f^U\left(X\right)-y|>i|Y=y)\right).$$

(55)

The variables $\{V^{\left(j\right)},j\in \mathbb{N}\}$ are centered, i.i.d. and uniformly bounded for all j (clearly, $V^{\left(j\right)}=V^{\left(j\right)}\left(U\right)$). For each $j\in \mathbb{N}$, the distributions of $V^{\left(j\right)}$ and $V\left(U\right)$ coincide, where $V\left(U\right)$ is introduced in (22). Thus, one has

$$\mathsf{var}{V}^{\left(j\right)}=\mathsf{var}V\left(U\right)={\sigma }^2\left(U\right),j\in \mathbb{N}.$$

(56)

According to the lemma on grouping independent random variables, for each $N\in \mathbb{N}$, the variables ${\sum }_{j\in D_k\left(N\right)}{V}^{\left(j\right)}$, $k=1,\dots ,K$, are independent. Since $N/♯D_k\left(N\right)\to K$ as $N\to \infty$, for $k=1,\dots ,K$, we come to the relation

$$\mathsf{E}\left(Q_N^2\right)=\mathsf{var}{Q}_N=\frac{N}{K^2}\sum _{k=1}^K\frac{1}{{(♯D_k\left(N\right))}^2}\sum _{j\in D_k\left(N\right)}\mathsf{var}{V}^{\left(j\right)}={\sigma }^2\left(U\right)\frac{1}{K^2}\sum _{k=1}^K\frac{N}{♯D_k\left(N\right)}\to {\sigma }^2\left(U\right),$$

as $N\to \infty$. Hence $\mathsf{E}{\eta }_N^2\to {\sigma }^2\left(U\right)$, $N\to \infty$. The goal of the third step has been achieved.

In view of the above approximations (in $L^2(\mathrm{\Omega },\mathcal{F},\mathsf{P})$) of the initial random variables ${\mathbb{Z}}_N$, introduced by (25), we conclude that $\mathsf{E}{\mathbb{Z}}_N^2\to {\sigma }^2\left(U\right)$, as $N\to \infty$. Namely, we apply the following elementary statement: if $\mathsf{E}{\alpha }_N^2\to 0$ and $\mathsf{E}{\beta }_N^2\to {\sigma }^2$ then $\mathsf{E}{({\alpha }_N+{\beta }_N)}^2\to {\sigma }^2$, as $N\to \infty$. Therefore, (24) is established. The proof of Lemma 1 is complete. □

Further we will also employ a result that immediately follows from Theorem 1.

Corollary 1.

Let the conditions of Lemma 1 be satisfied. Then the following relations hold:

$$\sqrt{N}\mathsf{E}\left({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{Err}\left(f^U\right)\right)\to 0,N\to \infty ,$$

(57)

$$\mathsf{var}\left(\sqrt{N}{\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)\right)\to {\sigma }^2\left(U\right),N\to \infty ,$$

(58)

where ${\sigma }^2\left(U\right)$ is a variance of the random variable $V\left(U\right)$ introduced in (22).

Proof. 

Condition (23) implies (20). Thus, according to Theorem 1, we have

$${\mathbb{Z}}_N\stackrel{\mathcal{D}}{\to }Z\sim N(0,{\sigma }^2\left(U\right)),N\to \infty ,$$

(59)

where ${\mathbb{Z}}_N$, $N\in \mathbb{N}$, are defined in (25). Due to Lemma 1 one has the uniform integrability of the sequence ${\left(\mathbb{Z}\right)}_{N\in \mathbb{N}}$. Consequently, relation (59) implies (57), i.e., $\mathsf{E}{\mathbb{Z}}_N\to \mathsf{E}Z=0$, as $N\to \infty$. Obviously,

$$\mathsf{var}\left(\sqrt{N}{\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)\right)$$

$$=\mathsf{E}{\left(\sqrt{N}({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{Err}\left(f^U\right)\right)}^2-{\left(\sqrt{N}\mathsf{E}({\widehat{\mathsf{Err}}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{Err}\left(f^U\right))\right)}^2.$$

Therefore, to obtain (58), it is sufficient to use Lemma 1 and take into account (57). The proof is complete. □

Note that (59) can be obtained directly under conditions of Lemma 1. For each $N\in \mathbb{N}$ and any $k=1,\dots ,K$, according to Lindeberg’s theorem applied to arrays $\{V^{\left(j\right)},j\in D_k\left(N\right)\}$ of centered i.i.d. uniformly bounded summands, where a sequence ${\left(V^{\left(j\right)}\right)}_{j\in \mathbb{N}}$ is introduced in (55), taking into account (56) one has

$$V_{N,k}:=\frac{1}{\sqrt{♯D_k\left(N\right)}}\sum _{j\in D_k\left(N\right)}{V}^{\left(j\right)}\stackrel{\mathcal{D}}{\to }{Z}_k\sim N(0,{\sigma }^2\left(U\right)),N\to \infty .$$

(60)

For every $N\in \mathbb{N}$, the random variables $V_{N,k}$, $k=1,\dots ,K$, are independent and $\mathsf{var}{V}_{N,k}={\sigma }^2\left(U\right)$. Since $N/♯D_k\left(N\right)\to K$ as $N\to \infty$, for $k=1,\dots ,K$, by virtue of (60) we come to relation

$$Q_N\stackrel{\mathcal{D}}{\to }Z\sim N(0,{\sigma }^2\left(U\right)),N\to \infty ,$$

(61)

where in view of (54) one has $Q_N=\frac{1}{K}{\sum }_{k=1}^K\sqrt{\frac{N}{♯D_k\left(N\right)}}{V}_{N,k}$, $N\in \mathbb{N}$. Applying (61) and Slutsky’s lemma, we arrive at (59).

Also note that relation (29) can be easily derived from (36) and (39) without employment of [32].

4. Forward Selection of Relevant Factors

Now we can turn to the sequential selection of factors based on MDR-EFE method. At the first step one searches for $j_1\in T$ a point where the function ${\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\left\{i\right\}}\right)$ attains the minimum over all $i\in T$. If there are several such points, then we take, e.g., one with the smallest index value. Recall that according to (17) (more precisely, after regularization), the random variable ${\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\left\{i\right\}}\right)$ is in fact a function of ${\widehat{f}}_{PA}^{\left\{i\right\}}$, which is a forecast of the function $f^{\left\{i\right\}}$. Then this procedure is repeated, namely, if at $(k-1)$-th step the set $S_{k-1}:=\{j_1,\dots ,j_{k-1}\}$ is constructed, where $k\in \{2,\dots ,r\}$, then $j_k\in T\setminus S_{k-1}$ is selected at step k in such a way that given $j_1,\dots ,j_{k-1}$ the function ${\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{S_{k-1},i\}}\right)$ takes the minimum value over $i\in T\setminus S_{k-1}$ for $i=j_k$. It is convenient to assume that an empty set is taken at the zero step. Then at each next step one new element is added to the previously constructed sets. If at some step there are several minimum points of the considered function then we take only one of them, e.g., with the minimal index.

Thus, for each $N\in \mathbb{N}$ the random sets $S_k\left(N\right)=S_k(N,\omega ):=\{j_1,\dots ,j_k\}$ arise, where $k=1,\dots ,r$ and $j_m=j_m(N,\omega )$, $m=1,\dots ,r$. By construction one can write

$$j_k(N,\omega )\in J_k(N,\omega ):=\mathsf{arg}\underset{i\in T\setminus S_{k-1}(N,\omega )}{min}{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{S_{k-1}(N,\omega ),i\}}\right),$$

where $S_0:=⌀$ and $\{⌀,i\}:=\left\{i\right\}$. In other words the choice $j_k(N,\omega )$ at step k means that, for $i\in T\setminus S_{k-1}(N,\omega )$,

$${\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{S_k(N,\omega )}\right)\le {\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{S_{k-1}(N,\omega ),i\}}\right),$$

(62)

moreover, $j_k(N,\omega )=min\{i:i\in J_k(N,\omega )\}$, $k=1,\dots ,r$. If the joint distribution of X and Y is known, then instead of the described scheme for constructing random sets, $S_k(N,\omega )$ we turn to considering the non-random “oracle” sets $T_k=\{i_1,\dots ,i_k\}$, where $k=1,\dots ,r$,

$$i_k\in \mathsf{arg}\underset{i\in T\setminus T_{k-1}}{min}\mathsf{Err}\left(f^{\{T_{k-1},i\}}\right),$$

(63)

$T_0:=⌀$, and the functional $\mathsf{Err}$ is introduced by formula (2). If there are several $i_k$ satisfying (63) we take among them that one which has the minimal value.

For $k\in \{1,\dots ,r\}$ and $i\in T\setminus T_k$ introduce

$$C_{k,i}:=\mathsf{Err}\left(f^{\{T_{k-1},i\}}\right)-\mathsf{Err}\left(f^{T_k}\right).$$

By construction of the sets $T_k$ we have $C_{k,i}\ge 0$, where $k=1,\dots ,r$ and $i\in T\setminus T_k$. We call a model, satisfying condition (1), regular whenever the following relation is true:

$$C_{k,i}>0,k=1,\dots ,r,i\in T\setminus T_k.$$

(64)

In other words, for each $k=1,\dots ,r$, a point $i_k$ in (63) is determined uniquely. Further we employ the penalty function introduced in (11). We also use its strongly consistent estimate of type (48) with

$${\widehat{p}}_N:=\frac{1}{W_N}\sum _{j\in W_N}\mathbb{I}\{Y^{\left(j\right)}=y\},$$

(65)

$W_N\subset \{1,\dots ,N\}$ and $♯W_N\to \infty$ as $N\to \infty$.

Theorem 2.

Let the considered model (1) with a collection of relevant factors having cardinality $r<p$, be regular, i.e., let (64) take place. Then, for the random sets $S_r\left(N\right)$ introduced above, the following relation is valid

$$\mathsf{P}(S_r\left(N\right)=T_r)\to 1,N\to \infty ,$$

(66)

where $T_r$ is defined by means of (63) for $k=1,\dots ,r$. In other words, with probability close to one, the described procedure of forward selection based on statistical estimates of the error functional leads to the “oracle” collection $T_r$, when N is large enough.

Proof. 

For a random set $S_r(N,\omega )=\{j_1(N,\omega ),\dots ,j_r(N,\omega )\}$, where $j_k(N,\omega )$ is an element taken at k-th step, one has

$$\mathsf{P}(\omega :S_r(N,\omega )=T_r)\ge \mathsf{P}(\omega :j_1(N,\omega )=i_1,\dots ,j_r(N,\omega )=i_r).$$

Note that

$$\mathsf{P}(\omega :j_1(N,\omega )=i_1,\dots ,j_r(N,\omega )=i_r)\ge \mathsf{P}\left(\bigcap _{k=1}^r{A}_k\left(N\right)\right),$$

where

$$A_k\left(N\right):=\bigcap _{i\in T\setminus T_{k-1}}\left\{{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{T_k}\right)<{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{T_{k-1},i\}}\right)\right\},$$

$k=1,\dots ,r$. Thus, we obtain:

$$\mathsf{P}\left(\bigcap _{k=1}^r{A}_k\left(N\right)\right)=1-\mathsf{P}\left(\bigcup _{k=1}^r{\overline{A}}_k\left(N\right)\right)\ge 1-\sum _{k=1}^r\mathsf{P}\left({\overline{A}}_k\left(N\right)\right)$$

$$\ge 1-\sum _{k=1}^r\sum _{i\in T\setminus T_{k-1}}\mathsf{P}\left({\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{T_k}\right)\ge {\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{T_{k-1},i\}}\right)\right),$$

(67)

where, as usual, $\overline{A}:=\mathrm{\Omega }\setminus A$ for $A\subset \mathrm{\Omega }$. Then, for $k=1,\dots ,r$, $i\in T\setminus T_{k-1}$ and $N\in \mathbb{N}$, we get

$$\begin{array}{ccc}\hfill {\Delta }_{k,i}\left(N\right)& :=& {\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{T_k}\right)-{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{T_{k-1},i\}}\right)\hfill \\ & =& ({\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{T_k}\right)-\mathsf{E}{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{T_k}\right))+(\mathsf{E}{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{T_k}\right)-\mathsf{Err}\left(f^{T_k}\right))\hfill \\ & +& (\mathsf{Err}\left(f^{T_k}\right)-\mathsf{Err}\left(f^{\{T_{k-1},i\}}\right))+(\mathsf{Err}\left(f^{\{T_{k-1},i\}}\right)-\mathsf{E}{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{T_{k-1},i\}}\right))\hfill \\ & +& (\mathsf{E}{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{T_{k-1},i\}}\right)-{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^{\{T_{k-1},i\}}\right)).\hfill \end{array}$$

(68)

For $U\subset T$, set

$$Z_N\left(U\right):={\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^U\right)-\mathsf{E}{\widehat{Err}}_{K,N,{\epsilon }_N}\left(f^U\right).$$

For any $k=1,\dots ,K$, $i\in T\setminus T_{k-1}$ and each $\delta \in (0,1)$ in light of formula (57) of Corollary 1, for all N large enough ($N\ge N_2(\delta ,k,i)$) it holds

$$\mathsf{P}({\Delta }_{k,i}\left(N\right)\ge 0)\le \mathsf{P}(\sqrt{N}|Z_N\left(T_k\left(N\right)\right)|+\sqrt{N}|Z_N\left(\{T_{k-1}\left(N\right),i\}\right)|\ge \sqrt{N}{C}_{k,i}-\delta )$$

$$\le \mathsf{P}\left(\sqrt{N}\left|Z_N\left(T_k\left(N\right)\right)\right|\ge \frac{(1-\delta )\sqrt{N}{C}_{k,i}}{2}\right)+\mathsf{P}\left(|Z_N\left(\{T_{k-1}\left(N\right),i\}\right)|\ge \frac{(1-\delta )\sqrt{N}{C}_{k,i}}{2}\right),$$

where $C_{k,i}$ are introduced in (66), ${\Delta }_{k,i}\left(N\right)$ is defined by (68).

Applying the Bienaymé - Chebyshev inequality and taking into account Formula (58) of Corollary 1, for each $U\subset T$ and any $c>0$, we come, for a centered random variable $Z_N\left(U\right)$, to the relation

$$\mathsf{P}(\sqrt{N}|Z_N\left(U\right)|\ge c\sqrt{N})\le \frac{N\mathsf{var}{Z}_N\left(U\right)}{N{c}^2}\sim \frac{\mathsf{var}V\left(U\right)}{N{c}^2},N\to \infty ,$$

(69)

where $V\left(U\right)$ is determined by Formula (22). According to (64), for $k\in \{1,\dots ,r\}$ and $i\in T\setminus T_k$, one has $C_{k,i}>0$. Therefore, for all N large enough ($N\ge N_3(\delta ,k,i)$), the following inequality takes place:

$$\mathsf{P}({\Delta }_{k,i}\left(N\right)\ge 0)\le \frac{4(\mathsf{var}V\left(T_k\right)+\mathsf{var}V\left(\{T_{k-1},i\}\right)}{N{(1-\delta )}^2{C}_{k,i}^2}.$$

(70)

For a fixed $m\in \mathbb{N}$, one can change the summation order over i and y to write Formula (22) as follows:

$$V\left(U\right)=\sum _{y=-m}^m\frac{\mathbb{I}\{Y=y\}}{\mathsf{P}(Y=y)}W(y,U),$$

where

$$W(y,U)=\sum _{0\le i<\left|y\right|+m}\left(\mathbb{I}\left\{\right|f^U\left(X\right)-y|>i\}-\mathsf{P}\left(\right|f^U\left(X\right)-y|>i|Y=y)\right).$$

(71)

Thus, for any $U\subset T$, one has

$$\left|V\left(U\right)\right|\le 2m\sum _{y=-m}^m\frac{\mathbb{I}\{Y=y\}}{\mathsf{P}(Y=y)}.$$

Consequently, we come to the inequality

$$\mathsf{var}V\left(U\right)\le \mathsf{E}{V}^2\left(U\right)\le 4m^2\sum _{y=-m}^m\frac{1}{\mathsf{P}(Y=y)}=:a,$$

where $a=a(m,{\left(\mathsf{P}(Y=y)\right)}_{y\in \mathbb{Y}})$. We see that $\mathsf{var}V\left(T_k\right)+\mathsf{var}V\left(\{T_{k-1},i\}\right)\le 2a$ for all $k\in \{1,\dots ,r\}$, $i\in T\setminus T_{k-1}$ and $N\in \mathbb{N}$. For each $\delta \in (0,1)$, any $k\in \{1,\dots ,r\}$, $i\in T\setminus T_{k-1}$ and all N large enough, we get the following bound:

$$\mathsf{P}({\Delta }_{k,i}\left(N\right)\ge 0)\le \frac{8a}{N{(1-\delta )}^2{C}_{k,i}^2}.$$

Hence, for each $\delta \in (0,1)$ and all N large enough, by virtue of (67) the following inequality holds:

$$\mathsf{P}(S_r\left(N\right)=T_r)\ge 1-\frac{8ar}{N{(1-\delta )}^2{C}_0^2}\left(p+1-\frac{r+1}{2}\right),$$

(72)

where $C_0^2:={min}_{k=1,\dots ,r,i\in T\setminus T_{k-1}}{C}_{k,i}^2>0$ according to (64). Thus relation (72) implies the validity of (66). □

Now note that according to (69) the following relation is true:

$$\mathsf{P}(\sqrt{N}|Z_N\left(U\right)|\ge c\sqrt{N})=O\left(\frac{1}{N}\right),N\to \infty .$$

(73)

The question arises whether this probability decreases like $C/N$ where C is a positive constant or more rapidly. The answer depends on the variance of the random variable $V\left(U\right)$ given by Formula (22). In view of (70) we will determine when the variable $V\left(U\right)$ is degenerate, i.e., equal to a constant a.s. This is also of independent interest for the CLT established in Section 6 of [32] and given above as Theorem 1. The following result provides a simple characterization of the $V\left(U\right)$ degeneracy.

Lemma 3.

For an arbitrary set $U\subset T$, the variance of the random variable $V\left(U\right)$, appearing in Formula (22), is zero if and only if, for every $y\in \mathbb{Y}$, there is $k_0\left(y\right)\in \{0,\dots ,m+|y\left|\right\}$ such that

$$\mathsf{P}\left(\right|f^U\left(X\right)-y|=k_0\left(y\right),Y=y)=\mathsf{P}(Y=y).$$

(74)

Thus, for each $y\in \mathbb{Y}$, on the set $\{Y=y\}$ the random variable $f^U\left(X\right)$ does not necessarily take a constant value. Moreover, the values of $k_0\left(y\right)$ need not coincide for different y.

Proof. 

For $y=0,\dots ,m$ and a random variable $W(y,U)$, introduced by Formula (71), one can write

$$\begin{array}{ccc}\hfill W(y,U)& =& \sum _{0\le i<y+m}\left(\mathbb{I}\left\{\right|f^U\left(X\right)-y|>i\}-\mathsf{P}\left(\right|f^U\left(X\right)-y|>i|Y=y)\right)\hfill \\ & =& \sum _{0\le i<y+m}\sum _{i<k\le m+y}\left(\mathbb{I}\left\{\right|f^U\left(X\right)-y|=k\}-\mathsf{P}\left(\right|f^U\left(X\right)-y|=k|Y=y)\right)\hfill \\ & =& \sum _{k=1}^{m+y}\sum _{i=0}^{k-1}\left(\mathbb{I}\left\{\right|f^U\left(X\right)-y|=k\}-\mathsf{P}\left(\right|f^U\left(X\right)-y|=k|Y=y)\right)\hfill \\ & =& \sum _{k=1}^{m+y}k\left(\mathbb{I}\right\{|f^U\left(X\right)-y|=k\}-\mathsf{P}\left(\right|f^U\left(X\right)-y|=k|Y=y\left)\right)\hfill \\ & =& \sum _{k=1}^{m+y}k\mathbb{I}\left\{\right|f^U\left(X\right)-y|=k\}-\mathsf{E}\left(\right|f^U\left(X\right)-y\left|\right|Y=y).\hfill \end{array}$$

In a similar way we consider $y=-m,\dots ,-1$. Thus, for all $y\in \mathbb{Y}$, one gets

$$W(y,U)=\sum _{k=1}^{m+\left|y\right|}k\mathbb{I}\left\{\right|f^U\left(X\right)-y|=k\}-\mathsf{E}\left(\right|f^U\left(X\right)-y\left|\right|Y=y).$$

Recall that $\mathsf{P}(Y=y)>0$ for all $y\in \mathbb{Y}$. If, for some $y,k,j\in \mathbb{Y}$, $k\ne j$, we have

$$\mathsf{P}\left(\right|f^U\left(X\right)-y|=k,Y=y)>0,\mathsf{P}\left(\right|f^U\left(X\right)-y|=j,Y=y)>0,$$

then on the events $\left\{\right|f^U\left(X\right)-y|=k,Y=y\}$ and $\left\{\right|f^U\left(X\right)-y|=j,Y=y\}$ the variable $W(y,U)$ takes different values. Therefore, $V\left(U\right)$ takes different values on these events. Hence $\mathsf{var}V\left(U\right)>0$, if (74) is not valid. Thus (74) is a necessary condition to guarantee that $\mathsf{var}V\left(U\right)=0$. Suppose now that, (74) holds. In this case we get

$$\mathsf{E}\left(\right|f^U\left(X\right)-y\left|\right|Y=y)=k_0\left(y\right),y\in \mathbb{Y}.$$

Clearly, $k_0\left(y\right)$ depends on U as well. We see that $V\left(U\right)$ on each set $\{Y=y\}$ takes (up to the set of measure zero) the value $\frac{1}{\mathsf{P}(Y=y)}(k_0\left(y\right)-k_0\left(y\right))=0$, $y\in \mathbb{Y}$. Therefore, $\mathsf{var}V\left(U\right)=0$. Note that $k_0\left(y\right)$ need not coincide for different $y\in \mathbb{Y}$. The proof is complete. □

5. Concluding Remarks

The established asymptotical result (Theorem 2) is rather qualitative in nature, since relation (66) assumes increasing values of N. Relation (72) is more precise. However, (72) demonstrates that, loosely speaking, one has to employ $N>>rp$. As previously, we assume that assumption (A), introduced on page 2, is valid. Evidently, the sequential choice of relevant variables based on statistical estimators of the error functional (of response approximation), is attractive for implementation, although suboptimal. In this regard Theorem 2 shows that under certain conditions, forward (random) selection with a high probability leads to the same collection of factors, which is provided by the sequential procedure with known joint distribution of the vector of factors X and the response Y. In the future work, it would be reasonable to supplement the theoretical results by computer simulations (see, e.g., [45]).

Consideration of the proximity of the results of optimal and suboptimal procedures requires a separate study. In addition, we note that within the framework of linear models, estimates of the probability of correct identification of relevant factors are considered, e.g., in [46,47]. Theorem 2 does not assume the linearity of stochastic model. Presumably for the first time, in our work a forward selection of relevant factors affecting the non-binary random response is treated on the base of MDR-EFE method. It would be interesting to extend the conditions allowing to establish relation (66). Moreover, stability problems of FS deserve special attention, see, e.g., [48,49,50]. Algorithms stability for classification problems in the framework of random trees is treated in [51].

Finally, we emphasize that the problem of statistical estimation of the cardinality of a set of relevant factors appearing in definition (1) is very important and complex. Along with dealing with the deterministic number of selected factors, there is a research approach based on developing the rules for stopping the procedures used to identify the relevant set. In this regard, we indicate, e.g., article [52], dedicated to information methods for selecting relevant factors. The study of non-discrete stochastic models is also of undoubted interest, see, e.g., [53].

Further it would be interesting to study other functionals than (2) to measure the quality of a response approximation by means of functions defined on various collections of factors. One can also consider a random number of observations. In this regard we refer, e.g., to [27,54].