Nonparametric Estimation for High-Dimensional Space Models Based on a Deep Neural Network

Abstract

With high dimensionality and dependence in spatial data, traditional parametric methods suffer from the curse of dimensionality problem. The theoretical properties of deep neural network estimation methods for high-dimensional spatial models with dependence and heterogeneity have been investigated only in a few studies. In this paper, we propose a deep neural network with a ReLU activation function to estimate unknown trend components, considering both spatial dependence and heterogeneity. We prove the compatibility of the estimated components under spatial dependence conditions and provide an upper bound for the mean squared error ($MSE$). Simulations and empirical studies demonstrate that the convergence speed of neural network methods is significantly better than that of local linear methods.

1. Introduction

Spatial data arise in many fields, including environmental science, econometrics, epidemiology, image analysis, oceanography, geography, geology, plant ecology, archaeology, agriculture, and psychology. Spatial correlation and spatial heterogeneity are two significant features of spatial data. Various spatial modeling methods have been applied to explore the effect of spatial heterogeneity. Notably, numerous local spatial techniques have been proposed to accommodate spatial heterogeneity. For example, Hallin et al. [1] and Biau and Cadre [2] proposed a local linear method for the modeling of spatial heterogeneity. Bentsen et al. [3] used a graph neural network architecture to extract spatial dependencies with different update functions to learn temporal correlations.

Spatial data often exhibit high dimensionality, a large scale, heterogeneity, and strong complexity. These challenges often make traditional statistical methods ineffective. Statistical machine learning methods can effectively address such challenges. Du et al. [4] pointed out the issues of traditional spatial data being large-scale and generally complex and summarized the effectiveness and application potential of four advanced machine learning methods—support vector machine (SVM)-based kernel learning, semi-supervised and active learning, ensemble learning, and deep learning—in handling complex spatial data. Farrell et al. [5] highlighted the challenges that high-dimensional spatial data, large volumes of data, and multicollinearity among covariates pose to traditional statistical models in variable selection. Three machine learning algorithms—maximum entropy (MaxEnt), random forests (RFs), and support vector machines (SVMs)—were employed to mitigate the issues of multicollinearity in high-dimensional spatial data. Nikparvar et al. [6] pointed out that the properties of spatially explicit data are often ignored or inadequately addressed in machine learning applications within spatial domains. They argued that the future prospects for spatial machine learning are very promising.

Statistical machine learning methods have advanced rapidly, while theoretical ones are not well established. Schmidt-Hieber [7] investigated the following nonparametric model:

$${\mathit{Y}}_i=f_0\left({\mathit{X}}_i\right)+{ϵ}_i,i=1,\dots ,n,$$

(1)

where the noise variables ${ϵ}_i$ are assumed to be i.i.d., ${\mathit{X}}_i\in {[0,1]}^d,i=1,\dots ,n,$ are independently and identically distributed, ${\mathit{Y}}_i\in \mathbb{R},i=1,\dots ,n$ and are independently and identically distributed. It was shown that estimators based on sparsely connected deep neural networks with the ReLU activation function and a properly chosen network architecture achieve minimax rates of convergence (up to log n-factors) under a general composition assumption on the regression function.

Considering the dependencies and heterogeneity of spatial models, we study nonparametric high-dimensional spatial models as follows:

$$Y_{\mathit{i}}=m\left({\mathit{X}}_{\mathit{i}}\right)+R_{\mathit{i}},$$

(2)

where $\mathit{i}\in {\Lambda }_{\mathit{n}}=\left\{\left(i_1,i_2\dots ,i_N\right):i_j=1,2,\dots ,n_j,j=1,2,\dots N\right\},Y_{\mathit{i}}\in \mathbb{R},{\mathit{X}}_{\mathit{i}}\in {\mathbb{R}}^d,m\left({\mathit{X}}_{\mathit{i}}\right)$ represents the trend function, $R_{\mathit{i}}$ satisfies the $\alpha$-mixing condition (the definition of the $\alpha$-mixing condition can be found at the beginning of Section 2.2), $n=n_1+\dots +n_N.$ For example, Y denotes the hourly ozone concentrations; $\mathit{X}$ is a $5\times 1$ vector that consists of the following explanatory variables observed at each station: wind speed, air pressure, air temperature, relative humidity, and elevation. The observation locations are recorded in longitude $i_1$ and latitude $i_2$. In this case, $d=5$ and $N=2$; see [8].

Under general assumptions, we prove the consistency of the estimator and provide bounds for the mean squared error $MSE$. In the simulation aspect, a comparison with the local linear regression method demonstrates that the neural network method converges much faster than does the local linear regression. In the empirical study, considering the air pollution index, air pollutants, and environmental factors, the effectiveness of the neural network is demonstrated through a comparison with the local linear regression method, especially in small sample cases.

Throughout the rest of the paper, bold letters are used to represent vectors; for example, $\mathit{x}:={\left(x_1,\dots ,x_d\right)}^{\top }$. We define ${\left|\mathit{x}\right|}_{\infty }:={max}_i\left|x_i\right|,{\left|\mathit{x}\right|}_0:={\sum }_i\mathbf{I}\left(x_i\ne 0\right),$ where $\mathbf{I}$ represents the indicator function:

$$\begin{array}{c}\hfill {\mathbf{I}}_{\mathrm{A}}\left(w\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}w\notin A,\hfill \\ 1\hfill & \mathrm{if}w\in A,\hfill \end{array}\right.\end{array}$$

${\left|\mathit{x}\right|}_0$ denotes the total number of ${\mathit{x}}_i$, which is not equal to zero. ${\left|\mathit{x}\right|}_p:={\left({\sum }_{i=1}^d{\left|x_i\right|}^p\right)}^{1/p}$, and we write ${\left|f\right|}_p:={\left|f\right|}_{L^p\left(D\right)}$ as the $L^p$ norm on D; D is some domain, and different situations may be different. For two sequences ${\left(a_n\right)}_n$ and ${\left(b_n\right)}_n$, we write $a_n\lesssim b_n$ if there exists a constant C such that for all n, $a_n\le C{b}_n$. Moreover, $a_n\asymp b_n$ means $a_n\lesssim b_n$ and $b_n\lesssim a_n$. ${log}_2$ denotes the logarithm base 2, $\ln$ denotes the logarithm base e, $\lceil x\rceil$ represents the smallest integer $\ge x$, and $\lfloor x\rfloor$ represents the largest integer $\le x$.

2. Nonparametric High-Dimensional Space Model Estimation

2.1. Mathematical Modeling of Deep Network Features

Definition 1. 

Fitting a multilayer neural network requires the choice of an activation function $\sigma :\mathbb{R}\to \mathbb{R}$ and the network architecture. Motivated by its importance in deep learning, we study the rectifier linear unit (ReLU) activation function; see [7].

$$\sigma \left(x\right)=\mathsf{max}(x,0).$$

For $\mathbf{v}=\left(v_1,\dots ,v_r\right)\in {\mathbb{R}}^r$, the displacement activation function ${\sigma }_{\mathbf{v}}:{\mathbb{R}}^r\to {\mathbb{R}}^r$ is defined as follows:

${\sigma }_{\mathbf{v}}:{\mathbb{R}}^r\to {\mathbb{R}}^r$,

$${\sigma }_{\mathbf{v}}\left(\begin{array}{c}{y}_1\\ ⋮\\ y_r\end{array}\right)=\left(\begin{array}{c}\sigma \left(y_1-v_1\right)\\ ⋮\\ \sigma \left(y_r-v_r\right)\end{array}\right).$$

The neural network architecture $(L,\mathbf{p})$ consists of a positive integer L known as the number of hidden layers or depth and a width vector $\mathbf{p}=\left(p_0,\dots ,p_{L+1}\right)\in {\mathbb{N}}^{L+2}$. A neural network with the network structure $(L,\mathbf{p})$ is any function of the following form:

$$f:{\mathbb{R}}^{p_0}\to {\mathbb{R}}^{p_{L+1}},\mathbf{x}\mapsto f\left(\mathbf{x}\right)=W_L{\sigma }_{{\mathbf{v}}_L}{W}_{L-1}{\sigma }_{{\mathbf{v}}_{L-1}}\cdots W_1{\sigma }_{{\mathbf{v}}_1}{W}_0\mathbf{x},$$

(3)

where ${\mathit{W}}_{\mathit{i}}$ is a $p_{i+1}\times p_i$ weight matrix and ${\mathbf{v}}_i\in {\mathbb{R}}^{pi}$ is a displacement vector, where $i=1,2,\dots ,L$. Therefore, the network function is constructed by alternating matrix vector multiplications and the action of nonlinear activation functions ${\sigma }_{\mathbf{v}}$. In Equation (3), the shift vectors can also be omitted by considering the input as $(\mathbf{x},1)$ and augmenting the weight matrices with an additional row and column. To fit the network to data generated by a d-dimensional nonparametric regression model, it is required to have $p_0=d$ and $p_{L+1}=1$.

Given a network function $f\left(\mathbf{x}\right)=W_L{\sigma }_{{\mathbf{v}}_L}{W}_{L-1}{\sigma }_{{\mathbf{v}}_{L-1}}\cdots W_1{\sigma }_{{\mathbf{v}}_1}{W}_0\mathbf{x}$, the network parameters are the elements of the matrices ${\left(W_j\right)}_{j=0,\dots ,L}$ and the vectors ${\left({\mathbf{v}}_j\right)}_{j=1,\dots ,L}$. These parameters need to be estimated/learned from the data. In this context, “estimate” and “learn” can be used interchangeably, as the process of estimating the parameters from data is often referred to as learning in the context of neural networks and machine learning.

The purpose of this paper is to consider a framework that encompasses the fundamental characteristics of modern deep network architectures. In particular, in this paper, we allow for a large depth L and a significant number of potential network parameters without requiring an upper bound on the number of network parameters for the main results. Consequently, this approach deals with high-dimensional settings that have more parameters than training data. Another characteristic of trained networks is that the learned network parameters are typically not very large; see [7]. In practice, the weights of trained networks often do not differ significantly from the initialized weights. As all elements in orthogonal matrices are bounded by 1, the weights of trained networks also do not become excessively large. However, existing theoretical results often demand that the size of the network parameters tends to infinity. To be more consistent with what is observed in practice, all parameters considered in this paper are bounded by 1. By projecting the network parameters at each iteration onto the interval $[-1,1]$, this constraint can be easily incorporated into deep learning algorithms.

Let ${\left|W_j\right|}_{\infty }$ denote the maximum element norm of $W_j$, and let us consider the network function space with a given network structure and network parameters bounded by 1 as follows:

$$\mathcal{F}(L,\mathbf{p}):=\left\{f\mathrm{in}\mathrm{the}\mathrm{form}\mathrm{of}\left(2.1\right):\underset{j=0,\dots ,L}{max}{∥W_j∥}_{\infty }\vee {\left|{\mathbf{v}}_j\right|}_{\infty }\le 1\right\},$$

(4)

where ${\mathbf{v}}_0$ is a vector with all components being 0.

In this work, we model the network sparsity assuming that there are only a few nonzero/active network parameters. If $\left|\right|W_j{\left|\right|}_0$ denotes the number of nonzero entries of $W_j$ and ${\left|\right|\left|f\right|}_{\infty }{\left|\right|}_{\infty }$ stands for the supnorm of the function $\mathit{x}\to {\left|f\left(\mathbf{x}\right)\right|}_{\infty }$, then the s-sparse networks are given by

$$\begin{array}{cc}\hfill \mathcal{F}(L,\mathbf{p},s)& :=\mathcal{F}(L,\mathbf{p},s,F)\hfill \\ & :=\left\{f\in \mathcal{F}(L,\mathbf{p}):\sum _{j=0}^L{∥W_j∥}_0+{\left|{\mathbf{v}}_j\right|}_0\le s,{∥{\left|f\right|}_{\infty }∥}_{\infty }\le F\right\},\hfill \end{array}$$

where F is a constant; the upper bound on the uniform norm of the function f is often unnecessary and is thus omitted in the notation. Here, we consider cases where the number of network parameters s is very small compared to the total number of parameters in the network.

For any estimate ${\widehat{m}}_n$ that returns a network in class $\mathcal{F}(L,\mathbf{p},s,F)$, the corresponding quantity is defined as follows:

$$\begin{array}{cc}\hfill & {\Delta }_n\left({\widehat{m}}_n,m\right)\hfill \\ \hfill & :=E_m\left[\frac{1}{n}\sum _{i\in {\Lambda }_{\mathit{n}}}{\left(Y_{\mathit{i}}-{\widehat{m}}_n\left({\mathit{X}}_{\mathit{i}}\right)\right)}^2-\underset{f\in \mathcal{F}(L,\mathbf{p},s,F)}{inf}\frac{1}{n}\sum _{i\in {\Lambda }_{\mathit{n}}}{\left(Y_{\mathit{i}}-f\left({\mathit{X}}_{\mathit{i}}\right)\right)}^2\right].\hfill \end{array}$$

(5)

The sequence ${\Delta }_n\left({\widehat{m}}_n,m\right)$ measures the discrepancy between the expected empirical risk of ${\widehat{m}}_n$ and the global minimum of this class over all networks. The subscript m in $E_m$ indicates the sample expectation with respect to the nonparametric regression model generated by the regression function m. Notice that ${\Delta }_n\left({\widehat{m}}_n,m\right)\ge 0$, and ${\Delta }_n\left({\widehat{m}}_n,m\right)=0$ if ${\widehat{m}}_n$ is an empirical risk minimizer.

Therefore, ${\Delta }_n\left({\widehat{m}}_n,m\right)$ is a critical quantity that, together with the minimax estimation rates, determines the convergence rate of ${\widehat{m}}_n$.

To evaluate the statistical performance of ${\widehat{m}}_n$ under general assumptions, the mean squared error of the estimator is defined as

$$R\left({\widehat{m}}_n,m\right):=E_m\left[{\left({\widehat{m}}_n\left(\mathit{X}\right)-m\left(\mathit{X}\right)\right)}^2\right].$$

(6)

2.2. Estimation and Theoretical Properties

In order to obtain asymptotic results, we will assume throughout this paper that ${\mathit{X}}_{\mathit{i}}$, $\mathit{i}$$\in {\Lambda }_{\mathit{n}}$ satisfies the following $\alpha$-mixing condition: there exists a function $\phi \left(t\right)\downarrow$ as $t\to \infty$ with $\phi \left(0\right)=1$, such that whenever $\mathrm{\Xi }$, $\widetilde{\left(\mathrm{\Xi }\right)}$⊆${\Lambda }_{\mathit{n}}$ are finite sets, it is the case that

$$\begin{array}{cc}& \tau (\mathcal{B}\left(\mathrm{\Xi }\right),\mathcal{B}\widetilde{\left(\mathrm{\Xi }\right)})\hfill \\ & =sup\left\{\right|Pr\left(AB\right)-Pr\left(B\right)|A\in \mathcal{B}\left(\left(\mathrm{\Xi }\right)\right),B\in \mathcal{B}\widetilde{\left(\mathrm{\Xi }\right)}\}\hfill \\ & \le \psi (card\left(\mathrm{\Xi }\right),\widetilde{\left(\mathrm{\Xi }\right)})\phi \left(d(\mathrm{\Xi },\mathrm{\Xi })\right),\hfill \end{array}$$

where $\mathcal{B}\left(\mathrm{\Xi }\right)$ denotes the Borel $\sigma$-field generated by $\{X_i,i\in \mathrm{\Xi }\}$, card$\left(\mathrm{\Xi }\right)$ is the cardinality of $\mathrm{\Xi }$, and d($\mathrm{\Xi },\widetilde{\left(\mathrm{\Xi }\right)}$) = $min\{||i-i^{{}^{\prime }}||:i\in \mathrm{\Xi },i^{{}^{\prime }}\in \widetilde{\left(\mathrm{\Xi }\right)}\}$ is the distance between $\mathrm{\Xi }$ and $\widetilde{\left(\mathrm{\Xi }\right)}$, where $\left|\right|i\left|\right|={(i_1^2+\cdots +i_N^2)}^{\frac{1}{2}}$ stands for the Euclidean norm and $\psi :{\mathbb{N}}^2\to {\mathbb{R}}^{+}$ is a symmetric positive function that is nondecreasing in each variable; see [8].

The theoretical performance of neural networks depends on the underlying function class, and a classic approach in nonparametric statistics is to assume that the regression function is $\beta$-smooth. In this paper, we assume that the regression function $m\left({\mathit{X}}_{\mathit{i}}\right)$ is a composition of multiple functions, i.e.,

$$m=g_q\circ g_{q-1}\circ \dots \circ g_1\circ g_0,$$

where $g_i:{\left[a_i,b_i\right]}^{d_i}\to {\left[a_{i+1},b_{i+1}\right]}^{d_{i+1}}$. We denote the components of $g_i$ as $g_i={\left(g_{ij}\right)}_{j=1,\dots ,d_{i+1}}^{\top }$, and we let $t_i$ be the maximum variable that each $g_i$ depends on. Thus, each $g_i$ is a function with $t_i$ variables.

If all partial derivatives up to order $\lfloor \beta \rfloor$ of a function exist and are bounded, the $\lfloor \beta \rfloor$-th order partial derivatives are $\beta$-$\lfloor \beta \rfloor$ Hölder, where $\lfloor \beta \rfloor$ represents the largest integer strictly less than $\beta$. Then, the ball of $\beta$-Hölder functions with radius K is defined as follows:

$$\begin{array}{cc}\hfill {\mathcal{C}}_r^{\beta }(D,K)=& \left\{f:D\subset {\mathbb{R}}^r\to \mathbb{R}:\right.\hfill \\ & \left.\sum _{\mathit{\alpha }:\left|\mathit{\alpha }\right|<\beta }{∥{\partial }^{\mathit{\alpha }}f∥}_{\infty }+\sum _{\mathit{\alpha }:\left|\mathit{\alpha }\right|=\lfloor \beta \rfloor }\underset{\begin{array}{c}\mathbf{x},\mathbf{y}\in D\\ \mathbf{x}\ne \mathbf{y}\end{array}}{sup}\frac{\left|{\partial }^{\mathit{\alpha }}f\left(\mathbf{x}\right)-{\partial }^{\mathit{\alpha }}f\left(\mathbf{y}\right)\right|}{{|\mathbf{x}-\mathbf{y}|}_{\infty }^{\beta -\lfloor \beta \rfloor }}\le K\right\},\hfill \end{array}$$

where we use multi-index notation, i.e., ${\partial }^{\mathit{\alpha }}={\partial }^{{\alpha }_1}\dots {\partial }^{{\alpha }_r}$, where $\mathit{\alpha }=({\alpha }_1,\dots ,{\alpha }_r)\in {\mathbb{N}}^r$; see [7].

We assume that each function $g_{ij}$ has Hölder smoothness ${\beta }_i$. Since $g_{ij}$ is also a function of $t_i$ variables, $g_{ij}\in {\mathcal{C}}_{t_i}^{{\beta }_i}\left({\left[a_i,b_i\right]}^{t_i},K_i\right)$, the underlying function space is then defined as

$$\begin{array}{c}\hfill \mathcal{G}(q,\mathbf{d},\mathbf{t},\mathit{\beta },K):=\left\{m=g_q\circ \dots \circ g_0:g_i={\left(g_{ij}\right)}_j:{\left[a_i,b_i\right]}^{d_i}\to {\left[a_{i+1},b_{i+1}\right]}^{d_{i+1}},\right.\\ \hfill \left.g_{ij}\in {\mathcal{C}}_{t_i}^{{\beta }_i}\left({\left[a_i,b_i\right]}^{t_i},K\right),\left|a_i\right|,\left|b_i\right|\le K\right\},\end{array}$$

where $\mathbf{d}:=\left(d_0,\dots ,d_{q+1}\right),\mathbf{t}:=\left(t_0,\dots ,t_q\right),\mathit{\beta }:=\left({\beta }_0,\dots ,{\beta }_q\right)$.

Theorem 1. 

We consider the nonparametric regression model with d variables for the composite regression function $m=g_q\circ g_{q-1}\circ \dots \circ g_1\circ g_0$ in the class $\mathcal{G}(q,\mathbf{d},\mathbf{t},\mathit{\beta },K)$, as described in Equation (2). Let ${\widehat{m}}_n$ be an estimator from the function class $\mathcal{F}\left(L,{\left(p_i\right)}_{i=0,\dots ,L+1},s,F\right)$ satisfying the following conditions:

(1) 

$F\ge max(K,1)$,

(2) 

${\sum }_{i=0}^q{log}_2\left(4t_i\vee 4{\beta }_i\right){log}_{2}n\le L\lesssim n{\varphi }_n$,

(3) 

$n{\varphi }_n\lesssim {min}_{i=1,\dots ,L}{p}_i$,

(4) 

$s\asymp n{\varphi }_n\ln n$,

where ${\varphi }_n$ is a positive sequence; then, there exist constants C and $C^{\prime }$ depending only on $q,\mathbf{d},\mathbf{t},\mathit{\beta },F$, such that if ${\Delta }_n\left({\widehat{m}}_n,m\right)\le C\varphi nL{ln}^{2}n$, then

$$R\left({\widehat{m}}_n,m\right)\le C^{\prime }{\varphi }_{n}L{ln}^{2}n,$$

(7)

if ${\Delta }_n\left({\widehat{m}}_n,m\right)\ge C\varphi nL{\ln }^{2}n$, then

$$\frac{1}{C^{\prime }}{\Delta }_n\left({\widehat{m}}_n,m\right)\le R\left({\widehat{m}}_n,m\right)\le C^{\prime }{\Delta }_n\left({\widehat{m}}_n,m\right).$$

(8)

To minimize ${\varphi }_{n}L{\ln }^{2}n$, let ${\Delta }_n\left({\widehat{m}}_n,m\right)\le C{\varphi }_n{\ln }^{3}n$; then,

$$R\left({\widehat{m}}_n,m\right)\le C^{\prime }{\varphi }_n{\ln }^{3}n.$$

The convergence rate in Theorem 1 depends on ${\varphi }_n$ and ${\Delta }_n\left({\widehat{m}}_n,m\right)$. The following reasoning shows that ${\varphi }_n$ serves as a lower bound for the supremum infimum estimation risk over this class. For any empirical risk minimizer, where the definition of the ${\Delta }_n$ term becomes zero, the following corollary holds.

Corollary 1. 

Let ${\tilde{m}}_n\in arg{min}_{f\in \mathcal{F}(L,\mathbf{p},s,F)}{\sum }_{i\in {\Lambda }_{\mathit{n}}}{\left(Y_{\mathbf{i}}-m\left({\mathit{X}}_{\mathit{i}}\right)\right)}^2$ be an empirical risk minimizer under the same conditions as in Theorem 1. There exists a constant $C^{\prime }$, depending only on $q,\mathbf{d},\mathbf{t},\mathit{\beta },F$, such that

$$R\left({\tilde{m}}_n,m\right)\le C^{\prime }{\varphi }_{n}L{\ln }^{2}n.$$

(9)

Condition (1) in Theorem 1 is very mild and states only that the network functions should have at least the same supremum norm as the regression function. From the other assumptions in Theorem 1, it becomes clear that there is a lot of flexibility in selecting a good network architecture as long as the number of active parameters s is taken to be in the right order.

In a fully connected network, the number of network parameters is ${\sum }_{i=0}^L{p}_i{p}_{i+1}$. This implies that Theorem 1 requires a sparse network. More precisely, the network must have at least ${\sum }_{i=1}^L{p}_i-s$ completely inactive nodes, meaning that all incoming signals are zero. Condition (4) chooses $s\asymp n{\varphi }_n\ln n$ to balance the mean squared error and variance. From the proof of this theorem (Appendix B), convergence rates for various orders of s can also be derived.

Deep learning excels over other methods only in the large sample regime. This suggests that the method may be adaptable to the underlying structures in the data. This may produce rapid convergence rates, but with larger constants or remainders, which can lead to relatively poor performance in small sample scenarios.

The proof of the risk bounds in Theorem 1 is based on the following oracle-type inequality.

Theorem 2. 

Let us consider the d-dimensional nonparametric regression model given by Equation (2) with an unknown regression function m, where $F\ge 1$ and ${\left|m\right|}_{\infty }\le F$, let ${\widehat{m}}_n$ be an arbitrary estimator taking values in the class $\mathcal{F}(L,\mathbf{p},s,F)$, and let

$$\begin{array}{cc}\hfill & {\Delta }_n\left({\widehat{m}}_n,m\right)\hfill \\ \hfill & :=E_m\left[\frac{1}{n}\sum _{i\in {\Lambda }_{\mathit{n}}}{\left(Y_{\mathbf{i}}-{\widehat{m}}_n\left({\mathit{X}}_{\mathit{i}}\right)\right)}^2-\underset{f\in \mathcal{F}(L,\mathbf{p},s,F)}{inf}\frac{1}{n}\sum _{i\in {\Lambda }_{\mathit{n}}}{\left(Y_{\mathbf{i}}-f\left({\mathit{X}}_{\mathit{i}}\right)\right)}^2\right],\hfill \end{array}$$

(10)

and for any $\epsilon \in (0,1]$, there exists a constant $C_{\epsilon }$, depending only on ε, such that

$${\tau }_{\epsilon ,n}:=C_{\epsilon }{F}^2\frac{(s+1)ln\left(n{(s+1)}^L{p}_0{p}_{L+1}\right)}{n},$$

and we have

$$\begin{array}{cc}& {(1-\epsilon )}^2{\Delta }_n\left({\widehat{m}}_n,m\right)-{\tau }_{\epsilon ,n}\le R\left({\widehat{m}}_n,m\right)\hfill \\ & \le {(1+\epsilon )}^2\left(\underset{f\in \mathcal{F}(L,\mathbf{p},s,F)}{inf}{∥f-m∥}_{\infty }^2+{\Delta }_n\left({\widehat{m}}_n,m\right)\right)+{\tau }_{\epsilon ,n}.\hfill \end{array}$$

In the context of oracle-type inequalities, an increase in the number of layers can lead to a deterioration in the upper bound on the risk. In practice, it has also been observed that having too many layers can result in a decline in performance. We refer to Section 4.4 in He et al. [9] and He and Sun [10] for more details.

The proof relies on two specific properties of the ReLU activation function rather than other activation functions. The first property is its projection property, which is expressed as

$$\sigma \circ \sigma =\sigma ,$$

where the composite of the ReLU activation function is considered, given that the foundation of approximation theory lies in constructing smaller networks to perform simpler tasks, which may not all require the same network depth. To combine these subnetworks, it is necessary to synchronize the network depth by adding hidden layers that do not alter the output. This can be achieved by selecting weight matrices in the network (assuming an equal width for consecutive layers) and by utilizing the projection property of the ReLU activation function, given by $\sigma \circ \sigma =\sigma$. This property is beneficial not only theoretically but also in practice, as it greatly aids in passing a result to deeper layers through skip connections.

Next, we prove that ${\varphi }_n$ serves as a lower bound for the supremum infimum estimation risk over class $\mathcal{G}(q,\mathbf{d},\mathbf{t},\mathit{\beta },K)$ with $t_i\le min\left(d_0,\dots ,d_{i-1}\right)$. This means that in the composition of functions, no additional dimensions are added at deeper abstract layers. In particular, this approach avoids the case where $t_i$ exceeds the input dimension $d_0$.

Theorem 3. 

Let us consider the nonparametric regression model (2), where ${\mathit{X}}_{\mathit{i}}$ is drawn from a distribution with a Lebesgue density on ${[0,1]}^d$, and the lower and upper bounds of this distribution are positive constants. For any nonnegative integer q, arbitrary dimension vectors $\mathbf{d}$ and $\mathbf{t}$, and for all i such that $t_i\le min\left(d_0,\dots ,d_{i-1}\right)$, and any smoothness vector β, along with all sufficiently large constants $K>0$, there exists a positive constant c such that

$$\underset{{\widehat{m}}_n}{inf}\underset{m\in \mathcal{G}(q,\mathbf{d},\mathbf{t},\mathit{\beta },K)}{sup}R\left({\widehat{m}}_n,m\right)\ge c{\varphi }_n,$$

where inf is taken over all estimators ${\widehat{m}}_n$.

By combining the supremum infimum lower bound with the oracle-type inequality, we can easily obtain the following result.

Lemma 1. 

Given $\beta ,K>0$, and $d\in \mathbb{N}$, there exist constants $c_1,c_2$ depending only on $\beta ,K,d$, such that for $\epsilon \le c_2$, we have

$$s\le c_1\frac{{\epsilon }^{-d/\beta }}{Lln(1/\epsilon )},$$

and then, for any width vector $\mathbf{p}$, where $p_0=d$ and $p_{L+1}=1$, we know that

$$\underset{m\in {\mathcal{C}}_d^{\beta }\left({[0,1]}^d,K\right)}{sup}\underset{f\in \mathcal{F}(L,\mathbf{p},s)}{inf}{∥f-m∥}_{\infty }\ge \epsilon .$$

2.3. Suboptimality of Wavelet Series Estimation

In this section, we show that wavelet series estimators are unable to take advantage of the underlying composition structure in the regression function and achieve, in some setups, much slower convergence rates. Wavelet estimation is susceptible to the curse of dimensionality, whereas neural networks can achieve faster convergence rates.

We consider a compressed wavelet system $\left({\psi }_{\mathit{\lambda }},\mathit{\lambda }\in \mathbf{\Lambda }\right)$, restricted to $L^2[0,1]$ from $L^2\left(\mathbb{R}\right)$, as referred to in Cohen et al. [11]. Here, $\mathbf{\Lambda }=\left\{(j,k):j=-1,0,1,\dots ;k\in I_j\right\}$, and ${\psi }_{-1,k}:=\varphi (·-k)$ denotes the shift-scaled function. For any function $f\in L^2{[0,1]}^d$, we have

$$f\left(\mathbf{x}\right)=\sum _{\left({\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_d\right)\in \mathbf{\Lambda }\times \dots \times \mathbf{\Lambda }}{d}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f\right)\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(x_r\right),$$

and the convergence on $L^2[0,1]$ entails wavelet coefficients.

$$d_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f\right):=\int f\left(\mathbf{x}\right)\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(x_r\right)d\mathbf{x}.$$

To construct a counterexample, it is sufficient to consider the nonparametric regression model $Y_{\mathbf{i}}=m\left({\mathit{X}}_{\mathit{i}}\right)+R_{\mathit{i}},$ $\mathit{i}\in {\Lambda }_{\mathit{n}}=\left\{\left(i_1,i_2,\dots ,i_N\right):i_j=1,2,\dots ,n_j,j=1,2,\dots ,N\right\},$ ${\mathit{X}}_i:=\left(U_{i,1},\dots ,U_{i,d}\right)\sim U{[0,1]}^d$. The empirical wavelet coefficients are obtained; furthermore,

$${\widehat{d}}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)=\frac{1}{n}\sum _{i\in {\Lambda }_{\mathit{n}}}{Y}_i\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(U_{i,r}\right).$$

Since $E\left[{\widehat{d}}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)\right]=d_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)$, an unbiased estimate for the wavelet coefficients is obtained; furthermore,

$$\begin{array}{cc}\hfill Var\left({\widehat{d}}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)\right)& =\frac{1}{n}Var\left(Y_1\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(U_{1,r}\right)\right)\hfill \\ & \ge \frac{1}{n}E\left[Var\left(Y_1\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(U_{1,r}\right)\mid U_{1,1,},\dots ,U_{1,d}\right)\right]\hfill \\ & =\frac{1}{n}.\hfill \end{array}$$

We study the estimators of the following form

$${\widehat{f}}_n\left(\mathbf{x}\right)=\sum _{\left({\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_d\right)\in I}{\widehat{d}}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(x_r\right),$$

(11)

and for any subset $I\subset \Lambda \times \dots \times \Lambda$, we have

$$\begin{array}{cc}\hfill & R\left({\widehat{f}}_n,f_0\right)=\sum _{\left({\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_d\right)\in I}E\left[{\left({\widehat{d}}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)-d_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)\right)}^2\right]+\sum _{\left({\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_d\right)\in I^c}{d}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}{\left(f_0\right)}^2\hfill \\ \hfill & \ge \sum _{\left({\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_d\right)\in \Lambda \times \dots \times \Lambda }\frac{1}{n}\wedge d_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}{\left(f_0\right)}^2.\hfill \end{array}$$

(12)

$\psi \in L^2\left(\mathbb{R}\right)$ possesses compact support; thus, without loss of generality, we assume that $\psi$ is zero outside $\left[0,2^q\right]$ for some integer $q>0$.

Lemma 2. 

For any integer $q>0$, $\nu :=⌈{log}_{2}d⌉+1$, and any $0<\alpha \le 1$ and $K>0$, there exists a nonzero constant $c(\psi ,d)$, which depends solely on d and the properties of the wavelet function ψ. Thus, for any j, we can find a function $f_{j,\alpha }\left(\mathbf{x}\right)=h_{j,\alpha }\left(x_1+\dots +x_d\right)$, where $h_{j,\alpha }\in {\mathcal{C}}_1^{\alpha }([0,d],K)$, such that for all $p_1,\dots ,p_d\in \left\{0,1,\dots ,2^{j-q-\nu }-1\right\}$, we have

$$d_{\left(j,2^{q+\nu }{p}_1\right)\dots \left(j,2^{q+\nu }{p}_d\right)}\left(f_{j,\alpha }\right)=c(\psi ,d)K{2}^{-\frac{j}{2}(2\alpha +d)}.$$

Theorem 4. 

If ${\widehat{m}}_n$ represents a wavelet estimator with compact support ψ and an arbitrary index set I,

$${\widehat{m}}_n\left(\mathbf{x}\right)=\sum _{\left({\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_d\right)\in I}{\widehat{d}}_{{\mathit{\lambda }}_1\dots {\mathit{\lambda }}_d}\left(f_0\right)\prod _{r=1}^d{\psi }_{{\mathit{\lambda }}_r}\left(x_r\right),$$

therefore, for any $0<\alpha \le 1$ and any Hölder radius $K>0$, we have

$$\underset{m\left(\mathbf{x}\right)=h\left({\sum }_{r=1}^d{x}_r\right),h\in {\mathcal{C}}_1^{\alpha }([0,d],K)}{sup}R\left({\widehat{m}}_n,m\right)\gtrsim n^{-\frac{2\alpha }{2\alpha +d}}.$$

As a result, the convergence rate of the wavelet series estimation is slower than $n^{-2\alpha /(2\alpha +d)}$. If d is large, this rate becomes significantly slower. Therefore, wavelet estimation is sensitive to the curse of dimensionality, while neural networks can achieve rapid convergence.

3. Simulation Experiments and Case Study

3.1. Simulation Experiments

In this section, we conduct a comparative study through 100 repeated experimental simulations, evaluating the mean squared error of the estimation using both the local linear regression method and deep neural networks with the ReLU activation function. We consider the following models.

Model 1:

$$Y_{\mathit{i}}=2+2{\mathit{X}}_{\mathit{i}}\beta +e_{\mathit{i}},$$

where ${\mathit{X}}_{\mathit{i}}$ follows a uniform distribution on $[-1,1]$, $\beta$ is the coefficient, simulated from a uniform distribution on $[0,1]$, and $e_{\mathit{i}}$ is the noise variable following a standard normal distribution.

A neural network with a depth of 3 and a width of 32 was created, where the first two layers are fully connected layers, and the output layer uses the ReLU activation function.

We consider four scenarios with the same sample size but different dimensions.

$$\mathrm{Scenario}1:\dim =3,n=[200,600,1000,1400,1800,2200];$$

$$\mathrm{Scenario}2:\dim =5,n=[200,600,1000,1400,1800,2200];$$

$$\mathrm{Scenario}3:\dim =8,n=[200,600,1000,1400,1800,2200];$$

$$\mathrm{Scenario}4:\dim =10,n=[200,600,1000,1400,1800,2200];$$

where dim represents the dimension and n is the sample size. For each scenario, the mean squared error $\left(MSE\right)$ is calculated to compare the performance of the local linear regression method and the deep neural network method. The $MSE$ in this study is defined as follows:

$$MSE=\frac{1}{n}\sum _{i=1}^n{(y^{\left(i\right)}-{\widehat{y}}^{\left(i\right)})}^2.$$

(13)

Table 1 presents the estimates at different values of n.

Table 1. Various dimensional MSE values of two methods for Model 1.

	Scenario 1		Scenario 2		Scenario 3		Scenario 4
$\mathit{n}$	$\mathit{NE}$	$\mathit{DNN}$	$\mathit{NE}$	$\mathit{DNN}$	$\mathit{NE}$	$\mathit{DNN}$	$\mathit{NE}$	$\mathit{DNN}$
200	0.1452	0.0363	0.9120	0.0695	4.4421	0.1119	7.5481	0.1520
600	0.0438	0.0150	0.2422	0.0280	1.3758	0.0488	2.7657	0.0655
1000	0.0242	0.0110	0.1343	0.0207	0.8347	0.0375	1.7616	0.0473
1400	0.0186	0.0097	0.0964	0.0187	0.6102	0.0336	1.2408	0.0414
1800	0.0146	0.0080	0.0755	0.0171	0.4768	0.0298	1.0533	0.0358
2200	0.0112	0.0075	0.0613	0.0153	0.4156	0.0258	0.9199	0.0357

In this table, $NE$ represents the $MSE$ of the nonparametric estimation method, and $DNN$ represents the $MSE$ of the deep neural network method. It is evident that as $MSE$ approaches 0, the estimation accuracy increases. From Table 1, we observe that for the same dimension, as the sample size increases, $DNN$ tends towards 0. For the same sample size, $DNN$ is significantly smaller than $NE$, and with the increase in dimension, the superiority of the neural network method over the local linear regression method becomes more pronounced. Therefore, the neural network method achieves much higher estimation accuracy, especially for large sample sizes and high dimensions.

As shown in Figure 1, with the increase in dimension, the performance of the neural network fitting surpasses the local linear regression method, where the x-axis represents the sample size n, and the y-axis represents the mean squared error ($MSE$). In higher dimensions, the $MSE$ of the neural network method approaches almost zero, which is attributed to the avoidance of the curse of dimensionality by deep neural networks. This demonstrates that the convergence rate of the deep neural network is superior to that of the local linear regression method and approaches the optimal convergence rate.

Figure 1. $MS{E}_s$ of local linear regression method (dashed blue line) and the neural network method (solid orange line) of dimension 3, 5, 8 and 10. (a) Three dimensions. (b) Five dimensions. (c) Eight dimensions. (d) Ten dimensions.

Next, we consider high-dimensional spatial models with dependency structures to compare the $MS{E}_s$ of the two methods mentioned above.

Model 2:

$$Y_{i,j}=2sin\left(2\pi {\mathit{X}}_{i,j}\right)+R_{i,j},$$

where $i=1,\dots ,n_1$, $j=1,\dots ,n_2$, and ${\mathit{X}}_{i,j}$ follows a zero-mean second-order stationary process. $R_{i,j}$ follows a standard normal distribution. Similarly to the work of Cressie and Wikle [12], high-dimensional spatial processes ${\mathit{X}}_{i,j}$ are generated using spectral methods.

$${\mathit{X}}_{i,j}={(2/M)}^{1/2}\sum _{k=1}^{M}cos(w(1,k)\ast i+w(2,k)\ast j+r\left(k\right)).$$

In this case, $w(i,k)$ for $i=1,2$ follows a standard normal distribution and is independent of $r\left(k\right)$ for $k=1,\dots ,M$, where $r\left(k\right)$ are independently and identically distributed from a uniform distribution on $[-\pi ,\pi ]$. As $n\to \infty$, ${\mathit{X}}_{i,j}$ converges to a Gaussian random process. We consider the case with dimension 5 and sample sizes [200, 600, 1000, 1400, 1800, 2200]. The network structure is the same as in Model 1.

As shown in Table 2, it can be observed that the convergence rate of the high-dimensional spatial model with dependence is worse than the convergence rate of the high-dimensional spatial model without dependence. However, in comparison to the local linear regression method, the $MSE$ values of the neural network are much smaller, indicating that the neural network achieves better convergence performance.

Table 2. The $MSE$ values of both methods for Model 2.

n	200	600	1000	1400	1800	2200
$NE$	36.8606	32.4013	17.6990	14.0570	10.7740	5.1166
$DNN$	2.0305	2.0078	2.0074	2.0047	2.0041	2.0020

As shown in Figure 2, we see that in the case of large sample sizes and high-dimensional spatial models, the neural network achieves superior convergence compared to that of the local linear regression method, where the x-axis represents the sample size n, and the y-axis represents the mean squared error ($MSE$).

Figure 2. Comparison of $MS{E}_s$ between two methods for Model 2.

3.2. Case Study

To compare the consistency of the local linear regression method and deep neural network for high-dimensional spatial models, we consider the relationship between air pollution and respiratory diseases in the New Territories East of Hong Kong from 1 January 2000 to 15 January 2001, as studied by Wang et al. [13]. There is a dataset consisting of 821 observations, where we mainly consider the air pollution index $X_1$ and five pollutants, sulfur dioxide ($\mathrm{g}/{\mathrm{m}}^3$) $X_2$, inhalable particulate matter ($\mathrm{g}/{\mathrm{m}}^3$) $X_3$, nitrogen compounds ($\mathrm{g}/{\mathrm{m}}^3$) $X_4$, nitrogen dioxide ($\mathrm{g}/{\mathrm{m}}^3$) $X_5$, and ozone ($\mathrm{g}/{\mathrm{m}}^3$) $X_6$, as well as two environmental factors: temperature (${}^{°}$C) $X_7$ and relative humidity (%) $X_8$. In this section, we examine the relationship between the levels of chemical pollutants in the New Territories East of Hong Kong and the daily hospital admissions for respiratory diseases (Y). The specific parameter settings are the same as those in the numerical simulation part.

After dimensionless and standardization processing, we use 397 data points for the case study. Among these, 80% of the data are used to train the model, and the remaining 20% are used to evaluate the quality of the trained model. The $MSE$ values are shown in Table 3. It can be observed that the $MS{E}_2$ values are closer to 0, indicating that in real-world cases, the mean squared error of the deep neural network method is much smaller than that of the local linear regression method. Therefore, the deep neural network shows better convergence performance.

Table 3. The $MSE$ values of both methods for this case.

n	50	100	150	200	250	300
$NE$	3.1912	2.2667	2.0146	1.8464	1.7926	1.7485
$DNN$	1.7958	1.5066	1.3923	1.3338	1.2983	1.2901

Figure 3 presents a visual representation of the $MSE$ values for both methods, where the x-axis represents the sample size n, and the y-axis represents the mean squared error ($MSE$). From the graph, it is evident that the deep neural network method exhibits a faster convergence rate compared to that of the local linear regression method.

Figure 3. Comparison of $MS{E}_s$ between two methods for this case.

4. Conclusions

In this study, we employ neural networks with ReLU activation functions for nonparametric estimation in high-dimensional spaces. By constructing suitable network architectures, we estimate unknown trend functions and prove the consistency of the estimators while also comparing and analyzing the deep neural network approach with traditional nonparametric methods.

The focus is on high-dimensional space models with unknown error distributions. Considering the spatial dependencies and heterogeneity in the space models, a deep neural network with ReLU activation functions is used to estimate the unknown trend functions. Under general assumptions, the consistency of the estimators is established, and bounds for the mean squared error ($MSE$) are provided. The estimators exhibit a convergence rate that is related to the sample size but independent of the dimensionality d, thereby avoiding the curse of dimensionality. Moreover, the proposed estimators achieve convergence speeds close to optimality.

Considering the spatial dependencies in high-dimensional settings with large sample sizes, the deep neural network method outperforms traditional nonparametric estimation methods.