Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Abstract

There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Based on our approximation error analysis, we derive the minimax optimality of the deep neural network estimators with the general activation functions in both regression and classification problems.

1. Introduction

Neural networks are learning machines motivated by the architecture of the human brain. Neural networks are comprised of multiple hidden layers, and each of the hidden layers has multiple hidden nodes which consist of an affine map of the outputs from the previous layer and a nonlinear map called an activation function. Deep neural networks have been leading tremendous success in various pattern recognition and machine learning tasks such as object recognition, image segmentation, machine translation and others. For an overview on the empirical success of deep neural networks, we refer to the review paper [1] and recent book [2].

Inspired by the success of deep neural networks, many researchers have tried to give theoretical supports for the success of deep neural networks. Much of the work upto date has focused on the expressivity of deep neural networks, i.e., ability to approximate a rich class of functions efficiently. The well-known classical result on this topic is the universal approximation theorem, which states that every continuous function can be approximated arbitrarily well by a neural network [3,4,5,6,7]. But these results do not specify the required numbers of layers and nodes of a neural network to achieve a given approximation accuracy.

Recently, several results about the effects of the numbers of layers and nodes of a deep neural network to its expressivity have been reported. They provide upper bounds of the numbers of layers and nodes required for neural networks to uniformly approximate all functions of interest. Examples of a class of functions include the space of rational functions of polynomials [8], the Hölder space [9,10,11,12], Besov and mixed Besov spaces [13] and even a class of discontinuous functions [14,15].

The nonlinear activation function is a central part that makes neural networks differ from the linear models, that is, a neural network becomes a linear function if the linear activation function is used. Therefore, the choice of an activation function substantially influences on the performance and computational efficiency. Numerous activation functions have been suggested to improve neural network learning [16,17,18,19,20,21]. We refer to the papers [21,22] for an overview of this topic.

There are also many recent theoretical studies about the approximation ability of deep neural networks. However, most of the studies focus on a specific type of the activation function such as ReLU [9,10,13,14,15], or small classes of activation functions such as sigmoidal functions with additional monotonicity, continuity, and/or boundedness conditions [23,24,25,26,27] and m-admissible functions which are sufficiently smooth and bounded [11]. For definitions of sigmoidal and m-admissible functions, see [24] and [11], respectively. Thus a unified theoretical framework still lacks.

In this paper, we investigate the approximation ability of deep neural networks with a quite general class of activation functions. We derive the required numbers of layers and nodes of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Our specified class of activation functions and the corresponding approximation ability of deep neural networks include most of previous results [9,10,11,23] as special cases.

Our general theoretical results of the approximation ability of deep neural networks enables us to study statistical properties of deep neural networks. Schmidt-Hieber [10] and Kim et al. [28] proved the minimax optimality of a deep neural network estimator with the ReLU activation function in regression and classification problems, respectively. In this paper, we derive similar results for general activation functions.

This paper is structured as follows. In Section 2, we introduce some notions about deep neural networks. In Section 3, we introduce two large classes of activation functions. In Section 4, we present our main result on the approximation ability of a deep neural network with the general activation function considered in Section 3. In Section 5, we apply the result in Section 4 to the supervised learning problems of regression and classification. Conclusions are given in Section 6. The proofs of all results are given in Appendix.

Notation

We denote by $𝟙\left(·\right)$ the indicator function. Let $\mathbb{R}$ be the set of real numbers and $\mathbb{N}$ be the set of natural numbers. For a real valued vector $\mathbf{x}\equiv (x_1,\dots ,x_d)$, we let ${|\mathbf{x}|}_0:={\sum }_{j=1}^d𝟙(x_j\ne 0)$, ${|\mathbf{x}|}_p:={\left({\sum }_{j=1}^d{|x_j|}^p\right)}^{1/p}$ for $p\in [1,\infty )$ and ${|\mathbf{x}|}_{\infty }:={max}_{1\le j\le d}|x_j|$. For simplicity, we let $|\mathbf{x}|:={|\mathbf{x}|}_1.$ For a real valued function $f(x):\mathbb{R}\to \mathbb{R}$, we let $f^{\prime }(a),f^{″}(a)$ and $f^{‴}(a)$ are the first, second and third order derivatives of f at a, respectively. We let $f^{\prime }(a+):={lim}_{ϵ\downarrow 0}(f(a+ϵ)-f(a))/ϵ$ and $f^{\prime }(a-):={lim}_{ϵ\downarrow 0}(f(a-ϵ)-f(a))/ϵ$. For $x\in \mathbb{R}$, we write ${(x)}_{+}:=max\{x,0\}$.

2. Deep Neural Networks

In this section we provide a mathematical representation of deep neural networks. A neural network with $L\in \mathbb{N}$ layers, $n_l\in \mathbb{N}$ many nodes at the l-th hidden layer for $l=1,\dots ,L$, input of dimension $n_0$, output of dimension $n_{L+1}$ and nonlinear activation function $\sigma :\mathbb{R}\to \mathbb{R}$ is expressed as

$$N_{\sigma }(\mathbf{x}|\mathit{\theta }):={\mathsf{A}}_{L+1}\circ {\sigma }_L\circ {\mathsf{A}}_L\circ \cdots \circ {\sigma }_1\circ {\mathsf{A}}_1(\mathbf{x}),$$

(1)

where ${\mathsf{A}}_l:{\mathbb{R}}^{n_{l-1}}\to {\mathbb{R}}^{n_l}$ is an affine linear map defined by ${\mathsf{A}}_l(\mathbf{x})={\mathbf{W}}_l\mathbf{x}+{\mathbf{b}}_l$ for given $n_l\times n_{l-1}$ dimensional weight matrix ${\mathbf{W}}_l$ and $n_l$ dimensional bias vector ${\mathbf{b}}_l$ and ${\sigma }_l:{\mathbb{R}}^{n_l}\to {\mathbb{R}}^{n_l}$ is an element-wise nonlinear activation map defined by ${\sigma }_l(\mathbf{z}):={(\sigma (z_1),\dots ,\sigma (z_{n_l}))}^{\top }$. Here, $\mathit{\theta }$ denotes the set of all weight matrices and bias vectors $\mathit{\theta }:=\left(({\mathbf{W}}_1,{\mathbf{b}}_1),({\mathbf{W}}_2,{\mathbf{b}}_2),\dots ,({\mathbf{W}}_{L+1},{\mathbf{b}}_{L+1})\right),$ which we call $\mathit{\theta }$ the parameter of the neural network, or simply, a network parameter.

We introduce some notations related to the network parameter. For a network parameter $\mathit{\theta }$, we write $L(\mathit{\theta })$ for the number of hidden layers of the corresponding neural network, and write $n_{max}(\mathit{\theta })$ for the maximum of the numbers of hidden nodes at each layer. Following a standard convention, we say that $L(\mathit{\theta })$ is the depth of the deep neural network and $n_{max}(\mathit{\theta })$ is the width of the deep neural network. We let ${|\mathit{\theta }|}_0$ be the number of nonzero elements of $\mathit{\theta }$, i.e.,

$${|\mathit{\theta }|}_0:=\sum _{l=1}^{L+1}\left(|\mathrm{vec}({\mathbf{W}}_l){|}_0+{|{\mathbf{b}}_l|}_0\right),$$

where $\mathrm{vec}({\mathbf{W}}_l)$ transforms the matrix ${\mathbf{W}}_l$ into the corresponding vector by concatenating the column vectors. We call ${|\mathit{\theta }|}_0$ sparsity of the deep neural network. Let ${|\mathit{\theta }|}_{\infty }$ be the largest absolute value of elements of $\mathit{\theta }$, i.e.,

$${|\mathit{\theta }|}_{\infty }:=max\left\{\underset{1\le l\le L+1}{max}|\mathrm{vec}({\mathbf{W}}_l){|}_{\infty },\underset{1\le l\le L+1}{max}{|{\mathbf{b}}_l|}_{\infty }\right\}.$$

We call ${|\mathit{\theta }|}_{\infty }$ magnitude of the deep neural network. We let $\mathsf{in}(\mathit{\theta })$ and $\mathsf{out}(\mathit{\theta })$ be the input and output dimensions of the deep neural network, respectively. We denote by ${\mathsf{\Theta }}_{d,o}(L,N)$ the set of network parameters with depth L, width N, input dimension d and output dimension o, that is,

$${\mathsf{\Theta }}_{d,o}(L,N):=\mathit{\theta }:L(\mathit{\theta })\le L,n_{max}(\mathit{\theta })\le N,\mathsf{in}(\mathit{\theta })=d,\mathsf{out}(\mathit{\theta })=o.$$

We further define a subset of ${\mathsf{\Theta }}_{d,o}(L,N)$ with restrictions on sparsity and magnitude as

$${\mathsf{\Theta }}_{d,o}(L,N,S,B):=\{\mathit{\theta }\in {\mathsf{\Theta }}_{d,o}{(L,N):|\mathit{\theta }|}_0\le S,{|\mathit{\theta }|}_{\infty }\le B\}.$$

3. Classes of Activation Functions

In this section, we consider two classes of activation functions. These two classes include most of commonly used activation functions. Definitions and examples of each class of activation functions are provided in the consecutive two subsections.

3.1. Piecewise Linear Activation Functions

We first consider piecewise linear activation functions.

Definition 1.

A function $\sigma :\mathbb{R}\to \mathbb{R}$ is continuous piecewise linear if it is continuous and there exist a finite number of break points $a_1\le a_2\le \cdots \le a_K\in \mathbb{R}$ with $K\in \mathbb{N}$ such that ${\sigma }^{\prime }(a_k-)\ne {\sigma }^{\prime }(a_k+)$ for every $k=1,\dots ,K$ and $\sigma (x)$ is linear on $(-\infty ,a_1],[a_1,a_2],\dots ,[a_{K-1},a_K],[a_K,\infty )$.

Throughout this paper, we write “picewise linear” instead of “continuous picewise linear” for notational simplicity unless there is a confusion. The representative examples of piecewise linear activation functions are as follows:

• ReLU: $\sigma (x)=max\{x,0\}$.
• Leaky ReLU: $\sigma (x)=max\{x,ax\}$ for $a\in (0,1)$.

The ReLU activation function is the most popular choice in practical applications due to better gradient propagation and efficient computation [22]. In this reason, most of the recent results on the function approximation by deep neural networks are based on the ReLU activation function [9,10,13,14,15]. In Section 4, as Yarotsky [9] did, we extend these results to any continuous piecewise linear activation function by showing that the ReLU activation function can be exactly represented by a linear combination of piecewise linear activation functions. A formal proof for this argument is presented in Appendix A.1.

3.2. Locally Quadratic Activation Functions

One of the basic building blocks in approximation by deep neural networks is the square function, which should be approximated precisely. Piecewise linear activation functions have zero curvature (i.e., constant first-order derivative) inside each interval divided by its break points, which makes it relatively difficult to approximate the square function efficiently. But if there is an interval on which the activation function has nonzero curvature, the square function can be approximated more efficiently, which is a main motivation of considering a new class of activation functions that we call locally quadratic.

Definition 2.

A function $\sigma :\mathbb{R}\to \mathbb{R}$ is locally quadratic if there exits an open interval $(a,b)\subset \mathbb{R}$ on which σ is three times continuously differentiable with bounded derivatives and there exists $t\in (a,b)$ such that ${\sigma }^{\prime }(t)\ne 0$ and ${\sigma }^{″}(t)\ne 0$.

We now give examples of locally quadratic activation functions. First of all, any nonlinear smooth activation function with nonzero second derivative, is locally quadratic. Examples are:

• Sigmoid: ${\displaystyle \sigma (x)=\frac{1}{1+{\mathrm{e}}^{-x}}.}$
• Tangent hyperbolic: ${\displaystyle \sigma (x)=\frac{{\mathrm{e}}^x-{\mathrm{e}}^{-x}}{{\mathrm{e}}^x+{\mathrm{e}}^{-x}}.}$
• Inverse square root unit (ISRU) [18]: ${\displaystyle \sigma (x)=\frac{x}{\sqrt{1+a{x}^2}}}$ for $a>0$.
• Soft clipping [19]: ${\displaystyle \sigma (x)=\frac{1}{a}log\left(\frac{1+{\mathrm{e}}^{ax}}{1+{\mathrm{e}}^{a(x-1)}}\right)}$ for $a>0$.
• SoftPlus [22]: $\sigma (x)=log(1+{\mathrm{e}}^x)$.
• Swish [21]: ${\displaystyle \sigma (x)=\frac{x}{1+{\mathrm{e}}^{-x}}}$.

In addition, piecewise smooth function having nonzero second derivative on at least one piece, is also locally quadratic. Examples are:

• Rectified power unit (RePU) [12]: ${\displaystyle \sigma (x)=max\{x^k,0\}}$ for $k\in \mathbb{N}\backslash \{1\}$.
• Exponential linear unit (ELU) [17]: ${\displaystyle \sigma (x)=a({\mathrm{e}}^x-1)𝟙\left(x\le 0\right)+x𝟙\left(x>0\right)}$ for $a>0$.
• Inverse square root linear unit (ISRLU) [18]: ${\displaystyle \sigma (x)=\frac{x}{\sqrt{1+a{x}^2}}𝟙\left(x\le 0\right)+x𝟙\left(x>0\right)}$ for $a>0$.
• Softsign [16]: ${\displaystyle \sigma (x)=\frac{x}{1+|x|}.}$
• Square nonlinearity [20]:
  ${\displaystyle \sigma (x)=𝟙\left(x>2\right)+(x-x^2/4)𝟙\left(0\le x\le 2\right)+(x+x^2/4)𝟙\left(-2\le x<0\right)-𝟙\left(x<-2\right)}$.

4. Approximation of Smooth Functions by Deep Neural Networks

In this section we introduce the function class we consider and show the approximation ability of the deep neural networks with a activation function considered in Section 3.

4.1. Hölder Smooth Functions

We recall the definition of Hölder smooth functions. For a d-dimensional multiple index $\mathbf{m}\equiv (m_1,\dots ,m_d)\in {\mathbb{N}}_0^d$ where ${\mathbb{N}}_0:=\mathbb{N}\cup \{0\}$, we let ${\mathbf{x}}^{\mathbf{m}}:=x_1^{m_1}\cdots x_d^{m_d}$ for $\mathbf{x}\in {\mathbb{R}}^d$. For a function $f:\mathcal{X}\to \mathbb{R}$, where $\mathcal{X}$ denotes the domain of the function, we let ${\parallel f\parallel }_{\infty }:={sup}_{\mathbf{x}\in \mathcal{X}}|f(\mathbf{x})|$. We use notation

$${\partial }^{\mathbf{m}}f:=\frac{{\partial }^{|\mathbf{m}|}f}{\partial {\mathbf{x}}^{\mathbf{m}}}=\frac{{\partial }^{|\mathbf{m}|}f}{\partial x_1^{m_1}\cdots \partial x_d^{m_d}},$$

for $\mathbf{m}\in {\mathbb{N}}_0^d$ to denote a derivative of f of order $\mathbf{m}$. We denote by ${\mathcal{C}}^m(\mathcal{X})$, the space of m times differentiable functions on $\mathcal{X}$ whose partial derivatives of order $\mathbf{m}$ with $|\mathbf{m}|\le m$ are continuous. We define the Hölder coefficient of order $s\in (0,1]$ as

$${[f]}_s:=\underset{{\mathbf{x}}_1,{\mathbf{x}}_2\in \mathcal{X},{\mathbf{x}}_1\ne {\mathbf{x}}_2}{sup}\frac{|f({\mathbf{x}}_1)-f({\mathbf{x}}_2)|}{|{\mathbf{x}}_1-{\mathbf{x}}_2{|}^s}.$$

For a positive real value $\alpha$, the Hölder space of order $\alpha$ is defined as

$${\mathcal{H}}^{\alpha }(\mathcal{X}):=\left\{f\in {\mathcal{C}}^{⌊\alpha ⌋}(\mathcal{X}):{\parallel f\parallel }_{{\mathcal{H}}^{\alpha }(\mathcal{X})}<\infty \right\},$$

where ${\parallel f\parallel }_{{\mathcal{H}}^{\alpha }(\mathcal{X})}$ denotes the Hölder norm defined by

$${\parallel f\parallel }_{{\mathcal{H}}^{\alpha }(\mathcal{X})}:=\sum _{\mathbf{m}\in {\mathbb{N}}_0^d:|\mathbf{m}|\le ⌊\alpha ⌋}{\parallel {\partial }^{\mathbf{m}}f\parallel }_{\infty }+\sum _{\mathbf{m}\in {\mathbb{N}}_0^d:|\mathbf{m}|=⌊\alpha ⌋}{[{\partial }^{\mathbf{m}}f]}_{\alpha -⌊\alpha ⌋}.$$

We denote by ${\mathcal{H}}^{\alpha ,R}(\mathcal{X})$ the closed ball in the Hölder space of radius R with respect to the Hölder norm, i.e.,

$${\mathcal{H}}^{\alpha ,R}(\mathcal{X}):=\left\{f\in {\mathcal{H}}^{\alpha }(\mathcal{X}):{\parallel f\parallel }_{{\mathcal{H}}^{\alpha }(\mathcal{X})}\le R\right\}.$$

4.2. Approximation of Hölder Smooth Functions

We present our main theorem in this section.

Theorem 1.

Let $d\in \mathbb{N}$, $\alpha >0$ and $R>0$. Let the activation function σ be either continuous piecewise linear or locally quadratic. Let $f\in {\mathcal{H}}^{\alpha ,R}({[0,1]}^d)$. Then there exist positive constants $L_0$, $N_0$, $S_0$ and $B_0$ depending only on d, α, R and σ such that, for any $ϵ>0$, there is a neural network

$${\mathit{\theta }}_{ϵ}\in {\mathsf{\Theta }}_{d,1}(L_{0}log(1/ϵ),N_0{ϵ}^{-d/\alpha },S_0{ϵ}^{-d/\alpha }log(1/ϵ),B_0{ϵ}^{-4(d/\alpha +1)})$$

(2)

satisfying

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}|f(\mathbf{x})-N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{ϵ})|\le ϵ.$$

(3)

The result of Theorem 1 is equivalent to the results on the approximation by ReLU neural networks [9,10] in a sense that the upper bounds of the depth, width and sparsity are the same orders of those for ReLU, namely, depth $=O(log(1/ϵ))$, width $=O({ϵ}^{-d/\alpha })$ and sparsity $=O({ϵ}^{-d/\alpha }log(1/ϵ))$. We remark that each upper bound is equivalent to the corresponding lower bound established by [9] up to logarithmic factor.

For piecewise linear activation functions, Yarotsky [9] derived similar results to ours. For locally quadratic activation functions, only special classes of activation functions were considered in the previous work. Li et al. [12] considered the RePU activation function and Bauer and Kohler [11] considered sufficiently smooth and bounded activation functions which include the sigmoid, tangent hyperbolic, ISRU and soft clipping activation functions. However, soft plus, swish, ELU, ISRLU, softsign and square nonlinearity activation functions are new ones only considered in our results.

Even if the orders of the depth, width and sparsity are the same for both both piecewise linear and locally quadratic activation functions, the ways of approximating a smooth function by use of these two activation function classes are quite different. To describe this point, let us provide an outline of the proof. We first consider equally spaced grid points with length $1/M$ inside the d-dimensional unit hypercube ${[0,1]}^d$. Let ${\mathbb{G}}_{d,M}$ be the set of such grid points, namely,

$${\mathbb{G}}_{d,M}:=\{\frac{1}{M}(m_1,\dots ,m_d):m_j\in \{0,1,\dots ,M\},j=1,\dots ,d\}.$$

For a given Hölder smooth function f of order $\alpha$, we first find a “local” function for each grid that approximates the target function near the grid point but vanishes at apart from the grid point. To be more specific, we construct the local functions $g_{\mathbf{z}}$, $\mathbf{z}\in {\mathbb{G}}_{d,M}$ which satisfies:

$$\underset{x\in {[0,1]}^d}{sup}\left|f(\mathbf{x})-\sum _{\mathbf{z}\in {\mathbb{G}}_{d,M}}{g}_{\mathbf{z},M}(\mathbf{x})\right|\le C|{\mathbb{G}}_{d,M}{|}^{-\alpha /d},$$

(4)

for some universal constant $C>0$. The inequality (4) implies that the more grid points we used, the more accurate approximation we get. Moreover, the quality of approximation is improved when the target function is more smooth (i.e., large $\alpha$) and low dimensional (i.e., small d). In fact, $g_{\mathbf{z},M}(\mathbf{x})$ is given by a product of the Taylor polynomial $P_{\mathbf{z},M}(\mathbf{x}):={\sum }_{\mathbf{m}\in {\mathbb{N}}_0^d:|\mathbf{m}|\le \alpha }{\partial }^{\mathbf{m}}f(\mathbf{z})\frac{{(\mathbf{x}-\mathbf{z})}^{\mathbf{m}}}{\mathbf{m}!}$ at $\mathbf{z}$ and the local basis function ${\varphi }_{\mathbf{z},M}(\mathbf{x}):={\prod }_{j=1}^d(1/M-|x_j-z_j{|)}_{+}$, where $\mathbf{m}!:={\prod }_{j=1}^d{m}_j!$. By simple algebra, we have

$$\begin{array}{cc}\hfill P_M(\mathbf{x}):=\sum _{\mathbf{z}\in {\mathbb{G}}_{d,M}}{g}_{\mathbf{z},M}(\mathbf{x})& :=\sum _{\mathbf{z}\in {\mathbb{G}}_{d,M}}{P}_{\mathbf{z},M}(\mathbf{x}){\varphi }_{\mathbf{z},M}(\mathbf{x})\hfill \\ & =\sum _{\mathbf{z}\in {\mathbb{G}}_{d,M}}\sum _{\mathbf{m}:|\mathbf{m}|\le \alpha }{\beta }_{\mathbf{z},\mathbf{m}}{\mathbf{x}}^{\mathbf{m}}{\varphi }_{\mathbf{z},M}(\mathbf{x}),\hfill \end{array}$$

where ${\beta }_{\mathbf{z},\mathbf{m}}:={\sum }_{\tilde{\mathbf{m}}:\tilde{\mathbf{m}}\ge \mathbf{m},|\tilde{\mathbf{m}}|\le \alpha }{\partial }^{\tilde{\mathbf{m}}}f(\mathbf{z})\frac{{(-\mathbf{z})}^{\tilde{\mathbf{m}}-\mathbf{m}}}{\mathbf{m}!(\tilde{\mathbf{m}}-\mathbf{m})!}$.

The second stage is to approximate each monomial ${\mathbf{x}}^{\mathbf{m}}$ and each local basis function ${\varphi }_{\mathbf{z},M}(\mathbf{x})$ by deep neural networks. Each monomial can be approximated more efficiently by a deep neural network with a locally quadratic activation function than a piecewise linear activation function since each monomial has nonzero curvature. On the other hand, the local basis function can be approximated more efficiently by a deep neural network with a piecewise linear activation than a locally quadratic activation function since the local basis function is piecewise linear itself. That is, there is a trade-off in using either a piecewise linear or a locally quadratic activation function.

We close this section by giving a comparison of our result to the approximation error analysis of [11]. Bauer and Kohler [11] studies approximation of the Hölder smooth function of order $\alpha$ by a two layer neural network with m-admissible activation functions with $m\ge \alpha$, where a function $\sigma$ is called m-admissible if (1) $\sigma$ is at least $m+1$ times continuously differentiable with bounded derivatives; (2) a point $t\in \mathbb{R}$ exists, where all derivatives up to the order m of $\sigma$ are different from zero; and (3) $|\sigma (x)-1|\le 1/x$ for $x>0$ and $|\sigma (x)|\le 1/|x|$ for $x<0$. Our notion of locally quadratic activation functions is a generalized version of the m-admissibility.

In the proof of [11], the condition $m\ge \alpha$ is necessary because they approximate any monomial of order $\mathbf{m}$ with $|\mathbf{m}|\le \alpha$ with a two layer neural network, which is impossible when $m<\alpha$. We drop the condition $m\ge \alpha$ by showing that any monomial of order $\mathbf{m}$ with $|\mathbf{m}|\le \alpha$ can be approximated by deep neural network with a finite number of layers, which depends on $\alpha$.

5. Application to Statistical Learning Theory

In this section, we apply our results about the approximation error of neural networks to the supervised learning problems of regression and classification. Let $\mathcal{X}$ be the input space and $\mathcal{Y}$ the output space. Let $\mathcal{F}$ be a given class of measurable functions from $\mathcal{X}$ to $\mathcal{Y}$. Let ${\mathrm{P}}_0$ be the true but unknown data generating distribution on $\mathcal{X}\times \mathcal{Y}$. The aim of supervised learning is to find a predictive function that minimizes the population risk $\mathcal{R}(f):={\mathrm{E}}_{(\mathbf{X},Y)\sim {\mathrm{P}}_0}\ell (Y,f(\mathbf{X}))$ with respect to a given loss function ℓ. Since ${\mathrm{P}}_0$ is unknown, we cannot directly minimize the population risk, and thus any estimator $\widehat{f}$ inevitably has the excess risk which is defined as $\mathcal{R}(\widehat{f})-{inf}_{f\in \mathcal{F}}\mathcal{R}(f).$ For a given sample of size n, let ${\mathcal{F}}_n$ be a given subset of $\mathcal{F}$ called a sieve and let $({\mathbf{x}}_1,y_1),\dots ,({\mathbf{x}}_n,y_n)$ be observed (training) data of input–output pairs assumed to be independent realizations of $(\mathbf{X},Y)$ following ${\mathrm{P}}_0.$ Let ${\widehat{f}}_n$ be an estimated function among ${\mathcal{F}}_n$ based on the training data $({\mathbf{x}}_1,y_1),\dots ,({\mathbf{x}}_n,y_n).$ The excess risk of ${\widehat{f}}_n$ is decomposed to approximation and estimation errors as

$$\mathcal{R}({\widehat{f}}_n)-\underset{f\in \mathcal{F}}{inf}\mathcal{R}(f)=\underset{\mathrm{Estimation}\mathrm{error}}{\underbrace{\left[\mathcal{R}({\widehat{f}}_n)-\underset{f\in {\mathcal{F}}_n}{inf}\mathcal{R}(f)\right]}}+\underset{\mathrm{Approximation}\mathrm{error}}{\underbrace{\left[\underset{f\in {\mathcal{F}}_n}{inf}\mathcal{R}(f)-\underset{f\in \mathcal{F}}{inf}\mathcal{R}(f)\right]}}.$$

(5)

There is a trade-off between approximation and estimation errors. If the function class ${\mathcal{F}}_n$ is sufficiently large to approximate the optimal estimator $f^{*}:={argmin}_{f\in \mathcal{F}}\mathcal{R}(f)$ well, then the estimation error becomes large due to high variance. In contrast, if ${\mathcal{F}}_n$ is small, it leads to low estimation error but it suffers from large approximation error.

One of the advantages of deep neural networks is that we can construct a sieve which has good approximation ability as well as low complexity. Schmidt-Hieber [10] and Kim et al. [28] proved that a neural network estimator can achieve the optimal balance between the approximation and estimation errors to obtain the minimax optimal convergence rates in regression and classification problems, respectively. But they only considered the ReLU activation function. Based on the results of Theorem 1, we can easily extend their results to general activation functions.

The main tool to derive the minimax optimal convergence rate is that the complexity of a class of functions generated by a deep neural network is not affected much by a choice of an activation function, provided that the activation function is Lipschitz continuous. The function $\sigma :\mathbb{R}\to \mathbb{R}$ is Lipschitz continuous if there is a constant $C_{\sigma }>0$ such that

$$|\sigma (x_1)-\sigma (x_2)|\le C_{\sigma }|x_1-x_2|,$$

(6)

for any $x_1,x_2\in \mathbb{R}$. Here, $C_{\sigma }$ is called the Lipschitz constant. We use the covering number with respect to the $L_{\infty }$ norm ${\parallel ·\parallel }_{\infty }$ as a measure of complexity of function classes. We recall the definition of the covering number. Let $\mathcal{F}$ be a given class of real-valued functions defined on $\mathcal{X}$. Let $\delta >0$. A collection $\{f_j\in \mathcal{F}:j=1,\dots ,J\}$ is called a $\delta$-covering set of $\mathcal{F}$ with respect to the $L_{\infty }$ norm if for all $f\in \mathcal{F}$, there exists $f_j$ in the collection such that $\parallel f-f_j{\parallel }_{\infty }\le \delta$. The cardinality of the minimal $\delta$-covering set is called the $\delta$-covering number of $\mathcal{F}$ with respect to the $L_{\infty }$ norm which is denoted by $\mathcal{N}(\delta ,\mathcal{F},\parallel ·{\parallel }_{\infty }).$ That is,

$${\mathcal{N}(\delta ,\mathcal{F},\parallel ·\parallel }_{\infty }):=inf\left\{J\in \mathbb{N}:\exists f_1,\dots ,f_J\mathrm{such}\mathrm{that}\mathcal{F}\subset \bigcup _{j=1}^J{B}_{\infty }(f_j,\delta )\right\},$$

where $B_{\infty }(f_j,\delta ):=\{f\in \mathcal{F}:\parallel f-f_j{\parallel }_{\infty }\le \delta \}$. The following proposition provides the covering number of a class of functions generated by neural networks.

Proposition 1.

Assume that the activation function σ is Lipschitz continuous with the Lipschitz constant $C_{\sigma }$. Consider a class of functions generated by a deep neural network

$${\mathcal{F}}_{d,1}(L,N,S,B):=N_{\sigma }(·|\mathit{\theta }):\mathit{\theta }\in {\mathsf{\Theta }}_{d,1}(L,N,S,B).$$

For any $\delta >0$,

$$log\mathcal{N}\left(\delta ,{\mathcal{F}}_{d,1}(L,N,S,B),{\parallel ·\parallel }_{\infty }\right)\le 2L(S+1)log\left({\delta }^{-1}{C}_{\sigma }L(N+1)(B\vee 1)\right),$$

(7)

where$B\vee 1:=max\{B,1\}$.

The result in Proposition 1 is very similar to the existing results in literature, e.g., Theorem 14.5 of [29], Lemma 5 of [10] and Lemma 3 of [13]. We employ similar techniques used in [10,13,29] to obtain the version presented here. We give the proof of this proposition in Appendix B.

All of the activation functions considered in Section 3 except RePU satisfy the Lipschitz condition (6) and hence Proposition 1 can be applied. An interesting implication of Proposition 1 is that the complexity of the function class generated by deep neural networks is not affected by the choice of an activation function. Hence, the remaining step to derive the convergence rate of a neural network estimator is that approximation accuracies by various activation functions are the same as that of the ReLU neural network.

5.1. Application to Regression

First we consider the regression problem. For simplicity, we let $\mathcal{X}={[0,1]}^d.$ Suppose that the generated model is $Y|\mathbf{X}=\mathbf{x}\sim \mathsf{N}(f_0(\mathbf{x}),1)$ for some $f_0:{[0,1]}^d\to \mathbb{R}.$ The performance of an estimator is measured by the $L_2$ risk ${\mathcal{R}}_{2,f_0}(f)$, which is defined by

$${\mathcal{R}}_{2,f_0}(f):={\mathsf{E}}_{f_0,{\mathsf{P}}_{\mathbf{x}}}{(Y-f(\mathbf{X}))}^2:={\mathsf{E}}_{Y|\mathbf{X}\sim \mathsf{N}(f_0(\mathbf{X}),1),\mathbf{X}\sim {\mathsf{P}}_{\mathbf{x}}}{(Y-f(\mathbf{X}))}^2,$$

where ${\mathsf{P}}_{\mathbf{x}}$ is the marginal distribution of $\mathbf{X}.$ The following theorem proves that the optimal convergence rate is obtained by the deep neural network estimator of the regression function $f_0$ for a general activation function.

Theorem 2.

Suppose that the activation function σ is either piecewise linear or locally quadratic satisfying the Lipschitz condition (6). Then there are universal positive constants $L_0$, $N_0$, $S_0$ and $B_0$ such that the deep neural network estimator obtained by

$${\widehat{f}}_n\in \underset{f\in {\mathcal{F}}_{\sigma ,n}}{argmin}\sum _{i=1}^n{(y_i-f({\mathbf{x}}_i))}^2,$$

with

$$\begin{array}{c}\hfill {\mathcal{F}}_{\sigma ,n}:=\left\{N_{\sigma }(·|\mathit{\theta }):{\parallel N_{\sigma }(·|\mathit{\theta })\parallel }_{\infty }\le 2R,\mathit{\theta }\in {\mathsf{\Theta }}_{d,1}\left(L_{0}logn,N_0{n}^{\frac{d}{2\alpha +d}},S_0{n}^{\frac{d}{2\alpha +d}}logn,B_0{n}^{\kappa }\right)\right\}\end{array}$$

for some $\kappa >0$ satisfies

$$\underset{f_0\in {\mathcal{H}}^{\alpha ,R}({[0,1]}^d)}{sup}\mathsf{E}\left[{\mathcal{R}}_{2,f_0}({\widehat{f}}_n)-\underset{f\in \mathcal{F}}{inf}{\mathcal{R}}_{2,f_0}(f)\right]\le C{n}^{-\frac{2\alpha }{2\alpha +d}}{log}^{3}n,$$

for some universal constant $C>0$, where the expectation is taken over the training data.

5.2. Application to Binary Classification

The aim of the binary classification is to find a classifier that predicts the label $y\in \{-1,1\}$ for any input $\mathbf{x}\in {[0,1]}^d$. An usual assumption on the data generating process is that $Y|\mathbf{X}=\mathbf{x}\sim 2\mathsf{Bern}(\eta (\mathbf{x}))-1$ for some $\eta :{[0,1]}^d\to [0,1]$, where $\mathsf{Bern}(p)$ denotes the Bernoulli distribution with parameter p. Note that $\eta (\mathbf{x})$ is the conditional probability function ${\mathsf{P}}_0(Y=1|\mathbf{X}=\mathbf{x}).$ A common approach is, instead of finding a classifier directly, to construct a real valued function f, a so-called classification function, and predict the label y based on the sign of $f(\mathbf{x})$. The performance of a classification function is measured by the misclassification error ${\mathcal{R}}_{01,\eta }(f)$, which is defined by

$${\mathcal{R}}_{01,\eta }(f):={\mathsf{E}}_{\eta ,{\mathsf{P}}_{\mathbf{x}}}𝟙(Yf(\mathbf{X})<0):={\mathsf{E}}_{Y|\mathbf{X}\sim 2\mathsf{Bern}(\eta (\mathbf{X}))-1,\mathbf{X}\sim {\mathsf{P}}_{\mathbf{x}}}𝟙(Yf(\mathbf{X})<0).$$

It is well known that the convergence rate of the excess risk for classification is faster than that of regression when the conditional probability function $\eta (\mathbf{x})$ satisfies the following condition: there is a constant $q\in [0,\infty ]$ such that for any sufficiently small $u>0$, we have

$${\mathsf{P}}_{\mathbf{x}}|\eta (\mathbf{X})-1/2|<u\le u^q.$$

(8)

This condition is called the Tsybakov noise condition and q is called the noise exponent [30,31]. When q is larger, the classification task is easier since the probability of generating vague samples become smaller. The following theorem proves that the optimal convergence rate can be obtained by the deep neural network estimator with an activation function considered in Section 3. As is done by [28], we consider the hinge loss ${\ell }_{\mathsf{hinge}}(z):=max\{1-z,0\}.$

Theorem 3.

Assume the Tsybakov noise condition (8) with the noise exponent $q\in [0,\infty ]$. Suppose that the activation function σ, which is either piecewise linear or locally quadratic satisfying the Lipschitz condition (6), is used for all hidden layers except the last one and the ReLU activation function is used for the last hidden layer. Then there are universal positive constants $L_0$, $N_0$, $S_0$ and $B_0$ such that the deep neural network estimator obtained by

$${\widehat{f}}_n\in \underset{f\in {\mathcal{F}}_{\sigma ,n}}{argmin}\sum _{i=1}^n{\ell }_{\mathsf{hinge}}(y_{i}f({\mathbf{x}}_i)),$$

with

$$\begin{array}{c}\hfill {\mathcal{F}}_{\sigma ,n}:=\left\{N_{\sigma }(·|\mathit{\theta }):{\parallel N_{\sigma }(·|\mathit{\theta })\parallel }_{\infty }\le 1,\mathit{\theta }\in {\mathsf{\Theta }}_{d,1}(L_{0}logn,N_0{n}^{\nu }{log}^{-3\nu }n,S_0{n}^{\nu }{log}^{-3\nu +1}n,B_0{n}^{\kappa })\right\},\end{array}$$

for $\nu :=d/\alpha (q+2)+d$ and some $\kappa >0$ satisfies

$$\underset{\eta \in {\mathcal{H}}^{\alpha ,R}({[0,1]}^d)}{sup}\mathsf{E}\left[{\mathcal{R}}_{01,\eta }({\widehat{f}}_n)-\underset{f\in \mathcal{F}}{inf}{\mathcal{R}}_{01,\eta }(f)\right]\le C{\left(\frac{{log}^{3}n}{n}\right)}^{\frac{\alpha (q+1)}{\alpha (q+2)+d}},$$

for some universal constant $C>0$, where the expectation is taken over the training data.

Note that the Bayes classifier $f^{*}:={argmin}_{f\in \mathcal{F}}{\mathcal{R}}_{01,\eta }(f)$ is given by

$$f^{*}(\mathbf{x})=2𝟙\left(2\eta (\mathbf{x})-1\ge 0\right)-1,$$

which is an indicator function. Since a neural network with the ReLU activation function can approximate indicator functions well [14,15,28], we use the ReLU activation function in the last layer in order to approximate the Bayes classifier more precisely and thus to achieve the optimal convergence rate.

6. Conclusions

In this study, we established the upper bounds of the required depth, width and sparsity of deep neural networks to approximate any Hölder continuous function for the general classes of activation functions. These classes of activation functions include most of the popularly used activation functions. The derived upper bounds of the depth, width and sparsity are optimal in a sense that they are equivalent to the lower bounds up to logarithmic factors. We used this generalization of the approximation error analysis to extend the statistical optimality of the deep neural network estimator in regression and classification problems, where the activation function is other than the ReLU.

Our construction of neural networks for approximation reveals that the piecewise linear activation functions are more efficient in approximating local basis functions while locally quadratic activation functions are more efficient in approximating polynomials. Hence if the activation function has both piecewise linear region and locally quadratic region, we could have a better approximation result. We leave the development of such activation functions as a future work.

Appendix A.1. Proof of Theorem 1 for Piecewise Linear Activation Functions

The main idea of the proof is that any deep neural network with the ReLU activation function can be exactly reconstructed by a neural network with a piecewise activation function whose proof is in the next lemma that is a slight modification of Proposition 1 (b) of [9].

Lemma A1.

Let σ be an any continuous peicewise linear activation function, and ρ be the ReLU activation function. Let $\mathit{\theta }\in {\mathsf{\Theta }}_{d,1}(L,N,S,B)$. Then there exists ${\mathit{\theta }}^{*}\in {\mathsf{\Theta }}_{d,1}(L,2N,4S+2LN+1,C_{1}B)$ such that

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}\left|N_{\sigma }(\mathbf{x}|{\mathit{\theta }}^{*})-N_{\rho }(\mathbf{x}|\mathit{\theta })\right|=0,$$

where $C_1>0$ is a constant depending on the activation function σ.

Proof. 

Let a be any break point of σ. Note that $\sigma (a-)\ne \sigma (a+)$. Let $r_0$ be the distance between a and the closest other break point. Then σ is linear on $[a-r_0,a]$ and $[a,a+r_0]$. Then for any $r>0$, the ReLU activation function $\rho (x):={(x)}_{+}$ is expressed as

$$\begin{array}{cc}\hfill \rho (x)& =\frac{\sigma \left(a+\frac{r_0}{2r}x\right)-\sigma \left(a-\frac{r_0}{2}+\frac{r_0}{2r}x\right)-\sigma (a)+\sigma \left(a-\frac{r_0}{2}\right)}{{\sigma }^{\prime }(a+)-{\sigma }^{\prime }(a-)\frac{r_0}{2r}}\hfill \\ & =:u_1\sigma \left(a+\frac{r_0}{2r}x\right)+u_2\sigma \left(a-\frac{r_0}{2}+\frac{r_0}{2r}x\right)+v\hfill \end{array}$$

(A1)

for any $x\in [-r,r]$, where we define $u_1:=1/(({\sigma }^{\prime }(a+)-{\sigma }^{\prime }(a-))\frac{r_0}{2r})$, $u_2:=-1/(({\sigma }^{\prime }(a+)-{\sigma }^{\prime }(a-))\frac{r_0}{2r})$ and $v:=(-\sigma (a)+\sigma (a-r_0/2))/(({\sigma }^{\prime }(a+)-{\sigma }^{\prime }(a-))\frac{r_0}{2r})$.

Let $\mathit{\theta }\equiv (({\mathbf{W}}_1,{\mathbf{b}}_1),\dots ,({\mathbf{W}}_{L+1},{\mathbf{b}}_{L+1}))\in {\mathsf{\Theta }}_{d,1}(L,N,S,B)$ be given. Since both input $\mathbf{x}\in {[0,1]}^d$ and the network parameter $\mathit{\theta }$ are bounded, we can take a sufficiently large r so that Equation (A1) holds for any hidden nodes of the network $\mathit{\theta }$. We consider the deep neural network ${\mathit{\theta }}^{*}\equiv (({\mathbf{W}}_1^{*},{\mathbf{b}}_1^{*}),\dots ,({\mathbf{W}}_{L+1}^{*},{\mathbf{b}}_{L+1}^{*}))\in {\mathsf{\Theta }}_{d,1}(L,2N),$ where we set

$$\begin{array}{cc}\hfill {\mathbf{W}}_l^{*}& :=\frac{r_0}{2r}\left(\begin{array}{cc}{u}_1{\mathbf{W}}_l& u_2{\mathbf{W}}_l\\ u_1{\mathbf{W}}_l& u_2{\mathbf{W}}_l\end{array}\right)\in {\mathbb{R}}^{2n_l\times 2n_{l-1}},\hfill \\ \hfill {\mathbf{b}}_l^{*}& :=\left(\begin{array}{c}a{\mathbf{1}}_{n_l}+\frac{r_0}{2r}(v{\mathbf{W}}_l{\mathbf{1}}_{n_{l-1}}+{\mathbf{b}}_l)\\ \left(a-\frac{r_0}{2}\right){\mathbf{1}}_{n_l}++\frac{r_0}{2r}(v{\mathbf{W}}_l{\mathbf{1}}_{n_{l-1}}+{\mathbf{b}}_l)\end{array}\right)\in {\mathbb{R}}^{2n_l},\hfill \end{array}$$

for $l=1,\cdots ,L$ and

$${\mathbf{W}}_{L+1}^{*}:=\left(\begin{array}{cc}{u}_1{\mathbf{W}}_{L+1}& u_2{\mathbf{W}}_{L+1}\end{array}\right),{\mathbf{b}}_{L+1}^{*}:=v.$$

Here, ${\mathbf{1}}_n$ denotes the n-dimensional vector of $1^{\prime }s$. Then by Equation (A1) and some algebra, we have that $N_{\sigma }(\mathbf{x}|{\mathit{\theta }}^{*})=N_{\rho }(\mathbf{x}|\mathit{\theta })$ for any $\mathbf{x}\in {[0,1]}^d$. For the sparsity of ${\mathit{\theta }}^{*}$, we note that

$$|\mathrm{vec}({\mathbf{W}}_l^{*}){|}_0+|{\mathbf{b}}_l^{*}{|}_0\le 4{|\mathrm{vec}({\mathbf{W}}_l)|}_0+2n_l$$

which implies that $|{\mathit{\theta }}^{*}{|}_0\le 4{|\mathit{\theta }|}_0+2L(\mathit{\theta })n_{max}(\mathit{\theta })+1$. □

Thanks to Lemma A1, to prove Theorem 1 for piecewise linear activation functions, it suffices to show the approximation ability of the ReLU networks, which is already done by [10] as in the next lemma.

Lemma A2

(Theorem 5 of [10]). Let ρ be the ReLU activation function. For any $f\in {\mathcal{H}}^{\alpha ,R}({[0,1]}^d)$ and any integers $m\ge 1$ and $M\ge max\left\{{(\alpha +1)}^d,(R+1)e^d\right\}$, there exists a network parameter $\mathit{\theta }\in {\mathsf{\Theta }}_{d,1}(L,N,S,1)$ such that

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}\left|N_{\rho }(\mathbf{x}|\mathit{\theta })-f(\mathbf{x})\right|\le (2R+1)(1+d^2+{\alpha }^2)6^{d}M{2}^{-m}+R{3}^{\alpha }{M}^{-\alpha /d},$$

(A2)

where $L=8+(m+5)(1+\lceil {log}_2(d\vee \alpha )\rceil )$, $N=6(d+⌈\alpha ⌉)M$, and $S=141{(d+\alpha +1)}^{3+d}M(m+6)$.

Theorem 1 for piecewise linear activation functions is a direct consequence of Lemmas A1 and A2, which is summarized as follows.

Proof of Theorem 1 for piecewise linear activation functions.

Let ρ be the ReLU activation function. By letting $M=3^d{(2R)}^{d/\alpha }{ϵ}^{-d/\alpha }$ and $m={log}_2\left(2(2R+1)(1+d^2+{\alpha }^2){18}^d{(2R)}^{d/\alpha }{ϵ}^{-d/\alpha -1}\right),$ Lemma A2 implies that there exists a network parameter ${\mathit{\theta }}^{\prime }$ such that ${sup}_{\mathbf{x}\in {[0,1]}^d}|N_{\rho }(\mathbf{x}|{\mathit{\theta }}^{\prime })-f(\mathbf{x})|\le ϵ$ with $L({\mathit{\theta }}^{\prime })\le L_0^{\prime }log\left(1/ϵ\right)$, $n_{max}({\mathit{\theta }}^{\prime })\le N_0^{\prime }{ϵ}^{-d/\alpha }$ and $|{\mathit{\theta }}^{\prime }{|}_0\le S_0^{\prime }{ϵ}^{-d/\alpha }log\left(1/ϵ\right)$ for some positive constants $L_0^{\prime }$, $N_0^{\prime }$, and $S_0^{\prime }$ depending only on α, d and R. Hence by Lemma A1, there is a network parameter $\mathit{\theta }$ producing the same output of the ReLU neural network $N_{\rho }(·|\mathit{\theta })$ with $L(\mathit{\theta })=L({\mathit{\theta }}^{\prime })$, $n_{max}(\mathit{\theta })=2n_{max}({\mathit{\theta }}^{\prime })$, ${|\mathit{\theta }|}_0\le 4{|{\mathit{\theta }}^{\prime }|}_0+2L({\mathit{\theta }}^{\prime })n_{max}({\mathit{\theta }}^{\prime })+1\le S_0{ϵ}^{-d/\alpha }log\left(1/ϵ\right)$ and ${|\mathit{\theta }|}_{\infty }\le B_0{|{\mathit{\theta }}^{\prime }|}_{\infty }$ for some $S_0>0$ depending only on α, d, R and σ, and some $B_0>0$ depending only on σ, which completes the proof. □

Appendix A.2. Proof of Theorem 1 for Locally Quadratic Activation Functions

Lemma A3.

Assume that an activation function σ is locally quadratic. There is a constant $K_0$ depending only on the activation function such that for any $K>K_0$ the following results hold.

(a) 

There is a neural network ${\mathit{\theta }}_2\in {\mathsf{\Theta }}_{1,1}(1,3)$ with $|{\mathit{\theta }}_2{|}_{\infty }\le K^2$ such that

$$\underset{x\in [-1,1]}{sup}\left|N_{\sigma }(x|{\mathit{\theta }}_2)-x^2\right|\le \frac{C_1}{K},$$

where $C_1>0$ is a constant depending only on σ.

(b) 

Let $A>0$. There is a neural network parameter ${\mathit{\theta }}_{\times ,A}\in {\mathsf{\Theta }}_{2,1}(1,9)$ with $|{\mathit{\theta }}_{\times ,A}{|}_{\infty }\le max\{K^2,2A^2\}$ such that

$$\underset{\mathbf{x}\in {[-A,A]}^2}{sup}|N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\times ,A})-x_1{x}_2|\le \frac{6A^2{C}_1}{K}.$$

(c) 

Let α be a positive integer. For any multi-index $\mathbf{m}\in {\mathbb{N}}_0^d$ with $|\mathbf{m}|\le \alpha$, there is a network parameter ${\mathit{\theta }}_{\mathbf{m}}\in {\mathsf{\Theta }}_{d,1}(⌈{log}_2\alpha ⌉,9\alpha )$ with $|{\mathit{\theta }}_{\mathbf{m}}{|}_{\infty }\le max\{K^2,C_2\}$ such that

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}|N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\mathbf{m}})-{\mathbf{x}}^{\mathbf{m}}|\le \frac{C_3}{K},$$

for some positive constants $C_2$ and $C_3$ depending only on σ and α.

(d) 

There is a network parameter ${\mathit{\theta }}_{1/2}\in {\mathsf{\Theta }}_{1,1}(⌈logK⌉,15)$ with $|{\mathit{\theta }}_{1/2}{|}_{\infty }\le max\{K^2,C_4\}$ such that

$$\underset{x\in [0,2]}{sup}\left|N_{\sigma }(x|{\mathit{\theta }}_{1/2})-\sqrt{x}\right|\le C_5\frac{logK}{K}$$

for some positive constants $C_4$ and $C_5$ depending only on σ.

(e) 

There is a network parameter ${\mathit{\theta }}_{abs}\in {\mathsf{\Theta }}_{1,1}(⌈logK⌉,15)$ with $|{\mathit{\theta }}_{abs}{|}_{\infty }\le max\{K^2,C_6\}$ such that

$$\underset{x\in [-1,1]}{sup}|N_{\sigma }(x|{\mathit{\theta }}_{\mathrm{abs}})-|x||\le \frac{C_7}{\sqrt{K}},$$

for some positive constants $C_6$ and $C_7$ depending only on σ.

Proof. 

Recall that there is an interval $(a,b)$ on which $\sigma (x)$ is three times continuously differentiable with bounded derivatives and there is $t\in (a,b)$ such that ${\sigma }^{\prime }(t)\ne 0$ and ${\sigma }^{″}(t)\ne 0$

Proof of (a). Take K large so that $2/K<min\{|t-b|,|t-a|\}$. Consider a neural network

$$N_{\sigma }(x|{\mathit{\theta }}_2):=\sum _{k=0}^2{(-1)}^{k-1}\frac{K^2}{{\sigma }^{″}(t)}\left(\genfrac{}{}{0pt}{}{2}{k}\right)\sigma \frac{k}{K}x+t.$$

(A3)

Since σ is three times continuously differentiable on $(a,b)$ and $(k-1)x/K+t\in (a,b)$ if $x\in [0,1]$, it can be expanded in the Taylor series with Lagrange remainder around t to have

$$\begin{array}{cc}\hfill N_{\sigma }(x|{\mathit{\theta }}_2)& =\frac{K^2}{{\sigma }^{″}(t)}\sum _{k=0}^2{(-1)}^k\left(\genfrac{}{}{0pt}{}{2}{k}\right)\left\{\sigma (t)+{\sigma }^{\prime }(t)\frac{kx}{K}+\frac{{\sigma }^{″}(t)}{2}\frac{{(kx)}^2}{K^2}+\frac{{\sigma }^{″}({\xi }_k)}{6}\frac{{(kx)}^3}{K^3}\right\}\hfill \\ & =\frac{K^2}{{\sigma }^{″}(t)}\left\{{\sigma }^{″}(t)\frac{x^2}{K^2}+\sum _{k=1}^2{(-1)}^k\left(\genfrac{}{}{0pt}{}{2}{k}\right)\frac{{\sigma }^{‴}({\xi }_k)}{6}\frac{{(kx)}^3}{K^3}\right\}\hfill \\ & =x^2+\frac{x^3}{6K{\sigma }^{″}(t)}\sum _{k=1}^2{(-1)}^k{k}^3\left(\genfrac{}{}{0pt}{}{2}{k}\right){\sigma }^{‴}({\xi }_k),\hfill \end{array}$$

where ${\xi }_k\in [t-k|x|/K,t+k|x|/K]\subset (a,b).$ Since the third order derivative is bounded on $(a,b)$, we get the desired assertion by retaking $K\leftarrow \sqrt{2/{\sigma }^{″}(t)}K$.

Proof of (b). The proof can be done straightforwardly by the polarization type identity:

$$x_1{x}_2=2A^2\left\{{\left(\frac{x_1+x_2}{2A}\right)}^2-{\left(\frac{x_1}{2A}\right)}^2-{\left(\frac{x_1}{2A}\right)}^2\right\}.$$

We construct the network as

$$N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\times ,A}):=2A^2\left\{N_{\sigma }\left(\frac{x_1+x_2}{2A}|{\mathit{\theta }}_2\right)-N_{\sigma }\left(\frac{x_1}{2A}|{\mathit{\theta }}_2\right)-N_{\sigma }\left(\frac{x_2}{2A}|{\mathit{\theta }}_2\right)\right\},$$

(A4)

where ${\mathit{\theta }}_2$ is defined in (A3). Since $(x_1+x_2)/2A,x_1/2A,x_2/2A\in [-1,1]$ for $\mathbf{x}\in {[-A,A]}^2$, the triangle inequality implies that $|N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\times ,A})-x_1{x}_2|\le 6A^2{C}_1/K$.

Proof of (c). Let $q:=⌈{log}_2\alpha ⌉$. We construct ${\mathit{\theta }}_{\mathbf{m}}$ as follows. Fix $\mathbf{x}\equiv (x_1,\dots ,x_d)\in {[0,1]}^d$. We first consider the affine map that transforms $(x_1,\dots ,x_d)$ to $\mathbf{z}\in {[0,1]}^{2^q}$ which is given by

$$\mathbf{z}:=(\underset{m_1\mathrm{times}}{\underbrace{x_1,\dots ,x_1}},\underset{m_2\mathrm{times}}{\underbrace{x_2,\dots ,x_2}},\dots ,\underset{m_d\mathrm{times}}{\underbrace{x_d,\dots ,x_d}},\underset{2^q-|\mathbf{m}|\mathrm{times}}{\underbrace{1,\dots ,1}}).$$

The first hidden layer of ${\mathit{\theta }}_{\mathbf{m}}$ pairs neighboring entries in $\mathbf{z}$ and applies the network ${\mathit{\theta }}_{\times ,A_1}$ defined in (b) with $A_1=1$ to each pair. That is, the first hidden layer of ${\mathit{\theta }}_{\mathbf{m}}$ produces

$$\left\{g_{1,j}:=N_{\sigma }((z_{2j-1},z_{2j})|{\mathit{\theta }}_{\times ,1}):j=1,\dots ,2^{q-1}\right\}.$$

Note that ${sup}_{1\le j\le 2^{q-1}}|g_{1,j}-z_{2j-1}{z}_{2j}|\le 6C_1/K$ and ${sup}_{1\le j\le 2^{q-1}}|g_{1,j}|\le 6C_1/K+1$, where $6C_1/K+1$ can be bounded by some constant $A_2>1$ depending only on $C_1$ and $K_0$. Then the second hidden layer of ${\mathit{\theta }}_{\mathbf{m}}$ pairs neighboring entries of $\left\{g_{1,j}:j=1,\dots ,2^{q-1}\right\}$ and applies ${\mathit{\theta }}_{\times ,A_2}$ to each pair to have

$$\left\{g_{2,j}:=N_{\sigma }((g_{1,2j-1},g_{1,2j})|{\mathit{\theta }}_{\times ,A_2}):j=1,\dots ,2^{q-2}.\right\}$$

Note that ${sup}_{1\le j\le 2^{q-2}}|g_{2,j}-g_{1,2j-1}{g}_{1,2j}|\le 6C_1{A}_2^2/K$ and ${sup}_{1\le j\le 2^{q-2}}|g_{2,j}|\le 6C_1{A}_2^2/K+1\le A_3$ for some $A_3>1$ depending only on $C_1$ and $K_0$. We repeat this procedure to produce $\left\{g_{k,j}:j=1,\dots ,2^{q-k}\right\}$ for $k=3,\dots ,q$ with

$$\underset{1\le j\le 2^{q-k}}{sup}|g_{k,j}-g_{k-1,2j-1}{g}_{k-1,2j}|\le \frac{6C_1{A}_k^2}{K},\underset{1\le j\le 2^{q-k}}{sup}\left|g_{k,j}\right|\le A_{k+1},$$

for some $A_{k+1}>1$, and we set $N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\mathbf{m}})$ equal to $g_{q,1}$.

By applying the triangle inequality repeatedly, we have

$$\begin{array}{cc}\hfill |g_{q,1}-{\mathbf{x}}^{\mathbf{m}}|& \le \left|g_{q,1}-g_{q-1,1}{g}_{q-1,2}\right|+\left|g_{q-1,1}-\prod _{j=1}^{2^{q-1}}{z}_j\right|\left|g_{q-1,2}\right|+\left|g_{q-1,2}-\prod _{j=2^{q-1}+1}^{2^q}{z}_j\right|\left|\prod _{j=1}^{2^{q-1}}{z}_j\right|\hfill \\ & \le \frac{6C_1{A}_q^2}{K}+A_q\left|g_{q-1,1}-\prod _{j=1}^{2^{q-1}}{z}_j\right|+\left|g_{q-1,2}-\prod _{j=2^{q-1}+1}^{2^q}{z}_j\right|\hfill \\ & \le \frac{6C_1{A}_q^2}{K}+(A_q+1)\frac{6C_1{A}_{q-1}^2}{K}+A_q{A}_{q-1}\left|g_{q-2,1}-\prod _{j=1}^{2^{q-2}}{z}_j\right|+A_q\left|g_{q-2,2}-\prod _{j=2^{q-2}+1}^{2\times 2^{q-2}}{z}_j\right|\hfill \\ & +A_{q-1}\left|g_{q-2,3}-\prod _{j=2\times 2^{q-2}+1}^{3\times 2^{q-2}}{z}_j\right|+\left|g_{q-2,4}-\prod _{j=3\times 2^{q-2}+1}^{4\times 2^{q-2}}{z}_j\right|\hfill \\ & \le \cdots \le \sum _{k=0}^{q-1}\left\{A_{q-k}^2\prod _{h=q-k+1}^q(A_h+1)\right\}\frac{6C_1}{K}\le C_1^{\prime }\frac{1}{K},\hfill \end{array}$$

for some $C_1^{\prime }>0$ depending only on $C_1$, $K_0$ and q. Since we set $\mathbf{x}$ arbitrary, the proof is done.

Proof of (d). By (b), it is easy to verify that there is a network ${\mathit{\theta }}_1\in {\mathsf{\Theta }}_{1,1}(1,6)$ with $|{\mathit{\theta }}_1{|}_{\infty }\le max\{K^2,2\}$ such that $|\sigma (x)-x|\le C_1^{\prime }/K$ for any $x\in [-1,1]$ and some constant $C_1^{\prime }>0$. The Taylor series with Lagrange remainder around 1 of $\sqrt{x}$ is given by

$$\sqrt{x}=\sum _{k=0}^J\frac{{(x-1)}^k}{k!}+\frac{1}{(J+1)!}\frac{d^{J+1}\sqrt{x}}{d{x}^{J+1}}{|}_{x=\xi }{(x-1)}^{J+1},$$

where $\xi \in [0,2],$ and thus

$$\underset{x\in [0,2]}{sup}\left|\sqrt{x}-\sum _{k=0}^J\frac{{(x-1)}^k}{k!}\right|\le C_1^{\prime }\frac{1}{(J+1)!}\le e{\frac{e}{J+1}}^{J+1}.$$

for some $C_1^{\prime }>0,$ where the last inequality is because $n!\ge {(n/e)}^{n}e$.

Now, we will construct a neural network ${\mathit{\theta }}_{p,J}$ that approximates the polynomial ${\sum }_{k=0}^J\frac{{(x-1)}^k}{k!}$ as follows. The first hidden layer computes $(N_{\sigma }(x-1|{\mathit{\theta }}_2)/2,N_{\sigma }(x-1|{\mathit{\theta }}_1))$ from the input x. Then

$$|(N_{\sigma }(x-1|{\mathit{\theta }}_2)/2,N_{\sigma }(x-1|{\mathit{\theta }}_1))-({(x-1)}^2/2,(x-1)){|}_{\infty }\le C_2^{\prime }\frac{1}{K},$$

for any $x\in [0,1]$ and some constant $C_2^{\prime }>0$. The next hidden layer computes $(N_{\sigma }((u,v)|{\mathit{\theta }}_{\times ,1+C_2^{\prime }/K})/3,N_{\sigma }(u+v|{\mathit{\theta }}_1))$ from the input $(u,v)$ from the first hidden layer. Using the triangle inequality, we have that the second hidden layer approximates the vector $({(x-1)}^3/3!,{(x-1)}^2/2+(x-1))$ by error $\le 2C_3^{\prime }/K$ for some $C_3^{\prime }>0$. Repeating this procedure, we construct the network ${\mathit{\theta }}_{p,J}\in {\mathsf{\Theta }}_{1,1}(J,15)$ which approximates ${\sum }_{k=0}^J\frac{{(x-1)}^k}{k!}$ by error $\le C_4^{\prime }J/K$ for some $C_4^{\prime }>0$. Taking $J=⌈logK⌉$, we observe that ${(e/J+1)}^{J+1}\le {(e/logK)}^{logK+1}\le eK/{(logK)}^{logK}\le 1/K$ for all sufficiently large K, which implies the desired result.

Proof of (e). Let $\zeta \in (0,1)$. Since for any $x\in \mathbb{R}$,

$$\sqrt{x^2+{\zeta }^2}-|x|\le \frac{{\zeta }^2}{\sqrt{x^2+{\zeta }^2}+|x|}\le \frac{{\zeta }^2}{\zeta }=\zeta ,$$

the function $\sqrt{x^2+{\zeta }^2}$ approximates the absolute value function $|x|$ by error ζ. For ${\mathit{\theta }}_2$ in (a) and ${\mathit{\theta }}_{1/2}$ in (d), we have that

$$\begin{array}{cc}\hfill \left|N_{\sigma }\left(N_{\sigma }(x|{\mathit{\theta }}_2)+{\xi }^2|{\mathit{\theta }}_{1/2}\right)-|x|\right|& \le |N_{\sigma }\left(N_{\sigma }(x|{\mathit{\theta }}_2)+{\zeta }^2|{\mathit{\theta }}_{1/2}\right)-\sqrt{x^2+{\zeta }^2}|+\zeta \hfill \\ & \le |N_{\sigma }\left(N_{\sigma }(x|{\mathit{\theta }}_2)+{\zeta }^2|{\mathit{\theta }}_{1/2}\right)-\sqrt{N_{\sigma }(x|{\mathit{\theta }}_2)+{\xi }^2}|\hfill \\ & +|\sqrt{N_{\sigma }(x|{\mathit{\theta }}_2)+{\zeta }^2}-\sqrt{x^2+{\zeta }^2}|+\zeta \hfill \\ & \le C_1^{\prime }\left(\frac{logK}{K}+\frac{1}{K\zeta }\right)+\zeta \hfill \end{array}$$

for some constant $C_1^{\prime }>0.$ We now set $\zeta =1/\sqrt{K}$ and $N_{\sigma }(x|{\mathit{\theta }}_{abs}):=N_{\sigma }(N_{\sigma }(x|{\mathit{\theta }}_2)+K^{-1}|{\mathit{\theta }}_{1/2})$. Since $(logK)/K=o(1/\sqrt{K})$, the proof is done. □

Proof of Theorem 1 for locally quadratic activation functions.

Recall that

$$P_M(\mathbf{x})=\sum _{\mathbf{z}\in {\mathbb{G}}_{d,M}}\sum _{\mathbf{m}\in {\mathbb{N}}_0^d:|\mathbf{m}|\le \alpha }{\beta }_{\mathbf{z},\mathbf{m}}{\mathbf{x}}^{\mathbf{m}}{\varphi }_{\mathbf{z},M}(\mathbf{x}).$$

Then by Lemma B.1 of [10],

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}|P_M(\mathbf{x})-f(\mathbf{x})|\le R{M}^{-\alpha }.$$

From the equivalent representation of the ReLU function ${(x)}_{+}=(x+|x|)/2$, we can easily check that the neural network $N_{\sigma }(x|{\mathit{\theta }}_{relu}):=N(x|{\mathit{\theta }}_{abs})+N_{\sigma }(x|{\mathit{\theta }}_1)/2$ with ${\mathit{\theta }}_{relu}\in {\mathsf{\Theta }}_{1,1}(⌈logK⌉,21)$ approximates the ReLU function by error $\le C_1^{\prime }/\sqrt{K}$ for some $C_1^{\prime }>0$, where ${\mathit{\theta }}_1\in {\mathsf{\Theta }}_{1,1}(1,6)$ is defined in the proof of (d) of Lemma A3 and ${\mathit{\theta }}_{abs}\in {\mathsf{\Theta }}_{1,1}(⌈logK⌉,15)$ is defined in (e) of Lemma A3. For $z\in (0,1)$ and $M\in \mathbb{N}$, we define

$$N_{\sigma }(x|{\mathit{\theta }}_{\varphi ,z,M}):=N_{\sigma }\left(1/M-N_{\sigma }((x-z)|{\mathit{\theta }}_{\mathrm{abs}})|{\mathit{\theta }}_{\mathrm{relu}}\right).$$

Then it approximates the function ${(1/M-|x-z|)}_{+}$ by error $\le C_2^{\prime }/\sqrt{K}$ for some $C_2^{\prime }>0.$ In turn, for $\mathbf{z}\in {\mathbb{G}}_{d,M}$, by invoking the similar construction used in (c) of Lemma A3 to approximates the product of d components, we can construct the network ${\mathit{\theta }}_{\varphi ,\mathbf{z},M}\in {\mathsf{\Theta }}_{1,1}(⌈logK⌉+⌈{log}_{2}d⌉,21d)$ with $|{\mathit{\theta }}_{\varphi ,\mathbf{z},M}{|}_{\infty }\le C_3^{\prime }{K}^2$ for some $C_3^{\prime }>0$ such that

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}\left|N(\mathbf{x}|{\mathit{\theta }}_{\varphi ,\mathbf{z},M})-\prod _{j=1}^d{\left(\frac{1}{M}-|x_j-z_j|\right)}_{+}\right|\le C_4^{\prime }\frac{1}{\sqrt{K}},$$

for some $C_4^{\prime }>0$. For each $\mathbf{m}\in {\mathbb{N}}_0^d$ with $|\mathbf{m}|\le \alpha$, we have the neural network ${\mathit{\theta }}_{\mathbf{m}}$ in (c) of Lemma A3 that approximates ${\mathbf{x}}^{\mathbf{m}}$. The number of these networks is $\left(\genfrac{}{}{0pt}{}{d+\alpha }{\alpha }\right)$, which is denoted by $A_{\alpha }$. Also there are $|{\mathbb{G}}_{d,M}{|=(M+1)}^d$ networks ${\mathit{\theta }}_{\varphi ,\mathbf{z},M}$ for $\mathbf{z}\in {\mathbb{G}}_{d,M}$. We need approximation of each product ${\mathbf{x}}^{\mathbf{m}}{\varphi }_{\mathbf{z},M}$, which requires additional $A_{\alpha }{(M+1)}^d$ many networks ${\mathit{\theta }}_{\times ,A}\in {\mathsf{\Theta }}_{2,1}(1,9)$, where ${\mathit{\theta }}_{\times ,A}$ is defined as in (A4) for some $A>1$ not depending on M and K. Finally we construct the output layer which computes the weighted sum of $\left\{N_{\sigma }\left((N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\mathbf{m}}),N_{\sigma }(\mathbf{x}|{\mathit{\theta }}_{\varphi ,\mathbf{z},M}))|{\mathit{\theta }}_{\times ,A}\right):\mathbf{m}\in {\mathbb{N}}_0^d,|\mathbf{m}|\le \alpha ,\mathbf{z}\in {\mathbb{G}}_{d,M}\right\}$. Letting ${\mathit{\theta }}_{f,K,M}$ be the network constructed above, we can check that

$$\underset{\mathbf{x}\in {[0,1]}^d}{sup}\left|N(\mathbf{x}|{\mathit{\theta }}_{f,K,M})-P_M(\mathbf{x})\right|\le C_5^{\prime }{A}_{\alpha }{(M+1)}^d\left(\frac{1}{K}+\frac{1}{\sqrt{K}}\right)\le C_6^{\prime }\frac{{(M+1)}^d}{\sqrt{K}},$$

for some positive constants $C_5^{\prime }$ and $C_6^{\prime }.$ In addition, we have $L({\mathit{\theta }}_{f,K,M})\le 1+(⌈logK⌉+⌈{log}_2(\alpha \vee d)⌉\le C_7^{\prime }⌈logK⌉$ and $n_{max}({\mathit{\theta }}_{f,K,M})\le C_8^{\prime }{A}_{\alpha }{(M+1)}^d$ for some positive constants $C_7^{\prime }$ and $C_8^{\prime }$. For sparsity of the network, we have

$$\begin{array}{cc}\hfill |{\mathit{\theta }}_{f,K,M}{|}_0& \le A_{\alpha }{(M+1)}^d|{\mathit{\theta }}_{\times ,A}{|}_0+{(M+1)}^d|{\mathit{\theta }}_{\varphi ,\mathbf{z},M}{|}_0+A_{\alpha }{|{\mathit{\theta }}_{\mathbf{m}}|}_0\hfill \\ & \le C_9^{\prime }⌈logK⌉{(M+1)}^d,\hfill \end{array}$$

for some $C_9^{\prime }>0.$ Taking $M+1={ϵ}^{-1/\alpha }$ and $K={ϵ}^{-2d/\alpha -2}$, we have

$${\mathit{\theta }}_{f,K,M}\in \mathsf{\Theta }\left(L_{0}log(1/ϵ),N_0{ϵ}^{-d/\alpha },S_0{ϵ}^{-d/\alpha }log(1/ϵ),B_0{ϵ}^{-4(d/\alpha +1)}\right),$$

so that ${\Vert P_M-N_{\sigma }(·|{\mathit{\theta }}_{f,K,M})\Vert }_{\infty }\le C_{10}^{\prime }ϵ$ for some $C_{10}^{\prime }>0.$ Since ${\Vert f-P_M\Vert }_{\infty }\le R{M}^{-\alpha }\le C_{11}^{\prime }ϵ$ for some $C_{11}^{\prime }>0,$ the proof is done. □