ReLU Neural Networks and Their Training

Abstract

Among various activation functions, the Rectified Linear Unit (ReLU) has become the most widely adopted due to its computational simplicity and effectiveness in mitigating the vanishing-gradient problem. In this work, we investigate the advantages of employing ReLU as the activation function and establish its theoretical significance. Our analysis demonstrates that ReLU-based neural networks possess the universal approximation property. In addition, we provide a theoretical explanation for the phenomenon of neuron death in ReLU-based neural networks. We further validate the effectiveness of this explanation through empirical experiments.

1. Introduction

In recent years, Artificial Neural Networks (ANNs) have emerged as a central paradigm in the fields of artificial intelligence and machine learning. They have found widespread applications in computer vision [1,2,3], natural language processing [4,5,6], and speech recognition [7,8], where they have achieved outstanding results.

Inspired by biological neural systems, ANNs are composed of multiple interconnected layers of artificial neurons. Each neuron performs a weighted summation of its inputs and passes the result through a nonlinear activation function to produce an output. By progressively combining multiple layers, neural networks can approximate highly complex nonlinear functions. So that the ReLU neural network exhibits powerful function approximation capabilities [9].

In 1986, Hinton introduced the Backpropagation (BP) algorithm [10], which provided an effective means of parameter optimization for neural networks. Meanwhile, Hecht-Nielsen and Robert theoretically proved the convergence of BP neural networks under certain conditions [11] in 1992, laying a solid theoretical foundation for the reliable training and application of ANNs.

2. ReLU Function and Related Works

As the AI field has continued to advance, it has become clear that the choice of activation function plays a critical role in enhancing the performance of neural networks. Activation functions not only provide neural networks with nonlinear modeling capabilities but also directly influence convergence speed, representational capacity, and generalization ability. Different activation functions can therefore lead to vastly different training dynamics and application outcomes. Statistical analyses of open-source platforms such as GitHub show that the Rectified Linear Unit (ReLU) is currently the most widely used activation function (see

Table 1. Usage frequency of various activation functions on GitHub (2025.7).

Rank	Function	Usage Count
1	ReLU	15.3 M
2	SoftMax	3.7 M
3	Tanh	3.1 M
4	Sigmoid	2.4 M
5	GELU	1.8 M
6	Swish	717 k
7	Leaky ReLU	635 k
8	Softplus	253 k

). ReLU is favored for its simplicity, computational efficiency, and effectiveness in mitigating the vanishing-gradient problem, making it the default choice in many neural network architectures.

The function was originally introduced by Professor Zheng Tang under the name ULR function in 1993 [12]. He proposed a multi-valued logic algebra system based on the ULR function and strictly proved its mathematical completeness, i.e., any multi-valued logic function can be represented by a combination of ULR functions [13]. Tang and his team conducted extensive theoretical and experimental studies on its potential. They proposed a learnable fuzzy network based on the ULR function [14]. Their achievements have had a far-reaching impact on the current development in complex control tasks such as intelligent manufacturing and autonomous driving. Later, Hinton reintroduced the function as ReLU (see Figure 1) and promoted its widespread use across various artificial intelligence tasks [15,16]. The ReLU function is popular because it is easy to differentiate, has a simple form, and is less likely to cause gradient vanishing or explosion issues in deep neural networks.

From ResNet to Transformer, ReLU and its variants have demonstrated exceptional versatility and practical value. Nevertheless, most applications of artificial neural networks are still primarily driven by empirical findings, with their deployment guided more by experimental success than by solid theoretical foundations ensuring generality and feasibility. Against this backdrop, we aim to study the mathematical principles underlying the approximation capability of neural networks using ReLU as the activation function (hereafter referred to as ReLU neural networks). Our goal is to establish a stronger theoretical basis for understanding their remarkable expressive power.

3. Main Result

3.1. Theorems and Corollaries

The first question is how to use mathematical models to quantify the expressive power of a ReLU neural network. For simplicity, we consider a ReLU neural network with an r-dimensional input and a one-dimensional output. Next, we will first provide some mathematical definitions to help explain this problem. In all the following definitions and formulas $R(·)$ denotes the ReLU function, i.e., $R\left(x\right)=max\{0,x\}$.

Definition 1. 

1. 

$A^r$ is the set of all affine functions from ${\mathbf{R}}^r$ to $\mathbf{R}$, which means

$$\forall A\in A^r,\exists w_A\in {\mathbf{R}}^{1\times r},b_A\in \mathbf{R},s.t.A\left(x\right)=w_A·x+b_A,\forall x\in {\mathbf{R}}^r.$$

(1)

2. 

${\mathbf{B}}^r$ is the Borel σ-field in ${\mathbf{R}}^r$.

3. 

${\mathbf{C}}^r$(res. ${\mathbf{M}}^r$) is the set of all continuous (res. Borel measurable) functions from ${\mathbf{R}}^r$ to $\mathbf{R}$.

4. 

Let S, T be subsets of the metric space $(\mathbf{X},\rho )$. We say S is ρ-dense in T when for any $ϵ>0$ and for all t in T, there is an s in S such that $\rho (s,t)<ϵ$.

5. 

A subset S of ${\mathbf{C}}^r$ is said to be uniformly dense on compact set in ${\mathbf{C}}^r$ if for every compact subset $\mathcal{K}\subset {\mathbf{R}}^r$, S is ${\rho }_{\mathcal{K}}$-dense in ${\mathbf{C}}^r$, where ${\rho }_{\mathcal{K}}(f,g)={\sup }_{x\in \mathcal{K}}|f\left(x\right)-g\left(x\right)|$.

6. 

$$\begin{array}{cc}\hfill {\Sigma }^r\left(R\right)=\left\{f\right|f\left(x\right)=\sum _{j=1}^q{\beta }_{j}R\left(A_j\left(x\right)\right),& \hspace{1em}x\in {\mathbf{R}}^r,{\beta }_j\in \mathbf{R},\hfill \\ & \hspace{1em}{A}_j\in A^r,q=1,2,3,\dots \}.\hfill \end{array}$$

7. 

$$\begin{array}{cc}\hfill \Sigma {\Pi }^r\left(R\right)=\left\{f\right|f\left(x\right)=\sum _{j=1}^q{\beta }_j\prod _{k=1}^{l_j}R\left(A_{jk}\left(x\right)\right),& \hspace{1em}x\in {\mathbf{R}}^r,{\beta }_j\in \mathbf{R},l_j\in \mathbf{N},\hfill \\ & \hspace{1em}{A}_{jk}\in A^r,q=1,2,3,\dots \}.\hfill \end{array}$$

It is readily seen that ${\Sigma }^r\left(R\right)$ is the mathematical model of a ReLU neural network with r-dimensional input and one-dimensional output.

After obtaining the definitions and formulas mentioned above, let us take a look at previous research results on the approximate property of neural networks. Hornik [9] proved the following conclusions in 1989 to show the universal approximation property of a normal neural network.

Theorem 1. 

For any continuous non-constant function G from $\mathbf{R}$ to $\mathbf{R}$, $\Sigma {\Pi }^r\left(G\right)$ is uniformly dense on compact set in ${\mathbf{C}}^r$. And for any continuous non-constant function, every r, and every probability measure μ on $({\mathbf{R}}^r,{\mathbf{B}}^r)$, $\Sigma {\Pi }^r\left(G\right)$ is ${\rho }_{\mu }$-dense in ${\mathbf{M}}^r$.

According to the theorem above, we obtain two corollaries.

Corollary 1. 

$\Sigma {\Pi }^r\left(R\right)$ is uniformly dense on compact set in ${\mathbf{C}}^r$.

Corollary 2. 

For every $r\in {\mathbf{Z}}_{+}$, every probability measure μ on $({\mathbf{R}}^r,{\mathbf{B}}^r)$, $\Sigma {\Pi }^r\left(R\right)$ is ${\rho }_{\mu }$-dense in ${\mathbf{M}}^r$.

Inspired by the theorem and corollaries above, we began studying neural networks that use ReLU as the activation function. The main result of our research is the theorem below.

Theorem 2. 

For every $r\in {\mathbf{Z}}_{+}$, every probability measure μ on $({\mathbf{R}}^r,{\mathbf{B}}^r)$, ${\Sigma }^r\left(R\right)$ is uniformly dense on compact set in ${\mathbf{C}}^r$, and it is ${\rho }_{\mu }$-dense in ${\mathbf{M}}^r$.

The theorem explains that the ReLU neural network and the Sigmoid neural network have the same expressive power. And we can easily prove the following corollaries.

Corollary 3. 

1. 

For every function g in ${\mathbf{M}}^r$, there is a compact set $\mathcal{K}\subset {\mathbf{R}}^r$ and a function $f\in {\Sigma }^r\left(R\right)$, such that for any real number $ϵ>0$ we have $\mu \left(\mathcal{K}\right)>1-ϵ$. And for every $x\in \mathcal{K}$ we have $\left|g\right(x)-f(x\left)\right|<ϵ$. Here r is an integer and μ is a probability measure on ${\mathbf{R}}^r$.

2. 

If a compact set $\mathcal{K}\subset {\mathbf{R}}^r$ satisfies $\mu \left(\mathcal{K}\right)=1$, then ${\Sigma }^r\left(R\right)$ is ${\rho }_p$-dense in ${\mathbf{L}}_p({\mathbf{R}}^r,\mu )$, for any $p\ge 1$ and any integer r.

3. 

If μ is a probability measure on ${[0,1]}^r$, then ${\Sigma }^r\left(R\right)$ is ${\rho }_p$-dense in ${\mathbf{L}}_p({[0,1]}^r,\mu )$, where $p\ge 1$ and r is arbitrary integer.

4. 

If μ puts mass 1 on a finite set of points, then for every $g\in {\mathbf{M}}^r$ and any $ϵ>0$ there is a function $f\in {\Sigma }^r\left(R\right)$ such that $\mu \{x:|f\left(x\right)-g\left(x\right)|<ϵ\}=1$.

5. 

For any Boolean function g and real number $ϵ>0$, there is a function $f\in {\Sigma }^r\left(R\right)$ such that ${max}_{x\in {\{0,1\}}^r}|g\left(x\right)-f\left(x\right)|<ϵ$.

In other words, a ReLU neural network can approximate a Boolean function with arbitrary precision limits as long as it has enough neurons.

Before proving the above, we need the following lemma for assistance.

Lemma 1. 

For arbitrary $a,b\in \mathbf{R}$, which $a<b$. Let f be a convex function on $[a,b]$. Assume f is derivable on the interval [a,b] and $|f^{\prime }|$ is bounded. Then for every $ϵ>0$, there exists $f_{ϵ}\in {\Sigma }^r\left(R\right)$ such that

$$\begin{array}{c}\hfill \underset{\lambda \in [a,b]}{\sup }|f\left(\lambda \right)-f_{ϵ}\left(\lambda \right)|<ϵ.\end{array}$$

A similar conclusion also holds for concave functions.

Corollary 4. 

For every $a,b$, which $a<b$, if f is a concave function on $[a,b]$, and $|f^{\prime }|$ is bounded. Then for any $ϵ>0$, there is an $f_{ϵ}\in {\Sigma }^r\left(R\right)$ such that

$$\begin{array}{c}\hfill \underset{\lambda \in [a,b]}{\sup }|f\left(\lambda \right)-f_{ϵ}\left(\lambda \right)|<ϵ.\end{array}$$

If the condition of a convex function is changed to a concave function, while keeping the other conditions unchanged, the conclusion in Lemma 1 still holds.

Lemma 2. 

For every $ϵ>0,M>0$, there is a function ${\cos }_{M,ϵ}\in {\Sigma }^1\left(R\right)$ such that

$$\begin{array}{c}\hfill \underset{\lambda \in [-M,+M]}{\sup }|{\cos }_{M,ϵ}\left(\lambda \right)-\cos \left(\lambda \right)|<ϵ.\end{array}$$

And this approximation can be extended to a triangular series.

Lemma 3. 

Let $g(·)={\sum }_{j=1}^Q{\beta }_j\cos \left(A_j(·)\right)$, $A_j\in A^r$, then for any compact set $\mathcal{K}\in {\mathbf{R}}^r$, and every real number $ϵ>0$, there is an $f\in {\Sigma }^r\left(R\right)$ such that

$$\begin{array}{c}\hfill \underset{x\in \mathcal{K}}{\sup }|g\left(x\right)-f\left(x\right)|<ϵ.\end{array}$$

The following lemma is about the density of ${\Sigma }^r\left(R\right)$ in ${\mathcal{C}}^r$.

Lemma 4. 

${\Sigma }^r\left(R\right)$ is uniformly dense on compact set in ${\mathbf{C}}^r$.

The last lemma we need is Lemma A.1 in the paper by Hornik [9]:

Lemma 5 

([9]). For any finite measure μ, ${\mathbf{C}}^r$ is ${\rho }_{\mu }$-dense in ${\mathbf{M}}^r$.

Artificial intelligence is a discipline that emphasizes practical applicability. Considering only the theoretical approximation capabilities of neural networks is insufficient for real-world applications. What matters more is the ability of ReLU neural networks to fit data in actual training and the approximation error under certain constraints. This motivates the following discussion: for any given training dataset, a single-hidden-layer ReLU neural network theoretically has the capacity to perfectly fit the training data.

Theorem 3. 

Let $\{x_1,x_2,\dots ,x_n\}$ be the set of different points in ${\mathbf{R}}^r$. And $g:{\mathbf{R}}^r\to \mathbf{R}$ is any function. Then there is a function $f\in {\Sigma }^r\left(R\right)$ such that $f\left(x_i\right)=g\left(x_i\right),i=1,2,\dots ,n$.

If we use ${\Sigma }^{r,k}\left(R\right)$ to donate the set

$$\begin{array}{c}\hfill \left\{f\right|f\left(x\right)={\Sigma }_{j=1}^q,{\beta }_{j}R\left(A_j\left(x\right)\right),x\in {\mathbf{R}}^r,{\beta }_j\in \mathbf{R},A_j(·)isaaffinefunctionfrom{\mathbf{R}}^{r}to{\mathbf{R}}^k\}.\end{array}$$

Then we can draw the following conclusion.

Corollary 5. 

Let $\{x_1,x_2,\dots ,x_n\}$ be the set of different points in ${\mathbf{R}}^r$. And $g:{\mathbf{R}}^r\to {\mathbf{R}}^k$ is any function. Then there is a function $f\in {\Sigma }^{r,k}\left(R\right)$ such that $f\left(x_i\right)=g\left(x_i\right),i=1,2,\dots ,n$.

3.2. Discussion

We have proven that for any given training set, there exists an $f\in {\Sigma }^r\left(R\right)$ that can fit it perfectly. Another question is how to find this f. The BP algorithm tells us that we can find a neural network with a locally minimal error. However, according to Hecht-Nielsen’s theory [11], we cannot guarantee that a single hidden-layer neural network architecture can converge to the global optimum. Some papers [17,18] also point out a phenomenon called Dying ReLU. Dying ReLU refers to a problem when ReLU neurons become inactive and only output 0 for any input. We can use our theory to explain this phenomenon.

If we use ${\sum }_{j=1}^q{\beta }_{j}R\left(A_j\left(x\right)\right)$ to donate a trained ReLU neural network, which ${\beta }_j\in \mathbf{R},A_j$ is an affine function for all $j=1,2,\dots ,q$. Let $S\subset {\mathbf{R}}^n$ be a dataset. Then the direct cause of Dying ReLU is that the affine function maps the dataset to the negative half-axis of the ReLU. In terms of a formula, it is expressed as

$$\begin{array}{c}\hfill \exists 1\le j\le q,s.t.A_j\left(x\right)\le 0,\forall x\in S.\end{array}$$

Here, $A_j\left(x\right)\le 0$ means that each component of $A_j\left(x\right)$ is less than 0. And we call all x that satisfy $A_j\left(x\right)\le 0$ the Dying Area of $A_j$, denoted as $D{y}_{A_j}$. So the Dying ReLU problem then becomes how to train a ReLU neural network so that the dataset does not fall into $D{y}_{A_j}$. It is easy to see that if $S\subset D{y}_{A_j}$ for some j, then the BP algorithm will no longer adjust the parameters of this neuron. Then the training will have no effect on it.

Previous researchers have proposed several different approaches to solve this problem. Some research teams have suggested using new activation functions, such as the leaky ReLU [19] and GELU [20] (see Figure 2). Avoiding the Dying Area might be a way to prevent neuron death.

Others believe that the initialization of parameters plays an important role in the training of neural networks [2,7]. A poor initialization parameter can slow down the training of the neural network, and may even cause some ReLU neurons to die before training begins. In the next subsection, we will conduct a set of experiments to help illustrate this point.

Some people are also thinking about approaching it from the perspective of network depth. They theoretically demonstrated that deep neural networks can achieve the function of shallow neural networks with fewer parameters [21]. We also conducted an experiment to compare the performance of different activation functions in deep neural networks.

3.3. Experiments

In this section, we will show the impact of different initialization parameter strategies on neural network training. Additionally, we will demonstrate the training differences between ReLU, Sigmoid, and Tanh in deep neural networks.

1. Datasets We evaluate our models on a subset of the COVID-19 Radiography Database from Kaggle (https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database, accessed on 20 November 2025), which originally contains 21,165 chest X-ray images across four categories. For our experiments, we sample 1059 images (953 for training and 106 for testing) while preserving class balance. All images are resized to 224 × 224 and normalized before training. We also validated the properties of some deep neural networks on the MNIST dataset to check whether the experimental conclusions were coincidental for a specific dataset.

2. Models We removed the residual connection structure from He Kaiming’s paper [2] and built each layer of the network using only fully connected layers. We implement two deep neural models (50 layers and 101 layers) for our experiments. Based on the above models, we designed three sets of experiments.

The first is to study the effect of the learning rate on different activation functions in deep neural networks. We used three learning rates (0.01, 0.001, 0.0001) and employed the Kaiming normal initialization strategy for parameters (gain = 1). This experiment was conducted on the MNIST dataset.

The second experiment investigates the impact of different initialization strategies on training 50-layer deep neural networks with various activation functions. This experiment was conducted on two datasets, with the learning rate fixed at 0.001.

The last experiment involved replacing the 50-layer deep neural network in the second experiment with a 101-layer deep neural network, while keeping all other hyperparameters unchanged.

For each experiment, three activation settings are evaluated: ReLU, Tanh, and Sigmoid. All experiments were trained for 50 epochs with a batch of 32 using the SGD optimizer. All experimental parameters are set as shown in

Table 2. Experimental Parameters.

Rank	Layers	Initialization	Learning Rate	Dataset
1	50	Kaiming normal	0.01, 0.001, 0.0001	MNIST
2	50	Kaiming normal & Xavier	0.001	COVID-19 & MNIST
3	101	Kaiming normal & Xavier	0.001	COVID-19 & MNIST

.

3. Evaluation Metrics Model performance is evaluated using Top-1 classification accuracy. We also record training loss curves to analyze convergence differences.

4. Experimental Procedure All controlled experiments are conducted, each using one activation function (ReLU, Sigmoid, or Tanh) and one method for parameter initialization (Kaiming or Xavier). All models are trained from scratch under identical settings, ensuring that any performance difference is solely due to the activation function.

5. Results and Discussion We report the performance of models trained with ReLU, Tanh, and Sigmoid activation functions under two different network depths. The results consistently demonstrate that the effect of activation functions becomes more pronounced as the network grows deeper.

• Analysis on the 50-layer network.
  In the shallower 50-layer architecture (see

  Table 3. The classification accuracy of different activation functions (50 layers on COVID-19).

  Rank	Function	Accuracy (%)
  1	ReLU	81.36
  2	Sigmoid	80.95
  3	Tanh	81.28

  and Figure 3 and Figure 4), all three activation functions achieve comparable accuracy and convergence behavior. Although the performance gap is relatively small, ReLU still produces slightly higher accuracy and faster convergence. This indicates that the vanishing-gradient issue is less severe at this depth, allowing Sigmoid and Tanh to remain competitive.
• Analysis on the 101-layer network (see Figure 5,

  Table 4. The classification accuracy of different activation functions (101 layers on COVID-19).

  Rank	Function	Accuracy (%)
  1	ReLU	73.16
  2	Sigmoid	67.18
  3	Tanh	62.85

  ).
  ReLU achieves the highest accuracy and fastest convergence in the 101-layer network, confirming its advantage in mitigating gradient vanishing. Sigmoid and Tanh show almost no learning progress during the first 20 epochs due to saturation in deep layers. After epoch 20, their curves diverge: Sigmoid recovers gradients more effectively and improves rapidly, while Tanh remains slower because of stronger saturation. Overall, activation choice becomes increasingly critical in deeper networks, with ReLU demonstrating the most stable and efficient training behavior.

By comparing the effects of different initializations on networks with the same depth and activation functions, it can be shown that parameter initialization does indeed affect the efficiency of model training.

4. Concluding Remark

Beyond the depth of the network architecture and the choice of initialization, our findings also shed light on several additional issues related to ReLU-based neural networks.

Increasing the number of parameters or neurons generally enhances the model’s generalization ability [22,23,24]. This is because a larger network reduces the likelihood that training samples fall into the Dying Area of ReLU units, thereby improving the effectiveness of gradient-based optimization. As a result, more neurons can actively participate in representing the target mapping function. Moreover, Theorem 2 provides an idealized upper bound connecting the number of effective neurons and the approximation error: the more effective neurons the network possesses, the smaller the expected approximation error becomes.

The quality of the training dataset is crucial for generalization performance [25,26]. Neural networks aim to approximate the underlying mapping represented by the dataset; thus, if the dataset poorly reflects the true characteristics of the real-world signal, the network cannot faithfully capture the actual data structure. Conversely, if the dataset contains substantial redundancy or meaningless samples, Theorem 3 implies that a significantly larger number of neurons would be required to fit the dataset adequately, leading to unnecessary computational and resource expenditure.

The future development of artificial intelligence should aim to enhance model capabilities while simultaneously minimizing energy consumption. We identify two potential research directions.

First, improving model architectures to maximize neuron utilization is essential. In other words, increasing the practical effectiveness of ReLU-based networks requires preventing as many neurons as possible from entering the Dying ReLU state, thereby enabling the same task to be accomplished with fewer neurons and parameters. However, the challenges in this direction are evident: aside from the transformer architecture, no substantially superior neural network design has been identified, and we still lack a clear theoretical understanding of why transformer-based models consistently exhibit such strong empirical performance.

Second, constructing higher-quality datasets through more precise data acquisition is crucial. If the collected data accurately reflects the latent structure of the target phenomenon, model training and learned representations will exhibit greater robustness and transferability. Yet this line of inquiry raises new and fundamental questions: What constitutes a high-quality dataset? And, for different application domains, what sampling strategies are required to reliably obtain such datasets? These questions remain open and merit careful investigation.

5. Mathematical Appendix

Proof of Lemma 1. 

Notice that convex function f satisfies for every $x,y\in [a,b]$, ${\displaystyle \frac{f(x)+f(y)}{2}}\ge f({\displaystyle \frac{x+y}{2}})$. We construct two linear function as below

$$\begin{array}{ccc}\hfill l_1(x)& =& {\displaystyle \frac{f(\frac{a+b}{2})-f(a)}{\frac{a+b}{2}-a}}(x-a)+f(a),\hfill \\ \hfill l_2(x)& =& {\displaystyle \frac{f(b)-f(\frac{a+b}{2})}{b-\frac{a+b}{2}}}(x-{\displaystyle \frac{a+b}{2}})+f({\displaystyle \frac{a+b}{2}}).\hfill \end{array}$$

Let $g_1(x)=\left\{\begin{array}{c}{l}_1(x),a\le x<{\displaystyle \frac{a+b}{2}};\hfill \\ l_2(x),{\displaystyle \frac{a+b}{2}}\le x\le b.\hfill \end{array}\right.$ be a piecewise linear function, then $f_1(x)=f(x)-g_1(x)$ is still a convex function on $[a,{\displaystyle \frac{a+b}{2}}]$ (res. on $[{\displaystyle \frac{a+b}{2}},b]$). We obtain the following inequality for estimating

$$\begin{array}{c}\hfill \underset{x\in [a,b]}{max}\left\{f(x)\right\}-\underset{x\in [a,b]}{min}\left\{f(x)\right\}\ge \underset{x\in [a,b]}{max}\left\{f_1(x)\right\}-\underset{x\in [a,b]}{min}\left\{f_1(x)\right\}.\end{array}$$

The equality holds if and only if f is a constant number. Similarly (Figure 6), we construct piecewise linear function $g_2$ based on $f_1$ (that is constructing piecewise linear functions similar to $g_1$ on $[a,{\displaystyle \frac{a+b}{2}}]$ and $[{\displaystyle \frac{a+b}{2}},b]$, respectively, and then concatenate them together). Let $f_2=f_1-g_2$, then

$$\begin{array}{c}\hfill \underset{x\in [a,b]}{max}\left\{f_1(x)\right\}-\underset{x\in [a,b]}{min}\left\{f_1(x)\right\}\ge \underset{x\in [a,b]}{max}\left\{f_2(x)\right\}-\underset{x\in [a,b]}{min}\left\{f_2(x)\right\}.\end{array}$$

Similarly, we can construct $f_3,f_4,$ and so on. We define ${\delta }_i={max}_{x\in [a,b]}f{(x)}_i-{min}_{x\in [a,b]}f{(x)}_i$, $i=1,2,\dots$ and $A_i^j=[a+{\displaystyle \frac{(j-1)(b-a)}{2^i}},a+{\displaystyle \frac{j(b-a)}{2^i}})$, $j=1,2,\dots ,2^i-1$ and $A_i^{2^i}=[a+{\displaystyle \frac{(2^i-1)(b-a)}{2^i}},b]$. Since $|f^{\prime }|$ is bounded, we assume $|f^{\prime }|<M$. Notice that $f_i$ is obtained by subtracting from f several piecewise linear functions on $[a,b]$, then we obtain the following estimation

$$\begin{array}{ccc}\hfill {\delta }_i& =& \underset{x\in [a,b]}{max}f{(x)}_i-\underset{x\in [a,b]}{min}f{(x)}_i\hfill \\ & =& \underset{j=1,2,\dots ,2^i}{max}\{\underset{x\in A_i^j}{max}{f}_i(x)-\underset{x\in A_i^j}{min}{f}_i(x)\}\hfill \\ & \le & \underset{j=1,2,\dots ,2^i}{max}\{\underset{x\in A_i^j}{max}f(x)-\underset{x\in A_i^j}{min}f(x)\}\hfill \\ & \le & \underset{j=1,2,\dots ,2^i}{max}{\mathsf{\int }}_{A_i^j}\left|f^{\prime }\right|dx\hfill \\ & \le & \underset{j=1,2,\dots ,2^i}{max}{\mathsf{\int }}_{A_i^j}Mdx\hfill \\ & =& {\displaystyle \frac{M(b-a)}{2^i}}.\hfill \end{array}$$

When $i>i_{ϵ}=[log({\displaystyle \frac{M(b-a)}{ϵ}})]$, ${\delta }_i<ϵ$, which means f satisfies

$$\begin{array}{c}\hfill \underset{\lambda \in [a,b]}{\sup }|f(\lambda )-f_{ϵ}(\lambda )|<ϵ.\end{array}$$

Here $f_{ϵ}(x)={\sum }_{j=1}^{i-1}{g}_j(x)$ is a piecewise linear function.

Now it remains to show that $f_{ϵ}\in {\Sigma }^r(R)$, we just need to show $g_j\in {\Sigma }^r(R)$. Notice that

$$\begin{array}{c}\hfill G_{g_1,a,b}(x)=\left\{\begin{array}{cc}f(a)(=g_1(a))\hfill & ,x<a,\hfill \\ g_1(x)\hfill & ,x\in [a,b],\hfill \\ f(b)(=g_1(b))\hfill & ,x>b.\hfill \end{array}\right.\end{array}$$

And we can express $G_{g_1,[a,b]}(x)$ as

$$\begin{array}{ccc}\hfill G_{g_1,[a,b]}(x)& =& {\displaystyle \frac{g_1(\frac{a+b}{2})-g_1(a)}{\frac{b-a}{2}}}R(x-a)-{\displaystyle \frac{2g_1(\frac{a+b}{2})-g_1(a)-g_1(b)}{\frac{b-a}{2}}}R(x-{\displaystyle \frac{a+b}{2}})\hfill \\ & & +{\displaystyle \frac{g_1(\frac{a+b}{2})-g_1(b)}{\frac{b-a}{2}}}R(x-b)+g_1(a)R(1).\hfill \end{array}$$

So $g_1\in {\Sigma }^r(R)$. Similarly, for $i\ge 2$

$$\begin{array}{c}\hfill g_i(x)=\sum _{j=1}^{2^{i-1}}{G}_{g_i,[a+{\displaystyle \frac{(j-1)(b-a)}{2^{i-1}}},a+{\displaystyle \frac{(j-1)(b-a)}{2^i}}]}(x).\end{array}$$

Now we show that for all $i=1,2,\dots ,2^{i-1}$, $g_i$ is an element of ${\Sigma }^r(R)$. Since ${\Sigma }^r(R)$ is an algebra, we have $f_{ϵ}\in {\Sigma }^r(R)$. □

Proof of Lemma 2. 

According to Lemma 1, there is a sequence of functions $\{f_k,k\in \mathbf{Z}\}\subset {\Sigma }^r(R)$, such that for $ϵ>0$

$$\begin{array}{c}\hfill \underset{\lambda \in [2k\pi +{\displaystyle \frac{\pi }{2}},2k\pi +{\displaystyle \frac{3\pi }{2}}]}{\sup }|f_k(\lambda )-\cos (\lambda )|<ϵ.\end{array}$$

And for all $\lambda \in (-\infty ,2k\pi +{\displaystyle \frac{\pi }{2}})\cup (2k\pi +{\displaystyle \frac{3\pi }{2}},\infty )$, $f_k(\lambda )=0$.

By the Corollary 3, it is easy to find a function sequence $\{g_k,k\in \mathbf{Z}\}$ in ${\Sigma }^r(R)$, which for every $ϵ>0$ we have estimation as below

$$\begin{array}{c}\hfill \underset{\lambda \in [2k\pi -{\displaystyle \frac{\pi }{2}},2k\pi +{\displaystyle \frac{\pi }{2}}]}{\sup }|g_k(\lambda )-\cos (\lambda )|<ϵ.\end{array}$$

And for any $\lambda \in (-\infty ,2k\pi -{\displaystyle \frac{\pi }{2}})\cup (2k\pi +{\displaystyle \frac{\pi }{2}},\infty )$, $g_k(\lambda )=0$.

Then for every $M>0$, choose $n\in \mathbf{N}$ that satisfies $2n\pi >M$. We have

$$\begin{array}{cc}& \underset{\lambda \in [-M,+M]}{\sup }|\sum _{k=-n}^n(f_k(\lambda )+g_k(\lambda ))-\cos (\lambda )|\hfill \\ <& \underset{k\in \mathbf{N},k\in [-n,n]}{\sup }\left\{\underset{\lambda \in [2k\pi -{\displaystyle \frac{\pi }{2}},2k\pi +{\displaystyle \frac{3\pi }{2}}]}{\sup }\{f_k\left(\lambda \right)+g_k(\lambda )-\cos \left(\lambda \right)\}\right\}\hfill \\ <& ϵ.\hfill \end{array}$$

So the function ${\cos }_{M,ϵ}\left(\lambda \right)={\sum }_{k=-n}^n(f_k\left(\lambda \right)+g_k\left(\lambda \right))$ is what we need. □

Proof of Lemma 3. 

Notice that $A_j\in A^r,j=1,2,\dots ,Q$ are continuous mapping and compact set $\mathcal{K}$ is bounded, there is a real number $M>0$, which for all $j=1,2,\dots ,Q$, $A_j(\mathcal{K})\subset [-M,M]$.

Since Lemma 2 tells us there is a function ${\cos }_{M,{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}}(·)\in {\Sigma }^1(R)$ such that

$$\begin{array}{c}\hfill \underset{\lambda \in [-M,+M]}{\sup }|\cos \left(\lambda \right)-{\cos }_{M,{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}}\left(\lambda \right)|<{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}.\end{array}$$

We can construct $f\left(x\right)$ as following

$$\begin{array}{c}\hfill f(x)=\sum _{j=1}^Q{\beta }_j{\cos }_{M,{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}}(A_j(x)).\end{array}$$

So that

$$\begin{array}{ccc}\hfill \underset{x\in \mathcal{K}}{\sup }|g(x)-f(x)|& =& \underset{x\in \mathcal{K}}{\sup }|\sum _{j=1}^Q{\beta }_j\cos (A_j(x))-f(x)|\hfill \\ & =& \underset{x\in \mathcal{K}}{\sup }\left|\sum _{j=1}^Q{\beta }_j\left(\cos (A_j(x))-{\cos }_{M,{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}}(A_j(x))\right)\right|\hfill \\ & \le & \underset{\lambda \in [-M,+M]}{\sup }\left|\sum _{j=1}^Q{\beta }_j\left(\cos (\lambda )-{\cos }_{M,{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}}(\lambda )\right)\right|\hfill \\ & \le & \sum _{j=1}^Q\left(\underset{\lambda \in [-M,+M]}{\sup }|{\beta }_j||\cos (\lambda )-{\cos }_{M,{\displaystyle \frac{ϵ}{|{\beta }_j|·Q}}}\left(\lambda \right)|\right)\hfill \\ & \le & \sum _{j=1}^Q{\displaystyle \frac{ϵ}{Q}}\hfill \\ & =& ϵ.\hfill \end{array}$$

Up to this point, we have proven Lemma 3. □

Proof of Lemma 4. 

With the trigonometric identity, we have

$$\begin{array}{c}\hfill 2\cos (\alpha )\cos (\beta )=\cos (\alpha +\beta )+\cos (\alpha -\beta ).\end{array}$$

Based on the formula above, we can transform elements in $\Sigma {\Pi }^r(\cos )$ into ones in ${\Sigma }^r(\cos )$. Furthermore, we can prove that $\Sigma {\Pi }^r(\cos )={\Sigma }^r(\cos )$. According to Theorem 1, we know $\Sigma {\Pi }^r(\cos )$ is uniformly dense on a compact set in ${\mathbf{C}}^r$. Then the proof is finished by Lemma 3. □

Proof of Theorem 2. 

On the one hand, Lemma 4 shows that ${\Sigma }^r(R)$ is uniformly dense on compact set in ${\mathbf{C}}^r$, so ${\Sigma }^r(R)$ is ${\rho }_{\mu }$-dense in ${\mathbf{C}}^r$ for any probability measure $\mu$ on $({\mathbf{R}}^r,{\mathbf{B}}^r)$. On the other hand, we know ${\mathbf{C}}^r$ is ${\rho }_{\mu }$-dense in ${\mathbf{M}}^r$ through Lemma 5. The triangle inequality implies that ${\Sigma }^r(R)$ is ${\rho }_{\mu }$-dense in ${\mathbf{M}}^r$. □

Proof of Theorem 3. 

First we find an affine function $\mathcal{A}\in A^r$ which makes $\mathcal{A}(x_i)(i=1,2,\dots ,n)$ all different from each other. Since the solution set of a system of linear equations ${\mathcal{A}}_{ij}(x_i)={\mathcal{A}}_{ij}(x_j),1\le i\ne j\le n$ is a real subset of ${\mathbf{R}}^r$, there must be a matrix A in ${\mathbf{R}}^{1\times r}$ such that $A{x}_i\ne A{x}_j,\forall i\ne j$. Then $\mathcal{A}(x)=Ax$ is what we need.

Let $y_i=\mathcal{A}(x_i)$, then $y_1,y_2,\dots ,y_n$ are different from each other. Without loss of generality, we assume $y_{i_1}<y_{i_2}<\dots <y_{i_n}$. Think about the function below.

$$\begin{array}{ccc}\hfill f_k(x)={\displaystyle \frac{g(x_k)}{y_k-y_{k-1}}}R(x-y_{k-1})& -& ({\displaystyle \frac{g(x_k)}{y_{k+1}-y_k}}+{\displaystyle \frac{g(x_k)}{y_k-y_{k-1}}})R(x-y_k)\hfill \\ & +& {\displaystyle \frac{g(x_k)}{y_{k+1}-y_k}}R(x-y_{k+1}).\hfill \end{array}$$

Specially, we define $y_{i_0}=min\{-2y_{i_1},2y_{i_1}\}$,$y_{i_{n+1}}=max\{2y_{i_n},-2y_{i_n}\}$. Meanwhile, $f_k(x)$ is a piecewise linear function that takes values between 0 and $g(x_k)$ on the interval $[y_{k-1},y_{k+1}]$. If and only if $x=y_k$, $f_k(x)=g(x_k)$. And $f_k$ remains 0 out of the interval $[y_{k-1},y_{k+1}]$. Let $F={\sum }_{k=1}^n{f}_k$, then $F(y_k)=g(x_k)$. Let $f=F\circ \mathcal{A}$, then for all $i=1,2,\dots ,n$, $f(x_i)=F(\mathcal{A}(x_i))=F(y_i)=g(x_i)$. So f is the function we need. □

Remark 1. 

It seems that the number of activation functions (neurons in the hidden layer) in the theorem above is approximately three times the number of sample points. However, in practice, we only need about the same number of activation functions as the number of sample points to achieve a perfect fit of the training set. Think about the function below.

$$\begin{array}{c}\hfill f_i(x)={\beta }_{k}R(x-y_k),k=1,2,\dots ,n-1.\end{array}$$

Here ${\beta }_1={\displaystyle \frac{g(x_2)-g(x_1)}{y_2-y_1}}$, ${\beta }_{k+1}={\displaystyle \frac{g(x_{k+2})-g(x_{k+1})}{y_{k+2}-y_{k+1}}}-{\sum }_{j=1}^k{\beta }_j,k=1,2,\dots ,n-1$.

And we can easily verify $F(x)=g(x_1)R(1)+{\sum }_{k=1}^{n-1}{\beta }_{k}R(x-y_k)$ satisfies $F(\mathcal{A}(x_i))=g(x_i)$.