On the Universally Optimal Activation Function for a Class of Residual Neural Networks

Abstract

While non-linear activation functions play vital roles in artificial neural networks, it is generally unclear how the non-linearity can improve the quality of function approximations. In this paper, we present a theoretical framework to rigorously analyze the performance gain of using non-linear activation functions for a class of residual neural networks (ResNets). In particular, we show that when the input features for the ResNet are uniformly chosen and orthogonal to each other, using non-linear activation functions to generate the ResNet output averagely outperforms using linear activation functions, and the performance gain can be explicitly computed. Moreover, we show that when the activation functions are chosen as polynomials with the degree much less than the dimension of the input features, the optimal activation functions can be precisely expressed in the form of Hermite polynomials. This demonstrates the role of Hermite polynomials in function approximations of ResNets.

1. Introduction

Consider a function approximation problem in which we use $f_1,\cdots ,f_k$ to approximate a target function g. Assume $g,f_i$ are bijective functions with the range consisting of $n(>k)$ elements from $\mathbb{R}$, and to obtain different functions, we sample elements of ranges of $g,f_i$ from the Gaussian distribution. When we only use linear combinations ${\sum }_{i=1}^k{w}_i{f}_i$ to approximate g, the expectation of minimal residual sum of squares (RSS) is achieved at $(1-\frac{k}{n})$. However, if we introduce a non-linear function $\sigma$ and use $\sigma ({\sum }_{i=1}^k{w}_i{f}_i)$ to approximate g, we can achieve an averaged RSS lower than $(1-\frac{k}{n})$. This paper quantitatively investigates the RSS that the non-linear approximator achieves, which is lower than its linear counterpart.

The function approximation problem revealed above can also be formulated in the terminology of neural networks. It is well-known that with only one hidden layer and choosing sigmoid as the activation function, we can use neural networks to approximate any continuous function [1]. However, this universal approximation theorem requires that the non-linear activation function satisfies certain regularity conditions and does not tell us which kind of activation function performs best in a certain problem. On the contrary, this paper shows that the Hermite polynomial is the optimal candidate to achieve the minimal RSS in the problem mentioned above.

Moreover, note that most of the previous works along this direction focused on providing upper and lower bounds [2] of the function approximation performance without closed-form characterizations. On the other hand, our formulation not only illustrates the role non-linearity plays in activation function, but more importantly, we provide the closed-form solutions for the non-linear activation function to achieve the minimal RSS. To the best of our knowledge, this is the first work that demonstrates how profitable that non-linear functions can be in a tight formulation.

In particular, our work is not restricted to any specific family of activation functions but only requires the activation function to be a perturbation of linear functions. This assumption allows us to study the non-linear gain when a non-linear term is added to the network. Such a gain can be expressed quantitatively up to the second-order approximation of $(1-\frac{k}{n})$. To simplify the technical analysis, we use the one-node neural network which outputs one-dimensional feature [3]. The network we consider can be regarded as a special ResNet model, which is widely used as a building block in deep neural network architectures. We use RSS to measure the network loss, from which we can obtain the universally optimal activation function for the given network, and it is found that the averaged loss is minimized when the activation function is the Hermite polynomial, which validates the previous empirical results [4].

The rest of this paper is organized as follows. In Section 3, we formulate the universally optimal activation function problem mathematically. Under specific assumptions, we derive the optimal solution in the form of the Hermite polynomial and establish the error rate in Section 4. Furthermore, in Section 5, numerical experiments are conducted to verify our theoretical results. Section 6 concludes the paper. The detail of proofs of our results are provided in Appendix A.

Throughout this paper, we use $X,\underline{X}$ and $\mathit{X}$ to represent random variables, random vectors and random matrices. In addition, we use $x,\underline{x},\mathit{x}$ (or $\mathbf{M}$) to represent a numerical value, numerical vector and numerical matrix. In addition, we use ${\mathbf{I}}_n$ to denote the n-dimensional identity matrix, and ${[·]}^{\mathrm{T}}$ is the transpose operation on matrices. Moreover, we use $1_{m=n}$ to denote the indicator function and $\gamma (i,j)\triangleq i+j+1(\mathrm{mod}2)$ which equals 1 if $i+j$ is even and 0 otherwise. Finally, $n!!$ is the double factorial which equals $n\times (n-2)\cdots \times 1$ if n is odd and $n\times (n-2)\cdots \times 2$ for even n. $(-1)!!=1$ and $(-3)!!=-1$ are defined specifically.

2. Related Works

To explore the influence of network structures, most existing theoretical works focus on the factor of layers and hidden units. For example, Kuri-Morales determines the minimal number of hidden units for a multilayer perceptron [5]. ReLU is used in [6], which focuses on studying the number of layers and hidden units. We notice that these works are restricted to a specific activation function and seldom explore the best performance that the non-linear activation function can achieve for a certain network architecture.

To compensate this shortcoming, our study on activation function adopts the approach of statistical learning theory [7], which regards the learning problem from the view of probability. Our theoretical result can explain previous empirical studies such as:

• Fixing the network structure, optimizing over parameterized activation function can improve the overall performance [8].
• Using Hermite orthogonal polynomial as an activation function works better than sigmoid under certain conditions [4].

3. Models and Methods

We consider one-node neural network, whose structure is shown in Figure 1. Its input is a k-dimensional random vector $\underline{X}=(X_1,\dots ,X_k)$ and it outputs a random variable Y. The one-node neural network is the output layer in a complex neural network for the regression task, where $\underline{X}$ is the extracted feature vector by previous layers. Although our study focuses on this specific network structure, we enlarge the choice of activation term $\sigma$ to find insights for the best choice of activation functions. Our goal is to predict Y using $\sigma ({\sum }_{i=1}^k{w}_i{X}_i)$. For a given sampling result, we have n data pairs $({\underline{x}}_1,y_1),\dots ,({\underline{x}}_n,y_n)$, and the ${\ell }_2$ loss function is given by ${\ell }_2(\mathit{x},\underline{y})={\Vert \underline{y}-\sigma ({\sum }_{i=1}^k{w}_i{\underline{x}}_i)\Vert }^2$ where ${\underline{x}}_i={(x_{1i},\dots ,x_{ni})}^2$ is the i-th column of the feature vector $\mathit{x}$ and $\underline{y}=(y_1,\dots ,y_n)$ is the label vector. $\sigma$ is applied to a vector in an elementwise way. The optimal weight for a given activation function $\sigma$ is the minimizer of ${\ell }_2(\mathit{x},\underline{y})$.

This type of network arises from function approximation problems, where a target function g, defined in a finite set, is estimated by $\sigma ({\sum }_{i=1}^k{w}_i{f}_i)$. Suppose the ranges of $f_i,g$ are with cardinality n; then, the functions themselves can be completely determined by n-dimensional vectors. Furthermore, we assume each element from the ranges of $f_i,g$ is sampled from a distribution, since we are considering the average performance over different $f_i,g$ pairs. Then, we can establish a correspondence between $f_i$ and ${\underline{x}}_i$, and between g and $\underline{y}$.

In our model, we suppose $\mathit{x},\underline{y}$ are drawn from a joint distribution G. That is, $\mathit{x},\underline{y}$ are samples of random matrix $\mathit{X}$ and random vector $\underline{Y}$. We are interested to find a function $\sigma$ which minimizes the expectation $\mathbb{E}\left[{\min }_{\underline{w}}{\ell }_2(\mathit{X},\underline{Y})\right]$. Our requirement of the activation function $\sigma$ is of the special form $\sigma \left(z\right)=z+ϵ\xi \left(z\right)$ where $ϵ$ is a small constant. Such a form of $\sigma$ can be regarded as a special kind of ResNet [9]. To draw a fair comparison between different non-linear terms $\xi$ for given $ϵ$, we apply the normalization constraint $\mathbb{E}\left[\xi \left(\mathit{X}\underline{w}\right)\right]=1$ to $\xi$, where $\mathit{X}\underline{w}={\sum }_{i=1}^k{w}_i{\underline{X}}_i$ is the matrix-vector product with $\underline{w}=(w_1,\dots ,w_k)$.

We formulate the universally optimal activation function $\sigma$ as follows:

Definition 1.

Assume $(\mathit{X},\underline{Y})$ follow distribution G, and $\mathcal{F}$ is a function space for $\sigma \left(z\right)$. Then, we define the averaged residual error $\mathrm{E}\left[\sigma \right]$ as

$$\begin{array}{cc}\hfill \mathrm{E}\left(\sigma \right)\triangleq & \mathbb{E}[\underset{\underline{w}}{\min }\Vert {\underline{Y}-\sigma \left(\mathit{X}\underline{w}\right)\Vert }^2]\hfill \end{array}$$

(1)

The function ${\sigma }^{*}$ which minimizes $\mathrm{E}\left(\sigma \right)$ is called the universally optimal activation function for the one-node neural network, i.e., ${\sigma }^{*}=\arg {\min }_{\sigma \in \mathcal{F}}\mathrm{E}\left(\sigma \right)$.

Definition 1 is a general formulation for any distribution space G. To obtain analytical insights, we should choose some specific distribution. Therefore, in the following analysis, we assume:

(1)

$\underline{Y}$ follows Gaussian distribution $\mathcal{N}(\mathbf{0},\frac{1}{n}{\mathbf{I}}_n)$.

(2)

$\mathit{X}$ is a $n\times k$ uniformly distributed random orthogonal matrix.

(3)

$\underline{Y}$ and $\mathit{X}$ are independent.

For assumption (2), the definition of uniformly distributed random orthogonal matrix is $\mathit{X}\triangleq {\mathit{X}}^{\prime }{({{\mathit{X}}^{\prime }}^{\mathrm{T}}{\mathit{X}}^{\prime })}^{-1/2}$ where elements of ${\mathit{X}}^{\prime }$ are i.i.d. $\mathcal{N}(0,1)$ random variables ([10] Proposition 7.1). Since ${\mathit{X}}^{\mathrm{T}}\mathit{X}={\mathbf{I}}_k$, $\mathit{X}$ is indeed an orthogonal matrix. The definition of $\mathit{X}$ can be regarded as the post-processing result of PCA (Principal Component Analysis) on network input ${\mathit{X}}^{\prime }$ and weight ${\underline{w}}^{\prime }$, because we make the transformation $\mathit{X}={\mathit{X}}^{\prime }{({{\mathit{X}}^{\prime }}^{\mathrm{T}}{\mathit{X}}^{\prime })}^{-1/2},\underline{w}={({{\mathit{X}}^{\prime }}^{\mathrm{T}}{\mathit{X}}^{\prime })}^{1/2}{\underline{w}}^{\prime }$. We see that ${\mathit{X}}^{\prime }{\underline{w}}^{\prime }=\mathit{X}\underline{w}$. For assumption (3), notice that when we project a random vector $\underline{Y}$ into a fixed linear subspace, $\underline{Y}$ is independent from the fixed space. When we are evaluating the representability of non-linear activation, we also choose a scenerario in which the input and output are independent to demonstrate that the non-linearity, instead of the correlation, helps to decrease the residual error. Based on the distribution space G by (1) –(3), for linear function space, we have the following conclusion:

Proposition 1.

We have $\mathrm{E}\left({\sigma }_0\right)=1-\frac{k}{n}$ where ${\sigma }_0\left(z\right)=z$ is the identity function.

Proposition 1 says that the averaged residue error for linear function is equal to $(1-\frac{k}{n})$, which is consistent with our intuition, since we use k degrees of freedom to estimate an arbitrary vector in n-dimensional space.

We have analyzed the averaged error for linear function. To extend our result to non-linear functions, we need to choose a proper function space $\mathcal{F}$. In this paper, we consider a local region which contains a non-linear function perturbed from the linear space. The distance from non-linear function to linear space is quantified by a positive value $ϵ$. Thus, we consider ${\mathcal{F}}_{{\sigma }_0}\left(ϵ\right)=\left\{\sigma \right|\Vert \sigma -{\sigma }_0\Vert \le ϵ\}$. Then, all smooth functions can be treated as ${\cup }_{ϵ>0}\mathcal{F}\left(ϵ\right)$. The norm of function is chosen as the expectation with respect to the distribution space G. That is, $\Vert \sigma -{\sigma }_0{\Vert }^2\triangleq \mathbb{E}[\Vert {(\sigma -{\sigma }_0)\left(\mathit{A}\underline{Y}\right)\Vert }^2]$ where $\mathit{A}=\mathit{X}{\mathit{X}}^{\mathrm{T}}$.

We are particularly interested in how the averaged error changes when $\sigma$ is contracted to ${\sigma }_0$ along a certain direction in $\mathcal{F}\left(\sigma \right)$. That is, given a function $\xi$, we can construct $\sigma ={\sigma }_0+ϵ\xi$ where $\xi \in {\mathcal{F}}_0=\left\{\xi \right|\Vert \xi \Vert \le 1\}$. Then, $\sigma \to {\sigma }_0$ is equivalent to $ϵ\to 0$.

To measure the change rate of $E\left[\sigma \right]$ when $\sigma \to {\sigma }_0$, we introduce the concept of the asymptotic error rate as follows:

Definition 2.

Let $\xi \in {\mathcal{F}}_0$; then, the asymptotic error rate for ξ is

$$\mathrm{C}\left[\xi \right]\triangleq \underset{\begin{array}{c}ϵ\to 0\\ \sigma ={\sigma }_0+ϵ\xi \end{array}}{\lim }\frac{\mathrm{E}\left[\sigma \right]-\mathrm{E}\left[{\sigma }_0\right]}{{ϵ}^2}$$

(2)

$\mathrm{C}\left[\xi \right]$ can represent the error change rate of perturbation from a linear function. If $\mathrm{C}\left[\xi \right]$ is negative for a given $\xi$, $\mathrm{E}\left[\sigma \right]$ decreases in the rate of $\left|\mathrm{C}\left[\xi \right]\right|$ along the perturbation direction $\xi$, which can be seen more clearly if we rewrite (2) as $\mathrm{E}\left[\sigma \right]=\mathrm{E}\left[{\sigma }_0\right]+\mathrm{C}\left[\xi \right]{ϵ}^2+o\left({ϵ}^2\right)$. In this form, we can also see that $\mathrm{C}\left[\xi \right]$ is the coefficient of the second-order term of $\mathrm{E}\left[\sigma \right]$.

To justify the definition of $\mathrm{C}\left[\xi \right]$, we need to show the limit in Equation (2) exists, which is guaranteed by the following proposition:

Proposition 2.

Let $\nabla \xi \left(\underline{z}\right)=\mathrm{diag}[{\xi }^{\prime }\left({\underline{z}}_1\right),\dots ,{\xi }^{\prime }\left({\underline{z}}_n\right)]$. Then, we have

$$\begin{array}{cc}\hfill \mathrm{C}\left[\xi \right]=& \hfill \mathbb{E}[\Vert {\xi \left(\mathit{A}\underline{Y}\right)\Vert }^2-{\Vert {\mathit{X}}^{\mathrm{T}}\xi \left(\mathit{A}\underline{Y}\right)\Vert }^2\\ \hfill & \hfill -\Vert {{\mathit{X}}^{\mathrm{T}}\nabla \xi \left(\mathit{A}\underline{Y}\right)(Y-\mathit{A}\underline{Y})\Vert }^2]\end{array}$$

(3)

Our goal is to obtain the fastest decreasing path $\xi$ from $\mathrm{E}\left[{\sigma }_0\right]$ to $\mathrm{E}\left[\sigma \right]$; this is equivalent to solving an optimization problem $\min \mathrm{C}\left[\xi \right]$ constraint by $\xi \in {\mathcal{F}}_0$. It is hard to optimize $\xi$ over ${\mathcal{F}}_0$ directly, and we consider ${\mathcal{F}}_{0,m}={\mathcal{F}}_0\cap P_m$ instead. $P_m$ consists of polynomials with the degree no greater than m, and we have ${\mathcal{F}}_0={\lim }_{m\to \infty }{\mathcal{F}}_{0,m}$.

For $\xi \in {\mathcal{F}}_{0,m}$, we have the following result:

Proposition 3.

If $\xi \left(z\right)={\sum }_{i=0}^m{\underline{q}}_i{z}^i$, and $m\ll k$, then we have

$$\begin{array}{cc}\hfill \mathrm{C}\left[\xi \right]& =(1-\frac{k}{n}){\underline{p}}^{\mathrm{T}}\mathbf{M}\underline{p}\hfill \\ \hfill where {\underline{p}}_i& =\frac{k^{i/2}}{n^{-1/2+i}}{\underline{q}}_i\hfill \\ \hfill {\mathbf{M}}_{ij}=& -\gamma (i,j)(i-1)(j-1)(i+j-3)!!\hfill \end{array}$$

(4)

The constraint $\xi \in {\mathcal{F}}_{0,m}$ is equivalent with

$$\begin{array}{cc}\hfill {\underline{p}}^{\mathrm{T}}\mathbf{N}\underline{p}& =1 with\hfill \\ \hfill {\mathbf{N}}_{ij}& =\gamma (i,j)(i+j-1)!!\hfill \end{array}$$

(5)

Proposition 3 gives a feasible approach to choose the optimal $\xi$ by solving a quadratic optimization problem. We make the assumption $m\ll k$ to write $M_{ij}$ in a concise form. This assumption requires that we should choose low degree polynomials as an activation function, and the number of nodes k should be relatively large to represent features. Since the computational cost is proportional to the degree of polynomials and feature dimensions are usually high, our assumption does not lose practicality.

4. Hermite Polynomials

One of our major results is that we find Proposition 3 leads naturally to the Hermite polynomials solutions. (Probabilists’) Hermite polynomials, denoted as $H_m\left(x\right),m=1,2,\dots$, are defined as a series of polynomials satisfying the orthogonal property $\mathbb{E}\left[H_m\left(X\right)H_n\left(X\right)\right]=n!1_{m=n}$ where X is the standard normal random variable. The degree of $H_m\left(x\right)$ is m. Using Hermite polynomials, the minimizer of $\mathrm{C}\left[\xi \right]$ is expressed in the following way:

Proposition 4.

Let

$${\xi }_m\left(z\right)\triangleq \frac{1}{\sqrt{m!n}}{H}_m\left(\frac{n}{k^{1/2}}z\right)$$

(6)

Then $\mathrm{C}\left[{\xi }_m\right]=-(1-\frac{k}{n})(m-1)\le \mathrm{C}\left[\xi \right]$ for all $\xi \in {\mathcal{F}}_{0,m}$.

Proposition 4 also gives the expression of the minimal asymptotic error rate, which has a linear relation with the degree of Hermite polynomials. This result is obtained under the assumption $m\ll k$. In a general case, when $m\ll k$ does not hold, by numerical simulation, we find that $-(1-\frac{k}{n})(m-1)<\mathrm{C}\left[\xi \right]$, while $\mathrm{C}\left[\xi \right]$ decreases in a linear way.

5. Experimental Validation and Discussions

To verify that the change rate of $\mathrm{C}\left[\xi \right]$ is linear as we increase the maximal degree of polynomials m, we conduct a simple experiment with $n=180,k=120$. We use ${\xi }_m\left(z\right)$ from Proposition 4 and normalized polynomials with only the highest degree ${\tilde{\xi }}_m\left(z\right)=\frac{{(nz/\sqrt{k})}^m}{\sqrt{(2m-1)!!n}}$ as an activation function. Using the Monte Carlo method to compute $\mathrm{C}\left[\xi \right]$ from Proposition 2, our experimental result is shown in Figure 2. As can be seen, the Hermite polynomial results coincide with the theoretical lower bound in Proposition 4. For ${\tilde{\xi }}_m$, it cannot achieve the minimum value, but we can still observe the nearly linear relationship between $\mathrm{C}[{\tilde{\xi }}_m]$ and degree m. From Equation (4), we also notice that for ${\tilde{\xi }}_m$, the highest degree term contributes $-(1-\frac{k}{n})\frac{{(m-1)}^2}{2m-1}$ to $\mathrm{C}\left[\xi \right]$, which is approximately $\frac{1}{2}\mathrm{C}\left[{\xi }_m\right]$. This phenomenon suggests that the contribution of each polynomial degree to $\mathrm{C}\left[\xi \right]$ is not evenly distributed, and higher degree terms contribute more.

6. Conclusions

In this paper, we have investigated the mechanism of non-linearity in a one-node neural network. If we add polynomial perturbation to linear activation function, the network loss is linearly decreased as the degree of polynomial is increased. Currently, this work only investigates a one-node neural network; how non-linearity works in neural networks with more neurons will be considered in the future.