A General Framework for Activation Function Optimization Based on Mollification Theory

Abstract

The deep learning paradigm is progressively shifting from non-smooth activation functions, exemplified by ReLU, to smoother alternatives such as GELU and SiLU. This transition is motivated by the fact that non-differentiability introduces challenges for gradient-based optimization, while an expanding body of research demonstrates that smooth activations yield superior convergence, improved generalization, and enhanced training stability. A central challenge, however, is how to systematically transform widely used non-smooth functions into smooth counterparts that preserve their proven representational strengths while improving differentiability and computational efficiency. To address this, we propose a general activation smoothing framework grounded in mollification theory. Leveraging the Epanechnikov kernel, the framework achieves statistical optimality and computational tractability, thereby combining theoretical rigor with practical utility. Within this framework, we introduce Smoothed ReLU (S-ReLU), a novel second-order continuously differentiable (C²) activation derived from ReLU that inherits its favorable properties while mitigating inherent drawbacks. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet-1K with Vision Transformers and ConvNeXt consistently demonstrate the superior performance of S-ReLU over existing ReLU variants. Beyond computer vision, large-scale fine-tuning experiments on language models further show that S-ReLU surpasses GELU, underscoring its broad applicability across both vision and language domains and its potential to enhance stability and scalability.

1. Introduction

The expressive capacity and optimization dynamics of artificial neural networks are largely influenced by activation functions, which introduce nonlinearity to transform linear computations into complex representations and capture complex data patterns. Over the past three decades, more than 400 activation functions have been proposed to enhance network performance and efficiency [1]. A clear evolutionary trend has emerged: a shift from nonsmooth activation functions toward more smooth ones. The key reason is that the non-differentiability of nonsmooth functions at certain points presents theoretical challenges for optimization algorithms and constitutes a practical bottleneck limiting performance improvements. In contrast, a growing body of theoretical work shows that smooth activation functions can deliver superior performance [2,3]. Specifically, studies on loss landscape visualization demonstrate that smooth functional curves facilitate the formation of flatter loss landscapes [4], which guide optimizers away from poor local minima, ensure better gradient information propagation [3], and significantly improve training stability. However, most existing smooth functions are exploratory modifications that address nonsmoothness but do not necessarily preserve the proven representational strengths of the original activation functions. Hence, developing a systematic framework for reshaping nonsmooth activation functions holds considerable value for machine learning research.

Our work addresses the three research questions outlined above. We first define a smoothing kernel and apply it to transform a non-smooth activation into a smooth one. After verifying its smoothness and approximation, we analyze its Lipschitz constant and establish the connection between smoothness and gradient stability. To address the third question, we propose S-ReLU as a polished variant of the traditional ReLU. We evaluate its effectiveness on Vision Transformer [5], its related derivatives [6,7], and ConvNeXt [8], across CIFAR-10, CIFAR-100 [9], and ImageNet-1K [10]. The results consistently show that S-ReLU outperforms baseline activation functions. Moreover, fine-tuning experiments on large language models (LLMs) using Direct Preference Optimization (DPO) [11] further demonstrate that S-ReLU surpasses GELU, highlighting its broad applicability in practical scenarios. Compared to heuristic smoothing approaches or sigmoid-based approximations, mollification theory offers a mathematically rigorous framework that guarantees uniform convergence to the original activation function. Crucially, it allows for precise control over the approximation error through the smoothing radius $\delta$. Furthermore, by selecting compact support kernels like the Epanechnikov kernel, this framework yields activation functions with polynomial closed-form expressions. This avoids the computational overhead of transcendental functions (e.g., exponential or error functions) commonly found in other smooth activations like GELU and SiLU, thereby balancing theoretical optimality with deployment efficiency. The principal contributions of this work are summarized below:

• First, we show that the mollification can smoothly transform non-smooth activation functions while preserving the desirable properties of the original. This ensures that the resulting activation not only inherits the advantages of the base function but also benefits from smoothness.
• Second, we establish that higher degrees of smoothness lead to greater stability of training gradients, and we derive a quantitative relationship between the two. This provides a theoretical foundation for subsequent studies on activation functions.
• Third, we introduce S-ReLU, a new activation function derived from ReLU via mollification. Extensive experiments across architectures, datasets, and tasks demonstrate that S-ReLU consistently outperforms existing activation functions and achieves state-of-the-art performance.

2. Related Work

The study of activation functions has evolved through several stages. Early works introduced compression functions such as Sigmoid and Tanh [12], which bound outputs to a finite interval but often saturate under extreme inputs, leading to gradient vanishing. To address these limitations, non-compression functions were proposed, with the rectified linear unit (ReLU) [13] marking a landmark contribution. By preserving a unit gradient in the positive region and truncating negative activations to zero, ReLU greatly improved optimization stability. However, its asymmetric form results in structural drawbacks: neurons can become permanently inactive (“dying ReLU”), and the absence of negative outputs introduces distributional bias in subsequent layers.

To overcome these issues, a variety of variants were developed. Leaky ReLU [14] introduces a fixed negative slope to maintain gradient flow for sub-zero inputs, while Parameterized ReLU (PReLU) [15] adapts this slope as a learnable parameter. The Exponential Linear Unit (ELU) [16] produces smooth negative outputs that reduce mean-shift effects, and the Continuously Differentiable Exponential Linear Unit (CELU) [17] simplifies the ELU parameterization for easier adjustment. More recently, smoother activations have gained traction for their theoretical and empirical benefits. Swish (SiLU) [18] leverages a smooth and differentiable form to improve gradient propagation and training stability, while Mish [19] combines unbounded positive outputs with moderated negative values, facilitating deeper signal transmission and enhancing generalization.

On the applied side, GELU [20] and its variants have become the dominant activations in large-scale vision and language architectures. They are the default in BERT and RoBERTa [21], used in ViT [5], and widely believed to support successive GPT models [22,23] and recent models like PaLM [24], LLaMA [25], and DeepSeek-V2 [26]. This reliance underscores the role of smooth activations in scaling and stabilizing deep learning. By contrast, a few open-source models, notably Mixtral [27], adopt SiLU, reflecting continued exploration of alternatives.

3. Motivation

Although piecewise linear activations such as ReLU are widely adopted for their simplicity and effectiveness, they suffer from intrinsic drawbacks that hinder further progress in deep neural networks. These include gradient instability arising from non-differentiability [2,28], inefficient information propagation that accelerates signal degradation in deep layers [3], and limited theoretical guarantees for regularization.

Our key idea is to build on the proven representational strengths of established activations and enhance them through systematic smoothing. By introducing smoothness while retaining the core advantages of the original function, we aim to improve differentiability, stability of gradient-based optimization, information propagation across layers, and regularization properties simultaneously. This perspective shifts the focus from designing entirely new exploratory functions to refining and elevating existing well-validated ones. Such an approach promises not only stronger and more stable performance in conventional deep learning tasks, but also provides a principled foundation for advancing modern large-scale architectures, including contemporary vision and language models.

4. Methodology

4.1. Smooth Activation Fuction Is Better

Research by Hayou et al. [3] indicate that the smoothness of an activation function is a key factor influencing the effective propagation of information in deep neural networks. A sufficient condition for an activation function to be smooth is that its second-order derivative can be piecewise represented as a sum of continuous functions. For smooth activation functions, the correlation between inter-layer neuron outputs converges to 1 at a rate of $O(1/l)$, with the specific formula being $1-c^l\sim {\beta }_q/l$, where c represents the correlation, l is the number of network layers, and ${\beta }_q$ is a coefficient determined by the target variance q and the activation function f. In contrast, for non-smooth functions like ReLU, this correlation converges at a rate of $O(1/l^2)$, with the specific relationship being $1-c^l\sim 9{\pi }^2/2l^2$. This research reveals that in deep networks employing non-smooth activation functions, the correlation of neuron outputs rapidly approaches 1 as the network depth increases. This high degree of correlation impedes the effective propagation of information, leading to unstable gradients and diminished expressive power, ultimately impairing model performance. Intuitively, the non-differentiability of functions like ReLU at the origin acts as a ’kink’ that introduces abrupt changes in the gradient flow. In deep networks, the accumulation of these sharp transitions can rapidly decorrelate the input–output relationship, causing information to degrade as it propagates through layers. Smoother activations, by contrast, offer a continuous transition of derivatives. This preserves the local geometry of the data manifold and ensures that gradients vary more predictably, thereby maintaining stronger signal correlations even in very deep architectures. Therefore, selecting activation functions that meet specific smoothness requirements is an important strategy for enhancing both the training efficiency and the final performance of deep learning models.

4.2. Smoothed Kernel Function

To smooth non-smooth activation functions, we can use the mollification method. This approach involves smoothing the target function by convolving it with a specific kernel. The goal is to create a smooth approximation of the original function without sacrificing its valuable attributes.

Definition 1. 

We define the smoothing kernel $\varphi \left(x\right)$ as follows:

$$\varphi \left(x\right)=\left\{\begin{array}{c}A{e}^{{\displaystyle \frac{1}{x^2-1}}},\left|x\right|<1\\ 0,\left|x\right|\ge 1\end{array}\right.$$

Here, constant A is defined as

$$A={\left({\int }_{-1}^1{e}^{{\displaystyle \frac{1}{x^2-1}}}dx\right)}^{-1}$$

Proposition 1. 

The smoothing kernel $\varphi \left(x\right)$ is normalized, i.e.,  ${\int }_{\mathbb{R}}\varphi \left(u\right)du=1$.

Proposition 2. 

$\varphi \left(x\right)\in C^{\infty }\left(\mathbb{R}\right)$, i.e., $\varphi \left(x\right)$ is infinitely differentiable on $\mathbb{R}$ and ${\varphi }^{\left(k\right)}\left(x\right)=0,$ $\left|x\right|\ge 1$. Furthermore, for each natural number k, ${\varphi }^{\left(k\right)}\left(x\right)$ is bounded on $\mathbb{R}$.

Detailed proofs of Propositions 1 and 2 can be found in Appendix A.

Definition 2. 

For any $\delta >0$ we define a family of smoothing kernels ${\left\{{\varphi }_{\delta }\right\}}_{\delta >0}$ generated by scaling the smoothing kernel $\varphi \left(x\right)$ as follows:

$${\varphi }_{\delta }\left(x\right)={\displaystyle \frac{1}{\delta }}\varphi \left({\displaystyle \frac{x}{\delta }}\right)$$

This family of functions has several important properties. Each ${\varphi }_{\delta }\left(x\right)$ is normalized, maintaining ${\int }_{\mathbb{R}}{\varphi }_{\delta }\left(x\right)dx=1$, and its support is scaled to the interval $[-\delta ,\delta ]$. As $\delta \to 0$, the family ${\varphi }_{\delta }\left(x\right)$ converges to the Dirac delta function in the sense of distributions. We can now employ this framework to smooth the target activation function.

4.3. Smoothness of the Mollified Activation Function

Definition 3. 

Let $f\left(x\right)$ be a locally integrable activation function. The convolution of $f\left(x\right)$ with the smoothing kernel ${\varphi }_{\delta }$ is called a smoothing of $f\left(x\right)$, denoted by $f_{\delta }\left(x\right)$ or $f\ast {\varphi }_{\delta }$, and defined as:

$$f_{\delta }\left(x\right)=(f\ast {\varphi }_{\delta })\left(x\right)={\int }_{\mathbb{R}}f\left(y\right){\varphi }_{\delta }(x-y)dy={\int }_{\mathbb{R}}f(x-y){\varphi }_{\delta }\left(y\right)dy$$

Remark 1. 

It is important to clarify that while the smoothed activation is defined via a convolution integral, this operation is performed analytically during the derivation phase (as shown later in Section 4.6). Consequently, the actual implementation relies on the resulting closed-form polynomial expression, avoiding the need for expensive numerical integration during training or inference. Thus, the convolution-based definition provides theoretical rigor without imposing a runtime computational burden.

Let $H(x,y)=f\left(y\right){\varphi }_{\delta }(x-y)$. Taking partial derivative of it, we can get:

$${\displaystyle \frac{{\partial }^k}{\partial x^k}}H(x,y)={\displaystyle \frac{{\partial }^k}{\partial x^k}}\left[f\left(y\right){\varphi }_{\delta }(x-y)\right]=f\left(y\right){\displaystyle \frac{{\partial }^k}{\partial x^k}}{\varphi }_{\delta }(x-y)=f\left(y\right){\varphi }_{\delta }^{\left(k\right)}(x-y)$$

This partial derivative exists for any $k\ge 1$, because ${\varphi }_{\delta }$ is a $C^{\infty }$ function according to Proposition 2. Let $x_0\in \mathbb{R}$ be a fixed, arbitrary point. We will consider values of x within a neighborhood of $x_0$. The function ${\varphi }_{\delta }^{\left(k\right)}$ is continuous and has compact support, which implies that it is bounded on the $\mathbb{R}$. Let

$$M_k=\underset{z\in \mathbb{R}}{sup}\left|{\varphi }_{\delta }^{\left(k\right)}\left(z\right)\right|<\infty$$

(1)

The support of the integrand with respect to y is the interval $[x-\delta ,x+\delta ]$. For any x in the chosen neighborhood, this interval is contained in the larger compact set $K=[x_0-1-\delta ,x_0+1+\delta ]$. From Equation (1), we obtain the following bound:

$$\left|{\displaystyle \frac{{\partial }^k}{\partial x^k}}H(x,y)\right|=\left|f\left(y\right){\varphi }_{\delta }^{\left(k\right)}(x-y)\right|\le \left|f\left(y\right)\right|·M_k·{\chi }_K\left(y\right)$$

where ${\chi }_K$ is the characteristic function of the set K, its specific form is as follows:

$${\chi }_K\left(y\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}y\in K\hfill \\ 0,\hfill & \mathrm{if}y\notin K\hfill \end{array}\right.$$

Since $f\in L_{\mathrm{loc}}^1\left(\mathbb{R}\right)$, it is integrable on the compact set K. This implies that the dominating function $\left|f\left(y\right)\right|·M_k·{\chi }_K\left(y\right)$ is integrable. This argument shows that for any order k, the control conditions required for the differential operator ${\displaystyle \frac{d^k}{d{x}^k}}$ are satisfied. According to the Lebesgue Dominated Convergence Theorem, we can move any order differential operator into the integral sign. Therefore, we can directly calculate the k-th derivative of $f_{\delta }$:

$$\begin{array}{cc}\hfill f_{\delta }^{\left(k\right)}\left(x\right)& ={\displaystyle \frac{d^k}{d{x}^k}}{f}_{\delta }\left(x\right)={\displaystyle \frac{d^k}{d{x}^k}}{\int }_{\mathbb{R}}f\left(y\right){\varphi }_{\delta }(x-y)dy={\int }_{\mathbb{R}}{\displaystyle \frac{{\partial }^k}{\partial x^k}}\left[f\left(y\right){\varphi }_{\delta }(x-y)\right]dy\hfill \\ & ={\int }_{\mathbb{R}}f\left(y\right){\varphi }_{\delta }^{\left(k\right)}(x-y)dy=(f\ast {\varphi }_{\delta }^{\left(k\right)})\left(x\right)\hfill \end{array}$$

This implies that the k-th derivative of $f_{\delta }$ exists for any positive integer k. Let

$$g_k\left(x\right):=f_{\delta }^{\left(k\right)}\left(x\right)=(f\ast {\varphi }_{\delta }^{\left(k\right)})\left(x\right)$$

Since the function ${\varphi }_{\delta }^{\left(k\right)}\left(x\right)$ is continuous with compact support, it is uniformly continuous on $\mathbb{R}$. Consider an arbitrary $h\in \mathbb{R}$, we then have

$$\begin{array}{cc}\hfill |g_k(x+h)-g_k\left(x\right)|& =\left|{\int }_{\mathbb{R}}f\left(y\right)[{\varphi }_{\delta }^{\left(k\right)}(x+h-y)-{\varphi }_{\delta }^{\left(k\right)}(x-y)]dy\right|\hfill \\ \hfill & \le {\int }_{\mathbb{R}}\left|f\left(y\right)\right|·|{\varphi }_{\delta }^{\left(k\right)}(x+h-y)-{\varphi }_{\delta }^{\left(k\right)}(x-y)|dy\hfill \end{array}$$

(2)

Then we establish a dominating function for the integrand. By the triangle inequality,

$$\left|f\left(y\right)\right|·|{\varphi }_{\delta }^{\left(k\right)}(x+h-y)-{\varphi }_{\delta }^{\left(k\right)}(x-y)|\le |f\left(y\right)|·(|{\varphi }_{\delta }^{\left(k\right)}(x+h-y)|+|{\varphi }_{\delta }^{\left(k\right)}(x-y)\left|\right)\le \left|f\left(y\right)\right|·2M_k·{\chi }_K\left(y\right)$$

Due to the uniform continuity of ${\varphi }_{\delta }^{\left(k\right)}$, when $h\to 0$, we have

$$\underset{h\to 0}{lim}|{\varphi }_{\delta }^{\left(k\right)}(x+h-y)-{\varphi }_{\delta }^{\left(k\right)}(x-y)|=0$$

(3)

Now we can apply Equation (3) and the Dominated Convergence Theorem to the Equality (2):

$$\underset{h\to 0}{lim}|g_k(x+h)-g_k\left(x\right)|\le {\int }_{\mathbb{R}}\underset{h\to 0}{lim}\left(\left|f\left(y\right)\right|·|{\varphi }_{\delta }^{\left(k\right)}(x+h-y)-{\varphi }_{\delta }^{\left(k\right)}(x-y)|\right)dy=0$$

Thus, we have proven that for any $k\ge 0$, the derivative $f_{\delta }^{\left(k\right)}\left(x\right)$ is continuous, which implies $f_{\delta }\left(x\right)\in C^{\infty }\left(\mathbb{R}\right)$. This establishes that the mollification process transforms the original activation function $f\left(x\right)$ into a smooth approximation $f_{\delta }\left(x\right)$.

4.4. Approximation Properties of the Mollified Activation Function

While achieving smoothness, a crucial question arises: to what degree does the new function $f_{\delta }\left(x\right)$ retain the core properties of the original function $f\left(x\right)$? Have we “distorted” the essence of the activation function in pursuit of smoothness? Let us discuss these questions next.

To analyze the approximation error, we start with the term $\left|f_{\delta }\left(x\right)-f\left(x\right)\right|$. Since the smoothing kernel integrates to 1, we can rewrite $f\left(x\right)$ in the following form:

$$f\left(x\right)=f\left(x\right)·1=f\left(x\right){\int }_{\mathbb{R}}{\varphi }_{\delta }\left(u\right)du={\int }_{\mathbb{R}}f\left(x\right){\varphi }_{\delta }\left(u\right)du$$

(4)

For the term $f_{\delta }\left(x\right)$, recall its definition and perform a substitution by letting $u=x-y$, which implies $y=x-u$. Thus,

$$f_{\delta }\left(x\right)={\int }_{-\infty }^{\infty }f\left(y\right){\varphi }_{\delta }(x-y)dy={\int }_{\infty }^{-\infty }f(x-u){\varphi }_{\delta }\left(u\right)(-du)={\int }_{-\infty }^{\infty }f(x-u){\varphi }_{\delta }\left(u\right)du$$

(5)

Combining Equations (4) and (5), we get:

$$|f_{\delta }\left(x\right)-f\left(x\right)|=\left|{\int }_{-\infty }^{\infty }f(x-u){\varphi }_{\delta }\left(u\right)du-{\int }_{-\infty }^{\infty }f\left(x\right){\varphi }_{\delta }\left(u\right)du\right|=\left|{\int }_{-\infty }^{\infty }[f(x-u)-f\left(x\right)]{\varphi }_{\delta }\left(u\right)du\right|$$

According to the triangle inequality for integrals and the fact that ${\varphi }_{\delta }\left(u\right)\ge 0$:

$$|f_{\delta }\left(x\right)-f\left(x\right)|\le {\int }_{-\infty }^{\infty }|f(x-u)-f\left(x\right)|{\varphi }_{\delta }\left(u\right)du={\int }_{-\delta }^{\delta }|f(x-u)-f\left(x\right)|{\varphi }_{\delta }\left(u\right)du$$

We consider the activation function $f:\mathbb{R}\to \mathbb{R}$ that is continuous. Let $D\subset \mathbb{R}$ be an arbitrary compact set. Since $u\in [-\delta ,\delta ]$, both x and $x-u$ lie within the larger compact set $D^{\prime }=D-[-\delta ,\delta ]=\{a-b\mid a\in D,b\in [-\delta ,\delta ]\}$. By the Heine-Cantor Theorem, a function that is continuous on a compact set is also uniformly continuous. Therefore, $f\left(x\right)$ is uniformly continuous on $D^{\prime }$. By the definition of uniform continuity, for any $ϵ>0$ given at the beginning, there must exist a $\Delta >0$ such that whenever $\left|u\right|<\Delta$, $\left|f\right(z-u)-f(z\left)\right|<ϵ$ holds for all $z\in D^{\prime }$. We choose our smoothing parameter $\delta$ to be less than $\Delta$ given by uniform continuity, i.e., $0<\delta <\Delta$. In this way, for all $u\in [-\delta ,\delta ]$ in our integral expression, we have $\left|u\right|\le \delta <\Delta$. Therefore, for these u, applying the property of uniform continuity yields the following:

$$\left|f\right(x-u)-f(x\left)\right|<ϵ$$

(6)

Substitute Equation (6) back into our integral:

$$|f_{\delta }\left(x\right)-f\left(x\right)|\le {\int }_{-\delta }^{\delta }|f(x-u)-f\left(x\right)|{\varphi }_{\delta }\left(u\right)du<{\int }_{-\delta }^{\delta }ϵ·{\varphi }_{\delta }\left(u\right)du=ϵ{\int }_{-\infty }^{\infty }{\varphi }_{\delta }\left(u\right)du=ϵ$$

In summary, we have proven that for any $ϵ>0$, there exists a $\Delta >0$ such that whenever $0<\delta <\Delta$, then $|f_{\delta }\left(x\right)-f\left(x\right)|<ϵ$ holds for all $x\in D$. From this, we can conclude the uniform convergence of the smoothed activation function $f_{\delta }\left(x\right)$ to the original activation function $f\left(x\right)$ on the set D.

By choosing a sufficiently small smoothing parameter $\delta$, we can make the smoothed activation function approximate the original function arbitrarily accurately. Uniform convergence plays a key role by ensuring that the smoothed activation function maintains the desirable characteristics of the original. It also ensures that the maximum error between the two functions approaches zero over any input interval of interest, thus avoiding unexpected deviations in specific regions.

4.5. Lipschitz Continuity Analysis

In the field of deep learning, an abstract mathematical concept—Lipschitz continuity—is increasingly becoming a key factor in building more reliable, robust, and generalizable neural network models. From theoretical analysis to practical applications, Lipschitz continuity provides a powerful tool for deeply understanding and effectively controlling the behavior of deep networks. Lipschitz continuity offers a stronger condition than standard continuity by constraining a function’s maximum rate of change. To understand this property, we will start with its formal definition.

Definition 4 

(Lipschitz Continuous Function [29]). A function $f:{\mathbb{R}}^d\to {\mathbb{R}}^K$ with domain $dom\left(f\right)\subseteq {\mathbb{R}}^d$ is said to be C-Lipschitz continuous with respect to an α-norm for some constant $C>0$ if the following condition holds for all $x,y\in dom\left(f\right)$:${\parallel f\left(x\right)-f\left(y\right)\parallel }_{\alpha }\le C{\parallel x-y\parallel }_{\alpha }$.

In our analysis, we concentrate on the smallest value of C that satisfies the aforementioned condition. This value is formally defined as the Lipschitz constant. Lipschitz continuity of the activation function is crucial for ensuring well-behaved optimization, thereby promoting efficient convergence during training. Ensuring model stability by controlling the Lipschitz property is an effective way to prevent gradient runaway [30,31,32,33]. The smaller the Lipschitz constant, the more stable the training gradients [29,34].

From the previous derivation, we know that the support set of the smoothing kernel function ${\varphi }_{\delta }\left(x\right)$ is $\left[-\delta ,\delta \right]$. The support set of a function derivative must be contained within the support set of the original function. Therefore, the support set of ${\varphi }_{\delta }^{\prime }\left(x\right)$ is also contained in $[-\delta ,\delta ]$. So we have

$$f_{\delta }^{\prime }\left(x\right)=(f\ast {\varphi }_{\delta }^{\prime })\left(x\right)={\int }_{-\infty }^{\infty }f\left(y\right){\varphi }_{\delta }^{\prime }(x-y)dy={\int }_{x-\delta }^{x+\delta }f\left(y\right){\varphi }_{\delta }^{\prime }(x-y)dy$$

For a continuous function $f\left(x\right)$, it must be bounded within the closed interval $[x-\delta ,x+\delta ]$, so we can find a constant $M>0$ that satisfies the condition $\left|f\right(y\left)\right|\le M$. Substitute into the above equation and simplify:

$$|f_{\delta }^{\prime }\left(x\right)|\le {\int }_{x-\delta }^{x+\delta }\left|f\left(y\right)\right||{\varphi }_{\delta }^{\prime }(x-y)|dy\le M{\int }_{x-\delta }^{x+\delta }\left|{\varphi }_{\delta }^{\prime }(x-y)\right|dy$$

(7)

Now let us calculate the integral ${\int }_{x-\delta }^{x+\delta }\left|{\varphi }_{\delta }^{\prime }(x-y)\right|dy$. We know that ${\varphi }_{\delta }^{\prime }\left(u\right)={\displaystyle \frac{1}{{\delta }^2}}{\varphi }^{\prime }\left({\displaystyle \frac{u}{\delta }}\right)$, by variable substitution $v=(x-y)/\delta$:

$${\int }_{x-\delta }^{x+\delta }|{\varphi }_{\delta }^{\prime }(x-y)|dy={\int }_{x-\delta }^{x+\delta }{\displaystyle \frac{1}{{\delta }^2}}\left|{\varphi }^{\prime }\left({\displaystyle \frac{x-y}{\delta }}\right)\right|dy={\displaystyle \frac{1}{\delta }}{\int }_{-1}^1\left|{\varphi }^{\prime }\left(v\right)\right|dv$$

(8)

The value of this integral ${\int }_{-1}^1\left|{\varphi }^{\prime }\left(v\right)\right|dv$ is completely determined by the kernel function $\varphi \left(x\right)$ we initially chose. It is a constant that is independent of f, $\delta$, and x; we denote this constant as $C_{\varphi }$. Combining Equations (7) and (8), we can conclude that

$$|f_{\delta }^{\prime }\left(x\right)|\le M·\left({\displaystyle \frac{1}{\delta }}{C}_{\varphi }\right)={\displaystyle \frac{M·C_{\varphi }}{\delta }}$$

This shows that the Lipschitz constant of $f_{\delta }\left(x\right)$, $L_{\delta }$, satisfies: $L_{\delta }\le {\displaystyle \frac{M·C_{\varphi }}{\delta }}$. An increase in the value of $\delta$ leads to a broader refinement range for the original activation function, consequently enhancing its overall smoothness. Additionally, increasing the value of $\delta$ lowers the Lipschitz upper bound of the enhanced activation function, thereby promoting a more stable training gradient. Therefore, we can conclude: the higher the smoothness, the higher the training gradient stability, and there is a quantitative relationship between the two, where the specific quantitative relationship is determined by constants M and $C_{\varphi }$. This conclusion provides a principled guiding framework for the design of activation functions in the future.

4.6. Reshaping Relu(S-Relu): From Relu to Better

In Section 4.1, we provide evidence that smoother activation functions are advantageous for enhancing both the efficiency of the training process and the final performance of the model. Section 4.3 and Section 4.4 further establish that activation functions constructed via smoothing theory possess desirable smoothness and approximation properties; specifically, they are infinitely differentiable and can approximate the original activation function arbitrarily closely, thereby preserving its favorable characteristics. Building upon these findings, we next apply the proposed methodology to a concrete instance. In particular, we select the widely used ReLU function as the basis for mollification. Regarding the choice of kernel, we strike a balance between theoretical rigor and engineering practicality by adopting the Epanechnikov kernel, which is theoretically optimal for minimizing the Mean Integrated Squared Error. While the Gaussian kernel is a common choice for smoothing, we select the Epanechnikov kernel for two strategic reasons. First, convolving a ReLU with a Gaussian kernel results in expressions involving the Error Function (erf) or exponentials, which are computationally expensive transcendental operations. In contrast, the Epanechnikov kernel is a polynomial; its convolution with ReLU yields a simple piecewise polynomial (S-ReLU), which can be computed using only basic arithmetic operations. Second, the Epanechnikov kernel has a finite support $[-\delta ,\delta ]$. This ensures that the smoothing effect is strictly local—S-ReLU is exactly linear (preserving ReLU’s properties) outside this interval. Gaussian smoothing, with its infinite support, asymptotically alters the function over the entire domain, theoretically sacrificing the strict linearity and sparsity of the original ReLU.

The standard ReLU activation function computes the maximum of zero and its input, as given by $f\left(x\right)=\mathrm{ReLU}\left(x\right)=max(0,x)$. We select the Epanechnikov kernel function, denoted as ${\varphi }_{\delta }\left(x\right)$, parameterized by the smoothing radius $\delta$. This kernel acts as a weighting function with compact support on the interval $[-\delta ,\delta ]$. Its normalized form is:

$${\varphi }_{\delta }\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{3}{4\delta }}\left(1-{\displaystyle \frac{x^2}{{\delta }^2}}\right)\hfill & ,\mathrm{if}\left|x\right|\le \delta \hfill \\ 0\hfill & ,\mathrm{if}\left|x\right|>\delta \hfill \end{array}\right.$$

Applying the preceding theory yields the smoothed activation function S-ReLU:

$$f_{\delta }\left(x\right)=(f\ast {\varphi }_{\delta })\left(x\right)={\int }_{R}max(0,y){\varphi }_{\delta }(x-y)dy=\left\{\begin{array}{cc}0\hfill & ,\mathrm{if}x\le -\delta \hfill \\ {\displaystyle \frac{x}{2}}+{\displaystyle \frac{3x^2}{8\delta }}+{\displaystyle \frac{3\delta }{16}}-{\displaystyle \frac{x^4}{16{\delta }^3}}\hfill & ,\mathrm{if}-\delta <x<\delta \hfill \\ x\hfill & ,\mathrm{if}x\ge \delta \hfill \end{array}\right.$$

Remark 2. 

While introducing smoothness often incurs a computational premium, the proposed S-ReLU minimizes this overhead through its design. Unlike widely used smooth activations such as GELU [20] or SiLU [18], which rely on computationally expensive transcendental functions, S-ReLU is constructed as a piecewise polynomial function. Within the smoothing interval $(-\delta ,\delta )$, it requires only basic arithmetic operations (addition, multiplication) and avoids the latency associated with approximating infinite series. Consequently, S-ReLU maintains a computational efficiency comparable to leaky ReLU variants, making it highly suitable for both large-scale training and latency-sensitive inference scenarios.

Differentiating the S-ReLU function twice yields the following:

$$f_{\delta }^{\prime \prime }\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{3({\delta }^2-x^2)}{4{\delta }^3}}\hfill & ,\mathrm{if}-\delta <x<\delta \hfill \\ 0\hfill & ,\mathrm{if}\left|x\right|\ge \delta \hfill \end{array}\right.$$

This indicates that S-ReLU is an activation function with continuous second derivatives, which meets the definition of a smooth function in Section 4.1. We specifically target second-order differentiability ($C^2$) rather than higher-order or infinite smoothness ($C^{\infty }$) for two pragmatic reasons. First, from an optimization perspective, $C^2$ continuity ensures a well-defined and continuous Hessian matrix, which is sufficient to guarantee stability of curvature-based optimization dynamics and avoid abrupt changes in the loss landscape. Second, achieving $C^{\infty }$ typically requires kernels with infinite support, which leads to computationally expensive transcendental functions. In contrast, restricting smoothness to $C^2$ allows us to utilize the Epanechnikov kernel, yielding a computationally efficient piecewise polynomial form while still satisfying the rigorous requirements for gradient stability. We use the uniform error metric to quantify how well S-ReLU approximates the target function. The calculation result is given as follows:

$$\left|\right|\mathrm{S}-\mathrm{ReLU}\left(x\right)-{\mathrm{ReLU}\left(x\right)\left|\right|}_{\infty }=\underset{x\in \mathbb{R}}{sup}|\mathrm{S}-\mathrm{ReLU}\left(x\right)-\mathrm{ReLU}\left(x\right)|={\displaystyle \frac{3\delta }{16}}$$

This result clearly demonstrates that, compared to ReLU, the approximation error of S-ReLU is controllable and proportional to the smoothing radius $\delta$. This is a valuable property, as it allows us to precisely control the extent of the approximation error introduced by the smoothing operation by choosing the value of $\delta$. A detailed proof is provided in Appendix B. Further details regarding S-ReLU are available in the appendices. Specifically, Appendix C presents the Python-style pseudocode, while Appendix D contains a more in-depth discussion of its characteristics. Finally, we discuss Lipschitz continuity. Based on the theory in Section 4.5, we can calculate the Lipschitz constant for each activation function.

Fact 1. 

Lipschitz constant of GELU is 1.084; Lipschitz constant of SiLU is 1.100; Lipschitz constant of Mish is 1.089. Lipschitz constant of S-ReLU is 1.000.

Remark 3. 

The Lipschitz constants presented in Fact 1 reveal a fundamental structural advantage of S-ReLU. Theoretically, an activation function f with a Lipschitz constant $L>1$ (such as GELU and SiLU) acts as an expansive operator. In deep neural networks, where gradients are computed via the chain rule involving products of layer Jacobians, an expansive activation can cause the gradient magnitude to grow exponentially with depth d (upper bounded by $O\left(L^d\right)$), potentially leading to gradient explosion or optimization instability. In contrast, S-ReLU guarantees $L=1.000$, making it a strictly non-expansive operator. This ensures that the gradient norm is never mathematically amplified by the activation layer itself ($\parallel \nabla f\left(x\right)\parallel \le \parallel \nabla x\parallel$). This property provides a quantifiable theoretical guarantee for training stability in deep architectures, distinguishing S-ReLU from existing smooth variants.

The detailed proof can be found in Appendix E. S-ReLU’s Lipschitz constant of 1 ensures that its output never changes more rapidly than its input. This property acts as a vital safeguard for stabilizing gradient flow in the network, which helps prevent the problem of exploding gradients and makes the model more robust to small input perturbations—a clear advantage over other activation functions.

4.7. Theoretical Connection: From Mollification to Optimization

The mathematical properties derived from our mollification framework translate directly into the empirical improvements observed in the subsequent experiments. Specifically, the guarantee of $C^2$ continuity ensures a continuous Hessian matrix, which effectively smooths out the loss landscape. This regularity allows second-order information—often implicitly utilized by adaptive optimizers like Adam—to guide the optimization trajectory more robustly than is possible with the singular Hessian of non-smooth functions. Furthermore, by selecting the normalized Epanechnikov kernel, we mathematically constrain the Lipschitz constant to exactly 1.000. This ensures that S-ReLU acts as a strictly non-expansive operator, thereby preventing the exponential accumulation of gradient magnitudes in deep layers, which serves as a crucial theoretical advantage over expansive activations like GELU or SiLU. Finally, the uniform convergence property guarantees that S-ReLU approximates the representational sparsity of ReLU with a controllable error bound, allowing the model to retain the favorable linear regions of the original function while eliminating the optimization difficulties caused by the non-differentiable singularity at the origin.

5. Experiments

Experimental Setup. We conducted experiments on the CIFAR-10, CIFAR-100, and ImageNet-1K datasets to evaluate the effectiveness of S-ReLU for image classification, as well as on LLM fine-tuning with human preference datasets SHP [35], HH [36], and GPT-2 [22]. We compare against representative ReLU variants, including GELU, ELU, PReLU, CELU, SiLU, and Mish. Detailed training configurations and hyperparameters are provided in Appendix F. For image classification, we use CIFAR-10, CIFAR-100 [9], and ImageNet-1K [10], which cover small- to large-scale benchmarks with increasing category numbers and image resolutions. For LLM fine-tuning, we adopt the human preference datasets SHP [35] and HH [36], which are widely used for aligning model outputs with human judgments. We also fine-tune GPT-2 (137M parameters); given its moderate size, we perform full parameter fine-tuning rather than LoRA-based tuning. We compare S-ReLU against representative ReLU variants, including GELU [20], ELU [16], PReLU [15], CELU [17], SiLU [18], and Mish [19], as detailed in Section 2 and Section 3. All experiments were conducted on 4 × A100 GPUs using the AdamW optimizer (weight decay 0.05), truncated normal initialization, gradient clipping norm of 1.0, cross-entropy loss, and a cosine annealing learning rate scheduler with linear warmup. For all transformer- and CNN-based architectures, we set $\delta =0.001$ (see Section 5.3 for sensitivity analysis). Each experiment was repeated three times, and we report mean and standard deviation. Additional hyperparameter details are provided in Appendix F.

Datasets. In the experiments on image classification tasks, we use three datasets, ranked by the number of categories: CIFAR-10, CIFAR-100 [9], and ImageNet-1K [10]. In the experiments on the Large-Scale Language Model (LLM) fine-tuning task, we use two human preference datasets: SHP [35] and HH [36].

Baselines. We experimentally compared the performance of S-ReLU with several typical ReLU correction methods described in Section 2 and Section 3: GELU [20], ELU [16], PReLU [15], CELU [17], SiLU [18], and Mish [19].

Experimental hyperparameters. For all transformer-based and CNN-based architectures, we directly set $\delta$ to 0.001. See Section 5.3 for sensitivity analysis of parameter delta. Detailed experimental hyperparameters are provided in Appendix F.

5.1. Task of Image Classification

Evaluation of ViTs on CIFAR-10, CIFAR-100, and ImageNet-1K. To comprehensively evaluate the proposed activation, we conducted experiments on ViT, DeiT and TNT. CIFAR-10 and CIFAR-100 were selected to examine the sensitivity of activation functions under different data distributions, while ImageNet-1K, with its larger image resolution and broader category coverage, was employed to assess performance in more challenging large-scale scenarios.

As summarized in Table 1, S-ReLU consistently outperforms all existing ReLU variants across every dataset and architecture. The gains are evident on both small-scale benchmarks(CIFAR-10/100), where S-ReLU achieves markedly higher accuracy, and on the large-scale ImageNet-1K dataset, where it surpasses strong baselines under more demanding conditions. These results demonstrate not only the superior fitting ability of S-ReLU, but also its strong generalization capacity, showing that the advantages of smoothing are preserved across data regimes of different scales and complexities.

Table 1. Test accuracy on CIFAR-10, CIFAR-100, and ImageNet-1K over 100 epochs.

Dataset	Model	Activation Functions
		GELU	ELU	PReLU	CELU	SiLU	Mish	S-ReLU
CIFAR-10	ViT-Tiny	$70.4\pm 0.2$	$66.4\pm 0.5$	$78.0\pm 0.6$	$66.5\pm 0.6$	$68.6\pm 0.3$	$68.7\pm 0.3$	$\mathbf{81}.\mathbf{0}\pm \mathbf{0}.\mathbf{6}$
	DeiT-Tiny	$72.4\pm 0.7$	$67.6\pm 0.6$	$75.4\pm 0.1$	$67.7\pm 0.8$	$69.9\pm 0.5$	$70.2\pm 0.6$	$\mathbf{81}.\mathbf{1}\pm \mathbf{0}.\mathbf{3}$
	TNT-Small	$73.7\pm 0.5$	$69.5\pm 0.6$	$75.8\pm 0.3$	$68.7\pm 0.2$	$71.1\pm 0.7$	$71.6\pm 0.8$	$\mathbf{84}.\mathbf{8}\pm \mathbf{0}.\mathbf{2}$
	Average	$72.2\pm 0.5$	$67.8\pm 0.6$	$76.4\pm 0.3$	$67.6\pm 0.5$	$69.9\pm 0.5$	$70.2\pm 0.6$	$\mathbf{82}.\mathbf{3}\pm \mathbf{0}.\mathbf{4}$
CIFAR-100	ViT-Tiny	$32.6\pm 0.8$	$28.9\pm 0.1$	$43.2\pm 1.0$	$28.9\pm 0.2$	$31.2\pm 0.6$	$30.6\pm 0.8$	$\mathbf{51}.\mathbf{2}\pm \mathbf{0}.\mathbf{6}$
	DeiT-Tiny	$46.6\pm 0.9$	$56.9\pm 0.0$	$50.0\pm 0.5$	$40.5\pm 0.5$	$43.5\pm 0.6$	$43.8\pm 1.0$	$\mathbf{57}.\mathbf{1}\pm \mathbf{0}.\mathbf{2}$
	TNT-Small	$47.5\pm 0.8$	$43.6\pm 0.3$	$49.0\pm 0.7$	$43.0\pm 0.5$	$45.0\pm 0.9$	$45.5\pm 0.8$	$\mathbf{61}.\mathbf{6}\pm \mathbf{0}.\mathbf{4}$
	Average	$42.2\pm 0.8$	$43.1\pm 0.1$	$47.4\pm 0.7$	$37.5\pm 0.4$	$39.9\pm 0.7$	$40.0\pm 0.9$	$\mathbf{56}.\mathbf{6}\pm \mathbf{0}.\mathbf{3}$
ImageNet-1K	ViT-Tiny	$53.9\pm 0.3$	$37.2\pm 0.6$	$56.8\pm 0.3$	$37.6\pm 0.5$	$46.1\pm 0.7$	$46.9\pm 1.1$	$\mathbf{56}.\mathbf{6}\pm \mathbf{0}.\mathbf{4}$
	DeiT-Tiny	$61.7\pm 0.4$	$49.1\pm 0.7$	$60.8\pm 0.4$	$48.9\pm 0.8$	$58.5\pm 0.7$	$58.9\pm 0.3$	$\mathbf{64}.\mathbf{9}\pm \mathbf{0}.\mathbf{4}$
	Average	$57.8\pm 0.4$	$43.2\pm 0.7$	$58.8\pm 0.4$	$43.3\pm 0.7$	$52.3\pm 0.7$	$52.9\pm 0.7$	$\mathbf{60}.\mathbf{8}\pm \mathbf{0}.\mathbf{4}$

Evaluation of ConvNeXt on CIFAR-10, CIFAR-100 and ImageNet-1K. Beyond transformer families, we investigate the universality of S-ReLU in convolutional networks by testing it on ConvNeXt, a leading convolutional model designed with modern principles to rival Transformers. This setting provides a stringent test of whether the performance improvements of S-ReLU are tied to specific architectural choices or generalize broadly across models.

Experimental results on CIFAR-10 and CIFAR-100 with 100 training epochs demonstrate that ConvNeXt models equipped with S-ReLU achieve consistently higher classification accuracy than those with existing activation functions. Notably, on the more challenging ImageNet-1K benchmark, S-ReLU continues to surpass GELU, SiLU, Mish, and other baselines, establishing new performance levels for ConvNeXt. These results confirm that the advantages of S-ReLU are not restricted to Transformer-based architectures but extend robustly to convolutional networks as well. Together with our earlier findings on Vision Transformers, these results highlight that S-ReLU delivers both architectural universality and strong generalization ability (Table 2).

Table 2. Testaccuracy of experiments conducted on ConvNeXt-tiny for 100 epochs.

Dataset	Model	Activation Functions
		GELU	ELU	PReLU	CELU	SiLU	Mish	S-ReLU
CIFAR-10	ConvNeXt	64.9 ± 0.4	59.8 ± 0.5	64.6 ± 1.4	59.8 ± 0.5	60.6 ± 0.2	61.4 ± 0.4	89.9 ± 0.1
CIFAR-100	ConvNeXt	36.6 ± 0.3	30.3 ± 0.4	35.2 ± 0.5	30.5 ± 0.2	35.0 ± 0.9	35.3 ± 0.7	66.8 ± 0.1
ImageNet-1K	ConvNeXt	72.9 ± 0.3	71.7 ± 0.5	72.9 ± 0.5	71.8 ± 0.9	72.3 ± 0.7	72.8 ± 0.6	73.1 ± 0.2

5.2. Task of Large Language Model (LLM) Fine-Tuning

To test the generalizability of our proposed S-ReLU activation function outside of computer vision, we also evaluated its performance in the increasingly important field of Large Language Models. Specifically, we fine-tune GPT-2 on the SHP and HH datasets using DPO. Importantly, both datasets pose challenges of stability and nuanced representation, making them suitable testbeds for evaluating whether activation functions like S-ReLU can enhance optimization robustness and expressive capacity. Specifically, we employ the standard DPO objective function [11], which optimizes the policy ${\pi }_{\theta }$ directly from human preferences without explicit reward modeling. The loss function is defined as:

$${\mathcal{L}}_{DPO}({\pi }_{\theta };{\pi }_{\mathrm{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[log\sigma \left(\beta log{\displaystyle \frac{{\pi }_{\theta }\left(y_w\right|x)}{{\pi }_{\mathrm{ref}}\left(y_w\right|x)}}-\beta log{\displaystyle \frac{{\pi }_{\theta }\left(y_l\right|x)}{{\pi }_{\mathrm{ref}}\left(y_l\right|x)}}\right)\right]$$

where $y_w$ and $y_l$ denote the preferred and dispreferred completions, respectively, and $\beta$ is the KL-divergence penalty coefficient. Regarding the evaluation metrics reported in Table 3, we define the Implicit Reward for a given input x and output y as $r(x,y)=\beta log{\displaystyle \frac{{\pi }_{\theta }\left(y\right|x)}{{\pi }_{\mathrm{ref}}\left(y\right|x)}}$. Chosen Reward and Rejected Reward refer to the average implicit rewards assigned to the ground-truth preferred ($y_w$) and dispreferred ($y_l$) responses in the test set. Preference Accuracy is calculated as the percentage of cases where the model assigns a higher implicit reward to the chosen response than to the rejected one. Because DPO relies on a reference strategy that may diverge from the true data distribution, we begin with supervised fine-tuning (SFT) to reduce this gap and then apply DPO with different penalty coefficients $\beta \in 0.1,2,5$.

Table 3. Metrics comparison between S-RELU and GELU in the task of LLM fine-tuning.

$\mathit{\beta }$ Value	Function	Evaluation Metrics
		Chosen Reward	Rejected Reward	Margin Reward ↑	Preference Accuracy ↑
$\beta =0.1$	S-RELU	0.1501 ± 0.0005	0.0517 ± 0.0003	0.0984 ± 0.0008	0.5938 ± 0.0000
	GELU	−0.2037 ± 0.0012	−0.2696 ± 0.0007	0.0659 ± 0.0019	0.5313 ± 0.0012
$\beta =2$	S-RELU	0.3861 ± 0.0018	0.3329 ± 0.0012	0.0532 ± 0.0030	0.5156 ± 0.0005
	GELU	−1.6600 ± 0.0003	−1.7060 ± 0.0052	0.0460 ± 0.0055	0.5080 ± 0.0022
$\beta =5$	S-RELU	1.1062 ± 0.0025	-0.7602 ± 0.0016	1.8664 ± 0.0041	0.5103 ± 0.0017
	GELU	−3.6484 ± 0.0031	−4.8299 ± 0.0003	1.1815 ± 0.0034	0.5012 ± 0.0001

Table 3 reports the mean and standard deviation for each evaluation metric, averaged over several experimental runs. Compared with GELU, S-ReLU consistently achieves higher chosen rewards, lower rejected rewards, larger reward margins, and improved preference accuracy. These results indicate that S-ReLU not only enhances the model’s ability to assign higher utility to preferred responses while penalizing non-preferred ones, but also strengthens its discriminative margin and alignment with human feedback. The consistent performance gains on all metrics confirm that introducing smoothness is an effective strategy, especially since it maintains the representational capabilities of the initial activation function. This demonstrates that the advantages of S-ReLU generalize robustly from vision tasks to LLM preference optimization, thereby confirming its potential as a broadly applicable activation function for deep learning.

5.3. Sensitivity Analysis

In this section, we focus on the impact of different $\delta$ on the final results. We set $\delta$ at 0.0001, 0.001, 0.01, 0.1, 0.2, 0.5, 1, 5, and 10, and perform experiments on the CIFAR10 and CIFAR100 datasets using ViT-tiny. We perform three runs and report the mean and standard deviation in Table 4.

Table 4. Test accuracy of ViT on CIFAR-10 with different sensitivity parameters for 100 epochs.

$\mathit{\delta }$-Value	0.0001	0.001	0.01	0.1	0.2	0.5	1	5	10
CIFAR-10	80.2 ± 0.3	81.0 ± 0.6	80.1 ± 0.2	80.4 ± 0.3	78.2 ± 0.2	80.4 ± 0.2	73.9 ± 0.3	63.4 ± 0.5	58.5 ± 0.3
CIFAR-100	50.3 ± 0.2	51.2 ± 0.6	48.7 ± 0.3	48.7 ± 0.5	48.5 ± 0.5	45.5 ± 0.3	41.4 ± 0.6	43.8 ± 1.0	25.7 ± 0.2

The sensitivity analysis presented reveals that S-ReLU achieves peak performance at $\delta =0.001$ across both CIFAR-10 ($81.0\%$) and CIFAR-100 ($51.2\%$), with performance remaining remarkably robust within the magnitude range of ${10}^{-4}$ to ${10}^{-1}$. While increasing $\delta$ beyond $0.5$ causes a theoretically consistent drop in accuracy due to excessive smoothing of the rectified non-linearity essential for feature representation, the stability observed in the lower range confirms that S-ReLU is not hypersensitive to hyperparameter perturbations. Consequently, the consistent optimality of $\delta \approx 0.001$ across datasets of varying complexity establishes it as a reliable universal default, thereby eliminating the need for extensive grid search and significantly enhancing the practical usability of the activation function for real-world deployments.

5.4. Empirical Verification of Gradient Stability

In Section 4.5, we theoretically derived that the Lipschitz constant of S-ReLU is inversely proportional to the smoothing parameter $\delta$, specifically bounded by $L_{\delta }\propto 1/\delta$. This theoretical insight implies that increasing the smoothing radius should mathematically constrain the maximum rate of change of the activation function, thereby suppressing the magnitude of gradients during backpropagation. To empirically validate this property and quantify the relationship between smoothness and optimization stability, we monitored the average global $L_2$ Gradient Norm of the ViT model trained on CIFAR-10 over 100 epochs under varying $\delta$ configurations, with the detailed results reported in Table 5.

Table 5. Test gradient norm of ViT on CIFAR-10 with different sensitivity parameters for 100 epochs.

$\delta$-Value	0.001	0.01	0.1	0.5	1	5	10
Gradient Norm	5.25	4.49	4.02	3.71	3.66	1.90	1.49

The experimental data demonstrates a strict inverse correlation between the smoothing parameter and the gradient norm, providing strong empirical corroboration for our theoretical analysis. Specifically, at a small smoothing radius of $\delta =0.001$, the gradient norm remains relatively high at $5.25$, exhibiting behavior similar to the standard ReLU which permits sharper gradient updates; however, as $\delta$ increases to $0.5$ and beyond, the gradient norm decreases monotonically, dropping to $3.71$ at $\delta =0.5$ and further to $1.49$ at $\delta =10$. These findings confirm that $\delta$ effectively acts as a control knob for optimization dynamics, where a larger $\delta$ enforces a tighter Lipschitz constraint to suppress gradient magnitudes and promote stability, whereas a smaller $\delta$ allows for more aggressive updates akin to non-smooth activations.

5.5. Computational Efficiency Analysis

To evaluate the practical deployment efficiency of S-ReLU relative to standard baselines, we conducted a rigorous micro-benchmark. We measured the average execution time for forward and backward passes, as well as the peak memory footprint with the detailed results summarized in Table 6.

Table 6. Test Computational Efficiency of ViT on CIFAR-10 for 100 epochs.

Computational Cost	Model	Activation Functions
		GELU	ELU	PReLU	CELU	SiLU	Mish	S-ReLU
Forward Time (ms)	Vit	59.2	57.7	57.5	53.3	54.2	56.2	55.2
Backward Time (ms)	Vit	148.0	155.8	189.5	154.9	154.6	148.8	152.9
Peak Memory (GB)	Vit	11.53	11.07	9.50	11.06	12.25	11.05	11.12

The benchmarking results highlight that S-ReLU achieves a highly competitive computational profile, demonstrating superior or comparable efficiency to existing state-of-the-art activations. Specifically, in the forward pass, S-ReLU (55.2 ms) outperforms widely used functions such as GELU (59.2 ms), ELU (57.7 ms), and PReLU (57.5 ms); this efficiency gain is primarily attributed to S-ReLU’s piecewise polynomial formulation, which avoids the latency associated with computationally expensive transcendental operations. Furthermore, in terms of backward propagation and memory usage, S-ReLU maintains performance parity with efficient baselines like SiLU and CELU while avoiding the significant computational overhead observed in PReLU (which requires 189.5 ms for the backward pass), thereby confirming that the proposed smoothing framework incurs negligible additional cost and is well-suited for high-throughput practical applications.

6. Discussion

Developing effective activation functions has long been a central problem in machine learning. While non-smooth activations suffer from well-documented limitations, smooth activations have emerged as a promising direction. Yet, a general methodology for systematically smoothing non-smooth activations remains absent. In this work, we introduce a mathematically rigorous and practically effective framework based on mollification theory to smooth non-smooth activations while preserving their desirable properties. Within this framework, we establish a quantitative relationship between smoothness and gradient stability, offering a theoretical foundation for advancing activation function design. Building on this, we derive a new activation function, S-ReLU, as a polished variant of ReLU. Across image classification and LLM fine-tuning tasks, S-ReLU consistently outperforms existing rectified ReLU variants. Our findings position S-ReLU as a strong new member of the family of high-performance activation functions, while also opening avenues for future research on smooth and stable architectures.

Limitations and Future Work

The results point to both theoretical and practical opportunities for advancing activation research. First, we employed the Epanechnikov kernel as the smoothing kernel due to its strong theoretical characterization, but alternative choices may yield improvements. In particular, shifting kernel design from fixed theoretical selection to dynamic, data-driven, or learnable formulations could provide greater flexibility and performance. Second, the applicability of mollification to other architectures, such as Kolmogorov–Arnold Networks (KANs), remains an open question. Third, although this work establishes the link between smoothness and gradient stability and analyzes the Lipschitz constant, deeper theoretical investigations are needed to understand the impact of smoothing on representational capacity, generalization bounds, and convergence. Fourth, beyond fixed or learnable parameterizations, the temporal scheduling of the smoothing parameter $\delta$ offers a promising direction. Similarly to learning rate annealing, a dynamic strategy—starting with a larger $\delta$ to ensure a convex, easy-to-optimize landscape during the early training phase, and gradually annealing it towards zero to recover the sparsity and efficiency of ReLU—could potentially combine the optimization benefits of smoothing with the representational sparsity of the original function. We plan to investigate such scheduling strategies in future work. Then, while S-ReLU demonstrates strong empirical results in image classification and large-scale language model fine-tuning, its extension to broader domains holds promise for advancing future studies. Finally, while this study validates S-ReLU across the currently dominant backbones—including CNNs, Vision Transformers, and LLMs—we acknowledge that the landscape of deep learning is vast. Future work could extend the evaluation of S-ReLU to other specialized architectures, such as Diffusion Models for generative tasks or RNNs for time-series forecasting, to further verify its universality.