Batch Gradient Learning Algorithm with Smoothing $L_1$ Regularization for Feedforward Neural Networks

Abstract

Regularization techniques are critical in the development of machine learning models. Complex models, such as neural networks, are particularly prone to overfitting and to performing poorly on the training data. $L_1$ regularization is the most extreme way to enforce sparsity, but, regrettably, it does not result in an NP-hard problem due to the non-differentiability of the 1-norm. However, the $L_1$ regularization term achieved convergence speed and efficiency optimization solution through a proximal method. In this paper, we propose a batch gradient learning algorithm with smoothing $L_1$ regularization (BGS$L_1$) for learning and pruning a feedforward neural network with hidden nodes. To achieve our study purpose, we propose a smoothing (differentiable) function in order to address the non-differentiability of $L_1$ regularization at the origin, make the convergence speed faster, improve the network structure ability, and build stronger mapping. Under this condition, the strong and weak convergence theorems are provided. We used N-dimensional parity problems and function approximation problems in our experiments. Preliminary findings indicate that the BGS$L_1$ has convergence faster and good generalization abilities when compared with BG$L_{1/2}$, BG$L_1$, BG$L_2$, and BGS$L_{1/2}$. As a result, we demonstrate that the error function decreases monotonically and that the norm of the gradient of the error function approaches zero, thereby validating the theoretical finding and the supremacy of the suggested technique.

1. Introduction

Artificial neural networks (ANNs) are computational networks based on biological neural networks. These networks form the basis of the human brain’s structure. Similar to neurons in a human brain, ANNs also have neurons that are interconnected to one another through a variety of layers. These neurons are known as nodes. The human brain is made up of 86 billion nerve cells known as neurons. They are linked to 1000 s of other cells by axons. Dendrites recognize stimulation from the external environment as well as inputs from sensory organs. These inputs generate electric impulses that travel quickly through the neural network. A neuron can then forward the message to another neuron to address the issue or not forward it at all. ANNs are made up of multiple nodes that mimic biological neurons in the human brain (See Figure 1). A feedforward neural network (FFNN) is the first and simplest type of ANN, and now it contributes significantly and directly to our daily lives in a variety of fields, such as education tools, health conditions, economics, sports, and chemical engineering [1,2,3,4,5].

Figure 1. (a) The biological neural networks (b) The artificial neural network structure.

The most widely used learning strategy in FFNNs is the backpropagation method [6]. There are two methods for training the weights: batch and online [7,8]. In the batch method, the weights are modified after each training pattern is presented to the network, whereas in the online method, the error is accumulated during an epoch and the weights are modified after the entire training set is presented.

Overfitting in mathematical modeling is the creation of an analysis that is precisely tailored to a specific set of data and, thus, may fail to fit additional data or predict future findings accurately [9,10]. An overfitted model is one that includes more parameters than can be justified by the data [11]. Several techniques, such as cross-validation [12], early stopping [13], dropout [14], regularization [15], big data analysis [16], or Bayesian regularization [17], are used to reduce the amount of overfitting.

Regularization methods are frequently used in the FFNN training procedure and have been shown to be effective in improving generalization performance and decreasing the magnitude of network weights [18,19,20]. A term proportional to the magnitude of the weight vector is one of the simplest regularization penalty terms added to the standard error function [21,22]. Many successful applications have used various regularization terms, such as weight decay [23], weight elimination [24], elastic net regularization [25], matrix regularization [26], and nuclear norm regularization [27].

Several regularization terms are made up of the weights, resulting in the following new error function:

$$E\left(W\right)=\check{E}\left(W\right)+\lambda {\Vert W\Vert }_q^q$$

(1)

where $\check{E}\left(w\right)$ is the standard error function depending on the weights $w$, $\lambda$ is the regularization parameter, and ${\Vert ·\Vert }_q$ is the q-norm is given by

$${\Vert W\Vert }_q={\left({\left|w_1\right|}^q+{\left|w_2\right|}^q+\cdots +{\left|w_N\right|}^q\right)}^{\frac{1}{q}}$$

where ($0\le q\le 2$). The gradient descent algorithm is a popular method for solving this type of problem (1). The graphics of the $L_0$, $L_{1/2}$, $L_1$, $L_2$, elastic net, and $L_{\infty }$ regularizers in Figure 2 show the sparsity. The sparsity solution, as shown in Figure 2, is the first point at which the contours touch the constraint region, and this will coincide with a corner corresponding to a zero coefficient. It is obvious that the $L_1$ regularization solution occurs at a corner with a higher possibility, implying that it is sparser than the others. The goal of network training is to find $W^{\ast }$ so that $E\left(W\right)=\min E\left(W\right)$. The weight vectors’ corresponding iteration formula is

$$W^{new}=W-\eta \frac{dE\left(W\right)}{dW}$$

(2)

$L_0$ regularization has a wide range of applications in sparse optimization [28]. As the $L_0$ regularization technique is an NP-hard problem, optimization algorithms, such as the gradient method, cannot be immediately applied [29]. To address this issue, ref. [30] proposes smoothing $L_0$ regularization with the gradient method for training FFNN. According to the regularization concept, Lasso regression was proposed to obtain the sparse solution based on $L_1$ regularization to reduce the complexity of the mathematical model [31]. Lasso quickly evolved into a wide range of models due to its outstanding performance. To achieve maximally sparse networks with minimal performance degradation, neural networks with smoothed Lasso regularization were used [32]. Due to its oracle properties, sparsity, and unbiasedness, the $L_{1/2}$ regularizer has been widely utilized in various studies [33]. A novel method for forcing neural network weights to become sparser was developed by applying $L_{1/2}$ regularization to the error function [34]. $L_2$ regularization is one of the most common types of regularization since the 2-norm is differentiable and learning can be advanced using a gradient method [35,36,37,38]; with $L_2$ regularization, the weights provided are bounded [37,38]. As a result, $L_2$ regularization is useful for dealing with overfitting problems.

Figure 2. The sparsity property of different regularization (a) $L_0$ regularization, (b) $L_{1/2}$ regularization, (c) $L_1$ regularization, (d) $L_2$ regularization, (e) elastic net regularization, (f) $L_{\infty }$ regularization.

The batch update rule, which modifies all weights in each estimation process, has become the most prevalent. As a result, in this article, we concentrated on the gradient method with a batch update rule and smoothing $L_1$ regularization for FFNN. We first show that if certain Propositions 1–3 are met, the error sequence will be uniformly monotonous, and the algorithm will be weakly convergent during training. Secondly, if there are no interior points in the error function, the algorithm with weak convergence is strongly convergent with the help of Proposition 4. Furthermore, numerical experiments demonstrate that our proposed algorithm eradicates oscillation and increases the computation learning algorithm better than the standard $L_{1/2}$ regularization, $L_1$ regularization, $L_2$ regularization, and even the smoothing $L_{1/2}$ regularization methods.

The following is the rest of this paper. Section 2 discusses FFNN, the batch gradient method with $L_1$ regularization (BG$L_1$), and the batch gradient method with smoothing $L_1$ regularization (BGS$L_1$). The materials and methods are given in Section 3. Section 4 displays some numerical simulation results that back up the claims made in Section 3. Section 5 provides a brief conclusion. The proof of the convergence theorem is provided in “Appendix A”.

2. Network Structure and Learning Algorithm Methodology

2.1. Network Structure

A three-layer neural network based on error back-propagation is presented. Consider a three-layer structures consisting of $N$ input layers, $M$ hidden layers and 1 output layer. Given that $g :ℝ \to ℝ$ can be a transfer function for the hidden and output layers, this is typically, but not necessarily, a sigmoid function. Let $w_0={\left(w_{01},w_{02},\dots ,w_{0M}\right)}^T\in {ℝ}^M$ be the weight vector between all the hidden layers and the output layer, and let denoted $w_j={\left(w_{j1},w_{j2},\dots ,w_{jN}\right)}^T\in {ℝ}^N$ be the weight vector between all the input layers and the hidden layer $j\left(j=1,2,\dots ,M\right)$. To classify the offer, we write all the weight parameters in a compact form, i.e., $W=\left(w_0^T,w_1^T,\dots ,w_M^T\right)\in {ℝ}^{M+NM}$, and we have also given a matrix $V={\left(w_1,w_2,\dots ,w_M\right)}^T\in {\mathbb{M}}^{N\times M}$. Furthermore, we define a vector function

$$\mathit{F}\left(\mathit{x}\right)=\left(\mathit{f}({\mathit{x}}_{\mathbf{1}}\right),\mathit{f}({\mathit{x}}_{\mathbf{2}}),\dots ,\mathit{f}({\mathit{x}}_{\mathit{N}}){)}^{\mathit{T}}$$

(3)

where $x={(x_1,x_2,\dots ,x_N)}^T\in {ℝ}^N$. For any given input $\xi \in {ℝ}^M$ the output of the hidden neuron is $F\left(V \xi \right)$, and the final output of the network is

$$\mathit{y}=\mathit{f}\left({\mathit{w}}_{\mathbf{0}}·\mathit{F}\left(\mathit{V} \mathsf{\xi }\right)\right)$$

(4)

where $w_0·F\left(V \xi \right)$ represents the inner product between the vectors ${\mathit{w}}^{\mathbf{0}}$ and $F\left(V \xi \right)$.

2.2. Modified Error Function with Smoothing $L_1$ Regularization (BGS$L_1$)

Given that the training set is ${\{{\xi }^l,O^l\}}_{l=1}^L\subset {ℝ}^N\times ℝ$, where $O^l$ is the desired ideal output for the input ${\xi }^l$. The standard error function $E\left(W\right)$ without regularization term as following

$$\check{E}\left(W\right)=\frac{1}{2}{{\displaystyle \sum }}_{l=1}^L{\left[O^l-f\left(w_0·F\left(V{\xi }^l\right)\right)\right]}^2={{\displaystyle \sum }}_{l=1}^L{f}_l\left(w_0·F\left(V{\xi }^l\right)\right)$$

(5)

where $f_l\left(t\right)≔\frac{1}{2}{\left[O^l-f_l\left(t\right)\right]}^2$. Furthermore, the gradient of the error function is given by

$${\check{E}}_{w_0}\left(W\right)={{\displaystyle \sum }}_{l=1}^L{f}_l^{\prime }\left(w_0·F\left(V{\xi }^l\right)\right)F\left(V{\xi }^l\right)$$

(6)

$${\check{E}}_{w_j}\left(W\right)={{\displaystyle \sum }}_{l=1}^L{f}_l^{\prime }\left(w_0·F\left(V{\xi }^l\right)\right)w_{0l}{f}_l^{\prime }\left(w_j·{\xi }^l\right){\xi }^l$$

(7)

The modified error function $E\left(W\right)$ with $L_1$ regularization is given by

$$E\left(W\right)=\check{E}\left(W\right)+\lambda {{\displaystyle \sum }}_{j=1}^M\left|w_j\right|$$

(8)

where $\left|W \right|$ denoted the absolute value of the weights. The purpose of the network training is to find $W^{\ast }$ such that

$$E \left(W^{\ast }\right)=minE \left(W\right)$$

(9)

The gradient method is a popular solution for this type of problem. Since Equation (8) involves the absolute value, this is a combinatorial optimization problem, and the gradient method cannot be employed to immediately minimize such an optimization problem. However, in order to estimate the absolute value of the weights, we recognize the use of a continuous and differentiable function to replace $L_1$ regularization by smoothing in (8). The error function with smoothing $L_1$ regularization can then be adapted by

$$E\left(W\right)=\check{E}\left(W\right)+\lambda {{\displaystyle \sum }}_{j=1}^{M}h\left(w_j\right),$$

(10)

where $h\left(x\right)$ is any continuous and differentiable functions. Specifically, we use the following piecewise polynomial function as:

$$h\left(t\right)=\left\{\begin{array}{c} \left|t \right| if \left|t\right|\ge m\\ -\frac{1}{8m^3}{t}^4+\frac{3}{4m}{t}^2+\frac{3}{8}m , if \left| t\right|<m,\\ \end{array}\right.$$

(11)

where $m$ is a suitable constant. Then the gradient of the error function is given by

$$E_W\left(W\right)={\left(E_{w_0}^T\left(W\right), E_{w_1}^T\left(W\right) ,E_{w_2}^T\left(W\right), \cdots , E_{w_M}^T\left(W\right)\right)}^T$$

(12)

The gradient of the error function in (10) with respect to $w_j$ is given by

$$E_{w_j}\left(W\right)={\check{E}}_{w_j}\left(W\right)+\lambda h^{\prime }\left(w_j\right)$$

(13)

The weights $\left\{W^k\right\}$ updated iteratively starting from an initial value $W^0$ by

$$W^{k+1}=W^k+\Delta W^k, k=0,1,2,\cdots ,$$

(14)

and

$$\Delta w_j^k=-\eta \left[{\check{E}}_{w_j^k}\left(W\right)+\lambda h^{\prime }\left(w_j^k\right)\right]$$

(15)

where $\eta >0$ is learning rate, and $\lambda$ is regularization parameter.

3. Materials and Methods

It will be necessary to prove the convergence theorem using the propositions below.

Proposition 1.

$\left|f\left(t\right)\right|, \left|f\prime \left(t\right)\right|, \left|f″\left(t\right)\right|$ and $\left|F\left(t\right)\right|, \left|F\prime \left(t\right)\right|$ are uniformly bounded for $t\in ℝ$.

Proposition 2.

$\Vert w_0^k\Vert \left(k=0,1,\cdots \right)$ is uniformly bounded.

Proposition 3.

$\eta$ and $\lambda$ are chosen to satisfy: $0<\eta <1/\left(\lambda \mathbb{A}-C_1\right)$, where

$$\begin{array}{l}{C}_1=L\left(1+C_2\right)C_3\mathit{max}\left\{C_2,C_5\right\}+\frac{1}{2}L\left(1+C_2\right)C_3+\frac{1}{2}L{C}_3^2{C}_4^2{C}_5,\\ C_2=max\left\{\sqrt{\mathbb{B}}{C}_3,{\left(C_3{C}_4\right)}^2\right\},\\ C_3=\mathit{max}\left\{\underset{t\in ℝ}{\mathit{sup}}\left|f\left(t\right)\right|,\underset{t\in ℝ}{\mathit{sup}}\left|f^{\prime }\left(t\right)\right|, \underset{t\in ℝ}{\mathit{sup}}\left|f^{″}\left(t\right)\right|,\underset{t\in ℝ,1\le l\le L}{\mathit{sup}}\left|f_l^{\prime }\left(t\right)\right|,\underset{t\in ℝ,1\le l\le L}{\mathit{sup}}\left|f_l^{″}\left(t\right)\right|\right\},\\ C_4=\underset{1\le l\le L}{\mathit{min}}\Vert {\xi }^l\Vert , C_5=\underset{k\in \mathbb{N}}{\mathit{sup}}\Vert w_0^k\Vert .\end{array}$$

(16)

Proposition 4.

There exists a closed bounded region $\mathsf{\Theta }$ such that $\left\{W^k\right\}\subset \mathsf{\Theta }$, and set ${\mathsf{\Theta }}_0=\left\{W\in \mathsf{\Theta }: E_w\left(W\right)=0\right\}$ contains only finite points.

Remark 1.

Both the hidden layer and output layer have the same transfer function, is $tansig\left(·\right)$. A uniformly bounded weight distribution is shown in Propositions 1 and 2. Thus, Proposition 4 is reasonable. The Equation (10) and Proposition 1, $\left|f_l\left(t\right)\right|, \left|f_l^{″}\left(t\right)\right|, \left|f_l^{″}\left(t\right)\right|$ are uniformly bounded for Proposition 3. Regarding Proposition 2, we would like to make the following observation. This paper focuses mainly on simulation problems with $f\left(t\right)$ being a sigmoid function satisfying Proposition 1. Typically, simulation problems require outputs of 0 and 1, or −1 and 1. To control the magnitude of the weights $w_0$, one can change the desired output into 0 + α and 1 − α, or −1 + α and 1 − α, respectively, where α > 0 is a small constant. Actually, a more important reason of doing so is to prevent the overtraining, cf. [39]. In the case of sigmoid functions, when $f\left(t\right)$ is bounded, the weights for the output layer are bounded.

Theorem 1.

Let the weight $\left\{W^k\right\}$ be generated by the iteration algorithm (14) for an arbitrary initial value $W^0$, the error function $E\left(W\right)$ be defined by (10) and if propositions 1–3 are valid, then we have

I.

$E\left(W^{k+1}\right)\le E\left(W^k\right), k=0,1,\cdots ;$

II.

There exists $E^{\ast }\ge 0$ such that $\underset{k\to \infty }{\mathit{lim}}{E}_{w_j}\left(W^k\right)=E^{\ast }.$

III.

$\underset{k\to \infty }{\mathit{lim}}\Vert \Delta W^k\Vert =0,$

IV.

Further, if proposition 4 is also valid, we have the following strong convergence

V.

There exists a point $W^{\ast }\in {\mathsf{\Theta }}_0$ such that $\underset{k\to \infty }{\mathit{lim}}{W}^k=W^{\ast }$.

Note: 

It is shown in conclusion (I) and (II) that the error function sequence $\left\{E\left(W^k\right)\right\}$ is monotonic and has a limit (II). According to conclusions (II) and (IV), $E\left(W^k\right)$ and $E_W\left(W^k\right)$ are weakly converging. The strong convergence of { $W^k$} is mentioned in Conclusion (V).

We used the following strategy as a neuron selection criterion by simply computing the norm of the overall outgoing weights from the neuron number to ascertain whether a neuron number in the hidden units will survive or be removed after training. There is no standard threshold value in the literature for eliminating redundant weighted connections and redundant neurons from the initially assumed structure of neural networks. The sparsity of the learning algorithm was measured using the number of weights with absolute values of ≤0.0099 and ≤0.01, respectively, according to ref. [40]. In this study, we chose 0.00099 as a threshold value at random, which is less than the existing thresholds in the literature. This procedure is repeated ten times. Algorithm 1 describes the experiment procedure.

Algorithm 1 The learning algorithm
Input	Input the dimension $M$, the number $N$ of the nodes, the number maximum iteration number $K$, the learning rate $\eta$, the regularization parameter $\lambda$, and the sample training set is ${\{{\xi }^l,O^l\}}_{l=1}^L\subset {ℝ}^N\times ℝ$.
Initialization	Initialize randomly the initial weight vectors $w_0^0={(w_{0,1}^0,\cdots ,w_{0,M}^0)}^T\in {ℝ}^M$ and $w_j^0={(w_{j0}^0,\cdots ,w_{jN}^0)}^T\in {ℝ}^N$ $j\left(j=1,2,\dots ,M\right)$
Training	For $k=1,2,\cdots ,K$ do Compute the error function Equation (10). Compute the gradients Equation (15). Update the weights $w_0^0$ and $w_j^0 \left(1\le j\le M\right)$ by using Equation (14). end
Output	Output the final weight vectors $w_0^K$ and $w_j^K \left(1\le j\le M\right)$

4. Experimental Results

The simulation results for evaluating the performance of the proposed BGS$L_1$ algorithm are presented in this section. We will compare BGS$L_1$ performance to that of four common regularization algorithms: the batch gradient method with $L_{1/2}$ regularization (BG$L_{1/2}$), the batch gradient method with smoothing $L_{1/2}$ regularization (BGS$L_{1/2}$), the batch gradient method with $L_1$ regularization (BG$L_1$), and the batch gradient method with $L_2$ regularization (BG$L_2$). Numerical experiments on the N-dimensional parity and function approximation problems support our theoretical conclusion.

4.1. N-Dimensional Parity Problems

The N-dimensional parity problem is another popular task that generates a lot of debate. If the input pattern contains an odd number of ones, the output criterion is one; alternatively, the output necessity is zero. An N-M-1 architecture (N inputs, M hidden nodes, and 1 output) is employed to overcome the N-bit parity problem. The well-known XOR problem is simply a 2-bit parity problem [41]. Here, the 3-bit and 6-bit parity problems are used as an example to test the performance of BGS$L_1$. The network has three layers: input layers, hidden layers, and an output unit.

Table 1 shows the parameter settings for the corresponding network, where LR and RP are abbreviations for the learning rate and regularization parameter, respectively. Figure 3 and Figure 4 show the performance results of BG$L_2$, BG$L_1$, BG$L_{1/2}$, BGS$L_{1/2}$, and BGS$L_1$ for 3-bit and 6-bit parity problems, respectively. As illustrated by Theorem 1, the error function decreases monotonically in Figure 3a and Figure 4a, and the norm of the gradients of the error function approaches zero in Figure 3b and Figure 4b. According to the comparison results, our proposal demonstrated superior learning ability and faster convergence. This corresponds to our theoretical analysis. Table 2 displays the average error and running time of ten experiments, demonstrating that BGS$L_1$ not only converges faster but also has better generalization ability than others.

Table 1. The learning parameters for parity problems.

Problems	Network Structure	Weight Size	Max Iteration	LR	RP
3-bit parity	3-6-1	[−0.5, 0.5]	2000	0.009	0.0003
6-bit parity	6-20-1	[−0.5, 0.5]	3000	0.006	0.003

Figure 3. The performance results of five different algorithms based on 3-bit parity problem: (a) The curve of error function, (b) The curve of norm of gradient.

Figure 4. The performance results of five different algorithms based on 6-bit parity problem: (a) The curve of error function, (b) The curve of norm of gradient.

Table 2. Numerical results for parity problems.

Problems	Learning Algorithms	Average Error	Norm of Gradient	Time (s)
3-bit parity	BG$L_{1/2}$	3.7979 × 10−7	0.0422	1.156248
	BG$L_1$	5.4060 × 10−7	7.1536 × 10−4	1.216248
	BG$L_2$	9.7820 × 10−7	8.7826 × 10−4	1.164721
	BGS$L_{1/2}$	1.7951 × 10−8	0.0011	1.155829
	BGS$L_1$	7.6653 × 10−9	7.9579 × 10−5	1.135742
6-bit parity	BG$L_{1/2}$	8.1281 × 10−5	1.1669	52.225856
	BG$L_1$	3.8917 × 10−5	0.0316	52.359129
	BG$L_2$	4.1744 × 10−5	0.0167	52.196552
	BGS$L_{1/2}$	4.8349 × 10−5	0.0088	52.210994
	BGS$L_1$	4.1656 × 10−6	0.0015	52.106554

4.2. Function Approximation Problem

A nonlinear function has been devised to compare the approximation capabilities of the above algorithms:

$$G\left(x\right)=\frac{1}{2}x-\mathit{sin}\left(x\right)$$

(17)

where $x\in \left[-4,4\right]$ and chooses 101 training samples from an evenly spaced interval of [−4, 4]. The initial weight of the network is typically generated at random within a given interval; training begins with an initial point and gradually progresses to a minimum of error along the slope of the error function, which is chosen stochastically in [−0.5, 0.5]. The training parameters are as follows: 0.02 and 0.0005 represent the learning rate ($\eta$) and parameter regularization ($\lambda$), respectively. The stop criteria are set to 1000 training cycles.

The average error and norm of the gradient and running time of 10 experiments are presented in Table 3. Through the results obtained from Figure 5a,b, respectively, we see that BGS$L_1$ has a better mapping capability than BGS$L_{1/2}$, BG$L_{1/2}$, BG$L_1$ and BG$L_2$, with the error decreasing monotonically as learning proceeds and its gradient go to zero. Table 3 shows the preliminary results are extremely encouraging and that the speedup and generalization ability of BGS$L_1$ is better than BGS$L_{1/2}$, BG$L_{1/2}$, BG$L_1$ and BG$L_2$.

Table 3. Numerical results for function approximation problem.

Learning Algorithms	Average Error	Norm of Gradient	Time (s)
BG$L_{1/2}$	0.0388	0.3533	4.415500
BG$L_1$	0.0389	0.3050	4.368372
BG$L_2$	0.0390	0.3087	4.368503
BGS$L_{1/2}$	0.0386	0.2999	4.349813
BGS$L_1$	0.0379	0.2919	4.320198

Figure 5. The performance results of five different algorithms based on function approximate problem: (a) The curve of error function, (b) The curve of norm of gradient.

5. Discussion

Table 2 and Table 3, respectively, show the performance comparison of the average error and the average norms of gradients under our five methods over the 10 trials. Table 2 shows the results of N-dimensional parity problems using the same parameters, while Table 3 shows the results of function approximation problems using the same parameters.

The comparison convincing in Table 2 and Table 3 shows that the BGS$L_1$ is more efficient and has better sparsity-promoting properties than BG$L_{1/2}$, BG$L_1$, BG$L_2$ and even than the BGS$L_{1/2}$. In addition to that, Table 2 and Table 3 show that our proposed algorithm is faster than that of all numerical results. $L_{1/2}$ regularization is sparser than the traditional $L_1$ regularization solution. Recently, in ref. [34], the BGS$L_{1/2}$ also shows that the sparsity is better than that of BG$L_{1/2}$. The results of ref. [33] show that the $L_{1/2}$ regularization has been demonstrated to have the following properties: unbiasedness, sparsity, and oracle properties.

We obtained all three numerical results for five different methods using one hidden layer of FFNNs. Our new method proposed a sparsification technique for FFNNs can be extended to encompass any number of hidden layers.

6. Conclusions

$L_1$ regularization is thought to be an excellent pruning method for neural networks. $L_1$ regularization, on the other hand, is also an NP-hard problem. In this paper, we propose BGS$L_1$, a batch gradient learning algorithm with smoothing $L_1$ regularization for training and pruning feedforward neural networks, and we approximate the $L_1$ regularization by smoothing function. We analyzed some weak and strong theoretical results under this condition, and the computational results validated the theoretical findings. The proposed algorithm has the potential to be extended to train neural networks. In the future, we will look at the case of the online gradient learning algorithm with a smoothing $L_1$ regularization term.