Training Artificial Neural Networks Using a Global Optimization Method That Utilizes Neural Networks

Abstract

Perhaps one of the best-known machine learning models is the artificial neural network, where a number of parameters must be adjusted to learn a wide range of practical problems from areas such as physics, chemistry, medicine, etc. Such problems can be reduced to pattern recognition problems and then modeled from artificial neural networks, whether these problems are classification problems or regression problems. To achieve the goal of neural networks, they must be trained by appropriately adjusting their parameters using some global optimization methods. In this work, the application of a recent global minimization technique is suggested for the adjustment of neural network parameters. In this technique, an approximation of the objective function to be minimized is created using artificial neural networks and then sampling is performed from the approximation function and not the original one. Therefore, in the present work, learning of the parameters of artificial neural networks is performed using other neural networks. The new training method was tested on a series of well-known problems, a comparative study was conducted against other neural network parameter tuning techniques, and the results were more than promising. From what was seen after performing the experiments and comparing the proposed technique with others that have been used for classification datasets as well as regression datasets, there was a significant difference in the performance of the proposed technique, starting with 30% for classification datasets and reaching 50% for regression problems. However, the proposed technique, because it presupposes the use of global optimization techniques involving artificial neural networks, may require significantly higher execution time than other techniques.

1. Introduction

Artificial neural networks (ANNs) are parametric models [1,2,3] in machine learning, and they are widely used in pattern recognition problems. A series of practical problems from the fields of physics [4,5,6], chemistry [7,8,9], economics [10,11,12], medicine [13,14], etc., can be transformed to pattern recognition problems and then solved using artificial neural networks. Furthermore, neural networks have been used with success to solve differential equations [15,16,17], solar radiation prediction [18,19], spam detection [20,21,22], etc. Moreover, variations of artificial neural networks have been employed to solve agricultural problems [23,24], facial expression recognition [25], prediction of the speed of wind [26], the gas consumption problem [27], intrusion detection [28], hydrological systems [29], etc. Furthermore, Swales and Yoon discussed the application of artificial neural networks to investment analysis in their work [30].

A neural network can be denoted as a function $N(\overrightarrow{x},\overrightarrow{w})$ where the vector $\overrightarrow{x}$ stands for the input vector and the vector $\overrightarrow{w}$ is the set of the parameters of the neural network that should be estimated. The input vector is usually called pattern in the relevant literature and the vector $\overrightarrow{w}$ is usually called the weight vector. Artificial neural network training methods adjust the vector of weights in order to minimize the following quantity:

$$E\left(N\left(\overrightarrow{x},\overrightarrow{w}\right)\right)=\sum _{i=1}^M{\left(N\left({\overrightarrow{x}}_i,\overrightarrow{w}\right)-y_i\right)}^2$$

(1)

In the previous equation, what will be called training error, the set $\left(\overrightarrow{x_i},y_i\right),i=1,...,M$ is the input training dataset for the neural network with M patterns. The value $y_i$ is the expected output for the pattern $\overrightarrow{x_i}$. Equation (1) can be minimized with respect to the weight vector using any local or global optimization method such as the back-propagation method [31,32], the hill climbing method [33], the RPROP method [34,35,36], quasi Newton methods [37,38], simulated annealing [39,40], genetic algorithms [41,42], particle swarm optimization [43,44], differential optimization methods [45,46], evolutionary computation [47], the whale optimization algorithm [48], etc. Furthermore, Cui et al. suggested the usage of a new stochastic optimization algorithm that simulates the plant growing process for neural network training. Furthermore, recently, the bird mating optimizer [49] was suggested as a training method for artificial neural networks [50], and hybrid methods have been developed by various researchers to optimize the weight vector, such as the work of Yaghini et al. [51] that combined particle swarm optimization with a back-propagation algorithm to minimize the error function. Moreover, Chen et al. [52] used a hybrid technique that combines particle swarm optimization and cuckoo search [53] to optimize the weight vector of neural networks.

In addition, many researchers have addressed the issue of the initial values for the weights of neural networks, such as the incorporation of decision trees [54], an initialization method using Cauchy’s inequality [55], incorporation of discriminant learning [56], methods based on genetic algorithms [57], etc. A paper discussing all the aspects of weight initialization strategies was written by Narkhede et al. [58].

Moreover, various groups of researchers are dealing with the issue of constructing the structure of artificial neural networks, such as the incorporation of genetic algorithms [41], the usage of the grammatical evolution method [59] for the construction of neural networks [60], a construction and pruning approach to optimize the structure of ANNs [61], usage of cellular automata [62], etc. Furthermore, because the training of artificial neural networks with optimization methods requires a significantly longer computing time, parallel techniques have been developed that take advantage of modern parallel computing units [63,64,65].

Another area of research in the field of artificial neural networks that attracts a multitude of researchers is the problem of overfitting that occurs in many cases. In this problem, although the artificial neural network has achieved a satisfactory level of training, this is not reflected in unknown patterns that were not present during training. This set of patterns will be called the test set in the following. Commonly used methods that tackle the overfitting problem are weight sharing [66,67], methods that reduce the number of parameters (weight pruning) [68,69,70], the method of dropout [71,72], weight decaying methods [73,74], the Sarprop method [75], positive correlation methods [76], etc.

In this paper, the use of a recent global minimization technique [77] called NeuralMinimizer, is proposed to find the optimal set for the weights of artificial neural networks. This innovative global minimization technique constructs an approximation of the objective function to be minimized using a limited number of its samples. These limited samples form the training set of an artificial neural network that can be trained with any optimization method. Subsequently, the sampling for the continuation of the global optimization method is not performed by the objective function but by the previously trained artificial neural network. The samples obtained by artificial neural networks before being fed into the global minimization method are classified and those with the smallest functional value will finally be input into the global minimization method. From the experimental results, it was shown that this global minimization method requires a limited number of samples from the objective function to find the global minimum and is also more efficient than other techniques for discovery of the global minimum. Therefore, this paper proposes using artificial neural networks to train other artificial neural networks. This new procedure will be tested on a series of known problems in order to evaluate its effectiveness.

The rest of this article is organized as follows: Section 2 describes the proposed method, Section 3 lists the experimental datasets and the results obtained by the incorporation of various methods, and finally, Section 4 discusses some conclusions.

2. The Proposed Method

In this section, some basic principles for artificial neural networks are presented and then a new training method that incorporates a modified version of the NeuralMinimizer global optimization technique is outlined.

2.1. Preliminaries

Let us consider an artificial neural network with only one hidden layer in which the sigmoid function is used as an activation function. The output value for every node in this layer is calculated as:

$$o_i\left(x\right)=\sigma \left(p_{i}x+{\theta }_i\right),$$

(2)

where the value $p_i$ is the weight vector, and ${\theta }_i$ denotes the bias for the node $i.$ The sigmoid function is defined as:

$$\sigma \left(x\right)=\frac{1}{1+exp\left(-x\right)}$$

(3)

and it is graphically illustrated in Figure 1.

When the neural network has H processing nodes, the output can be formulated as:

$$N\left(x\right)=\sum _{i=1}^H{v}_i{o}_i\left(x\right),$$

(4)

where $v_i$ stands for the output weight for node i. Hence, by using one vector for all the parameters (weights and biases) the neural network can be written in the following form:

$$N\left(\overrightarrow{x},\overrightarrow{w}\right)=\sum _{i=1}^H{w}_{(d+2)i-(d+1)}\sigma \left(\sum _{j=1}^d{x}_j{w}_{(d+2)i-(d+1)+j}+w_{(d+2)i}\right)$$

(5)

where d is the dimension of vector $\overrightarrow{x}$. From Equation (5) we can conclude that number of elements in the weight vector is:

$$d_w=(d+2)H$$

(6)

2.2. The Modified NeuralMinimizer Method

In its original version, the NeuralMinimizer method employed RBF neural networks [78] to build a model of the objective function. Even though radial basis function (RBF) networks have been used with success in a variety of problems [79,80,81,82], it is not possible to apply them to the training of the parameters of an artificial neural network due to the large dimension of the problem, as shown in Equation (6). Hence, in the current work, the RBF network has been replaced by an artificial neural network that implements Equation (5). The training of the artificial neural network was performed using a local minimization technique that is not particularly demanding in calculations and storage space, such as limited memory BFGS (L-BFGS) [83]. Obviously, any other technique that is not extremely memory intensive could be used in its place. Such a technique could be the Adam method [84], the SGD method [85,86], or even a simple global minimization method such as a genetic algorithm with a limited number of chromosomes. The L-BFGS method is a variation of the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method [87] using a limited amount of computer memory. This local minimization method has found wide application in difficult and memory-intensive optimization problems such as image reconstruction [88], inverse eigenvalue problems [89], seismic waveform tomography [90], etc. Because of the application of this technique to large-dimensional problems, a number of modifications have been proposed that make use of modern parallel computing systems [91,92,93]. A numerical study on the limited memory BFGS methods is provided in the work of Morales [94]. In the original publication on the NeuralMinimizer optimization method, an RBF neural network was used to generate the approximation function of the objective function. However, this would not always be possible in cases where the objective function to be minimized is the error of an artificial neural network, because an artificial neural network usually has a large number of parameters, and this would require an extremely large storage space for training the global minimization method’s RBF neural network. Of course, in some cases with small artificial neural networks, an RBF neural network could be used as the training model in the global optimization method NeuralMinimizer. However, when an artificial neural network is used to approach large and complex problems (something extremely common) then a relatively under powered RBF neural network should be used. In those cases, the RBF network will not be able to be an efficient approximation of the artificial neural network, which is, of course, the original aim of the NeuralMinimizer method.

In the following, the main steps of the modified NeuralMinimizer method for the training of neural networks are listed. In these steps, the neural network used by the NeuralMinimizer method will be called $N_N(x,w)$.

• Initialization step.

  (a)

  SetH the number of weights of the neural network. In the current method the same number of weights was used for both $N(x,w)$ and $N_N(x,w)$ artificial neural networks.

  (b)

  Set $N_S$ as the samples that will be initially drawn from $N(x,w)$. At this stage, the training error for the artificial neural network will be used as an objective function to minimize

  (c)

  Set $N_T$ as the number of points that will be utilized as local minimization method starters in every iteration.

  (d)

  Set $N_R$ as the number of samples that will be drawn from the $N_N(x,w)$ network in each iteration.

  (e)

  Set $N_G$ as the maximum number of iterations allowed.

  (f)

  Set Iter = 0, the current iteration number.

  (g)

  Set $\left(w^{*},y^{*}\right)$ as the global minimum discovered by the method. Initially $y^{*}=\infty ,w^{*}=\left(0,0,\dots ,0\right)$

• Creation Step.

  (a)

  Set $T=\varnothing$, the used training set for the $N_N(x,w)$ neural network.

  (b)

  For $i=1,..,N_S$ do

  • Draw a new sample $w_i$ from $N(x,w)$.
  • Calculate $y_i=f\left(w_i\right)$ using Equation 1
  • $T=T\cup \left(w_i,y_i\right)$

  (c)

  EndFor

  (d)

  Train the $N_N(x,w)$ neural network on set T using the L-BFGS method.

• Sampling Step

  (a)

  Set $T_R=\varnothing$

  (b)

  For $i=1,..,N_R$do

  • Produce randomly a sample $\left(w_i,y_i\right)$ from $N_N(x,w)$ neural network
  • Set $T_R=T_R\cup \left(x_{i,}{y}_i\right)$

  (c)

  EndFor

  (d)

  Sort $T_R$ in ascending with respect to the values $y_i$

• Optimization Step.

  (a)

  For $i=1,..,N_T$ do

  • Get the next item $\left(w_i,y_i\right)$ from $T_R$.
  • Train the neural network $N\left(x,w_i\right)$ on the training set of the objective problem using the L-BFGS method and obtain the corresponding training error $y_i$.
  • Update  $T=T\cup \left(w_i,y_i\right)$
  • Train the network $N_N(x,w)$ again on the modified set T. In this step, the original training set used by $N_N(x,w)$ is updated to include the new discovered local minimum. This operation is used in order to construct a more accurate approximation of the real objective function.
  • If $y_i\le y^{*}$ then $w^{*}=w_i,y^{*}=y_i$
  • If the termination rule proposed in [95], then apply the produced network $N\left(x,w^{*}\right)$ on the test set of the objective problem, report the test error and terminate.

  (b)

  EndFor

• Set iter=iter+1
• Goto to Sampling step.

A flowchart of the proposed method is graphically outlined in Figure 2.

3. Experiments

The effectiveness of the proposed artificial neural network training technique was evaluated using a series of data sets from the relevant literature. These datasets have been studied by various researchers in the relevant literature and cover a wide range of research areas from physics to economics. These datasets are freely available from the following websites:

• The UCI repository, https://archive.ics.uci.edu/ (accessed on 12 July 2023) [96]
• The Keel repository, https://sci2s.ugr.es/keel/datasets.php (accessed on 17 June 2023) [97].
• The Statlib URL ftp://lib.stat.cmu.edu/datasets/index.html (accessed on 17 June 2023). This repository is used mainly for the regression datasets.

3.1. Experimental Datasets

The following classification datasets from the relevant literature were used in the experiments:

• Appendicitis, a dataset used for medical purposes; it was found in [98,99].
• Australian dataset [100], an economic dataset, related to bank transactions.
• Balance dataset [101], which is related to psychological experiments.
• Cleveland dataset, a medical dataset found in the following research papers [102,103].
• Bands dataset, related to printing problems [104].
• Dermatology dataset [105], a dataset related to dermatology problems.
• Hayes-Roth dataset [106].
• Heart dataset [107], a medical dataset used to detect heart diseases.
• HouseVotes dataset [108], related to the Congressional voting records of USA.
• Ionosphere dataset, related to measurements from the ionosphere an thoroughly studied in a series of research papers [109,110].
• Liverdisorder dataset [111,112], a medical dataset.
• Lymography dataset [113].
• Mammographic dataset [114], a medical dataset related to breast cancer diagnosis.
• Page Blocks dataset [115].
• Parkinsons dataset [116,117], a medical dataset applied to the Parkinson’s decease.
• Pima dataset [118], a medical dataset.
• Popfailures dataset [119], a dataset related to meteorological data.
• Regions2 dataset, a medical dataset for liver biopsy images [120].
• Saheart dataset [121], a medical dataset related to heart diseases.
• Segment dataset [122], a dataset related to image segmentation.
• Wdbc dataset [123], a dataset related to breast tumors.
• Wine dataset, a dataset related to chemical analysis of wines [124,125].
• Eeg datasets [126,127], an EEG dataset, and the following cases were used in the experiments:

  (a)

  Z_F_S,

  (b)

  ZO_NF_S

  (c)

  ZONF_S.

• Zoo dataset [128], suggested for the detection of the proper classes of animals.

A table showing the number of classes for every classification dataset is shown in

Table 1. Description for every classification dataset.

DATASET	CLASSES
Appendicitis	2
Australian	2
Balance	3
Cleveland	5
Bands	2
Dermatology	6
Hayes-Roth	3
Heart	2
Housevotes	2
Ionosphere	2
Liverdisorder	2
Lymography	4
Mammographic	2
Page Blocks	5
Parkinsons	2
Pima	2
Popfailures	2
Regions2	5
Saheart	2
Segment	7
Wdbc	2
Wine	3
Z_F_S	3
ZO_NF_S	3
ZONF_S	2
Zoo	7

.

The following regression datasets were used:

• Abalone dataset [129], proposed to predict the age of abalones.
• Airfoil dataset, a dataset provided by NASA [130], created from a series of aerodynamic and acoustic tests.
• Baseball dataset, a dataset using baseball games.
• BK dataset [131], used to predict the points scored in a basketball game.
• BL dataset, used in machine problems.
• Concrete dataset [132], a dataset proposed to calculate the compressive strength of concrete.
• Dee dataset, used to detect the electricity energy prices.
• Diabetes dataset, a medical dataset.
• Housing dataset [133].
• FA dataset, used to fit body fat to other measurements.
• MB dataset [131].
• Mortgage dataset. The goal is to predict the 30-year conventional mortgage rate.
• PY dataset, (pyrimidines problem) [134].
• Quake dataset, used to approximate the strength of a earthquake given its the depth of its focal point, its latitude and its longitude.
• Treasure dataset, which contains economic data information from the USA, where the goal is to predict the 1-month CD Rate.
• Wankara dataset, a weather dataset.

3.2. Experimental Setup

The proposed method was tested on the regression and classification problems mentioned previously, and it was compared against the results of several other well-known optimization methods in the relevant literature. For greater reliability of the experimental results, the 10-fold validation technique was employed for every classification or regression dataset. Every experiment was executed 30 times, with different initialization for the random generator each time. Furthermore, the srand48() random generator of the C-programming language was utilized. The used code was implemented in ANSI C++ using the freely available OPTIMUS optimization library available from https://github.com/itsoulos/OPTIMUS/(accessed on 18 July 2023). For the case of the classification datasets, the average classification error was measured for every method. For regression datasets, the average mean squared error was measured in the test set. The number of hidden nodes for the neural networks was set to $H=10$ for every method. All the experiments were performed using an AMD Ryzen 5950X with 128 GB of RAM. The running operating system was Debian Linux. The methods used in the experimental results are the following:

• A genetic algorithm with 200 chromosomes was used to train a neural network with H hidden nodes. This method was denoted as GENETIC in the tables holding the experimental results.
• A radial basis function (RBF) network [78] with H hidden nodes.
• The Adam optimization method [84]. Here, the method was used to minimize the train error of a neural network with H hidden nodes.
• The resilient back-propagation (RPROP) optimization method [34,35,36] was also employed to train a neural network with H hidden nodes.
• The NEAT method (NeuroEvolution of Augmenting Topologies ) [135].

The values used for every parameter are listed in

Table 2. Experimental settings.

PARAMETER	MEANING	VALUE
H	Number of weights	10
$N_S$	Start samples	50
$N_T$	Starting points	100
$N_R$	Samples drawn from the first network	$10\times N_T$
$N_G$	Maximum number of iterations	200

and they are similar to the values used in the original publication of the NeuralMinimizer method.

3.3. Experimental Results

The experimental results for the classification datasets are shown in

Table 3. Experimental results for the classification datasets. The numbers in cells denote average classification error of 30 independent runs.

DATASET	GENETIC	RBF	ADAM	RPROP	NEAT	PROPOSED
Appendicitis	18.10%	12.23%	16.50%	16.30%	17.20%	22.30%
Australian	32.21%	34.89%	35.65%	36.12%	31.98%	21.59%
Balance	8.97%	33.42%	7.87%	8.81%	23.14%	5.46%
Bands	35.75%	37.22%	36.25%	36.32%	34.30%	33.06%
Cleveland	51.60%	67.10%	67.55%	61.41%	53.44%	45.41%
Dermatology	30.58%	62.34%	26.14%	15.12%	32.43%	4.14%
Hayes Roth	56.18%	64.36%	59.70%	37.46%	50.15%	35.28%
Heart	28.34%	31.20%	38.53%	30.51%	39.27%	17.93%
HouseVotes	6.62%	6.13%	7.48%	6.04%	10.89%	5.78%
Ionosphere	15.14%	16.22%	16.64%	13.65%	19.67%	16.31%
Liverdisorder	31.11%	30.84%	41.53%	40.26%	30.67%	33.02%
Lymography	23.26%	25.31%	29.26%	24.67%	33.70%	25.64%
Mammographic	19.88%	21.38%	46.25%	18.46%	22.85%	16.37%
PageBlocks	8.06%	10.09%	7.93%	7.82%	10.22%	5.44%
Parkinsons	18.05%	17.42%	24.06%	22.28%	18.56%	14.47%
Pima	32.19%	25.78%	34.85%	34.27%	34.51%	25.61%
Popfailures	5.94%	7.04%	5.18%	4.81%	7.05%	5.57%
Regions2	29.39%	38.29%	29.85%	27.53%	33.23%	22.73%
Saheart	34.86%	32.19%	34.04%	34.90%	34.51%	34.03%
Segment	57.72%	59.68%	49.75%	52.14%	66.72%	37.28%
Wdbc	8.56%	7.27%	35.35%	21.57%	12.88%	5.01%
Wine	19.20%	31.41%	29.40%	30.73%	25.43%	7.14%
Z_F_S	10.73%	13.16%	47.81%	29.28%	38.41%	7.09%
ZO_NF_S	8.41%	9.02%	47.43%	6.43%	43.75%	5.15%
ZONF_S	2.60%	4.03%	11.99%	27.27%	5.44%	2.35%
ZOO	16.67%	21.93%	14.13%	15.47%	20.27%	4.20%

and those of the regression datasets are shown in

Table 4. Average regression error for the regression datasets.

DATASET	GENETIC	RBF	ADAM	RPROP	NEAT	PROPOSED
ABALONE	7.17	7.37	4.30	4.55	9.88	4.50
AIRFOIL	0.003	0.27	0.005	0.002	0.067	0.003
BASEBALL	103.60	93.02	77.90	92.05	100.39	56.16
BK	0.027	0.02	0.03	1.599	0.15	0.02
BL	5.74	0.01	0.28	4.38	0.05	0.0004
CONCRETE	0.0099	0.011	0.078	0.0086	0.081	0.003
DEE	1.013	0.17	0.63	0.608	1.512	0.30
DIABETES	19.86	0.49	3.03	1.11	4.25	1.24
HOUSING	43.26	57.68	80.20	74.38	56.49	18.30
FA	1.95	0.02	0.11	0.14	0.19	0.01
MB	3.39	2.16	0.06	0.055	0.061	0.05
MORTGAGE	2.41	1.45	9.24	9.19	14.11	3.50
PY	105.41	0.02	0.09	0.039	0.075	0.03
QUAKE	0.04	0.071	0.06	0.041	0.298	0.039
TREASURY	2.929	2.02	11.16	10.88	15.52	3.72
WANKARA	0.012	0.001	0.02	0.0003	0.005	0.002

. The column PROPOSED represents the usage of the proposed method to train a neural network with H hidden nodes. Furthermore, the Figure 3 shows a scatter plot and the Wilcoxon signed-rank test for the classification datasets. In the same direction, Figure 4 shows the scatter plot for the regression datasets.

The experimental results and their graphical representation demonstrate the superiority of the proposed technique over the others in terms of the average error, as measured in the test set. For example, in the case of datasets used for classification, the proposed method outperforms the remaining techniques in 19 out of 26 datasets (73% percent). Furthermore, in several cases, the percentage reduction in error exceeds 50%. For the classification problems, the immediate most effective training method after the proposed one is the genetic algorithm and, on average, the proposed technique achieves lower classification error than the genetic algorithm error by 24%. Moreover, in regression problems, the next most effective method after the proposed one is the RBF neural network with small differences from the ADAM optimizer. However, in the case of regression problems, the improvement in average error using the proposed technique exceeds 49%. Of course, the proposed technique is quite time-consuming, because it requires the continuous training of an artificial neural network.

4. Conclusions

In this work, the application of a recent global minimization method for the training of artificial neural networks was proposed. The application of this method was used in artificial neural networks both for classification problems and for regression problems. This new global minimization method constructs an approximation of the objective function using neural networks. This construction is performed with a limited number of samples from the objective function. However, each time a local minimization takes place, this approximation is readjusted. Subsequently, the sampling for the minimization is performed from the approximate function and not from the objective one, even taking samples from the approximation with the smallest function value in order to speed up the discovery of the global minimum. In this particular case, the artificial neural network of the global minimization method is used to train the artificial neural network. However, due to the large time and storage requirements of artificial neural networks, the RBF network of the original NeuralMinimizer method was replaced with an artificial neural network that was trained using the local minimization method L-BFGS. The new artificial neural network training technique is tested on a wide collection of classification and regression problems from the relevant literature and is shown to significantly improve the learning error over other established artificial neural network training techniques. This improvement is 25% on average for the case of classification problems and rises significantly to 50% for regression problems. The proposed method outperforms the other methods and models in the majority of cases. For example, in the classification datasets, the proposed method outperforms the genetic algorithm in 22 datasets, the RBF model in 21 datasets, the ADAM optimizer in 23 cases, the RPROP optimizer in 22 cases and finally, the NEAT method in 25 datasets.

Nevertheless, the proposed procedure can be extremely slow, especially as the size of the artificial neural network increases. The size of the artificial neural network directly depends on the dimension of the input dataset. Future improvements to the methodology may include the use of parallel programming techniques, such as parallel implementations of the L-BFGS optimization method, in order to accelerate the training of artificial neural networks by taking advantage of modern computing structures. Furthermore, in the present phase, as a minimization method in step 4 of the proposed training method, a local minimization method is used. Future extensions could explore the possibility of also using global minimization techniques in this step, although care should be taken to make use of parallel computing techniques to avoid long execution times.