Constructing the Bounds for Neural Network Training Using Grammatical Evolution

Abstract

Artificial neural networks are widely established models of computational intelligence that have been tested for their effectiveness in a variety of real-world applications. These models require a set of parameters to be fitted through the use of an optimization technique. However, an issue that researchers often face is finding an efficient range of values for the parameters of the artificial neural network. This paper proposes an innovative technique for generating a promising range of values for the parameters of the artificial neural network. Finding the value field is conducted by a series of rules for partitioning the original set of values or expanding it, the rules of which are generated using grammatical evolution. After finding a promising interval of values, any optimization technique such as a genetic algorithm can be used to train the artificial neural network on that interval of values. The new technique was tested on a wide range of problems from the relevant literature and the results were extremely promising.

1. Introduction

Artificial neural networks (ANNs) are widespread machine learning models, which base their dynamics on a series of parameters that are also called computational units or weights in the relevant literature [1,2]. Neural networks are widely used in a variety of scientific applications, such as problems from physics [3,4,5], solutions of differential equations [6,7], agriculture problems [8,9], chemistry problems [10,11,12], problems that appear in the area of economic sciences [13,14,15], problems related to medicine [16,17], etc. Commonly, the neural networks are expressed as a function $N(\overrightarrow{x},\overrightarrow{w})$, with the assumption that the vector $\overrightarrow{x}$ stands for the input vector to the neural network and the vector $\overrightarrow{w}$ is the parameter vector that should be estimated. The first vector is called pattern in the bibliography and the second one weight vector. Techniques that adjust the parameter vector $\overrightarrow{w}$ minimize the so-called training error, defined as:

$$E\left(N\left(\overrightarrow{x},\overrightarrow{w}\right)\right)=\sum _{i=1}^M{\left(N\left({\overrightarrow{x}}_i,\overrightarrow{w}\right)-y_i\right)}^2$$

(1)

In Equation (1), the set $\left(\overrightarrow{x_i},y_i\right),i=1,...,M$ is denoted as the training set of the neural network. The $y_i$ are considered the target outputs for the $\overrightarrow{x_i}$ points. As proposed in [18], neural networks can be expressed as the following summation:

$$N\left(\overrightarrow{x},\overrightarrow{w}\right)=\sum _{i=1}^H{w}_{(d+2)i-(d+1)}\sigma \left(\sum _{j=1}^d{x}_j{w}_{(d+2)i-(d+1)+j}+w_{(d+2)i}\right)$$

(2)

The number of processing units is denoted by H and the value d stands for the dimension of input vector $\overrightarrow{x}$. The function $\sigma \left(x\right)$ denotes the well-known sigmoid function expressed as:

$$\sigma \left(x\right)=\frac{1}{1+exp(-x)}$$

(3)

From the above equations one can calculate the total number of parameters in the neural network as:

$$n=\left(d+2\right)H$$

(4)

The training of neural networks has been performed by a variety of numerical methods in the recent literature, such as: the Back Propagation method [19], the RPROP method [20], the Adam optimizer [21], etc. Furthermore, more advanced optimization methods have been incorporated to optimize the parameter vector of neural networks by minimizing Equation (1) such as quasi-Newton methods [22], the Tabu search method [23], Simulated Annealing [24], genetic algorithms [25], particle swarm optimization (PSO) [26], Differential Evolution [27], Ant Colony Optimization [28], etc.

Moreover, recently, a series of hybrid optimization methods were proposed to tackle the minimization of the training error, such as a method that combines PSO and genetic algorithms [29,30], the incorporation of a particle swarm optimization algorithm and gravitational search algorithm [31], a combination of a genetic algorithm and controlled gradient method [32], etc.

In addition, an important topic for artificial neural networks such as parameter initialization has been studied by several researchers in recent years. In this area appeared papers such as initialization with decision trees [33], incorporation of the Cauchy’s inequality [34], discriminant learning [35], etc. Another interesting topic in the area of neural networks is weight decaying, where some regularization is applied to the parameters of the neural network to avoid the overfitting problem. For this topic, several research papers have been suggested, such as a method that incorporates positive correlation [36], the SarProp algorithm [37], usage of pruning techniques [38], etc.

This paper proposes a two-stage method for efficient training of the parameters of artificial neural networks. In the first stage, a value interval is constructed for the parameters of the neural network through grammatical evolution [39]. In the first stage of this work, after an initial estimate of the interval for the parameters of the neural network is made, a series of rules for partitioning and expanding the initial value interval are applied with the usage of grammatical evolution, until a promising value interval is found. In the second stage of the method, a genetic algorithm is applied to train the parameters of the artificial neural network. Training is performed within the optimal interval of values found in the first stage of the method. The proposed method utilizes a genetic algorithm in the second stage to train the neural network, but any other optimization method can be used instead of genetic algorithms. However, genetic algorithms are used because they are easily adaptable to many optimization problems as they are fault tolerant, and because they can be easily parallelized using modern computing techniques.

The proposed method seeks in its first stage to find, using an evolutionary rule technique, a range of values for the parameters of the artificial neural networks. This interval should on the one hand be small enough to speed up the method of the second stage that will be used to train the model, and on the other hand, the artificial neural network should have satisfactory generalization abilities within the interval of values of the first stage.

The rest of this article is organized as follows: in Section 2, the steps of the proposed method are outlined in detail, in Section 3, the datasets used in the conducted experiments as well the experimental results are presented, and finally, in Section 4, some conclusions are presented.

2. Method Description

In this section a detailed description of the grammatical evolution technique is provided, accompanied by the proposed language that creates intervals for the parameters of the neural networks. Subsequently, the first phase of the proposed technique is described, and finally, the second phase, with the application of a genetic algorithm to the outcome of the first phase, is thoroughly analyzed.

2.1. Grammatical Evolution

Grammatical evolution is an evolutionary process, where the chromosomes represent production rules of some provided BNF (Backus–Naur form) grammar [40], and this evolutionary process can evolve programs in the underlying language. Grammatical evolution has been used successfully in many real-world problems, such as function approximation [41,42], the solution of trigonometric equations [43], automatic music composition [44], neural network construction [45,46], creating numeric constraints [47], video games [48,49], estimation of energy demand [50], combinatorial optimization [51], cryptography [52], the evolution of decision trees [53], the automatic design of analog electronic circuits [54], etc. Any BNF grammar can be described as the set $G=\left(N,T,S,P\right)$ where

• N is the set of the non-terminal symbols. Every symbol in N has a series of production rules, used to produce terminal symbols.
• T is the set of terminal symbols.
• S denotes the start symbol of the grammar, with $S\in N$.
• P is the set of production rules, to create terminal symbols non-terminal symbols. These rules are in the form $A\to a$ or $A\to aB,A,B\in N,a\in T$.

The algorithm initiates from the symbol S and iteratively produces terminal symbols by replacing non-terminal symbols with the right hand of the selected production rule. Every production rule is selected using the following steps:

• Obtain the next element V from the current chromosome.
• Select the next production rule according to: Rule = V mod $N_R$, where $N_R$ is the total number of production rules for the current non-terminal symbol.

The grammar used in the current work is shown in Figure 1.

The symbols in <> represent the non-terminal symbols of the grammar. The numbers in parentheses in the right part of each rule indicate production rule sequence numbers. In the proposed language described by the grammar of the scheme, three distinct operations can be applied to parameters of the artificial neural network either on the left end of the parameter’s value range (described by <lcommand>) or on the right end of the value range (described by the <rcommand> symbol). These commands are

1.

NOTHING. This command means that no action takes place.

2.

EXPAND. With this command, the corresponding end of the value field is extended by 50% of the width of the field.

3.

DIVIDE. With this command, the corresponding end of the value field is shrunk by 50% of the width of the field.

As a complete example, consider a neural network with $H=2$ hidden nodes and the dimension of the objective problem is set to $d=2$. Hence, the total number of parameters in the neural network is $n=(d+2)H=8$. In addition, for reasons of simplicity, we consider that the initial range of values for all parameters of the artificial neural network is the interval [−2, 2] and is denoted as the set of intervals

$$I_N=\left\{\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right]\right\}$$

(5)

and also consider the chromosome

$$x=\left[9,8,6,4,15,9,16,23,7\right]$$

The steps to produce the final program

$$p_{\mathrm{test}}=(x7,\mathrm{EEXPAND},\mathrm{NOTHING}),(x1,\mathrm{DIVIDE},\mathrm{EXPAND})$$

are shown in

Table 1. Steps to produce a valid expression from the BNF grammar.

Expression	Chromosome	Operation
	9,8,6,4,15,9,16,23,8	9 mod 2=1
<expr>,<expr>	8,6,4,15,9,16,23,8	8 mod 2=0
(<xlist>,<lcommand>,<rcommand>),<expr>	6,4,15,9,16,23,8	6 mod 8=6
(x7,<lcommand>,<rcommand>),<expr>	4,15,9,16,23,8	4 mod 3=1
(x7,EXPAND,<rcommand>),<expr>	15,9,16,23,8	15 mod 3=0
(x7,EXPAND,NOTHING),<expr>	9,16,23,8	9 mod 2 =1
(x7,EXPAND,NOTHING),(<xlist>,<lcommand>,<rcommand>)	16,23,8	16 mod 8=0
(x7,EXPAND,NOTHING),(x1,<lcommand>,<rcommand>)	23,8	23 mod 3=2
(x7,EXPAND,NOTHING),(x1,DIVIDE,<rcommand>)	8	8 mod 3=2
(x7,EXPAND,NOTHING),(x1,DIVIDE,EXPAND)

.

After the application of the program $p_{\mathrm{test}}$ to the original interval $I_N$ of Equation (5), the new set has as follows:

$$I_N=\left\{\left[-4,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[0,4\right],\left[-2,2\right]\right\}$$

(6)

2.2. The First Phase of the Proposed Method

Before starting the process of creating partitioning rules and expanding the value interval of the neural network weights, an initial estimate of this interval should be made. In the present work, only a few steps of a genetic algorithm are applied to make an initial estimate of the value interval. At the end of the genetic algorithm, the chromosome with the best value is

$$IX=\left(I{X}_1,I{X}_2,\dots ,I{X}_n\right)$$

(7)

This, the initial interval set for the parameters of the neural network is calculated as

$$I_N=\left\{\left[-I{X}_1,I{X}_1\right],\left\{-I{X}_2,I{X}_2\right\},\dots ,\left[-I{X}_n,I{X}_n\right]\right\}$$

(8)

The first phase of the proposed technique will accept the set $I_N$ as input and seek to compute a new set by reducing the training error of the artificial neural network. The first step is mainly a genetic algorithm and the corresponding steps are as follows:

1.

Set $N_c$ as the number of chromosomes for the grammatical evolution.

2.

Set H as the number of weights for the neural network.

3.

Set $N_g$ as the maximum number of allowed generations.

4.

Set $p_s$ as the selection rate, with $p_s\le 1$.

5.

Set $p_m$ as the mutation rate, with $p_m\le 1$.

6.

Set $N_s$ as the number of randomly created neural networks, which will be used in the fitness calculation.

7.

Initialize randomly the $N_c$ chromosomes. Every chromosome is a set of integer numbers used to produce valid programs through grammatical evolution and the associated grammar of Figure 1.

8.

Set $f^{*}=\left[\infty ,\infty \right]$, the best discovered fitness. For this algorithm, we consider the fitness function $f_g$ of any given chromosome g as an interval $f_g=\left[f_{g,\mathbf{low}},f_{g,\mathbf{upper}}\right]$

9.

Set iter=0.

10.

For  $i=1,\dots ,N_c$do

(a)

Create for the chromosome $c_i$ the corresponding program $p_i$ using the grammar of Figure 1.

(b)

Apply the program $p_i$ to $I_N$ in order to produce the bounds $\left[\overrightarrow{L_{p_i}},\overrightarrow{R_{p_i}}\right]$.

(c)

Set  $E_{\min }=\infty ,E_{\max }=-\infty$

(d)

For  $j=1,\dots ,N_S$do

i.

Create randomly $g_j\in$$\left[\overrightarrow{L_{p_i}},\overrightarrow{R_{p_i}}\right]$ as a set for the parameters of neural network.

ii.

Calculate the associated training error $E_{g_j}={\sum }_{k=1}^M{\left(N\left({\overrightarrow{x}}_k,\overrightarrow{g_j}\right)-y_k\right)}^2$

iii.

If  $E_{g_j}\le E_{\min }$  then  $E_{\min }=E_{gj}$

iv.

If  $E_{g_j}\ge E_{\max }$  then  $E_{\max }=E_{g_j}$

(e)

EndFor

(f)

Set $f_i=\left[E_{\min },E_{\max }\right]$ as the fitness value for the chromosome $c_i$.

11.

EndFor

12.

Apply the selection procedure. Firstly, the chromosomes are sorted with correspondence to their fitness values. Since fitness is considered an interval, a fitness comparison function is required. For this reason, the operator $L^{*}\left(f_a,f_b\right)$ is used to compare two fitness values $f_a=\left[a_1,a_2\right]$ and $f_b=\left[b_1,b_2\right]$ as follows:

$$\begin{array}{ccc}\hfill L^{*}\left(f_a,f_b\right)& =& \left\{\begin{array}{cc}\mathrm{TRUE},\hfill & a_1<b_1,\mathrm{OR}\left(a_1=b_1\mathrm{AND}{a}_2<b_2\right)\hfill \\ \mathrm{FALSE},\hfill & \mathrm{OTHERWISE}\hfill \end{array}\right.\hfill \end{array}$$

(9)

In practice this means that the fitness value $f_a$ is considered smaller than $f_b$ if $L^{*}\left(f_a,f_b\right)=\mathrm{TRUE}$. The first $\left(1-p_s\right)\times N_c$ chromosomes with the lowest fitness values are copied to the next generation. The remaining chromosomes are substituted by chromosomes produced by the crossover procedure. During the selection process, for every new offspring, two chromosomes are selected as parents from the population using the well-known procedure of tournament selection.

13.

Apply the crossover procedure. For each pair $(z,w)$ of parents, two new chromosomes $\tilde{z}$ and $\tilde{w}$ are created using the one-point crossover, graphically shown in Figure 2.

14.

Apply the mutation procedure. For each element of every chromosome alter the corresponding element with probability $p_m$.

15.

Set iter=iter+1

16.

If $iter\le N_g$ goto step 10.

2.3. The Second Phase of the Proposed Method

The result of the first phase of the proposed method is an interval of values for the parameters of the artificial neural network. This interval of values can be used to minimize the error function of the network with the parameters taking values exclusively in the above interval of values. Any optimization method can be used for this purpose, but in this work, a genetic algorithm was chosen, the steps of which are given below.

1.

Initialization Step

(a)

Set $N_c$ as the number of chromosomes that participate in the genetic algorithm.

(b)

Set $N_g$ as the maximum number of allowed iterations.

(c)

Set H as the number of weights for the neural network.

(d)

Obtain the best interval S from the previous step of Section 2.2.

(e)

Initialize using uniform distribution the $N_C$ chromosomes in S.

(f)

Set $p_s$ as the selection rate, with $p_s\le 1$.

(g)

Set $p_m$ as the mutation rate, with $p_m\le 1$.

(h)

Set  iter=0.

2.

Fitness calculation Step

(a)

For  $i=1,\dots ,N_g$do

i.

Calculate the fitness $f_i$ of chromosome $g_i$ as $f_i={\sum }_{j=1}^M{\left(N\left({\overrightarrow{x}}_j,\overrightarrow{g_i}\right)-y_j\right)}^2$

(b)

EndFor

3.

Genetic operations step

(a)

Selection procedure: Initially, the chromosomes are sorted according to their fitness values. The first $\left(1-p_s\right)\times N_c$ chromosomes with the lowest fitness values are copied to the next generation. The remaining chromosomes are substituted by chromosomes produced by the crossover procedure. During the selection process, for every new offspring, two chromosomes are selected as parents from the population using the well-known procedure of tournament selection.

(b)

Crossover procedure: For each pair $(z,w)$ of selected parents, two chromosomes $\tilde{z}$ and $\tilde{w}$ are constructed using the following equations:

$$\widetilde{z_i}=a_i{z}_i+\left(1-a_i\right)w_i$$

$$\widetilde{w_i}=a_i{w}_i+\left(1-a_i\right)z_i$$

(10)

where $a_i$ is a random number with the property $a_i\in [-0.5,1.5]$ [55].

(c)

Mutation procedure: For each element of every chromosome, alter the corresponding element with probability $p_m$.

4.

Termination Check Step

(a)

Set  $iter=iter+1$

(b)

If $\mathrm{iter}\le N_g$ go to step 2, otherwise apply a local search procedure to the best chromosome of the population. In the current work, the BFGS variant of Powell [56] was used.

3. Experiments

The effectiveness of the proposed method was evaluated on a series of classification and regression datasets, used in the relevant literature. These datasets can be downloaded from the following online databases:

1.

UCI dataset repository, https://archive.ics.uci.edu/ml/index.php [57] (accessed on 4 November 2023).

2.

Keel repository, https://sci2s.ugr.es/keel/datasets.php [58] (accessed on 4 November 2023).

3.

The Statlib URL ftp://lib.stat.cmu.edu/datasets/index.html (accessed on 4 November 2023).

3.1. Experimental Datasets

The classification datasets used in this paper are the following:

1.

Appendicitis dataset, a medical purpose dataset, suggested in [59].

2.

Australian dataset [60], a dataset related to credit card transactions.

3.

Balance dataset [61], related to psychological states.

4.

Cleveland dataset, which is a medical dataset [62,63].

5.

Dermatology dataset [64], a medical dataset related to erythemato-squamous diseases.

6.

Heart dataset [65], a medical dataset related to heart diseases.

7.

Hayes roth dataset [66].

8.

HouseVotes dataset [67], related to votes in the U.S. House of Representatives Congressmen.

9.

Ionosphere dataset, used for classification of radar returns from the ionosphere [68,69].

10.

Liverdisorder dataset [70], a medical dataset related to liver disorders.

11.

Mammographic dataset [71], used to identify breast tumors.

12.

Parkinsons dataset, a medical dataset related to Parkinson’s disease (PD) [72].

13.

Pima dataset [73], used to detect the presence of diabetes.

14.

Popfailures dataset [74], a dataset related to climate measurements.

15.

Regions2 dataset, medical dataset related to hepatitis C [75].

16.

Saheart dataset [76], a medical dataset related to heart diseases.

17.

Segment dataset [77], an image processing dataset.

18.

Wdbc dataset [78], a medical dataset related to breast tumors.

19.

Wine dataset, used to detect the origin of wines [79,80].

20.

Eeg datasets, a medical dataset related to EEG measurements [81]. There are three different cases from this dataset used here denoted as Z_F_S, ZO_NF_S, ZONF_S.

21.

Zoo dataset [82], used to classify animals.

The descriptions of the used regression datasets are as follows:

1.

Abalone dataset [83], used to to predict the age of abalone from physical measurements.

2.

Airfoil dataset, derived from NASA [84].

3.

Baseball dataset, used to estimate the salary of baseball players.

4.

BK dataset [85], used to predict the points scored in a basketball game.

5.

BL dataset, an electrical engineering dataset.

6.

Concrete dataset [86].

7.

Dee dataset, used to predict the price of electricity.

8.

Diabetes dataset, a medical dataset.

9.

Housing dataset, provided in [87].

10.

FA dataset, used to predict the fit body fat.

11.

MB dataset, available from from Smoothing Methods in Statistics [85].

12.

MORTGAGE dataset, related to economic data from the USA.

13.

PY dataset, (Pyrimidines problem) [88].

14.

Quake dataset, used to predict the strength of earthquakes.

15.

Treasure dataset, related to economic data from the USA.

16.

Wankara dataset, a dataset related to weather.

3.2. Experimental Results

The proposed method was coded in ANSI C++, and the optimization methods were obtained from the freely available OPTIMUS computing environment, downloaded from https://github.com/itsoulos/OPTIMUS/ (accessed on 9 September 2023). The results were validated using the 10-fold validation technique in all datasets. The experiments were executed 30 times for each dataset using a different seed for the random generator each time. The average classification error is reported for the case of classification datasets and the average mean test error for the regression datasets. The machine used in the experiments was an AMD Ryzen 5950X with 128GB of RAM, running the Debian Linux operating system. The experimental settings are listed in

Table 2. Experimental parameters.

Parameter	Value
H	10
$N_C$	200
$N_S$	50
$N_t$	200
$p_s$	0.10
$p_m$	0.01

. The experimental results for the classification datasets are shown in

Table 3. Experimental results for classification datasets. The values in cells indicate average classification error as measured on test set. The numbers in parentheses denote the standard deviation.

Dataset	Genetic	Adam	Rprop	Neat	Proposed
Appendicitis	18.10% (6.32)	16.50% (7.73)	16.30% (5.27)	17.20% (4.12)	17.00% (6.23)
Australian	32.21% (5.99)	35.65% (5.83)	36.12% (5.52)	31.98% (6.03)	24.55% (4.64)
Balance	8.97% (2.64)	7.87% (3.09)	8.81% (2.36)	23.14% (4.16)	16.71% (3.98)
Cleveland	51.60% (6.39)	67.55% (6.98)	61.41% (9.10)	53.44% (7.26)	47.91% (4.78)
Dermatology	30.58% (4.75)	26.14% (3.11)	15.12% (2.40)	32.43% (4.74)	8.93% (2.36)
Hayes Roth	56.18% (6.97)	59.70% (5.41)	37.46% (4.41)	50.15% (4.43)	32.21% (2.58)
Heart	28.34% (4.78)	38.53% (4.45)	30.51% (3.63)	39.27% (4.14)	17.40% (2.52)
HouseVotes	6.62% (2.11)	7.48% (1.81)	6.04% (1.17)	10.89% (2.30)	3.48% (1.43)
Ionosphere	15.14% (2.57)	16.64% (3.20)	13.65% (2.45)	19.67% (4.28)	7.14% (1.10)
Liverdisorder	31.11% (4.59)	41.53% (4.74)	40.26% (3.99)	30.67% (3.12)	28.90% (2.91)
Lymography	23.26% (3.84)	29.26% (4.72)	24.67% (3.48)	33.70% (4.17)	17.86% (2.42)
Mammographic	19.88% (2.79)	46.25% (2.66)	18.46% (2.34)	22.85% (3.27)	17.32% (1.79)
Parkinsons	18.05% (3.16)	24.06% (3.28)	22.28% (2.79)	18.56% (1.87)	14.35% (1.79)
Pima	32.19% (4.82)	34.85% (4.26)	34.27% (4.24)	34.51% (4.67)	25.58% (2.55)
Popfailures	5.94% (1.71)	5.18% (1.79)	4.81% (1.81)	7.05% (2.87)	4.58% (1.32)
Regions2	29.39% (3.88)	29.85% (3.95)	27.53% (3.23)	33.23% (4.41)	28.32% (3.59)
Saheart	34.86% (4.90)	34.04% (4.74)	34.90% (4.75)	34.51% (5.57)	27.43% (3.88)
Segment	57.72% (2.71)	49.75% (3.01)	52.14% (4.85)	66.72% (4.74)	20.68% (2.17)
Wdbc	8.56% (2.90)	35.35% (5.06)	21.57% (4.55)	12.88% (3.48)	5.23% (1.66)
Wine	19.20 (2.66)	29.40% (3.37)	30.73% (3.78)	25.43% (3.19)	5.35% (1.74)
Z_F_S	10.73% (2.80)	47.81% (5.75)	29.28% (4.81)	38.41% (6.18)	6.56% (1.45)
ZO_NF_S	8.41% (2.35)	47.43% (5.79)	6.43% (2.35)	43.75% (6.98)	3.60% (1.05)
ZONF_S	2.60% (0.33)	11.99% (1.19)	27.27% (1.58)	5.44% (1.11)	2.21% (0.54)
ZOO	16.67% (3.28)	14.13% (2.52)	15.47% (3.10)	20.27% (6.47)	6.10% (1.67)
AVERAGE	23.60%	31.54%	25.65%	29.42%	16.23%

, while for the regression datasets, the results are shown in

Table 4. Experimental results for regression datasets. The values indicate average regression error as measured on test set. The numbers in parentheses denote the standard deviation.

Dataset	Genetic	Adam	Rprop	Neat	Proposed
ABALONE	7.17 (1.11)	4.30 (0.55)	4.55 (0.75)	9.88 (1.61)	4.48 (0.52)
AIRFOIL	0.003 (0.002)	0.005 (0.003)	0.002 (0.001)	0.067 (0.002)	0.002 (0.001)
BASEBALL	103.60 (15.85)	77.90 (16.59)	92.05 (23.51)	100.39 (22.54)	51.39 (10.13)
BK	0.027 (0.009)	0.03 (0.004)	1.599 (0.15)	0.15 (0.02)	0.02 (0.005)
BL	5.74 (1.64)	0.28 (0.11)	4.38 (0.19)	0.05 (0.02)	0.002 (0.001)
CONCRETE	0.0099 (0.001)	0.078 (0.013)	0.0086 (0.001)	0.081 (0.004)	0.004 (0.0006)
DEE	1.013 (0.21)	0.63 (0.08)	0.608 (0.07)	1.512 (0.79)	0.23 (0.05)
DIABETES	19.86 (5.57)	3.03 (0.35)	1.11 (0.57)	4.25 (1.92)	0.41 (0.08)
HOUSING	43.26 (5.84)	80.20 (8.82)	74.38 (6.85)	56.49 (5.65)	24.55 (4.48)
FA	1.95 (0.44)	0.11 (0.021)	0.14 (0.01)	0.19 (0.014)	0.01 (0.005)
MB	3.39 (0.40)	0.06 (0.03)	0.055 (0.03)	0.061 (0.03)	0.048 (0.03)
MORTGAGE	2.41 (0.26)	9.24 (1.24)	9.19 (1.62)	14.11 (3.34)	0.65 (0.16)
PY	5.41 (0.63)	0.09 (0.013)	0.039 (0.019)	0.075 (0.022)	0.025 (0.012)
QUAKE	0.040 (0.004)	0.06 (0.017)	0.041 (0.006)	0.298 (0.13)	0.038 (0.005)
TREASURY	2.929 (0.69)	11.16 (1.37)	10.88 (1.26)	15.52 (2.52)	0.84 (0.29)
WANKARA	0.012 (0.005)	0.02 (0.005)	0.0003 (0.001)	0.005 (0.001)	0.0002 (0.0001)
AVERAGE	12.30	11.70	12.44	12.70	5.17

. The following apply to the results tables:

1.

A genetic algorithm where the parameters have the values of Table 2 used to train a neural network with H hidden nodes. The results in the experimental tables are denoted by the label GENETIC.

2.

The Adam optimization method is used to train a neural network with H hidden nodes. The column ADAM denotes the results for this method.

3.

The RPROP method is used to train a neural network with H hidden nodes. The corresponding results are denoted by RPROP in the relevant tables.

4.

The NEAT method (NeuroEvolution of Augmenting Topologies) [89], where the maximum number of allowed generations is the same as in the case of the genetic algorithm.

5.

The proposed method (denoted as PROPOSED) was used with the experimental settings and are shown in Table 2.

6.

An extra line was also added to the experimental tables under the title AVERAGE. This line represents the average classification or regression error for all datasets.

In general, the proposed technique outperforms the rest of the techniques in the experiments. In fact, in many cases, the test error is reduced by more than 70%. Additionally, more experiments were conducted using the number of processing nodes (H) as well as the number of generations ($N_t$), to determine the stability of the proposed technique. The average classification error for the classification datasets using different values for the H parameter is shown graphically in Figure 3. Furthermore, the average classification error of the proposed method for different numbers of maximum generations $N_t$ is shown graphically in Figure 4. The proposed technique does not show significant changes in its behavior as the critical parameters change, and therefore, its stability to changes is evident. Moreover, the average execution time for different values of the $N_t$ parameter is outlined in Figure 5. Judging from the above experiments, it can be said that the choice of values for the critical parameters H and $N_t$ is a compromise between performance and speed for the proposed technique.

The numerical superiority of the proposed technique over the others in the performed experiments is presented graphically in Figure 6 and Figure 7, where Wilcoxon signed-rank tests were performed on the executed experiments. However, since the proposed technique consists exclusively of a series of genetic algorithms, it can be significantly accelerated by the use of parallel computing techniques, since one of the main characteristics of genetic algorithms is their ease of parallelization [90,91]. Programming techniques that can be used to parallelize the method could be the MPI programming interface [92] or the OpenMP programming library [93].

4. Conclusions

This paper proposed a two-step technique to efficiently find a reliable interval of values for the parameters of artificial neural networks. In the first stage with partitioning techniques and using grammatical evolution, a range of parameter values was searched, and in the second stage, a genetic algorithm trained the artificial neural network within the optimal range of values of the first stage. The proposed technique is highly general and was successfully applied to both data fitting and classification problems. In both cases, the experimental results demonstrate its superiority over other techniques appearing in the relevant literature on the training of artificial neural networks. In fact, in many cases in the performed experiments, the proposed method outperforms others that were tested in percentages that exceed 70%. In the future, a series of actions can be carried out to improve the results and speed up the proposed method, such as:

1.

There is a need for more efficient techniques for initializing the value space for artificial neural network parameters. In the present work, the optimal result from the execution of a limited number of steps by a genetic algorithm was used as an initial estimate of the value interval.

2.

In the present work, the same techniques as in any whole-chromosome genetic algorithm were used to perform the crossover and mutation operations. Research could be conducted at this point to find more focused crossover and mutation techniques for this particular problem.

3.

The present technique consists of two phases, in each of which a problem-adapted genetic algorithm is executed. This means that significant computational time is required to complete the algorithm. However, since genetic algorithms are inherently parallelizable, modern parallel programming techniques could be used here.