Adapting the Parameters of RBF Networks Using Grammatical Evolution

Abstract

Radial basis function networks are widely used in a multitude of applications in various scientific areas in both classification and data fitting problems. These networks deal with the above problems by adjusting their parameters through various optimization techniques. However, an important issue to address is the need to locate a satisfactory interval for the parameters of a network before adjusting these parameters. This paper proposes a two-stage method. In the first stage, via the incorporation of grammatical evolution, rules are generated to create the optimal value interval of the network parameters. During the second stage of the technique, the mentioned parameters are fine-tuned with a genetic algorithm. The current work was tested on a number of datasets from the recent literature and found to reduce the classification or data fitting error by over 40% on most datasets. In addition, the proposed method appears in the experiments to be robust, as the fluctuation of the number of network parameters does not significantly affect its performance.

1. Introduction

Many practical problems of the modern world can be thought of as data fitting problems, such as, for example, problems that appear in physics [1,2], problems related to chemistry [3,4], economic problems [5,6], medicine problems [7,8], etc. Radial basis function (RBF) networks are commonly used machine learning tools to handle problems of this nature [9,10]. Usually, an RBF network is expressed using the following equation:

$$y\left(\overrightarrow{x}\right)=\sum _{i=1}^k{w}_i\varphi \left(∥\overrightarrow{x}-\overrightarrow{c_i}∥\right)$$

(1)

where the symbols in the equation are defined as follows:

• The element $\overrightarrow{x}$ represents the input pattern from the dataset describing the problem. For the rest of this paper, the notation d will be used to represent the number of elements in $\overrightarrow{x}$.
• The parameter k denotes the number of weights used to train the RBF network, and the associated vector of weights is denoted as $\overrightarrow{w}$.
• The vectors $\overrightarrow{c_i},i=1,\dots ,k$ stand for the centers of the model.
• The value $y\left(\overrightarrow{x}\right)$ represents the value of the network for the given pattern $\overrightarrow{x}$.

The $\varphi \left(x\right)$ function, in most cases, represents the Gaussian function given by:

$$\varphi \left(x\right)=exp\left(-\frac{{\left(x-c\right)}^2}{{\sigma }^2}\right)$$

(2)

The main advantages of RBF networks are as follows:

• They have a simpler structure than other models used in machine learning, such as multilayer perceptron neural networks (MLPs) [11], since they have only one processing layer and therefore have faster training techniques, as well as faster response times.
• They can be used to efficiently approximate any continuous function [12].

RBF networks have been applied in a variety of problems, such as physics problems [13,14,15,16], solving differential equations [17,18,19], robotics problems [20,21], face recognition [22], digital communications [23,24], chemistry problems [25,26], economic problems [27,28,29], and network security problems [30,31]. Recently, a variety of papers have appeared proposing novel initialization techniques for these networks’ parameters [32,33,34]. Benoudjit et al. [35] discussed the effect of kernel widths on RBF networks. Moreover, Neruda et al. [36] presented a comparison of some learning methods for RBF networks. Additionally, a variety of pruning techniques [37,38,39] have been suggested in the literature for decreasing the number of parameters. Due to the widespread usage of RBF networks and because considerable computing time is often required for their effective training, in recent years, a series of techniques have been published [40,41] for the exploitation of parallel computing units to adjust their parameters.

In the same direction of research, other machine learning models have been proposed, such as support vector machines (SVM) [42,43] and decision trees [44,45]. Also, Wang et al. suggested an auto-encoder reduction method applied on a series of large datasets [46]. Various methods have been proposed in the same direction, such as the work of Agarwal and Bhanot [47], which proposed the use of the ABC algorithm [48] and the incorporation of the Firefly algorithm [49] to adapt an RBF network’s parameters. Furthermore, Gyamfi et al. [50] recently proposed a differential RBF network that incorporated partial differential equations, aiming to make the network more robust in the presence of noisy data. Also, Li et al. [51] proposed a multivariate ensemble-based hierarchical linkage strategy (ME-HL) for the evaluation of the system reliability of aeroengine cooling blades.

The parameters of an RBF network can be modified in order to minimize the following loss function, which is called the training error of the network:

$$E\left(y(x,g)\right)=\sum _{i=1}^m{\left(y\left({\overrightarrow{x}}_i,\overrightarrow{g}\right)-t_i\right)}^2$$

(3)

where the parameter m denotes the number of patterns and $t_i$ represent the expected output for pattern ${\overrightarrow{x}}_i$. The vector $\overrightarrow{g}$ represents the parameter set of the network.

A common method of calculating the parameters in these neural networks uses a technique to calculate the centers of the functions $\varphi \left(x\right)$, and then the vector of weights $\overrightarrow{w}$ is calculated as a solution of a linear system of equations. Typically, the method used to calculate the centers is the well-known k-means method [52]. In many cases, this way of estimating the parameters leads to over-fitting of the model so that it cannot generalize satisfactorily to unknown data. Furthermore, since there is no range of values for the parameters, there is the possibility that they will take extremely large or extremely small values, with the result that any generalizability of the model is lost. This work suggests a two-phase method to minimize the error of Equation (3). During the first phase, an attempt is made to bound the parameter values to intervals at which the training error is likely to be significantly reduced. The identification of the most promising intervals for the parameters is performed using a technique that utilizes grammatical evolution [53], which collects information from the training data. During the second phase, the parameters can be trained inside the best located range of the first phase using a global optimization method [54,55]. In the proposed approach, the widely used method of genetic algorithm [56,57,58] was used for the second phase of the process. The main contributions of the suggested approach are as follows:

• The first phase of the procedure seeks to locate a range of values for the parameters while also reducing the error of the network on the training dataset.
• The rules grammatical evolution uses in the first phase are simple and can be generalized to any dataset for data classification or fitting.
• The determination of the value interval is conducted in such a way that it is faster and more efficient to train the parameters with an optimization method during the second phase.
• After identifying a promising value interval from the first phase, any global optimization method can be used on that value interval to effectively minimize the network training error.

The rest of this paper is divided into the following sections. In Section 2, the proposed method is fully described. Section 3 presents the used datasets and the conducted experiments. Finally, in Section 4, we provide a discussion on the conducted experiments.

2. Method Description

This section starts with an extended presentation of the grammatical evolution technique and the grammar that is used to generate partition rules for the parameter set of RBFs. Afterwards, the first phase of the proposed methodology is extensively analyzed, and then the second phase is presented, where a genetic algorithm is applied to the outcome of the first phase.

2.1. Grammatical Evolution

Grammatical evolution is a genetic algorithm, where the chromosomes are integer numbers. Genetic algorithms were initially proposed by John Holland [59] and are biologicaly inspired algorithms. The algorithm starts by forming a population of potential solutions to an optimization problem. These solutions are called chromosomes, and they are gradually altered using the genetic operators of selection, crossover, and mutation [60]. The chromosomes in the grammatical evolution method stand for a series of production rules of any given BNF (Backus–Naur form) grammar [61]. Grammatical evolution has been applied with success in a variety of cases, such as function approximation [62,63], solving equations related to trigonometry [64], the automatic composition of music [65], the construction of neural networks [66,67], producing numeric constraints [68],video games [69,70], the estimation of energy demand [71], combinatorial optimization [72], and cryptography [73]. The BNF grammar can be used to describe the syntax of programming languages, and usually, it is defined as $G=\left(N,T,S,P\right)$, where:

• N is a set of the non-terminal symbols. A series of production rules is associated with every non-terminal symbol. The application of these production rules produces series of terminal symbols.
• T stands for the set of terminal symbols.
• S denotes the start symbol of the grammar and $S\in N$.
• P defines the set of production rules. These are rules that follow the following notations: $A\to a$ or $A\to aB,A,B\in N,a\in T$.

The algorithm begins using the symbol S and gradually creates series of terminal symbols with the assistance of the production rules. The production rules are selected through the following procedure:

• Denote with V the next element form of the current chromosome.
• The next production rule is calculated as: Rule = V mod R. The number R stands for the total number of production rules for the non-terminal symbol that is currently under processing.

Algorithm 1 shows the BNF grammar used by the proposed method. Each non-terminal symbol of the grammar is enclosed in a <> symbol. The numbers that are enclosed in parentheses represent the sequence numbers of production rules for every non-terminal symbol. Every RBF network with k weights is constructed by the following series of parameters:

• A series of vectors $\overrightarrow{c_i},i=1,\dots ,k$ that stand for the centers of the model.
• For every Gaussian unit, an additional parameter ${\sigma }_i$ is required.
• The output weight vector $\overrightarrow{w}$.

The number n is the total number of parameters of the problem. In the case of this paper, it is the total number of parameters of the RBF network. For the current work, the number n can be computed using the following formula:

$$n=(d+2)\times k$$

(4)

Algorithm 1 The BNF grammar used in the proposed method to produce intervals for the RBF parameters. By using this grammar in the first phase of the current work, the optimal interval of values for the parameters can be identified.

S::=<expr>   (0)  <expr> ::=  (<xlist> , <digit>,<digit>)  (0)                         |<expr>,<expr>                (1) <xlist>::=x1    (0)                          | x2 (1)                          .........                         | xn (n) <digit>  ::= 0 (0)                         | 1 (1)

The number n in the corresponding grammar is computed as follows:

• For each center $\overrightarrow{c_i},i=1,\dots ,k$, there are d variables. As a consequence, every center requires $d\times k$ parameters.
• Every Gaussian unit requires an additional parameter: ${\sigma }_i,i=1,\dots ,k$, which means k more parameters.
• The weight vector $\overrightarrow{w}$ used in the output has k parameters.

As an example of production, the chromosome $x=\left[9,8,6,4,15,9,16,23,8\right]$ is considered, where $d=2,k=2,n=8$. The steps to produce the final program $p_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}=(x7, 0, 1),(x1, 1, 0)$ are outlined in

Table 1. The series of steps used to compute a valid expression from the BNF grammar for a given chromosome.

Expression	Chromosome	Operation
	9, 8, 6, 4, 15, 9, 16, 23, 8	9 mod 2 = 1
<expr>,<expr>	8, 6, 4, 15, 9, 16, 23, 8	8 mod 2 = 0
(<xlist>,<digit>,<digit>),<expr>	6, 4, 15, 9, 16, 23, 8	6 mod 8 = 6
(x7,<digit>,<digit>),<expr>	4, 15, 9, 16, 23, 8	4 mod 2 = 0
(x7,0,<digit>),<expr>	15, 9, 16, 23, 8	15 mod 2 = 1
(x7,0,1),<expr>	9, 16, 23, 8	9 mod 2 = 1
(x7,0,1),(<xlist>,<digit>,<digit>)	16, 23, 8	16 mod 8 = 0
(x7,0,1),(x1,<digit>,<digit>)	23, 8	23 mod 2 = 1
(x7,0,1),(x1,1,<digit>)	8	8 mod 2 = 0
(x7,0,1),(x1,1,0)

. Every partition program consists of a series of partition rules. Each partition rule contains three elements:

• The variable for which its original interval will be partitioned, for example, $x_7$.
• An integer number with values 0 and 1 at the left margin of the interval. If this value is 1, then the left margin of the corresponding variable’s value field will be divided by two; otherwise, no change will be made.
• An integer number with values 0 and 1 at the right end of the range of values of the variable. If this value is 1, then the right end of the corresponding variable’s value field will be divided by two; otherwise, no change will be made.

Hence, for the example program $p_{\mathrm{test}}$, the two partition rules will divide the right end of the variable $x_7$ and the left end of the variable $x_1$.

2.2. The First Phase of the Proposed Algorithm

The purpose of this phase is to initialize the bounds of the RBF model and discover a promising interval for the corresponding values. For this initialization, the k-means algorithm [52] is used, which is also used for the traditional RBF network training technique. A description of this algorithm in a series of steps is shown in Algorithm 2.

Algorithm 2 The k-means algorithm.

Repeat (a) Define $S_j=,j=1\dots k$ (b) For every pattern $x_i,i=1,\dots ,m$ do Compute $j^{*}={min}_{i=1}^k\left\{D\left(x_i,c_j\right)\right\}$. Compute $S_{j^{*}}=S_{j^{*}}\cup \left\{x_i\right\}$. (c) EndFor (d) For every center $c_j,j=1\dots k$ do Denote as $M_j$ the number of points in set $S_j$ Compute $c_j$ as $$c_j=\frac{1}{M_j}\sum _{i=1}^{M_j}{x}_i$$ (e) EndFor Compute the quantities $s_j$ as $${\sigma }_j^2=\frac{{\sum }_{i=1}^{M_j}{\left(x_i-c_j\right)}^2}{M_j}$$ Stop the algorithm if centers $c_j$ do not change anymore.

Having calculated the centers $c_i$ and the corresponding variances ${\sigma }_i$, the algorithm continues to compute the vectors $\overrightarrow{L},\overrightarrow{R}$ with dimension n, which are used as the initial bounds of the parameters. The above vectors are calculated through the procedure of Algorithm 3.

Algorithm 3 The proposed algorithm used to locate the vectors $\overrightarrow{L},\overrightarrow{R}$

Set m = 0 Define $F>1$, the scaling factor. Define $B>0$, the initial upper bound for the weight vector $\overrightarrow{w}$. For  $i=1\dots k$  do (a) For  $j=1\dots d$ do Compute $L_m$=$-F\times c_{ij}$, $R_m$=$F\times c_{ij}$ Compute  $m=m+1$ (b) EndFor (c) Compute $L_m=-F\times {\sigma }_i$, $R_m=F\times {\sigma }_i$ (d) Compute  $m=m+1$ EndFor For  $j=1,\dots ,k$  do (a) Compute  $L_m=-B,R_m=B$ (b) Compute  $m=m+1$ EndFor

The range of values for the first $(d+1)\times k$ parameters is estimated by multiplying the parameter F by the values already estimated by the k-means algorithm. The bounds of the weight vector $\overrightarrow{w}$ are initialized using the value B. Subsequently, the genetic algorithm described here is performed to estimate the most promising range $\left[\overrightarrow{L},\overrightarrow{R}\right]$ for the RBF parameters:

1.

Define as $N_c$ the number of chromosomes that will participate in the the grammatical evolution procedure.

2.

Define as k the number of processing nodes of the used RBF model.

3.

Define as $N_g$ the number of allowed generations.

4.

Define as $p_s$ the used selection rate, with $p_s\le 1$.

5.

Define as $p_m$ the used mutation rate, with $p_m\le 1$.

6.

Define as $N_s$ the total number of RBF networks that will be created randomly in every fitness calculation.

7.

Initialize $N_c$ chromosomes as sets of random numbers.

8.

Set $f^{*}=\left[\infty ,\infty \right]$ as the fitness of the best chromosome. The fitness function $f_g$ of any provided chromosome g is considered an interval $f_g=\left[f_{g,\mathbf{l}\mathbf{o}\mathbf{w}},f_{g,\mathbf{u}\mathbf{p}\mathbf{p}\mathbf{e}\mathbf{r}}\right]$

9.

Set iter = 0.

10.

For  $i=1,\dots ,N_c$ do

(a)

Produce the partition program $p_i$ using the grammar of Figure 1 for the chromosome i.

(b)

Produce the bounds $\left[\overrightarrow{L_{p_i}},\overrightarrow{R_{p_i}}\right]$ for the partition program $p_i$.

(c)

Set  $E_{min}=\infty ,E_{max}=-\infty$

(d)

For  $j=1,\dots ,N_S$ do

i.

Create randomly a set of parameters $\overrightarrow{g_j}\in$$\left[\overrightarrow{L_{p_i}},\overrightarrow{R_{p_i}}\right]$

ii.

Calculate the error $E_{\overrightarrow{g_j}}={\sum }_{k=1}^M{\left(y\left(\overrightarrow{x_k},\overrightarrow{g_j}\right)-t_k\right)}^2$

iii.

If $E_{\overrightarrow{g_j}}\le E_{min}$, then $E_{min}=E_{\overrightarrow{g_j}}$

iv.

If $E_{\overrightarrow{g_j}}\ge E_{max}$, then $E_{max}=E_{\overrightarrow{g_j}}$

(e)

EndFor

(f)

Set the fitness $f_i=\left[E_{min},E_{max}\right]$

11.

EndFor

12.

Perform the procedure of selection. Initially, the chromosomes of the population are sorted according to their fitness values. Since the fitness values are intervals, the $L^{*}$ operator is defined as

$$\begin{array}{ccc}\hfill L^{*}\left(f_a,f_b\right)& =& \left\{\begin{array}{cc}\mathrm{T}\mathrm{R}\mathrm{U}\mathrm{E},\hfill & a_1<b_1,\mathrm{O}\mathrm{R}\left(a_1=b_1\mathrm{A}\mathrm{N}\mathrm{D}{a}_2<b_2\right)\hfill \\ \mathrm{F}\mathrm{A}\mathrm{L}\mathrm{S}\mathrm{E},\hfill & \mathrm{O}\mathrm{T}\mathrm{H}\mathrm{E}\mathrm{R}\mathrm{W}\mathrm{I}\mathrm{S}\mathrm{E}\hfill \end{array}\right.\hfill \end{array}$$

(5)

As a consequence, the fitness value $f_a$ is considered smaller than $f_b$ if $L^{*}\left(f_a,f_b\right)=\mathrm{T}\mathrm{R}\mathrm{U}\mathrm{E}$. The first $\left(1-p_s\right)\times N_c$ chromosomes with smaller fitness values are copied without changes to the next generation of the algorithm. The rest of the chromosomes are replaced by chromosomes created in the crossover procedure.

13.

Perform the crossover procedure. The crossover procedure will create new $p_s\times N_c$ chromosomes. For every pair of created offspring, two parents $(z,w)$ are selected from the current population using the tournament selection method. These parent will produce the offspring $\tilde{z}$ and $\tilde{w}$ using the one-point crossover method shown in Figure 1.

14.

Perform the mutation procedure. In this process, a random number $r\in \left[0,1\right]$ is drawn for every element of each chromosome. The corresponding element is changed randomly if $r\le p_m$.

15.

Set iter = iter + 1

16.

If $\mathrm{iter}\le N_g$, go to step 10.

2.3. The Second Phase of the Proposed Algorithm

The second phase utilizes a genetic algorithm to optimize the parameters of the RBF network. The optimization of the parameters uses as bounds the best interval produced in the first phase of the method. The layout of each chromosome is shown in Figure 2.

• Initialization Step

  (a)

  Define as $N_c$ the number of chromosomes.

  (b)

  Define as $N_g$ the total number of generations.

  (c)

  Define as k the number of processing nodes of the used RBF model.

  (d)

  Define as $S=\left[L_{\mathrm{best}},R_{\mathrm{best}}\right]$ the best-located interval of the first stage of the algorithm of Section 2.2.

  (e)

  Produce $N_C$ random chromosomes in S.

  (f)

  Define as $p_s$ the used selection rate, with $p_s\le 1$.

  (g)

  Define as $p_m$ the used mutation rate, with $p_m\le 1$.

  (h)

  Set iter = 0.

• Fitness calculation step

  (a)

  For $i=1,\dots ,N_g$, do

  i.

  Compute the fitness $f_i$ of each chromosome $g_i$ as $f_i={\sum }_{j=1}^m{\left(y\left(\overrightarrow{x_j},\overrightarrow{g_i}\right)-t_j\right)}^2$

  (b)

  EndFor

• Genetic operations step

  (a)

  Selection procedure. Initially, the population is sorted according to the fitness values. The first $\left(1-p_s\right)\times N_c$ chromosomes with the lowest fitness values remain intact. The rest of the chromosomes are replaced by offspring that will be produced during the crossover procedure.

  (b)

  Crossover procedure: For every two new offspring $\left(\tilde{z},\tilde{w}\right)$, there are two parents $(z,w)$ that are selected from the current population with the selection procedure of tournament selection. The offspring are produced through the following process:

  $$\begin{array}{ccc}\hfill \widetilde{z_i}& =& a_i{z}_i+\left(1-a_i\right)w_i\hfill \\ \hfill \widetilde{w_i}& =& a_i{w}_i+\left(1-a_i\right)z_i\hfill \end{array}$$

  (6)

  The value $a_i$ is a random number, where $a_i\in [-0.5,1.5]$ [74].

  (c)

  Perform the mutation procedure. In this process, a random number $r\in \left[0,1\right]$ is drawn for every element of each chromosome. The corresponding element is changed randomly if $r\le p_m$.

• Termination Check Step

  (a)

  Set  $iter=iter+1$

  (b)

  If $\mathrm{iter}\le N_g$, go to step 2.

The steps of the current algorithm are also outlined graphically in Figure 3 using a flowchart.

3. Experiments

3.1. Experimental Datasets

The proposed method was tested on a wide set of classification and regression problems found in the relevant literature. The method was compared against some other well-known machine learning models. The following databases were used to obtain the datasets:

• The UCI dataset repository, https://archive.ics.uci.edu/ml/index.php (accessed on 5 December 2023);
• The Keel repository, https://sci2s.ugr.es/keel/datasets.php (accessed on 5 December 2023) [75];
• The Statlib URL http://lib.stat.cmu.edu/datasets/ (accessed on 5 December 2023).

The classification datasets are listed in

Table 2. The classification datasets used in the experiments. The column DATASET denotes the number of the dataset, the column CLASSES stands for the number of classes in each dataset, and the column REFERENCE points to the bibliography where the use of the particular dataset is presented.

Dataset	Classes	Reference
APPENDICITIS	2	[76]
AUSTRALIAN	2	[77]
BALANCE	3	[78]
CLEVELAND	5	[79,80]
DERMATOLOGY	6	[81]
HAYES ROTH	3	[82]
HEART	2	[83]
HOUSEVOTES	2	[84]
IONOSPHERE	2	[85,86]
LIVERDISORDER	2	[87]
MAMMOGRAPHIC	2	[88]
PARKINSONS	2	[89]
PIMA	2	[90]
POPFAILURES	2	[91]
SPIRAL	2	[92]
REGIONS2	5	[93]
SAHEART	2	[94]
SEGMENT	7	[95]
WDBC	2	[96]
WINE	3	[97,98]
Z_F_S	3	[99]
ZO_NF_S	3	[99]
ZONF_S	2	[99]
ZOO	7	[100]

, and the regression datasets are listed in

Table 3. The regression datasets used in the experiments. The column DATASET denotes the number of the dataset, and the column REFERENCE points to the bibliography or URL (KEEL or STATLIB) where the use of the particular dataset is presented.

Dataset	Reference
ABALONE	[101]
AIRFOIL	[102]
BASEBALL	STATLIB
BK	[103]
BL	STATLIB
CONCRETE	[104]
DEE	KEEL
DIABETES	KEEL
FA	STATLIB
HOUSING	[105]
MB	[103]
MORTGAGE	KEEL
NT	[106]
PY	[107]
QUAKE	[108]
TREASURY	KEEL
WANKARA	KEEL

.

3.2. Experimental Results

The RBF model used in the experiments was implemented in ANSI C++ with the assistance of the open-source Armadillo library [109]. The optimization methods used were also freely available from the OPTIMUS software, available from https://github.com/itsoulos/OPTIMUS/ (accessed on 5 December 2023). For validation purposes, the 10-fold validation technique was used for all datasets and for all methods that participated in the experiments. Also, all the experiments were conducted 30 times, and the seed number of the random generator was different in each execution. The average classification error is reported for the classification datasets, and the average mean test error is reported for the regression datasets. The machine used in the experiments was an AMD Ryzen 5950X with 128 GB of RAM, and the operating system was Debian Linux. The values of the parameters used in the experiments are shown in

Table 4. The values used for the experimental parameters.

Parameter	Value
$N_c$	200
$N_g$	100
$N_s$	50
F	10.0
B	100.0
k	10
$p_s$	0.90
$p_m$	0.05

. The experimental results for the classification datasets are outlined in

Table 5. The first column denotes the name of the classification dataset, and the numbers in the cells represent the classification error for every method used in the experiments. The last row stands for the average classification error for all datasets.

Dataset	Rprop	Adam	Neat	Rbf-Kmeans	Genrbf	Proposed
Appendicitis	16.30%	16.50%	17.20%	12.23%	16.83%	15.77%
Australian	36.12%	35.65%	31.98%	34.89%	41.79%	22.40%
Balance	8.81%	7.87%	23.14%	33.42%	38.02%	15.62%
Cleveland	61.41%	67.55%	53.44%	67.10%	67.47%	50.37%
Dermatology	15.12%	26.14%	32.43%	62.34%	61.46%	35.73%
Hayes Roth	37.46%	59.70%	50.15%	64.36%	63.46%	35.33%
Heart	30.51%	38.53%	39.27%	31.20%	28.44%	15.91%
HouseVotes	6.04%	7.48%	10.89%	6.13%	11.99%	3.33%
Ionosphere	13.65%	16.64%	19.67%	16.22%	19.83%	9.30%
Liverdisorder	40.26%	41.53%	30.67%	30.84%	36.97%	28.44%
Mammographic	18.46%	46.25%	22.85%	21.38%	30.41%	17.72%
Parkinsons	22.28%	24.06%	18.56%	17.41%	33.81%	14.53%
Pima	34.27%	34.85%	34.51%	25.78%	27.83%	23.33%
Popfailures	4.81%	5.18%	7.05%	7.04%	7.08%	4.68%
Regions2	27.53%	29.85%	33.23%	38.29%	39.98%	25.18%
Saheart	34.90%	34.04%	34.51%	32.19%	33.90%	29.46%
Segment	52.14%	49.75%	66.72%	59.68%	54.25%	49.22%
Spiral	46.59%	48.90%	50.22%	44.87%	50.02%	23.58%
Wdbc	21.57%	35.35%	12.88%	7.27%	8.82%	5.20%
Wine	30.73%	29.40%	25.43%	31.41%	31.47%	5.63%
Z_F_S	29.28%	47.81%	38.41%	13.16%	23.37%	3.90%
ZO_NF_S	6.43%	47.43%	43.75%	9.02%	22.18%	3.99%
ZONF_S	27.27%	11.99%	5.44%	4.03%	17.41%	1.67%
ZOO	15.47%	14.13%	20.27%	21.93%	33.50%	9.33%
AVERAGE	26.56%	32.36%	30.11%	28.84%	33.35%	18.73%

, and the results for the regression datasets are listed in

Table 6. The first column denotes the name of the regression dataset, and the the numbers in the cells represent the regression error for every method used in the experiments. The last row stands for the average regression error for all datasets.

Dataset	Rprop	Adam	Neat	Rbf-Kmeans	Genrbf	Proposed
ABALONE	4.55	4.30	9.88	7.37	9.98	5.16
AIRFOIL	0.002	0.005	0.067	0.27	0.121	0.004
BASEBALL	92.05	77.90	100.39	93.02	98.91	81.26
BK	1.60	0.03	0.15	0.02	0.023	0.025
BL	4.38	0.28	0.05	0.013	0.005	0.0004
CONCRETE	0.009	0.078	0.081	0.011	0.015	0.006
DEE	0.608	0.630	1.512	0.17	0.25	0.16
DIABETES	1.11	3.03	4.25	0.49	2.92	1.74
HOUSING	74.38	80.20	56.49	57.68	95.69	21.11
FA	0.14	0.11	0.19	0.015	0.15	0.033
MB	0.55	0.06	0.061	2.16	0.41	0.19
MORTGAGE	9.19	9.24	14.11	1.45	1.92	0.014
NT	0.04	0.12	0.33	8.14	0.02	0.007
PY	0.039	0.09	0.075	0.012	0.029	0.019
QUAKE	0.041	0.06	0.298	0.07	0.79	0.034
TREASURY	10.88	11.16	15.52	2.02	1.89	0.098
WANKARA	0.0003	0.02	0.005	0.001	0.002	0.003
AVERAGE	11.71	11.02	11.97	10.17	12.54	6.46

. For the tables with the experimental results, the following applies:

• The column RPROP represents an artificial neural network [110,111]. This neural network has 10 processing nodes and was trained using the Rprop method [112].
• The column denoted as ADAM indicates the application of the Adam optimizer [113,114] to train an artificial neural network with 10 hidden nodes.
• The column NEAT (NeuroEvolution of Augmenting Topologies) [115] denotes the application of the NEAT method for neural network training.
• The RBF-KMEANS column denotes the original two-phase training method for RBF networks.
• The column GENRBF stands for the RBF training method introduced in [116].
• The column PROPOSED stands for the results obtained using the proposed method.
• In the experimental tables, an additional row was added with the title AVERAGE. This row contains the average classification or regression error for all datasets.

On average, the current work appears to be 30–40% more accurate than the next best method. In many cases, this percentage exceeds 70%. Moreover, in the vast majority of problems, the proposed technique significantly outperforms the next best available method in terms of test error. In order to validate the results, an additional experiment was executed on the classification datasets, where the number of nodes increased from 5 to 20, and the results are graphically outlined in Figure 4. From this experiment, one can draw two conclusions: firstly, the proposed technique has a significant advantage over the others to a large extent in terms of average classification error, and secondly, the proposed method is shown to be robust and not significantly dependent on an increase in the processing nodes, since 5–10 processing nodes are enough to achieve low classification errors.

However, the proposed technique consists of two stages, and in each of them, a genetic algorithm should be executed. This means that it is significantly slower in computing time compared to the rest of the techniques, and, of course, it needs more computing resources. This is graphically shown in Figure 5, where the average execution time for the ADAM method and the proposed method is shown for the classification datasets when the number of processing nodes increases from 5 to 20. As expected, the current work requires significantly more time than a simple optimization technique such as ADAM, since it consists of two sequential genetic algorithms.

Of course, since we are talking about genetic algorithms, the training time required could be significantly reduced by using parallel techniques that take advantage of modern parallel computing structures, such as the MPI interface [117] or the OpenMP library [118]. The superiority of the proposed technique is also reinforced by the statistical tests carried out on the experimental results and outlined in Figure 6.

In addition, an additional set of experiments was executed on the classification data. In this set of experiments, the critical parameter F took the values 3, 5 and 10. The aim of this set of experiments was to establish the sensitivity of the proposed method to changes in its parameters. The experimental results are presented in

Table 7. The following table presents experimental results from the use of the proposed technique in classification problems and for different values of the critical parameter F.

Dataset	$\mathit{F}=3$	$\mathit{F}=5$	$\mathit{F}=10$
Appendicitis	15.57%	16.60%	15.77%
Australian	24.29%	23.94%	22.40%
Balance	17.22%	15.39%	15.62%
Cleveland	52.09%	51.65%	50.37%
Dermatology	37.23%	36.81%	35.73%
Hayes Roth	35.72%	32.31%	35.33%
Heart	16.32%	15.54%	15.91%
HouseVotes	4.35%	3.90%	3.33%
Ionosphere	12.50%	11.44%	9.30%
Liverdisorder	28.08%	28.19%	28.44%
Mammographic	17.49%	17.15%	17.72%
Parkinsons	16.25%	15.17%	14.53%
Pima	23.29%	23.97%	23.33%
Popfailures	5.31%	5.86%	4.68%
Regions2	25.97%	26.29%	25.18%
Saheart	28.52%	28.59%	29.46%
Segment	44.95%	48.77%	49.22%
Spiral	15.49%	18.19%	23.58%
Wdbc	5.43%	5.01%	5.20%
Wine	7.59%	8.39%	5.63%
Z_F_S	4.37%	4.26%	3.90%
ZO_NF_S	3.79%	4.21%	3.99%
ZONF_S	2.34%	2.26%	1.67%
ZOO	11.90%	10.50%	9.33%
AVERAGE	19.03%	18.93%	18.73%

, and a statistical test of the results is presented in Figure 7. The results and the statistics test indicate that there is no significant difference in the efficiency of the method for different values of the critical parameter F.

4. Conclusions

In the current work, an innovative two-stage technique was proposed to efficiently train RBF artificial neural networks. In the first stage of the application, using grammatical evolution, the interval of values of the neural network parameters is partitioned so as to find a promising range that may contain low values of the training error. In the second stage, the neural network parameters are trained within the best range of values found in the first stage. The training of the parameters of the second phase is carried out using a genetic algorithm. The proposed method was applied on a wide series of well-known datasets from the relevant literature and was tested against a series of machine learning models. The new training technique was compared with the traditional method of training RBF networks, as well as with other machine learning models. From the experimental results, its superiority is evident in percentages that exceed 40%. However, since the proposed technique includes two genetic algorithms that are executed sequentially, the execution time required is longer compared to other techniques, especially for datasets with many patterns. An immediate solution to increase the speed of the method would be the use of parallel computing techniques, since genetic algorithms can, by nature, be directly parallelized.

Future improvements to the proposed method may include the following:

• The proposed method could be applied to other variants of artificial neural networks.
• Intelligent learning techniques could be used in place of the k-means technique to initialize the neural network parameters.
• Techniques could be used to dynamically determine the number of necessary parameters for the neural network. For the time being, the number of parameters is considered constant, but this has the consequence of resulting in over-training phenomena being observed in various datasets.
• Crossover and mutation techniques that focus more on the existing interval construction technique for model parameters could be implemented.
• Efficient termination techniques for genetic algorithms could be used to obtain the most efficient termination of techniques without wasting computing time on unnecessary iterations.
• Techniques that are based on parallel programming could be used to increase the speed of the method.