RbfCon: Construct Radial Basis Function Neural Networks with Grammatical Evolution

Abstract

Radial basis function networks are considered a machine learning tool that can be applied on a wide series of classification and regression problems proposed in various research topics of the modern world. However, in many cases, the initial training method used to fit the parameters of these models can produce poor results either due to unstable numerical operations or its inability to effectively locate the lowest value of the error function. The current work proposed a novel method that constructs the architecture of this model and estimates the values for each parameter of the model with the incorporation of Grammatical Evolution. The proposed method was coded in ANSI C++, and the produced software was tested for its effectiveness on a wide series of datasets. The experimental results certified the adequacy of the new method to solve difficult problems, and in the vast majority of cases, the error in the classification or approximation of functions was significantly lower than the case where the original training method was applied.

1. Introduction

A variety of real-world problems can be considered as classification and regression problems, handled by machine learning models that have been thoroughly studied in the literature. Such problems arise in physics [1,2], chemistry [3,4], economics [5,6], medicine [7,8], etc. One common machine learning model that is applied in many areas is the radial basis function (RBF) neural network [9,10]. These neural networks are commonly formed using the following mathematical expression:

$$y\left(\overrightarrow{x}\right)=\sum _{i=1}^k{w}_i\varphi \left(∥\overrightarrow{x}-\overrightarrow{c_i}∥\right)$$

(1)

The following notation is used in Equation (1):

• The vector $\overrightarrow{x}$ represents the input pattern with dimension d.
• The parameter k stands for the number of weights of the model. These weights are represented by vector $\overrightarrow{w}$.
• The vectors $\overrightarrow{c_i},i=1,\dots ,k$ represent the so-called centers of the network.
• The final output of the model for the input pattern $\overrightarrow{x}$ is denoted by $y\left(\overrightarrow{x}\right)$.
• The function $\varphi \left(x\right)$ in most cases is represented by the Gaussian function defined as

  $$\varphi \left(x\right)=exp\left(-{\displaystyle \frac{{\left(x-c\right)}^2}{{\sigma }^2}}\right)$$

  (2)
  This function is selected as the output function since the output value $\varphi \left(x\right)$ uses only the distance of vectors x and c.

RBF neural networks were extensively used and studied in recent literature and, among others, can find applications in problems derived from physics [11,12], robotics [13,14], security problems [15,16], image processing [17,18], etc. Due to the widespread use of these machine learning models and their use in both classification and data fitting problems, a number of techniques were presented in recent years that seek to more accurately identify their parameters. Such methods include techniques used to initialize the parameters of these models [19,20,21] or pruning methods for the optimal adaptation of the architecture of these models [22,23,24]. Also, global optimization methods were proposed to estimate the optimal set of parameters of RBF networks in a series of research papers [25,26,27].

Furthermore, a discussion on the kernel widths of RBF networks is provided in the work of Benoudjit et al. [28]. Paetz proposed a method [29] that can reduce the number of neurons in RBF networks with dynamic decay adjustment in order to increase the generalization abilities of these networks. Yu et al., in their work [30], proposed an incremental design technique for RBF networks in order to estimate the optimal architecture of these networks. Moreover, Alexandridis et al. [31] proposed the incorporation of the Particle Swarm Optimization method to effectively estimate the weights of RBF networks. Also, Neruda et al. [32] provided a detailed comparison of methods used to estimate the parameters of RBF networks. Additionally, due to the widespread usage of parallel programming techniques, a series of techniques that exploit parallel computing units were suggested in recent years for RBF network training [33,34].

In most cases, the set of parameters of the model is estimated using a two-phase method: during the first phase, the set of centers and variances in Equation (1) are calculated using the K-means algorithm [35]. In the second phase, the set of weights $\overrightarrow{w}$ is obtained by solving a linear system of equations. Although the previous procedure is capable of estimating the optimal set of parameters in a very short time, it, nevertheless, has a number of problems with regard to numerical stability, the accurate determination of the number of weights, over-fitting problems, etc. To calculate these parameters with the classical calculation method, it is necessary to solve a system of equations, but in many cases, the solution of such a system leads to numerical problems as determinants appear with a value quite close to zero. To tackle such problems, a novel method was proposed in this study that constructs the optimal architecture of RBF networks using a method that incorporates the Grammatical Evolution procedure [36]. The proposed technique is able to estimate both the optimal architecture of the neural network and the numerical values of its parameters, and in this study, both the algorithm used and the software created for this process, which is freely available from the internet, are presented in detail.

The software proposed in this work is fully implemented in the programming language ANSI C++ and is executable with a series of command line parameters with which the user can efficiently process the datasets at their disposal. The contribution of the proposed technique and the implemented software can be summarized as follows:

• The method can efficiently construct the structure of RBF networks and achieve optimal adjustment of their parameters.
• The resulting networks do not have the numerical problems caused by the traditional training technique of RBF networks.
• The created software provides an easy user interface for performing experiments and can be installed on almost any operating system.

The rest of this article is divided into the following sections: in Section 2, the proposed methodology and the used software are thoroughly discussed; in Section 3, the datasets incorporated in the conducted experiments are listed, followed by the experimental results; in Section 4, a discussion on the experimental results is provided; and finally, in Section 5, a series of conclusions is presented.

2. Materials and Methods

The original training method of RBF networks is presented here followed by the detail presentation of the Grammatical Evolution procedure. Afterwards, the proposed method is fully described accompanied by a full working example of the proposed software.

2.1. The Original Training Procedure of RBF Neural Networks

The typical training procedure of an RBF network $y\left(x\right)={\sum }_{i=1}^k{w}_i\varphi \left(∥x-c_i∥\right)$ involves two major steps: In the first step, the centers $\overrightarrow{c_i}$ and the variances are calculated using the K-means procedure, which is described in Algorithm 1. During the second step, a system of equations should be solved to calculate the values for weights $w_i,i=1,\dots ,k$ as follows:

• Set $W=w_{kj}$ the matrix of k weights, $\Phi ={\varphi }_j\left(x_i\right)$, and $T=\left\{t_i\right\}$, where $t_i,i=1,\dots ,M$ are the expected values for input patterns $x_i$.
• Solve

  $${\Phi }^T\left(T-\Phi W^T\right)=0$$

  (3)

  $$W^T={\left({\Phi }^T\Phi \right)}^{-1}{\Phi }^{T}T={\Phi }^{†}T$$

  (4)
  The matrix ${\Phi }^{†}={\left({\Phi }^T\Phi \right)}^{-1}{\Phi }^T$ denotes the the pseudo-inverse of $\Phi$, with

  $${\Phi }^{†}\Phi =I$$

  (5)

Algorithm 1 The used K-means Algorithm

Initialization (a) Set k the number of centers. (b) Read the patterns of the training dataset $x_i,i=1,\dots ,M$ (c) Set $S_j=\varnothing$, from $j=1,\dots ,k$. For every pattern $x_i,i=1,\dots ,M$ do (a) Set $j^{*}={\mathrm{argmin}}_{m=1}^k\left\{D\left(x_i,c_m\right)\right\}$ where the variable $j^{*}$ denotes the nearest center from $x_i$ (b) Set $S_{j^{*}}=S_{j^{*}}\cup \left\{x_i\right\}$. End For For each center $c_j,j=1..k$ do (a) Calculate and denote as $M_j$ the number of samples in $S_j$ (b) Update the center $c_j$ as $$c_j={\displaystyle \frac{1}{M_j}}\sum _{x_i\in S_j}{x}_i$$ End For If the centers $c_j$ did not change then terminate else goto step 2

2.2. The Used Construction Procedure

The Grammatical Evolution procedure is able to construct programs in the provided Backus Naur Form (BNF) grammar [37] with the assistance of a genetic algorithm that utilizes integer chromosomes. These chromosomes represent rule numbers from the underlying grammar, and the method was used successfully in a series of cases, such as: trigonometric problems [38], production of music [39], construction of neural networks [40,41], video games [42,43], credit classification [44], network security [45], etc. Furthermore, many researchers developed and published a series of software regarding Grammatical Evolution, such as the GEVA [46] which suggests a GUI environment, a statistical software written in R named gramEvol [47], a Matlab toolbox named GeLab [48], the GenClass software used in classification problems [49], the QFc software [50] used to construct artificial features, etc.

The proposed procedure constructs the structure of an RBF neural network using Grammatical Evolution and BNF grammar and an appropriately modified genetic algorithm [51,52] that guides the course of the procedure. The grammar functions used in the Grammatical Evolution procedure are BNF grammar expressed as sets $G=\left(N,T,S,P\right)$, where

• The symbol N stands for the set of non-terminal symbols;
• The symbol T represents the set of terminal symbols;
• S is a non-terminal symbol, used as the start symbol of the grammar;
• P is the set of production rules that are used to produce terminal symbols from non-terminal symbols.

For each chromosome, Grammatical Evolution initiates from the symbol S, and through a series of production steps, it creates programs with terminal symbols by substituting non-terminal symbols with the right hand of the selected production rule. The selection of the production rule is accomplished in two steps:

• Obtain the next element V from the under-processing chromosome.
• Select the next production rule according to the equation, Rule = V mod $N_R$, where the quantity $N_R$ stands for the total number of production rules for the non-terminal symbol that is under processing.

The required BNF grammar for the current algorithm is outlined in Figure 1. The terminal symbol exp represents the function $exp\left(x\right)$, and the terminal symbol pow stands for the power operator $x^y$.

The steps of the algorithm used to create RBF neural networks are listed below:

• Initialization Step.

  (a)

  Set the number of chromosomes $N_c$ and the number of allowed generations $N_g$.

  (b)

  Set the selection rate of the genetic algorithm, denoted as $p_c$ where $p_c\le 1$.

  (c)

  Set the mutation rate of the genetic algorithm, denoted as $p_m$, with $p_m\le$1.

  (d)

  Initialize the chromosomes as sets of positive random integers.

  (e)

  Set k = 0, the generation number.

• Main loop step.

  (a)

  Evaluate fitness.

  • For every chromosome $C_i,i=1,\dots ,N_c$ create the corresponding neural network using the grammar of Figure 1. Denote this network as $y_i\left(x\right)$.
  • Compute the fitness $f_i$ for the train set as

    $$f_i=\sum _{j=1}^M{\left(y_i\left({\overrightarrow{x}}_j\right)-t_j\right)}^2$$

    (6)

    The set $\left(\overrightarrow{x_j},t_j\right),j=1,\dots ,M$ represents the train dataset of the objective problem.

  (b)

  Apply the selection operator. A sorting of the chromosomes is performed according to their fitness values. The best $\left(1-p_s\right)\times N_c$ of them are copied without changes to the next generation. The rest of the chromosomes will be replaced by new chromosomes produced during the crossover procedure.

  (c)

  Apply the crossover operator. During this procedure, a series of $p_s\times N_c$ offsprings will be produced. For each pair of offsprings denoted as $\tilde{z}$ and $\tilde{w}$, two chromosomes $(z,w)$ should be selected from the original population using tournament selection. The production of the new chromosomes is conducted using the one-point crossover procedure. An example of this process is graphically outlined in Figure 2.

  (d)

  Perform mutation. For each element of every chromosome, randomly draw a number $r\in \left[0,1\right]$. If $r\le p_m$, then the corresponding element is replaced by a new randomly produced element.

  (e)

  Set k = k + 1.

• Check for the termination step. If $k\le N_g$, then go to step 2.
• Local Optimization step. Obtain the chromosome $C^{*}$ and produce the corresponding RBF neural network $y^{*}\left(x\right)$. The parameters of this neural network are obtained by minimizing with some local search procedure the training error defined as

  $$E\left(y^{*}\left(x\right)\right)=\sum _{j=1}^M{\left(y^{*}\left({\overrightarrow{x}}_j\right)-t_j\right)}^2$$

  (7)
• Application in test set. Apply the final model $y^{*}\left(x\right)$ to the test dataset and report the test error.

The steps of the proposed method are also outlined as a flowchart in Figure 3.

2.3. Installation Procedure

The software was coded entirely in ANSI C++, and the software is available freely from the following URL: https://github.com/itsoulos/RbfCon (accessed on 14 November 2024). The user should install the QT Library, which is accessible from https://qt.io (accessed on 14 November 2024) in order to compile the software. After downloading the RbfCon-master.zip file, the user should execute the following commands, under any Unix-based system:

• unzip RbfCon-master.zip;
• cd RbfCon-master;
• qmake (or qmake-qt5 in some Linux installations);
• make.

For Windows systems, the user alternatively could install the software using the RBFCON.msi located in the distribution, which is an installation file that installs the software without the need for the QT Library.

2.4. The Main Executable RbfCon

The final outcome of the compilation has a series of command line parameters that the user may adjust. The list of these parameters is as follows:

• $--$trainfile=<filename> The string parameter filename determines the name of the file containing the training dataset for the software. The user is required to provide this parameter to start the process. The format for this file is shown in Figure 4. The first number denoted as D in this figure determines the dimension of the input dataset, and the second number denoted as M represents the number of input patterns. Every line in the file contains a pattern and the required desired output.
• $--$testfile=<filename> The string parameter filename stands for the name of the test dataset used by the software. The user should provide at least the parameters trainfile and testfile in order to initiate the method. The format of the test dataset is the same as the training dataset.
• $--$chromosome_count=<count> This integer parameter denotes the number of chromosomes in the genetic algorithm (parameter $N_c$ of the algorithm). The default value for this parameter is 500.
• $--$chromosome_size=<size> The integer parameter size determines the size of each chromosome for the Grammatical Evolution process. The default value for this parameter is 100.
• $--$selection_rate=<rate> The double precision parameter rate stands for the selection rate of the Grammatical Evolution process. The default value for this parameter is 0.10 (10%).
• $--$mutation_rate=<rate> The double precision parameter rate represents the mutation rate for the Grammatical Evolution process. The default value for this parameter is 0.05 (5%).
• $--$generations=<gens> The integer parameter gens stands for the maximum number of allowed generations for the Grammatical Evolution procedure (parameter $N_g$ of the current algorithm). The original value is 500.
• $--$local_method=<method> The used local optimization method will be applied to the parameters of the RBF model when the Grammatical Evolution procedure finished. The available local optimization methods are as follows:

  (a)

  lbfgs. The method L-BFGS can be considered a variation in the Broyden–Fletcher–Goldfarb–Shanno (BFGS) optimization technique [53], which utilizes a minimum amount of memory. This optimization method was used in many cases, such as image reconstruction [54], inverse eigenvalue problems [55], seismic problems [56], training of deep neural networks [57], etc. Also, the series of modifications that take advantage of modern parallel computing systems were proposed [58,59,60].

  (b)

  bfgs. The BFGS variant of Powell was used here [61], when this option is enabled.

  (c)

  adam. This option denotes the application of the Adam optimizer [62] as the local optimization algorithm.

  (d)

  gradient. This option represents the usage of the Gradient Descent method [63] as the local optimization algorithm.

  (e)

  none. With this option, no local optimization method will be used after the Genetic Algorithm is terminated.

• $--$iterations=<iters> This integer parameter determines the number of consecutive applications of the proposed technique to the original dataset. The default value is 30.

2.5. Example of Execution

As a fully working example, consider the Wdbc dataset [64], which is located under EXAMPLES subdirectory. This dataset is used for the detection of breast tumors. An example of execution could be the following:

./RbfCon --trainfile=EXAMPLES/wdbc.train --testfile=EXAMPLES/wdbc.test --iterations=2

The output of this command is shown in Algorithm 2.

Algorithm 2 The output of the example run

Iteration:    1 TRAIN ERROR:          20.40039584 Iteration:    1 TEST  ERROR:        0.07733519754 Iteration:    1 CLASS ERROR:                 9.12% Iteration:    1 SOLUTION:    ((6.711)*(exp((−((pow(x13 − (−97.73),2))+   ((pow(x23 − (43.7),2))+((pow(x23 − (84.7),2))+   (pow(x23 − (91.3),2))))))/(2*pow((60.1),2))))+   (−0.0099969))+((−0.39)*(exp((−((pow(x25 − (9.336),2))+   ((pow(x18 − (6.8),2))+(pow(x4 − (6.52),2)))))/(2*pow((711.5),2))))+   (0.00185452))+((3.7)*(exp((−(pow(x3 − (−6.8),2)))/(2*pow((2.3),2))))+   (−0.001))+((−5.8)*(exp((−(pow(x2 − (−99969.10),2)))/   (2*pow((6342.3747),2))))+(−0.002)) Iteration:    2 TRAIN ERROR:          16.06705417 Iteration:    2 TEST  ERROR:        0.05654528154 Iteration:    2 CLASS ERROR:                 4.56% Iteration:    2 SOLUTION:    ((4.4)*(exp((−((pow(x27 − (3.6),2))+   (pow(x23 − (8.9893),2))))/(2*pow((49.9),2))))+(−0.004))+   ((−2.09)*(exp((−((pow(x23 − (−9.1),2))+   ((pow(x30 − (−40.4),2))+((pow(x4 − (98.5),2))+   (pow(x22 − (−4.4),2))))))/(2*pow((189.9),2))))+(0.007))+   ((−9.8)*(exp((−((pow(x2 − (24.4),2))+((pow(x28 − (2.04),2))+   (pow(x27 − (9.11),2)))))/(2*pow((3.1),2))))+(0.009))+   ((−67.49)*(exp((−(pow(x10 − (−2629940.4),2)))/(2*pow((098.5),2))))+(0.0094)) Average Train Error:          18.23372501 Average Test  Error:        0.06694023954 Average Class Error:                 6.84%

The program prints the train error, the test error, and the classification for every run, as well as the averages for these values. Also, in each execution, the main program shows the constructed RBF network as a string value.

The basic elements that the software requires when initializing the process are essentially the file containing the training data and the file containing the test data. Both of these files must have the proposed structure and the same number of patterns. The user can change other elements of the algorithm, such as the number of chromosomes, through a series of parameters, but this is not required. At the end, the software will display the average training error, the average error in the test set, and the average classification error.

3. Results

The new technique can be used to construct RBF networks for both classification and data fitting problems. For this reason, datasets covering many scientific areas are used from the following websites:

• The UCI repository, https://archive.ics.uci.edu/ (accessed on 14 November 2024) [65];
• The Keel repository, https://sci2s.ugr.es/keel/datasets.php (accessed on 14 November 2024) [66];
• The Statlib URL ftp://lib.stat.cmu.edu/datasets/index.html (accessed on 14 November 2024).

3.1. The Used Datasets

The following list contains the used classification datasets for the the conducted experiments:

• Appendicitis, a medical dataset that was studied in [67,68]. This problem has two distinct classes.
• Alcohol, a dataset related to alcohol consumption [69]. This dataset contains four distinct classes.
• Australian, an economic dataset [70]. The number of classes for this dataset is two.
• Bands, related to problems that occur in printing [71]. It has two classes.
• Cleveland, a medical dataset proposed in a series of research papers [72,73]. This dataset has five distinct classes.
• Dermatology, which is also a medical dataset originated in [74]. It has six classes.
• Ecoli, that is related to problems regarding proteins [75]. It has eight classes.
• Fert, used in demographics. It has two distinct classes.
• Haberman, a medical dataset related to the detection of breast cancer. This dataset has two classes.
• Hayes-roth dataset [76]. This dataset has three classes.
• Heart, a medical dataset about heart diseases [77]. This dataset has two classes.
• HeartAttack, which is a medical dataset related to heart diseases. It has two distinct classes.
• Hepatitis, a dataset used for the detection of hepatitis.
• Housevotes, used for the Congressional voting in USA [78].
• Ionosphere, used for measurements from the ionosphere [79,80]. It has two classes.
• Liverdisorder, which is a medical dataset [81,82] with two classes.
• Lymography [83], which has four classes.
• Magic, this dataset contains generated data to simulate registration of high energy gamma particles [84]. It has two classes.
• Mammographic, a medical dataset related to the presence of breast cancer [85].
• Parkinsons, a dataset used for the detection of Parkinson’s disease [86,87].
• Pima, a medical dataset that was useful for the detection of diabetes [88].
• Popfailures, a dataset contains climate measurements [89]. It has two classes.
• Regions2, a medical dataset that contains measurements from liver biopsy images [90]. It has five classes.
• Ring, which is a problem with 20 dimensions and two classes, related to a series of multivariate normal distributions.
• Saheart, a medical dataset related to heart diseases with two classes [91].
• Statheart, which is a also a medical dataset related to heart diseases.
• Spambase, a dataset used to detect spam emails from a large database. The dataset has two distinct classes.
• Spiral, an artificial dataset with two classes.
• Student, a dataset contains measurements from various experiments in schools [92].
• Tae, this dataset consist of evaluations of teaching performance. It has three classes.
• Transfusion, which is a medical dataset [93].
• Wdbc, a medical dataset used to detect the presence of breast cancer [94,95].
• Wine, a dataset contains measurements about the quality of wines [96,97], with three distinct classes.
• EEG dataset, which is a medical dataset about EEG measurements studied in a series of papers [98,99]. The following cases were obtained from this dataset: Z_F_S, ZO_NF_S, Z_O_N_F_S and ZONF_S.
• Zoo, which used to detect the class of some animals [100]. It contains seven classes.

The next list provided the regression datasets that was incorporated during the conducted experiments:

• Abalone, a dataset that used to predict the age of abalones with 8 features.
• Airfoil, a dataset provided by NASA [101] with 5 features.
• Auto, a dataset used to predict the fuel consumption with 7 features.
• BK, that contains measurements from a series of basketball games with 4 features.
• BL, a dataset used to record electricity experiments with 7 features.
• Baseball, a dataset with 16 features used to estimate the income of baseball players.
• Concrete, a dataset with 8 features used in civil engineering [102].
• DEE, a dataset with 6 featured that was to predict the electricity cost.
• FA, that contains measurements about the body fat with 18 features.
• HO, a dataset originated in the STATLIB repository with 13 features.
• Housing, used to predict the price of houses [103] with 13 features.
• Laser, which is a dataset with 4 features. It has been used in various laser experiments.
• LW, a dataset with 9 features used to record the weight of babies.
• MB, a dataset provide by from Smoothing Methods in Statistics [104] with 2 features.
• Mortgage, an economic dataset from USA with 15 features.
• NT, a dataset with 2 features used to record body temperatures [105].
• Plastic, a dataset with 2 features used to detect the pressure on plastics.
• PL, a dataset with 2 features provided by the STATLIB repository.
• Quake, a dataset used to measure the strength of earthquakes with 3 features.
• SN, a dataset that provides experimental measurements related to trellising and pruning. This dataset has 11 features.
• Stock, a dataset with 9 features used to approximate the prices of various stocks.
• Treasury, an economic dataset from USA that contains 15 features.
• TZ, which is a dataset originated in the STATLIB repository. It has 60 features.

3.2. Experimental Results

The implementation of the machine learning methods that participated in the conducted experiments was made using ANSI C++. The experiments were conducted 30 times, and the average classification or regression error as measured on the test was recorded. The classification error is measured as follows:

$$E_C\left(M\left(x\right)\right)=100\times {\displaystyle \frac{{\sum }_{i=1}^N\left(\mathrm{class}\left(M\left(x_i\right)\right)-y_i\right)}{N}}$$

(8)

where $M\left(x\right)$ denotes the used machine learning model, and set T denotes the train dataset. On the other hand, the regression error is calculated as follows:

$$E_R\left(M\left(x\right)\right)={\displaystyle \frac{{\sum }_{i=1}^N{\left(M\left(x_i\right)-y_i\right)}^2}{N}}$$

(9)

The well-known technique of ten-fold cross-validation was incorporated for the validation of the experimental results. The experiments were executed on an AMD Ryzen 5950X with 128 GB of RAM, running as the operating system the Debian Linux. The values for the parameters of the algorithm are presented in

Table 1. The values for the experimental parameters.

Parameter	Value
chromosome_count	500
chromosome_size	100
selection_rate	0.1
mutation_rate	0.05
generations	500

.

This particular set of parameters was used in a multitude of scientific publications and can be considered as a compromise between the efficiency of an evolutionary process such as the one proposed in this work and the speed necessary to extract the results of its application within a reasonable time.

The experimental results from the application of various method on the classification datasets are shown in Table 2 and for the regression datasets in Table 3. For all tables, the following notation was used:

• The column MLP(ADAM) represents the usage of the ADAM optimization method [62] to train an artificial neural network with $k=10$ processing nodes.
• The column MLP(BFGS) stands for the usage of the BFGS optimization method [61] in the training process of an artificial neural network with $k=10$ processing nodes.
• The column RBF denotes the usage of the original two-phase method for the training of an RBF neural network with $k=10$ weights. This method was described previously in Section 2.1.
• The column CRBF stands for the utilization of the current software using the parameters of Table 1, and as local search procedure, the none option was selected.
• The column CRBF(LBFGS) denotes the application of the current method using as local search procedure the L-Bfgs method.
• The column CRBF(BFGS) stands for the application of the current work using as local search procedure the Bfgs method.
• The row denoted as AVERAGE outlines the average classification or regression error for all datasets in the corresponding table.
• The row denoted as STDEV depicts the standard deviation for all datasets in the corresponding table.

Table 2. Experimental results for the classification datasets using the mentioned machine learning models.

Dataset	MLP (ADAM)	MLP (BFGS)	RBF	CRBF	CRBF (LBFGS)	CRBF(BFGS)
APPENDICITIS	16.50%	18.00%	12.23%	13.60%	14.10%	13.60%
ALCOHOL	57.78%	41.50%	49.38%	51.24%	49.32%	45.11%
AUSTRALIAN	35.65%	38.13%	34.89%	14.14%	14.26%	14.23%
BANDS	36.92%	36.67%	37.17%	35.75%	36.58%	36.03%
CLEVELAND	67.55%	77.55%	67.10%	49.14%	49.52%	50.28%
DERMATOLOGY	26.14%	52.92%	62.34%	45.20%	38.43%	36.66%
ECOLI	64.43%	69.52%	59.48%	54.18%	54.39%	53.03%
FERT	23.98%	23.20%	15.00%	15.20%	15.50%	15.90%
HABERMAN	29.00%	29.34%	25.10%	26.27%	27.07%	26.40%
HAYES-ROTH	59.70%	37.33%	64.36%	34.54%	39.00%	36.76%
HEART	38.53%	39.44%	31.20%	17.22%	17.44%	16.96%
HEARTATTACK	45.55%	46.67%	29.00%	22.60%	21.83%	21.53%
HEPATITIS	68.13%	72.47%	64.63%	54.25%	47.50%	48.75%
HOUSEVOTES	7.48%	7.13%	6.13%	3.05%	3.65%	3.05%
IONOSPHERE	16.64%	15.29%	16.22%	14.32%	13.40%	12.37%
LIVERDISORDER	41.53%	42.59%	30.84%	32.21%	31.71%	31.35%
LYMOGRAPHY	39.79%	35.43%	25.50%	26.36%	25.71%	21.49%
MAGIC	40.55%	17.30%	21.28%	22.18%	19.62%	20.35%
MAMMOGRAPHIC	46.25%	17.24%	21.38%	17.67%	19.02%	17.30%
PARKINSONS	24.06%	27.58%	17.41%	12.79%	13.37%	12.47%
PIMA	34.85%	35.59%	25.78%	24.13%	24.90%	24.15%
POPFAILURES	5.18%	5.24%	7.04%	6.98%	6.94%	6.80%
REGIONS2	29.85%	36.28%	38.29%	26.34%	26.81%	26.55%
RING	28.80%	29.24%	21.67%	11.08%	10.13%	10.33%
SAHEART	34.04%	37.48%	32.19%	29.52%	29.28%	29.19%
SPAMBASE	48.05%	18.16%	29.35%	16.95%	15.53%	15.60%
SPIRAL	47.67%	47.99%	44.87%	42.14%	43.88%	43.05%
STATHEART	44.04%	39.65%	31.36%	19.22%	19.15%	18.19%
STUDENT	5.13%	7.14%	5.49%	7.63%	6.33%	4.32%
TAE	60.20%	51.58%	60.02%	56.73%	56.33%	56.40%
TRANSFUSION	25.68%	25.84%	26.41%	25.15%	24.70%	24.36%
WDBC	35.35%	29.91%	7.27%	6.77%	6.52%	6.36%
WINE	29.40%	59.71%	31.41%	11.00%	11.65%	10.71%
Z_F_S	47.81%	39.37%	13.16%	11.13%	11.47%	10.57%
Z_O_N_F_S	78.79%	65.67%	48.70%	52.34%	49.32%	46.22%
ZO_NF_S	47.43%	43.04%	9.02%	11.08%	11.18%	8.90%
ZONF_S	11.99%	15.62%	4.03%	4.14%	3.94%	3.58%
ZOO	14.13%	10.70%	21.93%	11.60%	9.00%	10.90%
AVERAGE	37.23%	35.36%	30.23%	24.63%	24.17%	23.42%
STDEV	18.25%	18.36%	18.41%	15.92%	15.46%	15.25%

Table 2. Experimental results for the classification datasets using the mentioned machine learning models.

Dataset	MLP (ADAM)	MLP (BFGS)	RBF	CRBF	CRBF (LBFGS)	CRBF(BFGS)
APPENDICITIS	16.50%	18.00%	12.23%	13.60%	14.10%	13.60%
ALCOHOL	57.78%	41.50%	49.38%	51.24%	49.32%	45.11%
AUSTRALIAN	35.65%	38.13%	34.89%	14.14%	14.26%	14.23%
BANDS	36.92%	36.67%	37.17%	35.75%	36.58%	36.03%
CLEVELAND	67.55%	77.55%	67.10%	49.14%	49.52%	50.28%
DERMATOLOGY	26.14%	52.92%	62.34%	45.20%	38.43%	36.66%
ECOLI	64.43%	69.52%	59.48%	54.18%	54.39%	53.03%
FERT	23.98%	23.20%	15.00%	15.20%	15.50%	15.90%
HABERMAN	29.00%	29.34%	25.10%	26.27%	27.07%	26.40%
HAYES-ROTH	59.70%	37.33%	64.36%	34.54%	39.00%	36.76%
HEART	38.53%	39.44%	31.20%	17.22%	17.44%	16.96%
HEARTATTACK	45.55%	46.67%	29.00%	22.60%	21.83%	21.53%
HEPATITIS	68.13%	72.47%	64.63%	54.25%	47.50%	48.75%
HOUSEVOTES	7.48%	7.13%	6.13%	3.05%	3.65%	3.05%
IONOSPHERE	16.64%	15.29%	16.22%	14.32%	13.40%	12.37%
LIVERDISORDER	41.53%	42.59%	30.84%	32.21%	31.71%	31.35%
LYMOGRAPHY	39.79%	35.43%	25.50%	26.36%	25.71%	21.49%
MAGIC	40.55%	17.30%	21.28%	22.18%	19.62%	20.35%
MAMMOGRAPHIC	46.25%	17.24%	21.38%	17.67%	19.02%	17.30%
PARKINSONS	24.06%	27.58%	17.41%	12.79%	13.37%	12.47%
PIMA	34.85%	35.59%	25.78%	24.13%	24.90%	24.15%
POPFAILURES	5.18%	5.24%	7.04%	6.98%	6.94%	6.80%
REGIONS2	29.85%	36.28%	38.29%	26.34%	26.81%	26.55%
RING	28.80%	29.24%	21.67%	11.08%	10.13%	10.33%
SAHEART	34.04%	37.48%	32.19%	29.52%	29.28%	29.19%
SPAMBASE	48.05%	18.16%	29.35%	16.95%	15.53%	15.60%
SPIRAL	47.67%	47.99%	44.87%	42.14%	43.88%	43.05%
STATHEART	44.04%	39.65%	31.36%	19.22%	19.15%	18.19%
STUDENT	5.13%	7.14%	5.49%	7.63%	6.33%	4.32%
TAE	60.20%	51.58%	60.02%	56.73%	56.33%	56.40%
TRANSFUSION	25.68%	25.84%	26.41%	25.15%	24.70%	24.36%
WDBC	35.35%	29.91%	7.27%	6.77%	6.52%	6.36%
WINE	29.40%	59.71%	31.41%	11.00%	11.65%	10.71%
Z_F_S	47.81%	39.37%	13.16%	11.13%	11.47%	10.57%
Z_O_N_F_S	78.79%	65.67%	48.70%	52.34%	49.32%	46.22%
ZO_NF_S	47.43%	43.04%	9.02%	11.08%	11.18%	8.90%
ZONF_S	11.99%	15.62%	4.03%	4.14%	3.94%	3.58%
ZOO	14.13%	10.70%	21.93%	11.60%	9.00%	10.90%
AVERAGE	37.23%	35.36%	30.23%	24.63%	24.17%	23.42%
STDEV	18.25%	18.36%	18.41%	15.92%	15.46%	15.25%

Table 3. Experimental results for the regression datasets using the series of mentioned machine learning models.

Dataset	MLP (ADAM)	MLP (BFGS)	RBF	CRBF	CRBF (LBFGS)	CRBF (BFGS)
ABALONE	4.30	5.69	7.37	6.14	5.35	5.32
AIRFOIL	0.005	0.003	0.27	0.004	0.004	0.002
AUTO	70.84	60.97	17.87	11.03	10.03	9.45
BK	0.025	0.28	0.02	0.02	0.02	0.03
BL	0.62	2.55	0.013	0.04	0.024	0.01
BASEBALL	77.90	119.63	93.02	66.74	65.80	64.52
CONCRETE	0.078	0.066	0.011	0.012	0.010	0.009
DEE	0.63	2.36	0.17	0.23	0.22	0.20
FA	0.048	0.43	0.015	0.013	0.014	0.011
HO	0.035	0.62	0.03	0.013	0.015	0.012
HOUSING	81.00	97.38	57.68	20.60	19.02	18.40
LASER	0.03	0.015	0.03	0.06	0.05	0.03
LW	0.028	2.98	0.03	0.011	0.011	0.011
MB	0.06	0.129	5.43	0.055	0.06	0.12
MORTGAGE	9.24	8.23	1.45	0.165	0.18	0.074
NT	0.006	0.129	13.97	0.006	0.006	0.006
PLASTIC	11.71	20.32	8.62	3.58	2.86	2.43
PL	0.32	0.58	2.118	0.064	0.026	0.024
QUAKE	0.117	0.29	0.07	0.036	0.036	0.036
SN	0.026	0.4	0.027	0.025	0.026	0.025
STOCK	180.89	302.43	12.23	8.31	6.90	6.25
TREASURY	11.16	9.91	2.02	0.0027	0.12	0.10
TZ	0.43	0.22	0.036	0.036	0.036	0.035
AVERAGE	19.54	27.64	9.67	5.10	4.82	4.66
STDEV	43.67	68.11	22.01	14.34	14.05	13.76

Table 3. Experimental results for the regression datasets using the series of mentioned machine learning models.

Dataset	MLP (ADAM)	MLP (BFGS)	RBF	CRBF	CRBF (LBFGS)	CRBF (BFGS)
ABALONE	4.30	5.69	7.37	6.14	5.35	5.32
AIRFOIL	0.005	0.003	0.27	0.004	0.004	0.002
AUTO	70.84	60.97	17.87	11.03	10.03	9.45
BK	0.025	0.28	0.02	0.02	0.02	0.03
BL	0.62	2.55	0.013	0.04	0.024	0.01
BASEBALL	77.90	119.63	93.02	66.74	65.80	64.52
CONCRETE	0.078	0.066	0.011	0.012	0.010	0.009
DEE	0.63	2.36	0.17	0.23	0.22	0.20
FA	0.048	0.43	0.015	0.013	0.014	0.011
HO	0.035	0.62	0.03	0.013	0.015	0.012
HOUSING	81.00	97.38	57.68	20.60	19.02	18.40
LASER	0.03	0.015	0.03	0.06	0.05	0.03
LW	0.028	2.98	0.03	0.011	0.011	0.011
MB	0.06	0.129	5.43	0.055	0.06	0.12
MORTGAGE	9.24	8.23	1.45	0.165	0.18	0.074
NT	0.006	0.129	13.97	0.006	0.006	0.006
PLASTIC	11.71	20.32	8.62	3.58	2.86	2.43
PL	0.32	0.58	2.118	0.064	0.026	0.024
QUAKE	0.117	0.29	0.07	0.036	0.036	0.036
SN	0.026	0.4	0.027	0.025	0.026	0.025
STOCK	180.89	302.43	12.23	8.31	6.90	6.25
TREASURY	11.16	9.91	2.02	0.0027	0.12	0.10
TZ	0.43	0.22	0.036	0.036	0.036	0.035
AVERAGE	19.54	27.64	9.67	5.10	4.82	4.66
STDEV	43.67	68.11	22.01	14.34	14.05	13.76

4. Discussion

In Table 2, for the APPENDICITIS dataset, the low error rates of the RBF and CRBF models (12.23% and 13.60%, respectively) indicate that radial basis function-based models are well-suited for this problem. This performance may be linked to their ability to handle non-linear relationships in the data. The higher error rates of the MLP models (above 16%) suggest they might require more tuning or larger datasets for optimal performance. In the ALCOHOL dataset, error rates range from 41.50% (MLP(BFGS)) to 57.78% (MLP(ADAM)). This wide variation implies that the dataset’s features are particularly challenging for most models, likely due to noise or high dimensionality. The intermediate error rates of the CRBF model (49.38%) show that while RBF-based models can handle some complexity, further adjustments are needed to make them competitive. In the AUSTRALIAN dataset, CRBF and its variations achieve exceptionally low error rates (14%), highlighting their strong adaptation to the dataset’s characteristics. In contrast, MLP(ADAM) and MLP(BFGS) exhibit significantly higher errors (35–38%), demonstrating their struggles to perform effectively on this problem. The CLEVELAND dataset, on the other hand, exemplifies complexity. MLP models show error rates as high as 77.55%, while CRBF models reduce the error to about 49%. This gap underscores the superiority of RBF-based approaches for data with complex, non-linear structures. Overall, the analysis reveals that CRBF models have a clear advantage in datasets with non-linear relationships and lower dimensionality. While MLP models are more flexible, they seem to require extensive parameter tuning to achieve comparable performance. Employing an RBF model with appropriate customization may be the optimal choice for many of the examined problems.

In Table 3, for the ABALONE dataset, MLP models show higher values, such as 4.3 for MLP(ADAM) and 5.69 for MLP(BFGS), while CRBF models perform better. The best result comes from CRBF(BFGS), with a value of 5.32, indicating that CRBF models adapt better to this dataset. In the AIRFOIL dataset, CRBF-based models exhibit exceptional performance, with CRBF(BFGS) achieving the best value of 0.002. In contrast, MLP models show higher values, such as 0.005 for MLP(ADAM), demonstrating that RBF-based models are more accurate in this case. In the AUTO dataset, there is a clear superiority of CRBF models. CRBF(BFGS) achieves the best value of 9.45, while MLP models record significantly higher values, such as 70.84 for MLP(ADAM). This indicates that MLP struggles to adapt effectively to this problem. In the BASEBALL dataset, CRBF models achieve lower error values, with the best result being 64.52 for CRBF(BFGS). In comparison, MLP(ADAM) shows a much higher value of 77.9, highlighting the advantage of CRBF models in this dataset. In the HOUSING dataset, the differences are even more pronounced. CRBF(BFGS) achieves the best performance with a value of 18.4, while MLP(ADAM) records 81, and MLP(BFGS) reaches 97.38. This stark contrast emphasizes the inability of MLP models to perform effectively on this dataset. In the PLASTIC dataset, CRBF models also deliver excellent performance. CRBF(BFGS) achieves the best result at 2.43, while MLP models show much higher values, such as 11.71 for MLP(ADAM). Overall, CRBF-based models demonstrate significantly lower error values across most datasets, indicating their ability to adapt to various data types. MLP models, although flexible, struggle in many cases, possibly due to the need for extensive parameter tuning or limitations in generalization. The overall analysis suggests that CRBF models are more reliable for regression problems, especially when the datasets involve non-linear relationships or high complexity.

Also, in Figure 5 and Figure 6, a statistical comparison between all methods is depicted.

In Figure 5, the Kruskal–Wallis test revealed statistically significant differences among the methods overall, with a general p-value of 0.0007, indicating that at least one method differs significantly from the others in terms of classification error rates. Pairwise comparisons provide further insights into the differences between specific methods. The comparison between MLP(ADAM) and MLP(BFGS) showed no statistically significant difference (p = 0.34), suggesting that the two methods perform similarly. In contrast, the comparison between MLP(ADAM) and RBF revealed a statistically significant difference (p = 0.0026), indicating that RBF achieves distinct error rates compared to MLP(ADAM). The comparison of MLP(ADAM) with CRBF showed a highly significant difference (p = $1.8\times {10}^{-7}$), highlighting the superiority of CRBF over MLP(ADAM). Similarly, the comparison between MLP(ADAM) and CRBF(LBFGS) demonstrated even greater statistical significance, with p = 4.6 × ${10}^{-8}$, confirming the advantage of CRBF(LBFGS). Additionally, the comparison of MLP(ADAM) with CRBF(BFGS) exhibited extremely high statistical significance (p = 1.8 × ${10}^{-8}$), favoring CRBF(BFGS). The subsequent comparisons of MLP(BFGS) with other methods also revealed statistically significant differences. The comparison between MLP(BFGS) and RBF yielded p = 0.011, indicating a notable difference, while the comparison of MLP(BFGS) with CRBF had p = 3.8 × ${10}^{-6}$, clearly demonstrating the superiority of CRBF. Comparisons of MLP(BFGS) with CRBF(LBFGS) and CRBF(BFGS) resulted in p-values of 1.9 × ${10}^{-6}$ and 5.3 × ${10}^{-7}$, respectively, confirming the statistical significance of the differences. Furthermore, comparisons of RBF with CRBF, CRBF(LBFGS), and CRBF(BFGS) all showed statistically significant differences, with p-values of 9.8 × ${10}^{-5}$, 3.9 × ${10}^{-5}$, and 4.3 × ${10}^{-6}$, respectively. Finally, the comparison between CRBF(LBFGS) and CRBF(BFGS) also revealed a statistically significant difference, with p = 0.0008. Overall, the results demonstrate clear performance differences among the methods, with CRBF consistently achieving superior results.

In Figure 6, the Kruskal–Wallis test revealed statistically significant differences among the methods on regression datasets, with an overall Kruskal–Wallis p-value of 0.0019. This indicates that at least one method performs significantly differently from the others. However, pairwise comparisons show mixed results. The comparison between MLP(ADAM) and MLP(BFGS) did not show a statistically significant difference (p = 0.17), suggesting similar performance between these two methods. Similarly, the comparison between MLP(ADAM) and RBF was not significant (p = 0.21). Comparisons of MLP(ADAM) with CRBF, CRBF(LBFGS), and CRBF(BFGS) yielded p-values of 0.086, 0.083, and 0.081, respectively, indicating trends toward differences but without statistical significance. The comparisons involving MLP(BFGS) against other methods yielded more pronounced results. Specifically, differences with CRBF, CRBF(LBFGS), and CRBF(BFGS) were statistically significant, with p = 0.034 for all three cases, indicating the superiority of the CRBF methods over MLP(BFGS). For the comparisons between RBF and the CRBF methods, statistically significant differences were observed. The comparison between RBF and CRBF had a p-value of 0.028, while comparisons with CRBF(LBFGS) and CRBF(BFGS) had p-values of 0.025 and 0.024, respectively, confirming the superior performance of the CRBF methods. Finally, the comparison between CRBF(LBFGS) and CRBF(BFGS) revealed a statistically significant difference, with p = 0.028, highlighting a notable distinction in the performance of these two methods. Overall, the results suggest that while MLP methods perform similarly, CRBF methods consistently achieve better results, with statistically significant differences observed in most comparisons against the other methods.

Summarizing the above results, one can see that the proposed technique for building and training RBF networks improves the performance of these networks by more than 20% on average on classification data and by more than 50% on regression data. In all datasets, the proposed method brought about a significant reduction in classification error in 80% of cases, and in many datasets, the reduction in classification or regression error was extremely significant.

5. Conclusions

A novel method that constructs the architecture of RBF neural networks with assistance was described in the present manuscript, accompanied by the relevant software. The method can construct the architecture of RBF networks using a process that incorporates the Grammatical Evolution procedure. Also, the used method can estimate the parameters of the network, which can be further modified by the application of a local optimization procedure. The proposed method can be applied without modifications to classification and regression datasets. The new method was compared against the traditional method used to train RBF networks, and the experimental results validated the efficiency of the proposed work in a wide series of classification and regression datasets found in recent literature.

The used software was coded in ANSI C++ with the assistance of the freely available library of QT in order to be portable on the majority of operating systems. The user can control the process by a series of simple command line options, such as the number of needed chromosomes for the genetic population or the selection rate used. Also, the software offers the possibility of improving the parameters of the RBF network with the application of some local optimization procedures. The experimental results indicated that the application of the local optimization algorithm can further reduce the test error in a series of datasets.

Future improvements to the software may include the periodic application of the local search procedure during the execution of the genetic algorithm or even the incorporation of global optimization procedures in order to reduce even further the test error of the current work. Also, since the method utilizes Genetic Algorithms, parallel optimization procedures, such as MPI [106] or OpenMP [107], can be used to reduce the execution time of the method. Finally, the software could possibly be extended to allow the user to input other grammar functions as parameters, or even to be able to use other output functions besides Gaussian.