Neural Network Algorithm with Dropout Using Elite Selection

Abstract

A neural network algorithm is a meta-heuristic algorithm inspired by an artificial neural network, which has a strong global search ability and can be used to solve global optimization problems. However, a neural network algorithm sometimes shows the disadvantage of slow convergence speed when solving some complex problems. In order to improve the convergence speed, this paper proposes the neural network algorithm with dropout using elite selection. In the neural network algorithm with dropout using elite selection, the neural network algorithm is viewed from the perspective of an evolutionary algorithm. In the crossover phase, the dropout strategy in the neural network is introduced: a certain proportion of the individuals who do not perform well are dropped and they do not participate in the crossover process to ensure the outstanding performance of the population. Additionally, in the selection stage, a certain proportion of the individuals of the previous generation with the best performance are retained and directly enter the next generation. In order to verify the effectiveness of the improved strategy, the neural network algorithm with dropout using elite selection is used on 18 well-known benchmark functions. The experimental results show that the introduced dropout strategy improves the optimization performance of the neural network algorithm. Moreover, the neural network algorithm with dropout using elite selection is compared with other meta-heuristic algorithms to illustrate it is a powerful algorithm in solving optimization problems.

1. Introduction

Optimization algorithms are applied to many fields to obtain the optimal results to improve performance or reduce cost. Deterministic approaches need to use a large amount of gradient information and are highly dependent on the selected initial point, which is easy to fall into a local minimum [1,2,3]. However, meta-heuristic algorithms do not rely on gradient information and are not easy to fall into local optimization [3,4,5], which shows strong search-ability.

The meta-heuristic algorithm is the product of the combination of a random algorithm and a local search algorithm. It mainly solves the global optimization problem by simulating the evolution law of nature or the wisdom of the group [6,7]. To a certain extent, it can search globally and find the approximate solution of the optimal solution. The process of a meta-heuristic algorithm is mainly divided into the following steps [8,9]. (1) Randomly generate candidate solutions as initial values. (2) Calculate the objective function values of the initial values. (3) According to the existing information, update the candidate solutions by crossover, mutation, and other methods to generate a new generation of candidate solutions. (4) The new candidate solutions enter the next iteration until the shutdown criterion is met. It is an iterative generation process. Through random initialization and crossover and mutation of candidate solutions, the exploration and development of the whole search space can be realized, and the optimal solution can be gradually searched [10,11].

Classical meta-heuristic algorithms include genetic algorithm (GA) [12], simulated annealing (SA) [13], particle swarm optimization (PSO) [14,15,16], harmony search (HS) [17], differential evolution (DE) [18,19], ant colony optimization (ACO) [20], and artificial bee colony optimization (ABC) [21]. These algorithms follow the principle of a meta-heuristic algorithm. For example, in a genetic algorithm, each independent variable is represented by a gene, and each individual is represented by a chromosome. Starting from an initial population, new chromosomes are generated through the process of chromosome crossover and mutation. After calculating the fitness, the individuals with poor fitness are eliminated, so as to promote the population evolution to produce better and better approximate solutions.

No meta-heuristic algorithm can be suitable for all types of optimization problems, so new meta-heuristic algorithms are constantly proposed, such as the neural network algorithm (NNA) [22], spotted hyena optimizer (SHO) [23], seagull optimization algorithm (SOA) [24], tunicate swarm algorithm (TSA) [25], elephant herding optimization (EHO) [26], sooty tern optimization algorithm (STOA) [27], chaotic neural network algorithm with competitive learning (CCLNNA) [28], monarch butterfly optimization (MBO) [29,30], earthworm optimization algorithm (EWA) [31], and moth search algorithm (MSA) [32]. Furthermore, in the literature [33], the Forest Optimization Algorithm (FOA) is proposed, inspired by nature’s process in the forest and it shows quite good accuracy compared with GA and PSO on the path generating four-bar mechanism in [34]. Additionally, the virus optimization algorithm (VOA) is an iteratively population-based algorithm inspired by the behavior of viruses attacking a living cell [35] and it is applied to the identification of elastoplastic properties of materials [36]. All the algorithms provide new solutions for solving different types of optimization problems.

Based on NNA, several improved algorithms are proposed and they are also applied to the practical problems. NNA is applied to improve the overall competitiveness of the single mixed refrigerant (SMR) process for synthetic natural gas (SNG) liquefaction, which saves energy and cost [37] and it is used to optimize parameters of the Fractional-Order-Proportional-Integral-Derivative (FOPID) controller [38]. An effective hybrid method named TLNNA, based on teaching–learning-based optimization (TLBO) is proposed to solve engineering optimization problems [39]. Grey wolf optimization with neural network algorithm (GNNA) is proposed by combining the improved grey wolf optimizer (GWO) and NNA, which significantly improves the performance [40]. A modified neural network algorithm (M-NNA) is adopted as the optimization algorithm in the Complex Fracture Network (CFN) optimization framework, with an optimization searching accuracy far better than the original algorithm [41]. A new methodology based on the combination of symbiosis organism search (SOS) and NNA is proposed for the optimal planning and operation of distributed generations (DGs) and capacitor banks (CBs) in the radial distribution networks (RDNs), with the results obtained helping to improve the annual energy loss mitigation and cost savings [42]. In the literature [43], a quasi-oppositional chaotic neural network algorithm (QOCNNA) is developed by combining NNA with chaotic local search (CLS) and quasi-oppositional-based learning (QOBL) approaches and it is more effective in improving the performance of RDNs.

This paper proposes a global optimization algorithm called the neural network algorithm with dropout using elite selection (DESNNA), which is a variant of NNA. NNA is an algorithm inspired by an artificial neural network. In NNA, each individual is regarded as a pattern solution. Firstly, the initial pattern solution and initial coefficient matrix are generated, and the coefficient matrix is applied to all pattern solutions each time, which is equivalent to the crossover operation between pattern solutions. However, the convergence speed of NNA is slow and sometimes falls into local optimization in complex situations. Therefore, the DESNNA is proposed to improve the convergence speed and performance of NNA. The main contributions of this paper are as follows:

(1)

NNA is analyzed from the perspective of an evolutionary algorithm, including crossover, mutation, and selection processes, which correspond to every step of NNA. It shows that NNA belongs to an evolutionary algorithm.

(2)

In the crossover stage of the DESNNA, similar to dropout in the neural network, the dropout strategy is applied to NNA: a certain proportion of the individuals are dropped and do not participate in the crossover process, which ensures the superiority of the individuals participating in the crossover process.

(3)

In the selection process of the DESNNA, some individuals who performed well in the previous generation are directly retained when updating the population, which increases the optimization ability of the algorithm without losing the diversity of the population.

The rest of this paper is organized as follows. The neural network algorithm is introduced in Section 2. The proposed DESNNA is introduced in detail in Section 3. The experiment and results of the DESNNA on the benchmark functions are presented in Section 4 and the conclusion and future work are stated in Section 5.

2. Neural Network Algorithm

2.1. Artificial Neural Network

The artificial neural network (ANN) is a complex structure based on biological neurons. ANN consists of neurons, which are simple processing units and weighted connections between these neurons. A typical structure is a multilayer perceptron (MLP), as shown in Figure 1. ANN receives a data set, starts the training process, and adjusts the connection weights between neurons [44]. The artificial neural network with dropout makes the activation value of a neuron stop working with a certain probability during the forward propagation, which can make the model more generalized because it does not rely too much on some local features [45,46,47]. The artificial neural network with dropout is shown in Figure 2, in which the dotted circle represents the dropped neurons. ANN has many advantages in the fields of medicine, robot, image processing, and so on [48,49,50,51]. The NNA draws on the idea of ANN forward propagation and its weight matrix is updated in each iteration [52,53].

2.2. The Introduction of Neural Network Algorithm

According to the NNA [22], a pattern solution is an array of 1 × D defined as $x=[x_1,x_2,\cdots ,x_D]$. The population of pattern solutions is a matrix with a size N × D, which can be defined as

$$X=\left[\begin{array}{cccc}{x}_1^1& x_2^1& \cdots & x_D^1\\ x_1^2& x_2^2& \cdots & x_D^2\\ ⋮& ⋮& ⋮& ⋮\\ x_1^N& x_2^N& \cdots & x_D^N\end{array}\right].$$

(1)

The cost of each pattern solution can be obtained by a fitness function. For example, the cost of the ith pattern solution is

$$C_i=f(x_1^i,x_2^i,\cdots ,x_D^i).$$

(2)

In NNA, weights are a square matrix with the size N × N, defined as

$$W=[W_1,W_2,\cdots ,W_{N_{pop}}]=\left[\begin{array}{cccc}{w}_1^1& w_1^2& \cdots & w_1^N\\ w_2^1& w_2^2& \cdots & w_2^N\\ ⋮& ⋮& ⋮& ⋮\\ w_N^1& w_N^2& \cdots & w_N^N\end{array}\right]=\left[\begin{array}{cccc}{w}_{11}& w_{21}& \cdots & w_{N1}\\ w_{12}& w_{22}& \cdots & w_{N2}\\ ⋮& ⋮& ⋮& ⋮\\ w_{1N}& w_{2N}& \cdots & w_{NN}\end{array}\right].$$

(3)

NNA is described in two stages as the following.

(1)

Initialization stage

Firstly, the number of the pattern solutions (N) and the maximum number of iterations are set. Then the initial population containing N pattern solutions is randomly generated between LB and UB. The cost of each individual in the population is obtained by the fitness function. The weight matrix is generated randomly, which satisfies the constraints:

$${\displaystyle \sum _{j=1}^N{w}_{ij}=1,\hspace{1em}i=1,2,\cdots ,N}.$$

(4)

$$w_{ij}\in U(0,1),\hspace{1em}i,j=1,2,\cdots ,N.$$

(5)

According to the cost, the target solution (X^Target) and the corresponding target weight (W^Target) should be set.

(2)

Cycle stage

Similar to the crossover process, the weight matrix multiplying pattern solutions generates the new pattern solutions using the following equation:

$$X_j^{New}(t+1)={\displaystyle \sum _{i=1}^N{w}_{ij}(t)\times }{X}_i(t),\hspace{1em}j=1,2,\cdots ,N.$$

(6)

$$X_i(t+1)=X_i(t)+X_i^{New}(t+1),\hspace{1em}i=1,2,\cdots ,N.$$

(7)

where t is an iteration index. Then the weight matrix should be updated, following Equation (8):

$$W_i(t+1)=W_i(t)+2\times rand\times (W^{Target}(t)-W_i(t)),\hspace{1em}i=1,2,\cdots ,N.$$

(8)

where the weight matrix (W) should always satisfy the constraints (4) and (5).

After the updating process, according to the modification factor β, check the bias condition and choose to either perform the bias operator according to Equations (9) and (10) or perform the transfer function operator by Equation (10). The Equation (9) is given below:

$$X(i,j)=LB+(UB-LB)\times rand,\hspace{1em}i=1,2,\cdots ,N_b.$$

(9)

where N_b = round(D × β) is the number of biased variables in the population of the new pattern solution and j is a random integer between 0 and D. The Equation (10) is given below:

$$W(i,j)=U(0,1),\hspace{1em}i=1,2,\cdots ,N_{wb}.$$

(10)

where N_wb = round(N × β) is the number of biased variables in the updated weight matrix. The transfer function operator makes new pattern solutions transfer from their positions to another position to generate better solutions. The transfer function operator on pattern solutions is defined as:

$$X_i^{*}(t+1)=X_i(t+1)+2\times rand\times (X^{\mathit{Target}}(t)-X_i(t)),\hspace{1em}i=1,2,\cdots ,N.$$

(11)

The bias operator is similar to the mutation operator, which can prevent premature convergence.

The cost of every pattern solution for the population is calculated and the minimum is chosen as the optimal value. The target solution (X^Target) and the target weight (W^Target) corresponding to the optimal value should be updated. Finally, β is reduced according to Equation (12):

$$\beta (t+1)=0.99\times \beta (t),\hspace{1em}t=1,2,\cdots ,Max\_iteration.$$

(12)

If the stopping condition is satisfied, the NNA will stop. Otherwise, go back to the beginning of the cycle stage. The process of NNA is as following Algorithm 1:

Algorithm 1. The implementation of the neural network algorithm (NNA).

01 Create random initial population X and weights W with constraints by Equations (4) and (5)

02 Calculate the cost of every pattern solution and set the target solution and target weight

03 For i = 1:max_iteration

04  Generate new pattern solutions Xt+1 by Equations (6) and (7)

05  Update the weights by Equation (8)

06   If rand ≤ β

07    Perform the bias operator for pattern solutions Xt+1 and weights Wt+1 by Equations (9) and (10)

08   Else

09    Perform the transfer function operator on Xt+1 by Equation (11)

10   End if

11  Calculate the cost of every pattern solution and find the optimal solution and weight

12  Reduce the modification factor β by Equation (12)

13 End for

3. The Neural Network Algorithm with Dropout Using Elite Selection

In order to improve the convergence performance of NNA, the DESNNA is proposed. This section is divided into three subsections, including viewing NNA from the perspective of an evolutionary algorithm, the introduced dropout strategy in the DESNNA, and the elite selection in the DESNNA.

3.1. NNA from the Perspective of Evolutionary Algorithm

Firstly, we view NNA from the perspective of an evolutionary algorithm. The main steps of an evolutionary algorithm include initialization, crossover, mutation, and selection. In NNA, each pattern solution is viewed as an individual of the population and, therefore, the pattern solution matrix is seen as a population.

Similar to the crossover process, when generating new pattern solutions, the weight matrix multiplies pattern solutions by Equation (6), which ties each pattern solution together. For instance, four pattern solutions generating the first new pattern solution can be expressed as:

$$X_1^{New}(t+1)=w_{11}{X}_1(t)+w_{21}{X}_2(t)+w_{31}{X}_3(t)+w_{41}{X}_4(t).$$

(13)

What can be seen from Equation (13) is that when the new pattern solution is generated, the process of linear combination between individuals using the values of the weight matrix is regarded as a crossover process. Figure 3 presents how NNA generates its new population of pattern solutions.

Resembling the mutation process, after checking the bias condition, the bias operator for the new pattern solution or updated weight matrix by Equation (9) or Equation (10) and the transfer function operator for pattern solution by Equation (11) perform random offset operation, which is regarded as a mutation process. Similar to the selection process, after calculating the cost of each pattern solution, the minimum is chosen as the optimal value. In conclusion, the whole process of NNA corresponds to the framework of the evolutionary algorithm.

3.2. The Introduced Dropout Strategy in the DESNNA

In order to overcome the low convergence speed problem, the idea of dropout in the neural network is applied to the crossover process. The principle of dropout in deep learning is to set a probability when samples are input into the neural network for training so that each neuron has a certain probability of death and does not participate in the network training. Dropout is similar to sexual reproduction in biological evolution. The power of genes lies in the ability to mix rather than the ability of a single gene. Sexual reproduction can not only pass down excellent genes but also reduce the joint adaptability between genes.

Inspired by dropout in the neural network, in the crossover process of the DESNNA, when calculating and generating new pattern solutions, the linear combination of different individuals in the population confirms new pattern solutions. Then, 10% of individuals who do not perform well are dropped and do not participate in the process. In order to achieve this, the pattern solutions that do not participate in the crossover process are set to zero. The implementation of concrete details is reflected in the following Equations (14) and (15):

$$X_{drop}(t)=0.$$

(14)

$$X_j^{New}(t+1)={\displaystyle \sum _{i=1}^N{w}_{ij}(t)\times X_i}(t),\hspace{1em}j=1,2,\cdots ,N.$$

(15)

where X₁, X₂, …, X_N includes 10% of X_drop. Therefore, after applying the dropout strategy to NNA, the crossover process is improved from Equation (6) to Equations (14) and (15). In this way, the individuals with poor performance corresponding to the pattern solutions are set to zero, which is equivalent to them not participating in the crossover process. This ensures the superiority of individuals in the crossover process and then ensures the superiority of the whole population. Consequently, introducing the dropout strategy can improve the convergence speed of the algorithm.

3.3. The Elite Selection in the DESNNA

To improve the convergence speed, the elite selection strategy is applied to the selection process. When calculating and generating new pattern solutions, the linear combination of different individuals in the population confirms new pattern solutions. Adding new pattern solutions to the old pattern solutions obtains a new population. Then the cost of every pattern solution of the new population is calculated, and then they are sorted. In the selection process, the top 15% of individuals with the highest fitness are saved for the next generation. Therefore, some of the better individuals are preserved in every evolutionary process, which makes the level of the whole population higher. In this way, the convergence performance of the algorithm is better. Meanwhile, the bias operator and the transfer function operator are performed as usual without losing the diversity of the population.

The process of the DESNNA is as following Algorithm 2:

Algorithm 2. The implementation of the neural network algorithm with dropout using elite selection (DES-NNA).

01 Create random initial population X and weights W with constraints by Equations (4) and (5)

02 Calculate the cost of every pattern solution and set the target solution and target weight

03 For i = 1:max_iteration

04  10% of individuals with the worst fitness corresponding pattern solution Xworst is set 0    Generate new pattern solutions Xt+1 by Equations (14), (15) and (7)

05  Update the weights by Equation (8)

06   If rand ≤ β

07    Perform the bias operator for pattern solutions Xt+1 and weights Wt+1 by Equations (9) and (10)

08   Else

09    Perform the transfer function operator on Xt+1 by Equation (11)

10   End if

11  Calculate the cost of every pattern solution and find the optimal solution and weight

12  Sort the cost of each pattern solution in the new population

13  Save the top 15% of individuals with the highest fitness to the next generation

14  Reduce the modification factor β by Equation (12)

15 End for

4. DESNNA for Global Optimization

In order to verify the performance of the DESNNA in solving numerical optimization, 18 benchmark functions extracted from the literature [54] are used for the experiment. The experimental results from the DESNNA are compared with those from NNA, which reflects the effectiveness of our improved strategy. Additionally, the results are compared with other meta-heuristics optimization algorithms.

4.1. Benchmark Functions

The test functions provided in the literature [54,55] have been applied to the experimental research of meta-heuristic algorithms. The definitions of benchmark functions F1 to F11 are shown in

Table 1. Benchmark functions F1 to F11.

Function	Name	Definition
F1	Shifted Sphere Function	${\displaystyle \sum _{i=1}^D{z}_i^2}+f\_bias,z=x-o$
F2	Shifted Schwefel Problem 2.21	${\max }_i\{\left|z_i\right|,1\le i\le D\}+f\_bias,z=x-o$
F3	Shifted Rosenbrock’s Function	${\displaystyle \sum _{i=1}^{D-1}(100(z_i^2}+z_{i+1}{)}^2+{(z_i-1)}^2)+f\_bias,z=x-o$
F4	Shifted Rastrigin’s Function	${\displaystyle \sum _{i=1}^D(z_i^2-10\cos (2\pi z_i})+10)+f\_bias,z=x-o$
F5	Shifted Griewank’s Function	${\displaystyle \sum _{i=1}^D\frac{z_i^2}{4000}}-{\displaystyle \prod _{i=1}^D\cos (\frac{z_i}{\sqrt{i}}})+1+f\_bias,z=x-o$
F6	Shifted Ackley’s Function	$-20\exp (-0.2\sqrt{\frac{1}{D}{\displaystyle \sum _{i=1}^D{z}_i^2}}-\exp (\frac{1}{D}{\displaystyle \sum _{i=1}^D\cos (2\pi z_i})+20+e+f\_bias$
F7	Schwefel’s Problem 2.22	${\displaystyle \sum _{i=1}^D\left|x_i\right|}+{\displaystyle \prod _{i=1}^D\left|x_i\right|}$
F8	Schwefel’s Problem 1.2	${\displaystyle \sum _{i=1}^D({\displaystyle \sum _{j=1}^i{x}_j}}{)}^2$
F9	Extended f10	$\begin{array}{l}\left({\displaystyle \sum _{i=1}^{D-1}{f}_{10}(x_i,x_{i+1})}\right)+f_{10}(x_D,x_1)\\ f_{10}={(x^2+y^2)}^{0.25}\cdot ({\sin }^2(50\cdot {(x^2+y^2)}^{0.1})+1)\end{array}$
F10	Bohachevsky	${\displaystyle \sum _{i=1}^D(x_i^2}+2x_{i+1}^2-0.3\cos (3\pi x_i)-0.4\cos (4\pi x_{i+1})+0.7$
F11	Schaffffer	${\displaystyle \sum _{i=1}^D{(x_i^2+x_{i+1}^2)}^{0.25}({\sin }^2}(50\cdot {(x_i^2+x_{i+1}^2)}^{0.1})+1)$

and the definitions of hybrid composition functions F12 to F18 are presented in

Table 2. Hybrid composition functions.

Function	First Function	Second Function	Weight Factor
F12	F9	F1	0.25
F13	F9	F4	0.25
F14	F5	F1	0.5
F15	F3	F4	0.5
F16	F9	F1	0.75
F17	F9	F3	0.75
F18	F9	F4	0.75

, which have higher complexity compared to the functions F1 to F11. The detailed procedure used to hybridize the first function with the second function is shown in the literature [54]. The properties of the benchmark functions are shown in

Table 3. Properties of F1 to F18.

Function	Range	Optimum	Unimodal/ Multimodal	Separable	Shifted	f_bias
F1	[−100, 100]D	0	U	Y	Y	−450
F2	[−100, 100]D	0	U	N	Y	−450
F3	[−100, 100]D	0	M	Y	Y	390
F4	[−5, 5]D	0	M	Y	Y	−330
F5	[−600, 600]D	0	M	N	Y	−180
F6	[−32, 32]D	0	M	Y	Y	−140
F7	[−10, 10]D	0	U	Y	N	−
F8	[−65.536, 65.536]D	0	U	N	N	−
F9	[−100, 100]D	0	U	N	N	−
F10	[−15, 15]D	0	U	Y	N	−
F11	[−100, 100]D	0	U	Y	N	−
F12	[−100, 100]D	0	U	N	Y	−450
F13	[−5, 5]D	0	M	N	Y	−330
F14	[−100, 100]D	0	U	N	Y	−630
F15	[−10, 10]D	0	M	Y	Y	60
F16	[−100, 100]D	0	U	N	Y	−450
F17	[−100, 100]D	0	M	N	Y	390
F18	[−5, 5]D	0	M	N	Y	−330

, and the optimal solutions of all benchmark functions are known.

In order to verify the performance of the DESNNA, we compare the DESNNA with the other six algorithms on these functions, including the NNA, CCLNNA, TSA, PSO, GA, and HS. The parameters of these applied algorithms are listed in

Table 4. Optimal values of user parameters used in the reported optimizers.

Methods	Parameters	Optimal Values
GA	N	50
	Pc	0.8
	Pm	0.2
PSO	N	50
	C1, C2	2
	w	0.9
HS	N	50
	HMCR	0.95
	PAR	0.3
CCLNNA	N	50
N	N	50
N	N	50
	rateOfSelect	0.15
	rateOfDropt	0.10

. For a fair comparison, the dimension of the benchmark function is set to 50 in this experiment. The maximum number of function evaluations (NFES) is used as the shutdown condition, which is set to 5000 times the dimension (D). Each algorithm runs independently of benchmark functions 30 times, taking the best error, the average error, the worst error, and the error standard deviation. The population size is uniformly set to 50. All the optimizers in this paper are coded in MATLAB R2019b.

4.2. Comparison between Improved DESNNA and NNA

In order to verify the effectiveness of the improvement strategy, this section focuses on comparing optimization results between the NNA and DESNNA. The experimental results of the NNA and DESNNA running on benchmark functions are shown in

Table 5. Experimental results obtained by the NNA and DESNNA.

Function	Methods	Best Error	Average Error	Worst Error	Error Standard Deviation
F1	NNA DESNNA	5.684 × 10−14	5.684 × 10−14	3.411 × 10−13	1.339 × 10−13
		0	0	1.137 × 10−13	5.971 × 10−14
F2	NNA DESNNA	7.209 × 10−2	3.251 × 10−1	1.425 × 100	2.901 × 10−1
		2.626 × 10−2	3.152 × 10−1	1.010 × 100	2.389 × 10−1
F3	NNA DESNNA	4.822 × 101	5.860 × 101	2.094 × 102	3.953 × 101
		4.822 × 101	4.822 × 101	4.822 × 101	3.493 × 10−13
F4	NNA DESNNA	4.547 × 10−13	1.825 × 100	7.960 × 100	2.653 × 100
		0	5.978 × 10−1	4.975 × 100	1.470 × 100
F5	NNA DESNNA	5.684 × 10−14	5.754 × 10−4	9.865 × 10−3	2.213 × 10−3
		0	0	5.684 × 10−14	3.077 × 10−14
F6	NNA DESNNA	2.154 × 10−11	7.012 × 10−11	1.863 × 10−10	3.857 × 10−11
		5.684 × 10−14	2.842 × 10−14	1.705 × 10−13	5.853 × 10−14
F7	NNA DESNNA	3.132 × 10−12	3.177 × 10−11	1.101 × 10−10	3.063 × 10−11
		6.335 × 10−15	6.297 × 10−14	5.688 × 10−13	1.072 × 10−13
F8	NNA DESNNA	9.448 × 10−4	5.364 × 10−3	2.151 × 10−2	5.002 × 10−3
		7.039 × 10−5	1.794 × 10−3	1.380 × 10−2	2.701 × 10−3
F9	NNA DESNNA	3.105 × 10−1	4.330 × 100	2.096 × 101	4.678 × 100
		1.418 × 10−2	3.460 × 100	1.369 × 101	4.206 × 100
F10	NNA DESNNA	0	3.701 × 10−17	2.220 × 10−16	8.417 × 10−17
		0	0	0	0
F11	NNA DESNNA	2.143 × 10−1	5.928 × 100	2.416 × 101	6.384 × 100
		1.787 × 10−2	2.126 × 100	1.356 × 101	3.941 × 100
F12	NNA DESNNA	4.151 × 10−5	2.217 × 10−2	6.596 × 10−1	1.204 × 10−1
		5.449 × 10−10	1.710 × 10−8	8.094 × 10−8	1.962 × 10−8
F13	NNA DESNNA	1.906 × 10−5	1.727 × 100	9.720 × 100	2.649 × 100
		2.314 × 10−8	1.017 × 100	7.469 × 100	2.367 × 100
F14	NNA DESNNA	2.274 × 10−13	8.102 × 10−3	2.061 × 10−1	3.772 × 10−2
		0	3.196 × 10−3	5.157 × 10−2	1.120 × 10−2
F15	NNA DESNNA	2.346 × 101	2.356 × 101	2.645 × 101	5.449 × 10−1
		2.346 × 101	2.346 × 101	2.346 × 101	5.357 × 10−10
F16	NNA DESNNA	3.345 × 10−2	2.657 × 100	8.910 × 100	2.854 × 100
		1.699 × 10−5	2.869 × 10−1	4.255 × 100	8.419 × 10−1
F17	NNA DESNNA	1.096 × 101	4.957 × 101	2.714 × 102	5.833 × 101
		1.061 × 101	2.680 × 101	1.434 × 102	3.733 × 101
F18	NNA DESNNA	1.152 × 10−2	2.101 × 100	1.293 × 101	3.218 × 100
		7.856 × 10−6	1.070 × 100	1.092 × 101	2.881 × 100

. The better results are highlighted in bold type. From Table 5, in terms of the best error, the average error, the worst error, and the error standard deviation, the DESNNA is superior to the NNA on all 18 benchmark functions. That is to say, the DESNNA has better search-ability and stability than the NNA, which means the applied improvement strategy improves the performance of the NNA.

In addition, the convergence performance between the DESNNA and NNA is compared. Several typical curves of the convergence process on the functions F2, F9, F11, and F14 are provided in Figure 4 to show the convergence performance. From Figure 4, the DESNNA converges faster than the NNA on functions. It can be seen from the characteristics of these curves that in the initial stage, the DESNNA has better convergence performance than the NNA and in fewer iterations, the DESNNA tends to find the value closer to the optimal value. With the increase of iterations, the DESNNA tends to converge to the optimal value. Therefore, the DESNNA has better convergence performance compared to the NNA.

4.3. Comparisons between the Improved DESNNA and Other Algorithms

The optimization performance between the DESNNA and five other algorithms containing the CCLNNA, TSA, PSO, GA, and HS are compared in this section.

Table 6. Experimental results obtained by the DESNNA and other methods.

Methods	Best Error	Average Error	Worst Error	Error Standard Deviation
F1
DESNNA	0	0	1.137 × 10−13	5.971 × 10−14
CCLNNA	7.135 × 10−9	3.697 × 10−8	1.215 × 10−7	2.172 × 10−8
TSA	5.684 × 10−14	5.684 × 10−14	1.137 × 10−13	5.382 × 10−14
PSO	1.137 × 10−13	1.137 × 10−13	5.116 × 10−13	2.635 × 10−13
GA	1.503 × 10−8	3.612 × 10−8	1.002 × 10−7	1.858 × 10−8
HS	1.694 × 103	2.893 × 103	3.857 × 103	5.336 × 102
F2
DESNNA	2.626 × 10−2	3.152 × 10−1	1.010 × 100	2.389 × 10−1
CCLNNA	8.691 × 10−3	2.016 × 10−2	3.789 × 10−2	6.979 × 10−3
TSA	1.070 × 10−8	1.568 × 10−6	7.838 × 10−6	2.323 × 10−6
PSO	2.019 × 101	2.019 × 101	2.696 × 101	3.218 × 100
GA	1.367 × 100	2.145 × 100	3.068 × 100	3.789 × 10−1
HS	4.165 × 101	4.724 × 101	5.265 × 101	2.566 × 100
F3
DESNNA	4.822 × 101	4.822 × 101	4.822 × 101	3.493 × 10−13
CCLNNA	4.822 × 101	5.123 × 101	1.308 × 102	1.507 × 101
TSA	4.841 × 101	4.863 × 101	4.882 × 101	1.484 × 10−1
PSO	2.430 × 1010	2.430 × 1010	3.757 × 1010	7.454 × 109
GA	4.822 × 101	4.825 × 101	4.861 × 101	8.374 × 10−2
HS	8.008 × 107	1.502 × 108	2.711 × 108	4.366 × 107
F4
DESNNA	0	5.978 × 10−1	4.975 × 100	1.470 × 100
CCLNNA	1.020 × 10−8	3.317 × 10−2	9.950 × 10−1	1.817 × 10−1
TSA	2.270 × 102	3.014 × 102	3.827 × 102	4.299 × 101
PSO	5.530 × 102	5.530 × 102	6.947 × 102	5.642 × 101
GA	8.955 × 100	2.030 × 101	5.373 × 101	9.087 × 100
HS	2.478 × 102	2.783 × 102	3.106 × 102	1.740 × 101
F5
DESNNA	0	0	5.684 × 10−14	3.077 × 10−14
CCLNNA	3.530 × 10−8	7.538 × 10−3	5.867 × 10−2	1.363 × 10−2
TSA	2.842 × 10−14	3.489 × 10−3	2.495 × 10−2	6.073 × 10−3
PSO	2.170 × 10−1	2.170 × 10−1	9.131 × 10−1	3.363 × 10−1
GA	3.800 × 10−10	1.802 × 10−3	3.680 × 10−2	6.969 × 10−3
HS	1.897 × 101	3.000 × 101	3.834 × 101	4.877 × 100
F6
DESNNA	5.684 × 10−14	2.842 × 10−14	1.705 × 10−13	5.853 × 10−14
CCLNNA	1.605 × 10−5	3.366 × 10−5	5.397 × 10−5	7.631 × 10−6
TSA	8.527 × 10−14	1.060 × 100	3.385 × 100	1.436 × 100
PSO	1.919 × 10−7	1.919 × 10−7	1.809 × 10−6	3.368 × 10−7
GA	9.199 × 10−5	1.027 × 100	1.945 × 100	6.774 × 10−1
HS	7.334 × 100	9.207 × 100	1.034 × 101	6.722 × 10−1
F7
DESNNA	6.335 × 10−15	6.297 × 10−14	5.688 × 10−13	1.072 × 10−13
CCLNNA	6.837 × 10−5	1.053 × 10−4	1.493 × 10−4	2.521 × 10−5
TSA	1.162 × 10−132	2.264 × 10−127	6.129 × 10−126	1.117 × 10−126
PSO	1.097 × 10−10	1.097 × 10−10	1.058 × 10−9	1.998 × 10−10
GA	4.951 × 10−1	2.597 × 100	4.642 × 100	1.114 × 100
HS	1.917 × 101	2.156 × 101	2.353 × 101	1.276 × 100
F8
DESNNA	7.039 × 10−5	1.794 × 10−3	1.380 × 10−2	2.701 × 10−3
CCLNNA	3.661 × 10−2	8.557 × 10−2	1.540 × 10−1	2.742 × 10−2
TSA	6.565 × 10−58	1.529 × 10−34	3.362 × 10−33	6.436 × 10−34
PSO	3.630 × 104	3.630 × 104	4.899 × 104	5.672 × 103
GA	9.505 × 10−1	9.095 × 100	4.847 × 101	1.405 × 101
HS	3.420 × 104	5.157 × 104	6.727 × 104	9.005 × 103
F9
DESNNA	1.418 × 10−2	3.460 × 100	1.369 × 101	4.206 × 100
CCLNNA	7.099 × 100	1.795 × 101	3.166 × 101	4.477 × 100
TSA	2.168 × 101	5.648 × 101	1.543 × 102	2.840 × 101
PSO	4.944 × 102	4.944 × 102	5.464 × 102	3.333 × 101
GA	2.670 × 101	3.818 × 101	4.814 × 101	4.947 × 100
HS	1.469 × 102	1.863 × 102	2.114 × 102	1.564 × 101
F10
DESNNA	0	0	0	0
CCLNNA	1.485 × 10−8	4.157 × 10−8	1.051 × 10−7	2.222 × 10−8
TSA	0	4.246 × 100	3.191 × 101	9.976 × 100
PSO	3.892 × 103	3.892 × 103	5.628 × 103	7.097 × 102
GA	5.249 × 100	8.968 × 100	1.727 × 101	2.900 × 100
HS	1.519 × 102	1.987 × 102	2.723 × 102	3.068 × 101
F11
DESNNA	1.787 × 10−2	2.126 × 100	1.356 × 101	3.941 × 100
CCLNNA	8.918 × 100	1.739 × 101	3.008 × 101	4.385 × 100
TSA	1.440 × 101	5.089 × 101	1.683 × 102	3.334 × 101
PSO	4.653 × 102	4.653 × 102	5.113 × 102	2.339 × 101
GA	2.475 × 101	3.564 × 101	4.380 × 101	4.903 × 100
HS	1.603 × 102	1.807 × 102	2.132 × 102	1.307 × 101
F12
DESNNA	5.449 × 10−10	1.710 × 10−8	8.094 × 10−8	1.962 × 10−8
CCLNNA	8.100 × 10−2	2.453 × 10−1	1.333 × 100	2.410 × 10−1
TSA	1.137 × 10−13	2.020 × 100	1.651 × 101	4.570 × 100
PSO	3.922 × 104	3.922 × 104	7.287 × 104	1.119 × 104
GA	7.491 × 10−2	4.669 × 100	1.343 × 101	3.597 × 100
HS	7.270 × 102	1.231 × 103	2.018 × 103	3.087 × 102
F13
DESNNA	2.314 × 10−8	1.017 × 100	7.469 × 100	2.367 × 100
CCLNNA	1.724 × 10−2	3.492 × 10−1	2.526 × 100	6.498 × 10−1
TSA	1.577 × 102	2.325 × 102	3.012 × 102	3.723 × 101
PSO	4.077 × 102	4.077 × 102	5.031 × 102	4.723 × 101
GA	1.084 × 101	2.201 × 101	3.664 × 101	5.255 × 100
HS	1.602 × 102	1.989 × 102	2.269 × 102	1.666 × 101
F14
DESNNA	0	3.196 × 10−3	5.157 × 10−2	1.120 × 10−2
CCLNNA	6.372 × 10−8	2.275 × 10−2	9.816 × 10−2	2.461 × 10−2
TSA	2.274 × 10−13	7.897 × 10−3	4.276 × 10−2	1.055 × 10−2
PSO	2.045 × 104	2.045 × 104	3.515 × 104	7.255 × 103
GA	2.863 × 10−7	1.402 × 10−2	5.987 × 10−2	1.993 × 10−2
HS	7.064 × 101	1.534 × 102	2.943 × 102	6.223 × 101
F15
DESNNA	2.346 × 101	2.346 × 101	2.346 × 101	5.357 × 10−10
CCLNNA	2.346 × 101	2.347 × 101	2.356 × 101	1.750 × 10−2
TSA	8.121 × 101	1.363 × 102	2.428 × 102	3.715 × 101
PSO	6.650 × 105	6.650 × 105	1.519 × 106	3.700 × 105
GA	2.546 × 101	4.732 × 101	1.409 × 102	2.594 × 101
HS	1.258 × 103	1.565 × 103	1.953 × 103	2.054 × 102
F16
DESNNA	1.699 × 10−5	2.869 × 10−1	4.255 × 100	8.419 × 10−1
CCLNNA	5.691 × 100	1.168 × 101	2.231 × 101	4.565 × 100
TSA	8.421 × 100	4.459 × 101	1.386 × 102	2.386 × 101
PSO	6.533 × 103	6.533 × 103	1.763 × 104	4.735 × 103
GA	1.568 × 101	2.582 × 101	3.621 × 101	5.827 × 100
HS	1.653 × 102	2.057 × 102	2.295 × 102	1.711 × 101
F17
DESNNA	1.061 × 101	2.680 × 101	1.434 × 102	3.733 × 101
CCLNNA	1.610 × 101	3.373 × 101	8.777 × 101	1.650 × 101
TSA	2.165 × 101	6.243 × 101	4.272 × 102	7.386 × 101
PSO	2.160 × 109	2.160 × 109	9.648 × 109	2.406 × 109
GA	3.540 × 101	5.233 × 101	9.555 × 101	1.336 × 101
HS	5.139 × 102	2.355 × 103	9.487 × 103	2.902 × 103
F18
DESNNA	7.856 × 10−6	1.070 × 100	1.092 × 101	2.881 × 100
CCLNNA	5.046 × 10−1	2.065 × 100	4.341 × 100	1.014 × 100
TSA	8.064 × 101	1.093 × 102	1.443 × 102	1.659 × 101
PSO	1.737 × 102	1.737 × 102	2.260 × 102	1.931 × 101
GA	2.264 × 101	3.461 × 101	4.626 × 101	6.337 × 100
HS	8.716 × 101	9.495 × 101	1.038 × 102	4.256 × 100

shows the experimental results obtained by the DESNNA and other methods.

From Table 6, in terms of the best error, the DESNNA outperforms CCLNNA on all functions except for F2. The DESNNA is superior to the TSA on functions F1, F3, F4, F5, F6, F9, F10, F11, F13, F14, F15, F16, F17, and F18. The DESNNA beats the PSO, GA, and HS on all functions. From Table 6, in terms of the average error, the DESNNA performs better than the CCLNNA on all functions except for F2, F4, and F13. TSA beats the DESNNA only on functions F2, F7, and F8. The DESNNA is superior to the PSO, GA, and HS on all functions. As for the worst error, the DESNNA outperforms CCLNNA on functions F1, F3, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15, and F16 and outperforms TSA on functions F1, F3, F4, F5, F6, F9, F10, F11, F12, F13, F14, F15, F16, F17, and F18. The DESNNA is superior to the PSO, GA, and HS on all functions. In terms of error standard deviation, the DESNNA outperforms CCLNNA on functions F1, F3, F5, F6, F7, F8, F9, F10, F11, F12, F14, F15, and F16. TSA beats the DESNNA only on functions F1, F2, F7, and F8, and GA beats the DESNNA only on F17. The DESNNA is superior to the PSO and HS on all functions. Clearly, the DESNNA shows better performance than other compared methods.

5. Conclusions and Future Work

In this paper, a new meta-heuristic algorithm called the neural network algorithm with dropout using elite selection strategy (DESNNA) is proposed, which is a new variant of the neural network algorithm (NNA). In the DESNNA, when a new population is generated from the previous population, a certain percentage of individuals who perform the worst are dropped and a certain proportion of the individuals of the previous generation with the best performance are retained and directly enter the next generation to ensure the outstanding performance of the population. In order to verify the effectiveness of the improved strategy, the DESNNA is used on 18 well-known benchmark functions. The experimental results showed that the DESNNA outperforms the NNA on all 18 benchmark functions and the DESNNA beats each other compared algorithm on more than 80% of benchmark functions. Therefore, the introduced dropout and elite selection strategy improved the optimization performance of the NNA, and the DESNNA is a powerful algorithm for solving optimization problems. This work can advance the state-of-the-art. The improved DESNNA can be applied to the single mixed refrigerant process for synthetic natural gas liquefaction or the optimal planning and operation of distributed generations and capacitor banks in the radial distribution networks, which can help to improve the annual energy loss mitigation and cost savings.

As for future research, because the NNA is inspired by the neural network, it can be optimized from the perspective of a neural network. Other new variants of the NNA should be proposed in future research, such as introducing back-propagation and gradient descent to update the weight matrix. The hybridization of the NNA with other meta-heuristic algorithms may form a better optimization algorithm. What is more, the DESNNA can be applied to solve real-world optimization problems in engineering, such as constrained engineering design problems.