Using Cuckoo Search Algorithm with Q-Learning and Genetic Operation to Solve the Problem of Logistics Distribution Center Location

Abstract

Cuckoo search (CS) algorithm is a novel swarm intelligence optimization algorithm, which is successfully applied to solve some optimization problems. However, it has some disadvantages, as it is easily trapped in local optimal solutions. Therefore, in this work, a new CS extension with Q-Learning step size and genetic operator, namely dynamic step size cuckoo search algorithm (DMQL-CS), is proposed. Step size control strategy is considered as action in DMQL-CS algorithm, which is used to examine the individual multi-step evolution effect and learn the individual optimal step size by calculating the Q function value. Furthermore, genetic operators are added to DMQL-CS algorithm. Crossover and mutation operations expand search area of the population and improve the diversity of the population. Comparing with various CS algorithms and variants of differential evolution (DE), the results demonstrate that the DMQL-CS algorithm is a competitive swarm algorithm. In addition, the DMQL-CS algorithm was applied to solve the problem of logistics distribution center location. The effectiveness of the proposed method was verified by comparing with cuckoo search (CS), improved cuckoo search algorithm (ICS), modified chaos-enhanced cuckoo search algorithm (CCS), and immune genetic algorithm (IGA) for both 6 and 10 distribution centers.

1. Introduction

Optimization problems have been one of the most important research topics in recent years. They exist in many domains, such as scheduling [1,2], image processing [3,4,5,6], feature selection [7,8,9] and detection [10], path planning [11,12], feature selection [13], cyber-physical social system [14,15], texture discrimination [16], saliency detection [17], classification [18,19], object extraction [20], shape design [21], big data and large-scale optimization [22,23], multi-objective optimization [24], knapsack problem [25,26,27], fault diagnosis [28,29,30], and test-sheet composition [31]. Metaheuristic algorithms [32], a theoretical tool, are based on nature-inspired ideas, which have been extensively used to solve highly non-linear complex multi-objective optimization problems [33,34,35]. Several popular metaheuristics with a stochastic nature are compared in some studies [36,37,38] with deterministic Lipschitz methods by using operational zones. Most of these metaheuristics methods are inspired by natural or physical processes, such as bat algorithm (BA) [39], biogeography-based optimization (BBO) [40], ant colony optimization (ACO) [41], earthworm optimization algorithm (EWA) [42], elephant herding optimization (EHO) [43,44], moth search (MS) algorithm [45], firefly algorithm (FA) [46], artificial bee colony (ABC) [47,48,49], harmony search (HS) [50,51], monarch butterfly optimization (MBO) [52,53], particle swarm optimization (PSO) [54,55], genetic programming [56], krill herd (KH) [57,58,59,60,61,62,63], immune genetic algorithm (IGA) [64], and cuckoo search (CS) [65,66,67,68,69].

Yang and Deb [69] proposed a metaheuristic optimization method named CS algorithm, which is inspired by smart incubation behavior of a type of birds called cuckoos in nature.

CS performs local search well in most cases, but sometimes it cannot escape from local optima, which restricts its ability to carry out full search globally. To enhance the ability of CS, Mlakar et al. [70] proposed a novel hybrid self-adaptively CS algorithm adding three features: a self-adaptively of cuckoo search control parameters, a linear population reduction, and a balancing of the exploration search strategies. Li et al. [71] enhanced the exploitation ability of the cuckoo search algorithm by using an orthogonal learning strategy. An improved discrete version of CS was presented by Ouaarab et al. [72].

On the other hand, most researchers agree that the performance of algorithms can be improved by using learning techniques. For example, Wang et al. [73] presented a new method to enhance learning speed and improved final performance, which directly tuned the Q-values to affect the action selection policy. Alex et al. [74] presented a new evolutionary cooperative learning scheme that is able to solve function approximation and classification problems, improving accuracy and generalization capabilities. A new CS algorithm named snap-drift cuckoo search (SDCS) was presented by Hojjat et al. [75]. In SDCS, a snap-drift learning strategy is employed to improve search operators. The snap-drift learning strategy provides an online trade-off between local and global search via two snap and drift modes.

Although much effort has been made to enhance the performance of CS, many of the variants fail to improve the performance of CS algorithm on certain complicated problems. Furthermore, there are few studies on optimizing the parameters of CS algorithm by using learning strategy. In this paper, we present an improved CS algorithm called dynamic step size cuckoo search algorithm (DMQL-CS) that adopts strategies with Q-Learning and genetic operator. Step size strategy of the traditional CS focused only on examining the individual fitness value based on the one-step evolution effect of individual, but ignored the evaluation of step size from the multi-step evolution effect, which is not conducive to the evolution of the algorithm. We use Q-Learning method to optimize the step size, in which the most appropriate step size control strategies are retained for the next generation. At the same time, their weights are adaptively adjusted by using learning rate, which is used to guide individuals to search for a better solution at the next evolution. In addition, crossover operation and mutation operation are added into the DMQL-CS algorithm to accelerate the convergence speed of the algorithm and expand the diversity of the population.

The present manuscript differs from other similar work insofar as the advantage of learning based on Q-Learning and genetic operators. Q-Learning considers the multi-step evolution effect of individual such that the most appropriate step size control strategies are retained for the next generation. For the proposed DMQL-CS approach, the outstanding work of the paper is mainly listed in the following two aspects:

(1)

In the DMQL-CS algorithm, the step size strategy is considered as an action which applies multiple step control strategies (linear decreasing strategy, non-linear decreasing strategy, and adaptively step-size strategy). In the DMQL-CS algorithm, according to multi-step effect of individual for a few steps forward, the optimal step size control strategy is learned. During each learning evolution step size, finally, the optimal individual and corresponding optimal step size strategy are derived by calculating the Q function value. The current individual continues to evolve through the step size obtained, which increases the adaptability of individual evolution.

(2)

The research introduces two genetic operators, crossover and mutation, into the DMQL-CS algorithm, intended for accelerating convergence. During crossover and mutation process, chromosomes are divided into pairs according to certain probability. We introduce the specifically designed crossover operation into problem of logistics distribution center location in this paper, which determines the performance of the algorithm to some extent. To improve the search ability of the CS algorithm, numerous strategies have been designed to adjust the crossover rate. In this work, a self-adaptive scheme is used to adjust the crossover rate. Genetic operators expand the search area of the population to improve the exploration and maintain the diversity of the population, which also helps to improve the exploration of the population of learners.

Finally, the DMQL-CS method was tested on 15 benchmark functions, CEC 2013 test suite, and the problem of logistics distribution center location. The experimental results compared with those of other approaches demonstrated the superiority of the proposed strategy. A series of simulation experiments showed that DMQL-CS performs more accurately and efficiently than other evolutionary methods in terms of the quality of the solution and convergence rate.

The remainder of this paper is organized as follows. In Section 2, the related work on cuckoo search is presented. Section 3 presents cuckoo search. The proposed DMQL-CS algorithm, including Q-Learning model, step size control model with Q-Learning, and genetic operator, is described in Section 4. The comparison with other methods, through 15 functions, CEC 2013 test suite, and the problem of logistics distribution center location, is given in Section 5. Finally, Section 6 concludes this paper and points out some future research directions.

2. Related Work

CS algorithm is capable of finding the best solutions by continuously using new and potentially better solution to replace a not-so-good cuckoo in the population, and it has been applied successfully to diverse fields. Recently, many CS variants have been developed to improve the performance of the CS algorithm. These variants can be generally divided into four categories: (1) parameter control [70]; (2) novel learning schemes [76]; (3) hybrid methods with other algorithm [74]; and (4) local search operator [77].

Due to the important influence of control parameters for the performance, much meaningful work has been done on the control parameter settings of CS algorithm. Initially, step size parameter control was investigated to improve the performance of CS algorithms. For instance, aiming at the faults that Cuckoo Search algorithm cannot acquire exact solutions and converges slowly in the later period, Ma et al. [78] proposed a self-adaptively step size adjustment cuckoo search algorithm (ASCS), which is an adaptively adjusted step size by using the distance between cuckoo nest location and the optimal nest location, which speeds up CS algorithm speed and improves the calculation accuracy. To balance the exploration and exploitation, Li and Yin [79] introduced two mutation rules and combined these two rules using a linear decreasing probability. Then, an adaptive parameter adjustment strategy was developed according to the relative success number of two newly added parameters in the previous iteration. Comparison results of the proposed algorithm show that this scheme is better than other algorithms. Two important factors, speed factor and aggregation factor, were defined by Yang et al. [80]. Then, according to these two factors, the step size and discovery probability were regulated. Experimental results show that the CS with improved step size and discovery probability has strong competitiveness in tackling numerical optimization problems. Li et al. [79] proposed the self-adaptive parameter CS algorithm, which uses two new mutation rules based on the rand and best individuals among the entire population. The self-adaptive parameter is set as a uniform random value based on the relative success number of the two new proposed parameters in the previous period, which enhance diversity of the population. Experimental results show that the proposed method performs better than twelve algorithms from the literature.

Li et al. [65] proposed an enhanced CS algorithm called dynamic CS with Taguchi opposition-based search and dynamic evaluation. The Taguchi search strategy provided random generalized learning based on opposing relationships to enhance the exploration ability of the algorithm. The dynamic evaluation strategy reduced the number of function evaluations, and accelerated the convergence property. Statistical comparisons of experimental results showed that the proposed algorithm makes an appropriate trade-off between exploration and exploitation. Li et al. [81] proposed a new cuckoo search algorithm extension based on self-adaptive knowledge learning, in which a learning model with individual history knowledge and population knowledge is introduced into the CS algorithm. Individuals constantly adjust their position according to historical knowledge and communicate in the optimization process. Statistical comparisons of the experimental results showed that the proposed algorithm is a competitive new type of algorithm. Hojjat et al. [75] presented a new CS algorithm, called snap-drift cuckoo search (SDCS), which first employs a learning strategy and then considers improved search operators. The snap-drift learning strategy provides an online trade-off between local and global search via two snap and drift modes. SDCS tends to increase global search to prevent algorithm of being trapped in local minima via snap mode and reinforces the local search to enhance the convergence rate via drift mode. Statistical comparisons of experimental results showed that SDCS is superior to modified CS algorithms in terms of convergence speed and robustness.

According to the rand and best individuals among the entire population, Cheng et al. [82] proposed an ensemble CS variant in which three different cuckoo search algorithms coexist in the entire search process, which compete to produce better offspring for numerical optimization. Then, an external archive is introduced to further maintain population diversity. Statistical comparisons of experimental results showed that the improved CS variant is superior to modified CS algorithms in terms of convergence speed and robustness. Wen et al. [83] proposed a new hybrid algorithm based on grey wolf optimizer and cuckoo search (GWOCS), which was developed to extract the parameters of different PV cell models with the experimental data under different operating conditions. Zhang et al. [84] proposed an ensemble CS variant that divides the population into two subgroups and adopts CS and DE for these two subgroups independently. These two subgroups can exchange useful information by division. These two algorithms can utilize each other’s advantages to complement their shortcomings, thus balancing the quality of solution and the computation consumption. Zhang et al. [85] devised a hybridization of CS and covariance matrix adaption evolution strategy (CMA_ES) to improve performance for the different optimization problems. Computational results demonstrate that the proposed algorithm outperforms other competitor algorithms. Tang et al. [86] introduced Gaussian distribution, Cauchy distribution, Levy distribution, and Uniform distribution, improving the performance of cuckoo search algorithm by the method of pair combination. Simulation results show that the hybrid distribution with Cauchy distribution and Levy distribution can make the CS algorithm perform better.

With respect to applications, CS has been extensively applied to many domains, such as neural networks [87], image processing [88], nonlinear systems [89,90], network structural optimization [91], agriculture optimization [92], engineering optimization [93], and scheduling [94]. These applications indicate that CS algorithm is an effective and efficient optimizer for solving some real-world problems.

3. Cuckoo Search

The cuckoo search algorithm [69] is a stochastic optimization algorithm that models brood parasitism of cuckoo birds. The algorithm is based on the obligate brood parasitic behavior found in some cuckoo nests by combining a model of this behavior with the principles of Lévy flights, which discard worst solutions and generate new ones after some certain iteration.

According to the mentioned characteristics, CS can be expressed as three idealized rules:

(1)

Each cuckoo lays one egg at a time, and places it in a randomly chosen nest.

(2)

The best nests with the highest-quality eggs (solutions) will be carried over to the next generations.

(3)

The number of available host nests is fixed, and the alien egg is discovered by the host bird with the probability $p_a\in [0,1]$. If the alien egg is discovered, the nest is abandoned and a new nest is built in a new location.

The CS algorithm is equiponderant to the integration of Lévy flights. The position of the ith nest is indicated by using D-dimensional vector $X_i=(x_{i1},x_{i2},\dots ,x_{id}),1\le i\le n$; a Lévy flight is performed:

$$X_i^{t+1}=x_i^t+a\otimes levy(\lambda ) (i=1,2,\dots ,n),$$

(1)

$$a=a_0\otimes (x_j^t-x_i^t)\pm$$

(2)

where α > 0 is the step size that is used to control the range of the random search, which should be related to the scales of the problem of interests, and step size information is more useful can be computed by Equation (2). The product $\otimes$ means entry-wise multiplications. $x_i^t$ and $x_j^t$ are two different solutions selected randomly. A new solution with the same number of cuckoos is generated after partial solutions are discarded. $levy(\lambda )$ with the random walk can be expressed in terms of a simple power-law equation.

$$levy(\beta )\sim \mu =t^{-1-\beta },0<\beta \le 2$$

(3)

where μ and t are two random numbers following the normal distribution and β often takes a fixed value of 1.5.

$$levy(\beta )\sim \frac{\varphi \times \mu }{|v{|}^{1/\beta }}$$

(4)

$$\varphi ={\left[\frac{\Gamma (1+\beta )\times \sin (\frac{\pi \times \beta }{2})}{\Gamma (\frac{1+\beta }{2})\times \beta \times 2^{\frac{\beta -1}{2}}}\right]}^{1/\beta }$$

(5)

where Γ is gamma function. μ and v are random numbers drawn from a normal distribution with mean of 0 and standard deviation of 1, which have an infinite variance with an infinite mean. Here, the consecutive jumps/steps of a cuckoo essentially form a random walk process that obeys a power-law step length distribution with a heavy tail. In Lévy flights random walk component, the new solution X_i is generated through Equation (6).

$$X_{g+1,i}=X_{g,i}+{\alpha }_0\frac{\varphi \times \mu }{|v{|}^{1/\beta }}(X_{g,i-}{X}_{g,best})$$

(6)

where X_g,best represents the best solution obtained thus far and α₀ is a scaling factor. The Lévy distribution is a process of random walk; after a series of smaller steps, Lévy flights can suddenly obtain a relatively larger step size. Lévy distribution is implemented at the initial stage of algorithm, which helps to jump out of the local optimum.

$$X_i^{t+1}=x_i^t+r(X_m^t-X_n^t)$$

(7)

where $X_m^t$ and $X_n^t$ are random solutions at the tth generation. r generates a random number between −1 and 1. The basic steps of the CS algorithm are summarized in Algorithm 1.

Algorithm 1 CS Algorithm.

(1) randomly initialize population of n host nests

(2) calculate fitness value for each solution in each nest

(3) while (stopping criterion is not meet do)

(4)   Generate $x_i^{t+1}$ as new solution by using Lévy flights;

(5)   Choose candidate solution $x_i^t$;

(6)   if $f(x_i^t)>f(x_i^{t+1})$

(7)    Replace $x_i^t$ with new solution $x_i^{t+1}$;

(8)   end if

(9)   Throw out a fraction (pa) of worst nests;

(10) Generate solution $k_i^{t+1}$ using Equation (3);

(11) if $f(x_i^t)>f(x_i^{t+1})$

(12)    Replace $x_i^t$ with new solution $x_i^{t+1}$;

(13) end if

(14) Rank the solution and find the current best.

(15) end while

4. Cuckoo Search Algorithm with Q-Learning and Genetic Operations

4.1. Q-Learning Model

Q-Learning model, a milestone in reinforcement learning research, is an enhanced learning method that is not constrained by the problem model. The optimal policy of Q-Learning is generated by executing the action with the highest expected Q-values, which is the action of maximizing the cumulative benefits with a discount. Control strategy of the optimal step size can be transformed into the optimal action for the agent. The Q function is defined as discounted. In general, the environment is the current state in which the agent makes decisions. The agent includes a set of feasible actions which affect both next state and reward. In fact, the Q-Learning is a mapping from state–action to prediction. The output for state vector s and action a are denoted by Q-value $Q(s,a)$:

$$Q(s_t,a_t)\leftarrow (1-a)Q(s_t,a_t)+a\left[r_{t+1}+\gamma \underset{a_{t+1}}{max}{Q(s}_{t+1}{,a}_{t+1})\right]$$

(8)

where $Q(s_t,a_t)$ represents the cumulative reward of action in the state of s at time t. $Q(s_{t+1},a_{t+1})$ indicates the cumulative reward of action in the state vector s at time t + 1. $r_{t+1}$ is the reward received for the action a at time t + 1. When $s_{t+1}$ is terminal, $Q(s_{t+1},a_{t+1})$ goes to zero, where a and $\gamma$ represent learning factors and discount factors, respectively ($0<a<1$, $0\le \gamma <1$). $\gamma$ determines the impact of lagging returns on optimal action. Q-Learning provides strong proof of convergence. The Q value will converge with probability 1 to Q when each state–action pair is repeatedly visited. The error of Q (s, a) must be reduced by $\gamma$ whenever it is updated. When each state–action pair is visited infinitely, the estimates of $Q_n(s,a)$ converge to real values of $Q(s,a)$ as $n\to \infty$.

4.2. Step Size Control Model by Using Q-Learning

In CS algorithm, the most important parameter is step size scaling factor with the typical characteristics of Lévy flight, in addition to the population size, the number of iterations, and the probability of discovery. Step size scaling factor is as suitable action that is selected to control an individual search process. The accuracy of selected parameter can be improved by predicting before making an action decision. When an individual selects an action, the advantages and disadvantages of various actions can be evaluated by the multi-step effect of individual. Q-Learning is helpful to learn the optimal step size control strategy and transform optimal step size control strategy into optimal action selected of agent.

During the iteration of CS algorithm, the fixed step size strategy cannot meet the dynamic requirements of the algorithm. Considering the aforementioned facts, at the later stage of the CS algorithm, we add three step size control methods in the iterative process: (1) Dynamic linear decreasing strategy (L1) is defined by Equation (9). (2) Dynamic non-linear decreasing strategy (L2) is defined by Equation (10). (3) Adaptive step-size strategy (L3) is defined by Equation (11). Each individual obtains the optimal step size control strategy via learning multiple steps forward, thus becomes close to the optimal solution. Therefore, we try to evaluate the step size control strategy by using multi-step evolution method, which increases the adaptability of individual evolution and improves the performance of the algorithm. The current best step size control strategy is selected to execute the next iteration by using Q-Learning method.

$$a=(a_1-a_0)\times (t_{\max }-t)/t_{\max }+a_1$$

(9)

$$a=(a_1-a_0)\cdot {(t/t_{\max })}^2+(a_0-a_1)\cdot (2\cdot t/t_{\max })+a_1\pm$$

(10)

$$a=a_0+(a_1-a_0)\cdot d_i$$

(11)

$$d_i=\frac{\left|\right|x_i-x_{best}\left|\right|}{d_{\max }}$$

(12)

where t_max expresses the total number of iterations, t is the current number of iterations, and d_max is the maximum distance between the optimal nest and all other nests. $a_0<a_1$, $a_0$ is the initial value of step size.

In Q-Learning algorithm, the agent receives feedback, which is called reward, for each action. When the state is set to s and the action is set to a, a set of actions is set to $H=\{a_1,a_2,\dots a_n\}$, the agent has n actions to choose from each state, and the maximum reward of discount for the agent is:

$$Q(s,a)=r(s_t,a_t)+\gamma \cdot \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })$$

(13)

where $r(s,a)$ is the immediate benefits for state s. $\underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })$ is the maximum return value that the agent select different actions at the next state $s^{\prime }$. $a^{\prime }$ is the action which is selected at the next state $s^{\prime }$. $\gamma$ is the discount factor. The benefits that the agent selecting action a receives is:

$$Q(a)=r(a)+\gamma \cdot Q(a^{(1)})+{\gamma }^2\cdot Q(a^{(2)})+\dots +{\gamma }^m\cdot Q(a^{(m)})$$

(14)

where m represents the number of steps forward, $a,a^{(i)}\in A,$ $1\le i\le m$. When $\gamma$ = 0, Q is reduced to one step forward. When $\gamma$ is close to 1, the lagging benefits of optimal action increase gradually. $r(a)$ is the immediate benefit that the agent selects action a, which expresses that individuals have evolved once, and new individuals use $(a^{(1)})$ to generate new individuals again. At this time, the benefit is recorded as $Q(a^{(1)})$. By analogy, after m evolution, a new individual is generated by using $(a^{(m)})$, and the corresponding benefit is recorded as $Q(a^{(m)})$.

n offspring will be generated after each evolution. These offspring are evolved again by adopting n strategies. $n^m$ offspring will be produced after m evolutions. Boltzmann distribution is used to calculate the probability of new individuals retained. Boltzmann distribution can be defined by Equation (15):

$$p(a_i)=e^{\frac{r(a_i)}{T}}/{\displaystyle {\sum }_i^n{e}^{\frac{r(a_i)}{T}}}$$

(15)

where $r(a_i)$ indicates the immediate benefits of the ith step strategy and $T$ represents the temperature.

The step size control strategy corresponding to the maximum probability is selected. The results of each generation are simplified by Boltzmann distribution. $f_p(a)$ is defined as the fitness function corresponding to parent individual in the population and $f_o(a)$ is the fitness function corresponding to the individual after adopting the parameter selection strategy. Substituting $r(a)=f_p(a)-f_o(a)$ into Equation (13) Equation (16) is obtained.

$$Q(a)=f_p(a)-(1-\gamma )\cdot f_o(a^{(1)})-\gamma \cdot (1-\gamma )\cdot f_o(a^{(2)})-\dots -{\gamma }^m\cdot f_o(a^{(m)})$$

(16)

where $\forall m$, ${\lim }_{\Upsilon \to 1}{\Upsilon }^m(1-\Upsilon )=0$ ${\lim }_{\Upsilon \to 1}{\Upsilon }^m=1$; according to Equation (17), it can be concluded that ${\lim }_{\Upsilon \to 1}Q(a)=f_p(a)-f_o(a{(}^m))$, $a^{\prime }=\arg {\max }_a\in A\lim \Upsilon \to 1Q(a)=\arg {\max }_a\in A(f_p(a)-f_o(a{(}^m)))$. The step size control strategy model with Q-Learning is described in Algorithm 2 and Figure 1.

Algorithm 2 Step size with Q-Learning.

(1) Each individual is expressed as (x, σ), and the number of learning steps M is set;

(2) Generate three new offspring for each individual by using the given step size control strategy (Linear decreasing strategy, non-linear decreasing strategy, adaptively step-size dynamic adjustment strategy), and set t = 1;

(3) Do while t < m

Each individual generates three offspring by using the given step size control strategy, as shown in Equations (9)–(12).

Calculate the probability of the newly generated offspring by using the Boltzmann distribution, and an individual is selected according to the probability.

t = t + 1;

(4) Calculate the corresponding Q value of each retained individual according to the three-step selection strategy. The step size corresponding to the step control strategy is retained when Q is maximized, the corresponding offspring are selected, and other offspring will be discarded.

4.3. Genetic Operation

4.3.1. Crossover Process

As we know, two of the most important operators are the crossover operator and mutation operator for genetic operation [95], which have a great influence on the behavior and performance of genetic operation. Therefore, these operations are introduced into the DMQL-CS algorithm. In crossover process, a parameter C_r is defined as the probability of crossover and chromosomes are divided into pairs. We introduce the specifically designed crossover operation into the problem of logistics distribution center location in this paper, and apply it to a pair of chromosomes G₁ and G₂, as illustrated in Figure 2. First, some genes are randomly selected in chromosome G₁, as those pointed to by a red arrow in the illustration. Then, these genes are found in chromosome G₂, as pointed to by a green arrow. If the same gene is not found in G₂, two genes are randomly selected as the crossover point. Generate one child as the combination of red-pointed genes in G₁ and the rest of blue genes in G₂, and generate another child as the combination of green-pointed genes in G₂ and the rest of blue genes in G₁. Finally, the optimal is found as an arc between any two nodes by using enumeration method, which keep the child to obtain the lowest objective value, and obtain chromosomes R₁ and R₂. Two chromosomes are selected from the parents and children with the smallest objective values to replace the parents.

At the same time, the crossover rate (C_r) is a critical factor for how the crossover operator behaves, which determines the performance of the algorithm to some extent. To improve the search ability of the algorithm, a substantial number of strategies have been designed to adjust the crossover rate. In this work, a self-adaptive scheme was used to adjust the crossover rate, which can be calculated as shown below.

$$C_r=1/[1+\exp (K_1\mho )]$$

(17)

where $\mho =f_{avg}-f_{\max }$, $f_{avg}$ is the average fitness, $f_{\max }$ is the max fitness, and $K_1$ is the scale factor between 0 and 1, $K_1$ = 0.02.

4.3.2. Mutation Process

A parameter C_m is defined as the mutation probability. The number r is randomly generated in the interval [0, 1]. If r < C_m, the ith chromosome G₁ is selected to perform the mutation operation and this process is repeated at each iteration. For illustration, we continue to use the problem of logistics distribution center location with 40 cities and 10 distribution centers. Two genes located on chromosome G₁ are randomly selected and their positions swapped to obtain a possible child. Then, the optimal is found as an arc between any two nodes by using enumeration method, which keeps the child obtaining the lowest objective value. Finally, we get chromosome R₁, as shown in Figure 3. If R₁ has a smaller objective value than G₁, G₁ is replaced with R₁, else G₁ is retained. A new generation of population is generated after the evaluation, crossover, and mutation operations.

4.3.3. Cuckoo Search Algorithm with Q-Learning Model and Genetic Operator

Introducing Q-Learning into CS algorithm helps to learn the optimal step size strategy and transform. Crossover and mutation strategies enable the nest to approach the historical optimal nest quickly, which can speed up the global convergence rate. The structure of the genetic operator cuckoo search algorithm with Q-Learning model (DMQL-CS) is described in Algorithm 3.

Algorithm 3 DMQL-CS Algorithm.

Input: Population size, NP; Maximum number of function evaluations, MAX_FES, LP

(1) Randomly initialize position of NP nest, FES = NP;

(2) Calculate the fitness value of each initial solution;

(3) while (stopping criterion is not meet do)

(4)     Select the best step size control strategy according to Algorithm 2;

(5)   Generate new solution $x_i^{t+1}$ with the new step size by Lévy flights;

(6)   Randomly choose a candidate solution $x_i^t$;

(7)   if $f(x_i^t)>f(x_i^{t+1})$

(8)    Replace $x_i^t$ with new solution $x_i^{t+1}$;

(9)   end if

(10)    Generate new solution $x_i^{t+1}$ by using crossover operator and mutation operator;

(11)    Throw out a fraction (pa) of worst nests, generate solution $k_i^{t+1}$ using Equation (3);

(12)    if $f(x_i^t)>f(x_i^{t+1})$

(13)    Replace $x_i^t$ with new solution $x_i^{t+1}$;

(14)    end if

(15)    Rank the solution and find the current best.

(16) end while

4.3.4. Analysis of Algorithm Complexity

To show the convergence effect of the algorithm, typical function Rastrigrin was selected to analyze the convergent process of DMQL-CS algorithm. Figure 4 shows the location distribution of cuckoo individuals in the search area with a population size of 10. Figure 4a describes the individual distribution at the first generation, Figure 4b describes the individual distribution at the 30th generation, Figure 4c describes the individual distribution at the 50th generation, and Figure 4d describes the individual distribution at the 80th generation. In Figure 4, it can be seen that the activity area of individuals keeps changing and gradually draws closer to the optimal solution during the evolution of the algorithm. It is worth noting that algorithm converged at the 80th generation, which indicates that Q-learning and genetic operation expand activity area of the population and improve the convergence performance of the DMQL-CS algorithm.

5. Results

5.1. Optimization of Functions and Parameter Settings

In this section, to check and verify the efficacy DMQL-CS algorithm, it is thoroughly investigated through benchmark evaluations from various respects. We tested our algorithms on two function groups: Group A and Group B. Group A contains fourteen different global optimization problems, as shown in

Table 1. Brief description of fifteen functions.

Type	Function	Name	Search Range	Acceptable Accuracy	Global Optimum
Unimodal	F1	Sphere	[−100, 100]	1 × 10−8	0
	F2	Rosenbrock	[−30, 30]	1 × 10−8	0
	F3	Step	[−100, 100]	1 × 10−8	0
	F4	Schwefel2.22	[−10, 10]	1 × 10−8	0
Multimodal Shifted multimodal	F5	Ackley	[−32, 32]	1 × 10−8	0
	F6	Rastrigin	[−5.12, 5.12]	10	0
	F7	Griewank	[−600, 600]	0.05	0
	F8	Generalized Penalized1	[−50, 50]	1 × 10−8	0
	F9	Generalized Penalized2	[−50, 50]	1 × 10−8	0
	F10	Shifted Schwefels Problem 1.2	[−100, 100]	1 × 10−8	−450
	F11	Shifted Rotated High Conditioned Elliptic Function	[−100, 100]	1 × 10−8	−450
	F12	Shifted Rosenbrock	[−100, 100]	2	390
	F13	Shifted Rotated Ackleys	[−32, 32]	2	−140
	F14	Shifted Griewanks	[−600, 600]	0.2	0
	F15	Shifted Rotated Rastrigin	[−5.12, 5.12]	10	−330

. Group B is the CEC 2013 test suite including 28 benchmark functions. To make a fair comparison, all experiments were carried out on a P4 Dual-core platform with a 1.75 GHz processor and 4 GB memory, running under the Windows 7.0 operating system. The algorithms were written in MATLAB R2017a. The following were set: maximum number of evaluation MAX_FES = NP × 10⁵, population size NP = 30, run time T = 30, and probability of foreign eggs p_a = 0.25.

5.2. Comparison with Other CS Variants and Rank Based Analysis

We compared the performance of DMQL-CS with four improved CS variants: CCS [68], GCS [96], CSPSO [97], and OLCS [71]. CCS is a modified Chaos enhanced Cuckoo search algorithm. GCS introduces Gaussian disturbance into the CS algorithm. CSPSO is a kind of algorithm combining CS with PSO. A new search strategy based on orthogonal learning strategy is used in OLCS to enhance the exploitation ability of CS algorithm. The parameter configurations of these algorithms are shown in

Table 2. The personal parameters of different algorithms.

Algorithms	Parameter Configurations
CCS [68]	pa = 0.2, a = 0.5, b = 0.2, xi = (0, 1)
GCS [96]	a = 1/3, pa = 0.25
CSPSO [97]	pa = 0.25, a = 0.1, W = 0.9~0.4, c1 = c2 = 2.0
OLCS [71]	pa = 0.2, a = 0.5, K = 9, Q = 3
DMQL-CS	pa = 0.25, M = 3, $\gamma$ = 0.5

according to corresponding references. Fifteen benchmark functions are shown in

Table 3. The optimization results obtained by CCS, GCS, CSPSO, OLCS, and DMQL-CS at D = 30.

Func	CCS	GCS	CSPSO	OLCS	DMQL-CS
F1	3.21 × 10−12 ± 2.09 × 10−12	4.34 × 10−10 ± 3.23 × 10−11	4.77 × 10−45 ± 3.65 × 10−44	2.89 × 10−106 ± 1.43 × 10−105	0.00 ± 0.00
F2	3.96 × 10−5 ± 8.01 × 10−5	1.54 × 10−1 ± 1.82 × 10−1	1.66 × 10−1 ± 2.95 × 100	1.45 × 10−7 ± 4.01 × 10−7	0.76 × 10−7 ± 5.12 × 10−7
F3	4.12 × 100 ± 3.11 × 100	7.09 × 100 ± 2.13 × 100	7.12 × 100 ± 3.31 × 100	0.00 ± 0.00	4.88 × 10−2 ± 5.19 × 10−1
F4	5.56 × 10−33 ± 3.21 × 10−32	4.11 × 10−24 ± 5.01 × 10−23	2.76 × 10−76 ± 4.43 × 10−76	4.09 × 10−34 ± 3.88 × 10−34	8.88 × 10−35 ± 5.78 × 10−34
F5	4.67 × 10−5 ± 3.21 × 10−6	4.11 × 10−14 ± 5.01 × 10−13	2.76 × 10−2 ± 4.43 × 10−2	7.21 × 10−15 ± 0.00	1.01 × 10−15 ± 2.87 × 10−14
F6	6.22 × 10−7 ± 1.12 × 10−5	3.13 × 10−7 ± 2.98 × 10−6	3.87 × 101 ± 2.01 × 101	0.00 ± 0.00	0.00 ± 0.00
F7	3.13 × 10−10 ± 1.11 × 10−10	2.87 × 10−11 ± 2.12 × 10−10	5.77 × 10−6 ± 3.03 × 10−6	0.00 ± 0.00	0.00 ± 0.00
F8	4.90 × 10−7 ± 2.77 × 10−7	3.96 × 10−7 ± 3.31 × 10−5	1.39 × 10−6 ± 1.17 × 10−5	1.88 × 10−8 ± 4.09 × 10−8	3.38 × 10−7 ± 2.99 × 10−7
F9	2.22 × 10−23 ± 1.05 × 10−22	3.04 × 10−22 ± 1.99 × 10−22	4.67 × 10−4 ± 2.89 × 10−6	4.39 × 10−29 ± 6.50 × 10−26	2.45 × 10−22 ± 6.89 × 10−22
F10	6.01 × 10−15 ± 3.77 × 10−16	3.66 × 10−15 ± 2.19 × 10−16	3.21 × 10−16 ± 5.33 × 10−16	3.21 × 10−15 ± 3.17 × 10−11	9.55 × 10−15 ± 7.09 × 10−13
F11	2.76 × 109 ± 5.77 × 109	2.81 × 109± 3.06 × 109	2.28 × 109 ± 9.02 × 108	5.63 × 106 ± 2.22 × 106	5.11 × 106 ± 3.90 × 106
F12	1.23 × 101 ± 2.77 × 100	1.42 × 101 ± 2.93 × 100	5.23 × 101 ± 2.91 × 10+1	2.65 × 101 ± 4.23 × 100	4.21 × 101 ± 1.09 × 101
F13	1.90 × 103 ± 3.97 × 103	5.88 × 103 ± 3.08 × 103	4.34 × 104 ± 1.88 × 103	4.70 × 103 ± 2.26 × 103	2.06 × 102 ± 3.77 × 101
F14	2.71 × 10−1 ± 2.09 × 100	4.01 × 10−1 ± 7.00 × 100	1.37 × 10−2 ± 8.01 × 10−2	0.00 ± 0.00	0.00 ± 0.00
F15	5.90 × 101 ± 3.78 × 100	7.88 × 101 ± 2.89 × 100	0.98 × 102 ± 3.56 × 101	3.65 × 101 ± 4.11	2.87 × 101 ± 4.77 × 101

,

Table 4. The ranking of different strategies according to the Friedman test.

	CCS	GCS	CSPSO	OLCS	MP-QL-CS
Friedman rank	3.18	3.82	4.31	2.53	2.44
Final rank	3	4	5	2	1

,

Table 5. The optimization results obtained by CCS, GCS, CSPSO, OLCS, and DMQL-CS at D = 50.

Func	CCS	GCS	CSPSO	OLCS	MP-QL-CS
F1	3.76 × 10−6 ± 2.21 × 10−6	2.78 × 10−8 ± 5.67 × 10−9	3.99 × 10−19 ± 6.43 × 10−18	4.45 × 10−29 ± 6.33 × 10−28	6.55 × 10−30 ± 2.90 × 10−28
F2	3.88 × 101 ± 3.09 × 101	2.89 × 101 ± 1.22 × 102	5.98 × 10−1 ± 2.99 × 10−1	3.10 × 101 ± 2.90 × 101	1.99 × 10−1 ± 4.56 × 10−1
F3	3.78 × 101 ± 2.66 × 100	3.67 × 101 ± 4.52× 100	5.34 × 100± 2.11× 100	0.00± 0.00	4.77 × 10−2 ± 3.21 × 10−12
F4	4.02 × 10−2 ± 1.55 × 10−2	3.78 × 10−2 ± 2.90 × 10−2	3.88 × 10−4 ± 1.89 × 10−4	4.65 × 10−5 ± 4.09 × 10−5-	4.90 × 10−7 ± 2.11 × 10−8
F5	1.89 × 10−2 ± 2.87 × 10−2	5.97 × 10−7 ± 5.22 × 10−7	4.77 × 10−2 ± 4.44 × 10−1	5.09 × 10−12 ± 4.89 × 10−14	2.99 × 10−12 ± 3.09 × 10−14
F6	5.34 × 10−1 ± 3.87 × 10−1	6.44 × 10−6 ± 3.72 × 10−6	9.28 × 103 ± 4.73 × 103	0.00 ± 0.00	3.98 × 10−2 ± 2.22 × 10−1
F7	3.12 × 10−2 ± 4.78 × 10−2	4.33 × 10−2 ± 9.21 × 10−2	6.34 × 10−2 ± 3.18 × 10−2	0.00 ± 0.00	0.00 ± 0.00
F8	6.67 × 10−5 ± 1.90 × 10−5	5.78 × 10−7 ± 3.77 × 10−7	8.90 × 10−7 ± 2.30 × 10−7	3.77 × 10−8 ± 7.56 × 10−8	1.77 × 10−4 ± 2.12 × 10−4
F9	5.78 × 10−3 ± 0.55 × 10−3	7.78 × 10−20 ± 6.23 × 10−20	3.66 × 10−1 ± 3.41 × 10−1	4.67 × 10−25 ± 1.23 × 10−26	3.90 × 10−10 ± 3.66 × 10−9
F10	5.78 × 10−10 ± 5.55 × 10−10	3.99 × 10−10 ± 2.98 × 10−10	7.90 × 10−10 ± 8.11 × 10−10	7.34 × 10−10 ± 5.45 × 10−9	2.78 × 10−10 ± 1.34 × 10−6
F11	3.66 × 1012 ± 3.89 × 1012	2.89 × 1012 ± 5.78 × 1012	2.90 × 107 ± 3.11 × 107	5.89 × 108 ± 9.90 × 108	4.89 × 1011 ± 3.67 × 1010
F12	4.89 × 103 ± 3.78 × 103	2.90 × 103 ± 2.22 × 103	5.98 × 103 ± 2.09 × 103	6.99 × 102 ± 3.90 × 101	2.97 × 103 ± 1.86 × 103
F13	4.89 × 105 ± 2.17 × 105	5.89 × 104 ± 1.12 × 104	5.33 × 105± 4.56 × 105	5.02 × 104 ± 2.09 × 104	1.09 × 103 ± 3.89 × 103
F14	6.98 × 101 ± 1.11 × 102	5.56 × 101 ± 2.98 × 102	5.89 × 102 ± 2.21 × 102	0.00 ± 0.00	2.90 × 101 ± 3.76 × 101
F15	6.25 × 102 ± 3.33 × 102	4.28 × 102 ± 1.77 × 102	3.45 × 103 ± 2.76 × 103	8.89 × 102 ± 4.11 × 102	2.22 × 102 ± 1.78 × 102

and

Table 6. The ranking of different strategies according to the Friedman test.

	CCS	GCS	CSPSO	OLCS	MP-QL-CS
Friedman rank	4.19	3.62	4.15	2.41	2.46
Final rank	5	3	4	1	2

at D = 30 and D = 50. All optimization algorithms were tested by using the same parameter settings: population size NP = 30, MAX_FES = 100,000 × D, probability switching parameter p_a = 0.25, and run time T = 30.

As shown in Table 3, the DMQL-CS find global optima 0.00 on the four benchmark functions F1, F6, F7, and F14 when D = 30. For unimodal functions F1–F5, the DMQL-CS algorithm achieves higher accuracy than other CS variants on functions F2, F4, and F5. DMQL-CS is only inferior to OLCS on F2. For multimodal problems F6–F11, DMQL-CS algorithm shows higher performance than the other CS variants on functions F6, F7, F8, and F11. For F10, the same solution is found by the four algorithms (CCS, GCS, CSPSO, and OLCS). For the shifted unimodal functions F13–F15, DMQL-CS is significantly better than CCS, GCS, OLCS, and CSPSO on F13, F14 and F15. For F12, CCS performs the best.

DMQL-CS still has outstanding optimization performance when D = 50, as shown in Table 5. From the results, it is apparent that the convergence precision of other algorithms drops rapidly, while the DMQL-CS algorithm achieves better performance than other CS variants on most functions. DMQL-CS and OLCS achieve the global optimum on function F7. DMQL-CS cannot get the minimum; even then, it is not inferior to other algorithms on F4, F5, F10, F12, F13, and F15. In addition, the DMQL-CS demonstrates a remarkable accuracy on benchmark F1 and F2. Comparing with the optimization results, we can conclude that the DMQL-CS optimization algorithm explored a larger search space than other CS variants. Moreover, it is important to point out that, regardless of the problem’s dimensionality, the DMQL-CS converges to the better solution on the shifted multimodal functions F13, F14 and F15. Therefore, these statistical tests confirmed that DMQL-CS algorithm with Q-Learning step size and genetic operators has a better overall performance than all other tested competitors. For a clearer observation that DMQL-CS performs best, Table 4 shows the ranking of the strategies in Table 3 according to the Friedman test. We can see that DMQL-CS obtains the best rank, OLCS ranks second, followed by CCS, GCS, and CSPSO. Table 6 shows the ranking of the five strategies according to the Friedman test. OLCS obtains the best rank, DMQL-CS ranks second, followed by GCS, CSPSO, and CCS.

To further demonstrate the convergence of DMQL-CS, the median convergence properties of five algorithms are illustrated in Figure 5. There is no obvious “evolution stagnation” for all algorithms. For the same population size and number of generations, the optimization performance of the four algorithms declines rapidly. However, DMQL-CS can get better convergence curve than CCS, GCS, CSPSO, and OLCS on F1–F2, F5–F6, F12, and F14. In Figure 5, DMQL-CS algorithm converged to the specified error threshold on function F1, which suggests that DMQL-CS algorithm has a faster convergence rate for the specified error threshold. Generally speaking, when M is too small, useful step size information will not be learned. When M is too large, the speed of Q-Learning will be slowed down. When the value of M is 3 or 5, the convergence performance of DMQL-CS can be improved for the ill-condition function F2, complex multimodal functions F5–F6, and Shifted multimodal functions F12 and F14. It is worth mentioning that the accuracy of OLCS is similar to that of DMQL-CS, but the convergence speed of DMQL-CS is much faster than that of OLCS. For multimodal function, all algorithms converge to the specified error threshold with the same number of successes. However, DMQL-CS has good reliability, stability, and faster convergence rate on functions F5–F6. For function F14, DMQL-CS algorithms can find the global optimum with 50,000 FES. As mentioned above, it can be clearly observed that DMQL-CS provided better performance than the four other CS versions, and achieves a promising solution on most test functions.

A series of comparisons proved the high efficiency of DMQL-CS. The performance ranking of the multiple algorithms of the test suite is listed in

Table 7. Rank results of each algorithm on F1–F8 for D = 30 and D = 50.

Dim	Algorithm	F1	F2	F3	F4	F5	F6	F7	F8
30	CCS	4	3	3	3	4	4	4	4
	GCS	5	4	4	2	3	3	3	3
	CSPSO	3	5	5	4	5	5	5	5
	OLCS	2	1	1	5	2	1	1	2
	DMQL-CS	1	2	2	1	1	1	1	1
50	CCS	5	5	5	5	4	4	3	3
	GCS	4	3	4	4	3	2	4	2
	CSPSO	3	2	3	3	5	5	5	4
	OLCS	2	4	1	2	2	1	1	1
	DMQL-CS	1	1	2	1	1	3	1	5

,

Table 8. Rank results of each algorithm on F9-F15 for D = 30 and D = 50.

Dim	Algorithm	F9	F10	F11	F12	F13	F14	F15
30	CCS	3	4	4	1	2	4	4
	GCS	4	3	5	2	4	3	3
	CSPSO	5	1	3	5	5	2	5
	OLCS	1	2	2	3	3	1	2
	DMQL-CS	2	5	1	4	1	1	1
50	CCS	4	3	5	4	4	3	2
	GCS	2	2	4	2	3	4	4
	CSPSO	5	5	1	5	5	5	3
	OLCS	1	4	2	1	2	1	5
	DMQL-CS	3	1	3	3	1	2	1

and

Table 9. Total rank and final rank on F1–F15 for D = 30 and D = 50.

Dim	Rank	Algorithms
		CCS	GCS	CSPSO	OLCS	DMQL-CS
30	Total rank	49	53	63	29	25
	Final rank	3	4	5	2	1
50	Total rank	60	47	55	30	29
	Final rank	5	3	4	2	1

. The average rankings of the five CS variants for functions F1–F8 are shown in Table 7 (D = 30 and D = 50). The average rankings of the five CS variants for functions F9–F15 are shown in Table 8 (D = 30 and D = 50). In competition ranking, if performances of algorithms are the same, they received the same rank. It can be seen in Table 7, Table 8 and Table 9 that the average ranking value of DMQL-CS on D = 30 is smaller than that of CCS, GCS, OLCS, and CSPSO. Therefore, the performance of DMQL-CS is better than the other CS variants. When D = 50, the results are similar to those when D = 30, with the average ranking value of DMQL-CS being smaller than those of CCS, GCS, OLCS, and CSPSO.

In Table 9, DMQL-CS has the best total rank when D = 30 and D = 50, i.e., 25 and 29, which means that DMQL-CS has the best performance on most of the test functions compared with other algorithms. OLCS has the second-best total rank at D = 30 and D = 50, i.e., 29 and 30. Obviously, OLCS has better performance than the three other algorithms on high-dimension test functions. Similarly, in Table 9, the order can be clearly observed: DMQL-CS > OLCS > CCS > GCS > CSPSO at D = 30; and DMQL-CS > OLCS > GCS > CCS > CSPSO at D = 50. Based on the analysis of the above, DMQL-CS has the best performance among all the algorithms on both D = 30 and D = 50.

5.3. Statistical Analysis of Performance for the CEC 2013 Test Suite

In this section, the CEC 2013 test suite is selected to test the effectiveness of three different algorithms (jDE [98], SaDE [99], and CLPSO [100]). These algorithms can be seen as representatives of the state-of-the-art algorithms for comparison, and the parameter configurations of these algorithms were set according to the corresponding references, as listed in

Table 10. The personal parameters of different algorithms.

Algorithms	Parameter Configurations
jDE [98]	F = 0.5, CR = 0.9
SaDE [99]	F~N(0.5, 0.3), CR0 = 0.5, CR~N(CRm, 0.1), LP = 50
CLPSO [100]	W = 0.7298, c = 1.49618, m = 7, pc = 0.05~0.5
DMQL-CS	pa = 0.25, PL = 20, $\eta$ = 0.015

.

Table 11. The mean and standard deviation (STD) of CEC 2013 test suite with four algorithms.

Function	Mean Std	Algorithms
		SaDE	jDE	CLPSO	DMQL-CS
CEC 2013-F1	Mean/Std	0.00/0.00	0.00/0.00	2.16 × 10−13/0.00	0.00/0.00
CEC 2013-F2	Mean/Std	4.21 × 105/1.21 × 105	1.27 × 105/6.86 × 105	2.99 × 107/2.10 × 106	1.12 × 105/4.88 × 105
CEC 2013-F3	Mean/Std	2.98 × 107/2.99 × 107	2.99 × 106/3.01 × 106	3.16 × 108/2.92 × 108	3.78 × 107/5.87 × 106
CEC 2013-F4	Mean/Std	3.22 × 103/2.98 × 103	0.97 × 101/1.88 × 101	4.87 × 104/1.09 × 103	8.09 × 100/7.90 × 100
CEC 2013-F5	Mean/Std	0.00/0.00	1.19 × 10−13/3.55 × 10−14	3.54 × 10−11/2.01 × 10-12	4.56 × 10−14/1.90 × 10-12
CEC 2013-F6	Mean/Std	2.78 × 101/5.66 × 101	1.23 × 101/9.78 × 100	3.56 × 101/1.00 × 101	5.62 × 10−2/9.89 × 100
CEC 2013-F7	Mean/Std	2.22 × 101/1.38 × 100	2.12 × 101/1.38 × 100	6.97 × 101/3.20 × 101	8.90 × 101/5.12 × 100
CEC 2013-F8	Mean/Std	2.11 × 101/6.45 × 101	2.01 × 100/6.11 × 100	2.09 × 101/3.73 × 10-2	2.07 × 101/1.78 × 10−2
CEC 2013-F9	Mean/Std	1.78 × 101/2.33 × 100	2.59 × 101/4.45 × 100	3.19 × 101/3.62 × 100	1.01 × 101/7.90 × 100
CEC 2013-F10	Mean/Std	2.73 × 10−1/2.33 × 10−1	4.37 × 10−2/4.73 × 10−2	3.99 × 101/2.01 × 100	2.77 × 10−3/6.89 × 10−2
CEC 2013-F11	Mean/Std	3.87 × 10−2/3.64 × 10−1	0.00/0.00	6.14 × 101/3.01× 101	0.00/0.00
CEC 2013-F12	Mean/Std	4.19 × 101/2.65× 101	5.56 × 101/1.49× 101	1.23 × 102/2.11× 101	9.01 × 100/4.67 × 101
CEC 2013-F13	Mean/Std	1.10 × 101/3.21 × 102	1.29 × 101/5.22× 101	1.78 × 102/3.91× 101	1.11 × 102/3.99 × 102
CEC 2013-F14	Mean/Std	7.34 × 100/2.22 × 100	1.06 × 10−4/4.24 × 10−3	5.77 × 102/4.79× 102	1.89 × 102/0.00
CEC 2013-F15	Mean/Std	4.90 × 103/5.67 × 102	5.03 × 103/6.02 × 102	6.01 × 103/4.25× 102	4.98 × 103/2.67 × 103
CEC 2013-F16	Mean/Std	2.24 × 100/5.12 × 10−1	3.05 × 100/2.26 × 10−1	3.22 × 100/2.21 × 10-1	1.18 × 100/2.05 × 10−1
CEC 2013-F17	Mean/Std	6.12 × 101/1.16 × 10−2	6.13 × 101/3.65 × 100	7.23 × 101/5.51 × 100	5.89 × 102/4.24 × 101
CEC 2013-F18	Mean/Std	1.02 × 102/2.57× 101	1.61 × 102/3.21 × 101	2.25 × 101/1.11× 101	1.78 × 101/1.12 × 101
CEC 2013-F19	Mean/Std	4.91 × 100/6.18 × 10−1	1.05 × 100/3.81 × 10−1	3.18 × 100/1.03 × 10−1	3.27 × 100/3.99 × 10−1
CEC 2013-F20	Mean/Std	1.10 × 101/5.23 × 100	1.15 × 101/4.09 × 100	1.41 × 101/2.55 × 100	1.02 × 101/5.12 × 100
CEC 2013-F21	Mean/Std	2.80 × 102/3.55 × 101	2.65 × 102/1.29 × 101	3.12 × 102/4.11 × 101	2.01 × 102/4.88 × 102
CEC 2013-F22	Mean/Std	1.25 × 103/2.23 × 102	1.17 × 103/2.23 × 102	7.35 × 102/1.01 × 102	8.23 × 102/6.98 × 102
CEC 2013-F23	Mean/Std	4.83 × 103/0.21 × 103	4.90 × 103/2.21 × 102	6.23 × 103/3.83 × 102	2.00 × 103/3.23 × 102
CEC 2013-F24	Mean/Std	2.35 × 102/2.56 × 100	2.21 × 102/5.56 × 101	2.89 × 102/7.38 × 100	2.67 × 102/3.21 × 100
CEC 2013-F25	Mean/Std	2.45 × 102/1.28 × 101	2.66 × 102/1.01 × 100	2.80 × 102/6.33 × 100	2.02 × 102/3.98 × 100
CEC 2013-F26	Mean/Std	2.23 × 102/2.09 × 103	2.17 × 102/2.51 × 101	1.97 × 102/1.66 × 10−1	1.04 × 102/1.09 × 102
CEC 2013-F27	Mean/Std	5.48 × 102/4.23 × 101	6.34 × 102/3.32 × 102	8.13 × 102/2.65 × 102	6.89 × 102/3.89 × 102
CEC 2013-F28	Mean/Std	3.05 × 102/0.00	3.05 × 102/0.00	3.03 × 102/2.98 × 10-3	3.02 × 102/3.20 × 102

summarizes the results of CEC 2013 test problems on 28 benchmark functions for 30-dimensional case. The rank was used to obtain the ranking of different algorithms on all problems, as shown in

Table 12. The ranking of different strategies according to the Friedman test.

	SaDE	jDE	CLPSO	DMQL-CS
Friedman rank	3.34	2.95	4.26	2.59
Final rank	3	2	4	1

. This means that DMQL-CS gets the first rank and outperforms jDE, SaDE, and CLPSO. The results in Table 11 indicate that with 80% certainty DMQL-CS has statistically higher accuracy than the other algorithms. Note that DMQL-CS obtains the global optimal value 0.00 on F1 and F11. DMQL-CS is significantly better than the three other algorithms, especially on functions CEC 2013-F1, CEC 2013-F2, and CEC 2013-F4. About basic Multimodal Function and composition Functions (CEC 2013-F6–CEC 2013-F28), the ability of DMQL-CS to find the optimal solution is slightly better than that of CLPSO. For functions CEC 2013-F5, CEC 2013-F7, CEC 2013-F17, CEC 2013-F22, and CEC 2013-F25, the performance of DMQL-CS is slightly worse than the other algorithms. For the Unimodal problem CEC 2013-F3, jDE obtains the best solution 2.99 × 10⁶. On Shift Rastrigin Function, SaDE and jDE can get better solutions of 1.10 × 10¹ and 1.06 × 10⁻⁴, respectively. For CEC 2013-F25–CEC 2013-F26, DMQL-CS is obviously better than SaDE and jDE. CLPSO has the weakest ability to find the optimal solution for 28 functions. From the above results, it can be seen that DMQL-CS with Q-Learning and genetic operations has a better overall performance than all other tested competitors on the CEC 2013 test suite.

Table 13. Comparisons between DMQL-CS and other algorithms for CEC 2013 test suite.

Function	SaDE	jDE	CLPSO	DMQL-CS
	Rank	Rank	Rank	Rank
CEC 2013-F1	1 (≈/=)	1 (≈/=)	4 (-)	1
CEC 2013-F2	4 (-)	2 (-)	3 (-)	1
CEC 2013-F3	2 (-)	1 (+)	4 (-)	3
CEC 2013-F4	3 (-)	2 (-)	4 (-)	1
CEC 2013-F5	1 (+)	3 (+)	2 (+)	4
CEC 2013-F6	3 (-)	2 (-)	4 (-)	1
CEC 2013-F7	2 (+)	1 (+)	3 (+)	4
CEC 2013-F8	4 (-)	1 (+)	3 (-)	2
CEC 2013-F9	2 (-)	3 (-)	4 (-)	1
CEC 2013-F10	2 (-)	3 (-)	4 (-)	1
CEC 2013-F11	3 (-)	1 (≈/=)	4 (-)	1
CEC 2013-F12	2 (-)	3 (-)	4 (-)	1
CEC 2013-F13	1 (+)	3 (+)	4 (+)	2
CEC 2013-F14	2 (+)	1 (+)	4 (-)	3
CEC 2013-F15	1 (+)	3 (-)	4 (-)	2
CEC 2013-F16	2 (-)	3 (-)	4 (-)	1
CEC 2013-F17	1 (+)	2 (+)	3 (+)	4
CEC 2013-F18	2 (-)	3 (-)	4 (-)	1
CEC 2013-F19	4 (-)	1 (+)	2 (+)	3
CEC 2013-F20	2 (-)	3 (-)	4 (-)	1
CEC 2013-F21	3 (-)	2 (-)	4 (-)	1
CEC 2013-F22	3 (+)	2 (+)	1 (+)	4
CEC 2013-F23	2 (-)	3 (-)	4 (-)	1
CEC 2013-F24	2 (+)	1 (+)	4 (-)	3
CEC 2013-F25	1(+)	2 (+)	3 (+)	4
CEC 2013-F26	4 (-)	3 (-)	2 (-)	1
CEC 2013-F27	1 (+)	2 (+)	4 (-)	3
CEC 2013-F28	3 (-)	3 (-)	2 (-)	1
Rank_Sun	63	60	96	56
Rank_Final	3	2	4	1

reports the rankings of the results between DMQL-CS and other algorithms. In Table 13, it can be seen that DMQL-CS performs the best among the four algorithms. DMQL-CS exhibits consistent ranks of the first in optimizing most of the functions. For a clearer observation that DMQL-CS performs best, Table 12 shows the ranking of the algorithms in according to the Friedman test. DMQL-CS obtains the best rank, jDE ranks second, followed by SaDE and CLPSO.

5.4. Application in the Problem of Logistics Distribution Center Location

5.4.1. Problem Description

The location of logistics distribution center is an important link in the logistics system. The location of distribution center determines the efficiency of the entire logistics network system and the utilization of resources. The location selection model of logistics distribution center is a nonlinear model with more complex constraints and non-smooth characteristics, which can be attributed to HP-hard problem. The problem of logistics distribution center location can be described as: m cargo distribution centers are searched in n demand points, so that the distance between m searched distribution centers and other n cargo demand points is the shortest. At the same time, the following constraint conditions must be met: the supply of goods in the distribution center can meet the requirements of the cargo demand points; the goods required for a cargo demand point can only be provided by one distribution center; and the cost of transporting the goods to the distribution center is not considered. According to the above assumptions, the mathematical model of the problem for logistics distribution center location can be described as:

$$\min (\cos t)={\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n(nee{d}_j\cdot dis{t}_{i,j}\cdot {\mu }_{i.j})}}$$

(18)

$$s.t.\hspace{1em}\hspace{1em}{\displaystyle \sum _{i=1}^m{\mu }_{i,j}}=1,i\in M,j\in N$$

(19)

$${\mu }_{i,j}\le h_j,i\in M,j\in N$$

(20)

$${\displaystyle \sum _{t=1}^m{h}_i}\le p,i\in M$$

(21)

$$h_j\in \left\{0,1\right\},i\in M$$

(22)

$${\mu }_{i,j}\in \left\{0,1\right\},i\in M,j\in N$$

(23)

$$M=\left\{j|j=1,2,\dots ,m\right\}\hspace{1em}\hspace{1em}N=\left\{j|j=1,2,\dots ,n\right\}$$

(24)

where Equation (18) is the objective function, cost represents the transportation cost, m is the number of logistics distribution center, n determines the number of goods demand point, nest_j is the demand quantity of demand point j, and dist_i,_j indicates the distance between distribution center i and goods demand point j. When u_i,_j is equal to 1, the goods of demand point j are distributed by distribution point i. Equations (19)–(24) are the constraints. Equation (19) defines that a demand point of goods can only be distributed by a distribution center. Equation (20) indicates that each demand point of goods must have a distribution center to distribute goods. Equation (21) represents the number of goods demand points for a distribution center. Equations (22)–(24) are the relevant definitions.

5.4.2. Analysis of Experimental Results

To verify the performance of the DMQL-CS algorithm in solving the problem of logistics distribution center location, 40 demand points were adopted. The geographical position coordinates and demands are shown in

Table 14. The geographical position coordinates and demands.

No	Coordinates		Demand	No	Coordinates		Demand	No	Coordinates		Demand	No	Coordinates		Demand
	x	y			x	y			x	y			x	y
1	97	28	94	11	91	96	85	21	111	117	92	31	125	66	45
2	100	56	11	12	39	90	54	22	63	42	99	32	169	49	98
3	45	67	50	13	50	101	25	23	67	105	98	33	31	188	31
4	150	197	88	14	67	66	87	24	160	156	88	34	86	42	91
5	105	48	80	15	157	54	66	25	100	125	47	35	90	21	79
6	24	158	29	16	104	35	82	26	35	48	47	36	46	53	47
7	88	61	93	17	169	95	48	27	143	172	34	37	62	30	84
8	55	105	10	18	48	39	78	28	94	56	33	38	163	176	52
9	120	120	18	19	115	61	16	29	57	73	43	39	190	141	10
10	43	105	38	20	154	174	49	30	25	127	100	40	170	30	77

. To make a fair comparison, all experiments were carried out on a P4 Dual-core platform with a 1.75 GHz processor and 4 GB memory, running under the Windows 7.0 operating system. The algorithms were written by MATLAB R2017a. The maximum number of iterations, population size, and the times of running were set to 30,000, 15, and 30, respectively. The probability of foreign eggs being found was = 0.25.

To further verify the efficiency of the DMQL-CS algorithm, the effectiveness of the proposed method was verified by comparing the standard cuckoo search algorithm (CS) [69], the improved cuckoo search algorithm (ICS) [101], a modified chaos-enhanced cuckoo search algorithm (CCS) [68], and the immune genetic algorithm (IGA) [64]. Figure 5 shows the average convergence curve and optimal convergence curve of DMQL-CS algorithm for running 20, 30, and 50 times, respectively, in 40 cities and six distribution centers. The six optimal distribution center points and optimal routes found by DMQL-CS algorithm are also shown in Figure 6. Figure 7 shows the average convergence curve and optimal convergence curve of DMQL-CS algorithm running 20, 30, and 50 times, respectively, in 40 cities and 10 distribution centers.

Table 15. The distribution scheme for six distribution centers in 40 cities.

Distribution Center	Distribution Scope
10	33, 6, 30, 12, 13, 8, 23
21	11, 25, 9
20	4, 27, 38, 24, 39
22	14, 29, 3, 36, 26, 18, 37, 7
1	28, 25, 16, 19, 34, 35
15	31, 17, 32, 40

shows distribution ranges for six distribution centers in 40 cities, and

Table 16. The distribution scheme for 10 distribution centers in 40 cities.

Distribution Center	Distribution Scope
30	6, 33
23	8,12, 13, 10
14	3, 29
18	26, 36, 22, 37
11	-
28	7, 34, 2, 19, 31, 5
21	25, 9
1	16, 35
20	4, 27, 38, 24, 39
15	17, 32, 40

shows distribution ranges for 10 distribution centers in 40 cities.

For the first set of experiments, the DMQL-CS algorithm was run 20, 30, and 50 times independently in 40 cities for six distribution centers. As shown in Figure 6, the average convergence curve can converge at 30 iterations. It indicates that the fitness value decreases rapidly for the logistics distribution center location method based on DMQL-CS algorithm at early stage of the algorithm. The optimal distribution cost and average distribution cost obtained by the DMQL-CS algorithm are 4.5013 × 10⁴ and 4.8060 × 10⁴, respectively, which indicates that DMQL-CS has high solution accuracy for six distribution centers and reduces the cost of logistics distribution. The optimal distribution center points found in Figure 3 are: 10, 21, 20, 22, 1, and 15.

For the second set of experiments, the DMQL-CS algorithm was run 20, 30, and 50 times independently for 40 cities and 10 distribution centers. As shown in Figure 7, the average convergence curve can converge at 20 iterations. The optimal distribution cost and average distribution cost obtained by the DMQL-CS algorithm are 2.9811 × 10⁴ and 3.0157 × 10⁴, respectively, which indicates that DMQL-CS has high solution accuracy not only for six distribution centers but also for 10 distribution centers. The 10 optimal distribution centers and distribution addressing schemes are shown in Table 16 and Figure 7. The optimal distribution center points are: 30, 23, 14, 18, 11, 28, 21, 1, 20, and 15.

Due to limited space, only three comparison algorithms (CS [69], CCS [68], and IGA [64]) are listed in this paper. IGA algorithm introduced crossover and variation strategy into immune algorithm, which improves performance of the immune algorithm. In this experiment, the convergence curves and optimal distribution scheme diagrams of 6 and 10 distribution centers in 40 cities are shown, respectively. Figure 8 and Figure 9 show the convergence curves and optimal distribution centers scheme for the IGA algorithm for 6 and 10 distribution centers. Figure 10 and Figure 11 show the convergence curves and optimal distribution centers scheme for the CS algorithm for 6 and 10 distribution centers. Figure 12 and Figure 13 show the convergence curves and optimal distribution centers scheme for CCS algorithm for 6 and 10 distribution centers. The six optimal distribution centers and distribution addressing schemes for these algorithms are shown in

Table 17. Comparisons between DMQL-CS and other algorithms for six distribution centers.

CS		IGA		CCS
D-C	Distribution Scope	D-C	Distribution Scope	D-C	Distribution Scope
3	30, 12, 10, 13, 29, 14, 36, 26	10	33, 6, 30, 12, 8, 23, 13	23	33, 6, 30, 10, 12, 13, 8, 11
11	8, 23, 6, 33, 25, 21, 9	22	26, 36, 3, 18, 29, 14, 37, 35	21	9, 25
22	18, 37, 7	21	25, 11, 9	22	26, 36, 3, 29, 14, 18, 37
1	34, 35, 28, 2, 5, 16, 19	2	7, 34, 28, 19, 31, 1, 16, 5, 19	16	1, 35, 34, 7, 28, 2, 5, 19
15	31, 32, 17, 40	20	4, 27, 24, 38, 39	15	31, 17, 32, 40
20	27, 4, 38, 24, 39	17	15, 32, 40	20	4, 27, 24, 38, 39

, while the 10 optimal distribution centers and distribution addressing schemes are shown in

Table 18. Comparisons between DMQL-CS and other algorithms for 10 distribution centers.

CS		IGA		CCS
D-C	Distribution Scope	D-C	Distribution Scope	D-C	Distribution Scope
6	30, 33	30	6, 33	6	33
8	10, 12, 13, 23, 19	23	12, 10, 13, 8	10	30, 12, 13, 8
18	3, 26, 36, 22, 37	14	29, 3, 26, 36, 18, 22	23	11
11	-	1	34, 37, 35, 16	14	3, 29
21	25, 9	2	7, 28,5,19,31	22	36, 26, 18, 37
28	14, 7, 34, 2, 19, 31	11	-	25	21, 9
16	5	25	21, 9	7	34, 28, 2,19
1	35	24	17, 20, 38, 39	16	1, 35, 5
20	4, 27, 38, 24, 39	15	17, 32, 40	15	31, 17, 32, 40
15	17, 32, 40	4	-	20	4, 27, 38, 24, 39

.

For the third set of experiments, the CS algorithm was run 20, 30 and 50 times independently for the 40 city and six distribution center example. As shown in Figure 10, both the average convergence curve and the optimal convergence curve can converge at 80 iterations. The optimal distribution cost and average distribution cost obtained by the CS algorithm are 4.9629 × 10⁴ and 6.1392 × 10⁴, respectively. As shown in Figure 11, average convergence curve can converge at 100 iterations for 10 distribution centers. The optimal distribution cost and average distribution cost obtained by the CS algorithm are 3.2435 × 10⁴ and 3.9502 × 10⁴, which indicates that logistics distribution location strategy of CS algorithm is the worst in both the optimal convergence curve and the average convergence curve. Convergence curve of CCS algorithm can converge at 20 iterations in both the optimal convergence curve and the average convergence curve. CCS algorithm is much inferior to DMQL-CS algorithm in solving accuracy for 6 and 10 distribution centers. Although the IGA algorithm can converge, it has a lot of noise for the average convergence curve. The convergence effect of IGA is worse compared with CCS algorithm. The results of standard deviation indicate that DMQL-CS has a better robustness than the other algorithms. The optimal distribution centers and distribution addressing schemes are shown in Table 17 and Table 18. According to Table 17 and Table 18, the optimal distribution center points found by CS algorithm for 6 and 10 distribution centers are (3, 11, 22, 1, 15, 20) and (6, 8, 18, 11, 21, 28, 16, 1, 20, 15). The optimal distribution center points found by IGA algorithm for 6 and 10 distribution centers are (10, 22, 21, 2, 20, 17) and (30, 23, 14, 1, 2, 11, 25, 24, 15, 4). The optimal distribution center points found by CCS algorithm for 6 and 10 distribution centers are (23, 22, 21, 16, 15, 20) and (6, 10, 23, 14, 22, 25, 7, 16, 15, 20).

To further analyze the effectiveness of DMQL-CS algorithm, DMQL-CS was compared with four algorithms: CS [69], CCS [68], ICS [101], and IGA [64]. The comparison results with average fitness value (Mean), the best fitness value (Best), the worst fitness value (Worst), standard deviation (Std), and running time (Time) are shown in

Table 19. Comparisons between DMQL-CS and other algorithms for 6 and 10 distribution centers in 40 cities.

Algorithm	Distribution Points		Algorithms
		Best	Mean	Worst	Std	Time (s)
CS	6	4.9629 × 104	6.1392 × 104	7.9211 ×104	2.4874 × 105	4.5187
	10	3.2435 × 104	3.9502 × 104	4.3961 × 104	3.9872 × 105	4.5530
CCS	6	4.7913 × 104	4.9009 × 104	5.2085 × 104	4.9009 × 105	4.9486
	10	3.1619 × 104	3.3815 × 104	3.4209 × 104	3.3815 × 104	4.8706
IGA	6	5.2032 × 104	5.3008 × 104	5.3814 × 104	5.9226 × 105	4.4255
	10	3.5424 × 104	3.6460 × 104	3.6980 × 104	9.0172 × 105	4.5235
ICS	6	4.5748 × 104	4.6187 × 104	4.6919 × 104	5.8622 × 104	4.7245
	10	3.1034 × 104	3.2197 × 104	3.3113 × 104	8.0172 × 104	4.7811
DMQL-CS	6	4.5013 × 104	4.8060 × 104	4.9253 ×104	1.2763 × 104	4.6255
	10	2.9811 × 104	3.0157 × 104	3.2132 ×104	2.7651 × 104	4.6509

. It can be seen that the average distribution cost of DMQL-CS at six distribution centers is 4.8060 × 10⁴ which is 13,332 lower than CS, and the average distribution cost in 10 distribution centers is 3.0157 × 10⁴, which is 9345 lower than CS. Therefore, DMQL-CS is obviously superior to CS algorithm. Although ICS can provide far better final results for most of the cases, it takes more execution time because of the use of more expensive exploration operation during the initial phases. For 6 and 10 distribution centers, there is not much difference between ICS and DMQL-CS for average distribution cost, but the optimal distribution cost of DMQL-CS is significantly higher than that of ICS algorithm. Meanwhile, from the standard deviation and running time data, it can be known that DMQL-CS has better robustness. The IGA algorithm achieved the worst performance compared with other comparison algorithms, except for the CS algorithm. For the six distribution centers, the average value obtained by IGA algorithm is 5.3008 × 10⁴, which is 4948 more than DMQL-CS. The average value obtained by IGA algorithm is 3.6460 × 10⁴, which is 6330 more than DMQL-CS for 10 distribution centers. CCS algorithm obtains the third best performance for 6 and 10 distribution center, respectively. In summary, the results of DMQL-CS are better than the comparison algorithms in terms of optimal value, worst value, average value, or running time. The reason may be that the Q-learning step size strategy improves the precision of the algorithm. The crossover and mutation operator accelerate the convergence speed of the algorithm. Overall, the selection method of logistics distribution center based on cuckoo search algorithm with Q-Learning and genetic operation has better optimal value compared with the five other algorithms for both 6 and 10 distribution centers, which indicates that the selection strategy based on DMQL-CS has higher solution accuracy and wider range of optimization. Meanwhile, it can be seen in Table 19 that the running time of DMQL-CS is significantly lower than the four other algorithms, and the number of iterations is significantly reduced. In general, DMQL-CS algorithm can select the address of logistics distribution center more quickly and accurately compared with the comparison algorithm. Finally, we can say that our proposed algorithm interestingly outperforms other competitive algorithms in terms of convergence rate and robustness.

6. Conclusions

In this study, we constructed a model of CS with Q-Learning and genetic operators, and then solved the address of logistics distribution center with DMQL-CS algorithm in which adopts Q-Learning scheme to learn the individual optimal step size strategy according to the effect of individual multi-steps. The most appropriate step size control strategy is chosen as a parameter for the current step size evolution of the cuckoo, which increases the adaptability of individual evolution. At the same time, to accelerate the convergence of the algorithm, genetic operators and hybrid operations are added to DMQL-CS algorithm. Crossover and mutation operations expand the search area of the population, and accelerate the convergence of the DMQL-CS algorithm.

To verify the performance of DMQL-CS, DMQL-CS was employed to solve fifteen benchmark test functions and CEC 2013 test suit. The results show that the proposed DMQL-CS algorithm clearly outperforms the standard CS algorithm. Comparing with some improved CS variants and DE variants, we found that DMQL-CS algorithm outperforms the other algorithms on a majority of benchmarks. In addition, the effectiveness of the proposed method was further verified by comparing with CS, ICS, CCS, and IGA for both 6 and 10 distribution centers.

In the future, we will focus our research work on the study of special cases to strengthen the algorithm in more complex conditions. We will determine how to generalize our work to handle combinatorial optimization problems and to extend DMQL-CS optimization algorithms to in the realistic engineering areas and feature selection for machine learning [102].