A Reinforcement Learning-Based Multi-Strategy Differential Evolution Algorithm for Agricultural UAV Path Planning

Abstract

In the realm of precision agriculture, agricultural UAV path planning is challenging because the UAV must avoid obstacles, follow uneven terrain, and satisfy multiple flight constraints simultaneously. Differential evolution (DE) has been widely adopted for this problem because of its simple structure and effective optimization capability. However, existing DE-based methods often become trapped in local optima and cannot effectively balance exploration and exploitation in complex search environments. To address these issues, this paper proposes a reinforcement learning-based multi-strategy differential evolution algorithm, named PPOMSDE. By introducing Proximal Policy Optimization (PPO) to construct a multi-dimensional state pool and an action pool, PPOMSDE enables adaptive strategies for individuals, improving strategy selection during the search process. An independent multi-buffer is adopted to ensure strict data isolation and efficient learning to avoid strategy confusion. In addition, an adaptive triplet mechanism which partitions the population into fitness-based tiers (best, medium, and worst) assigns different control parameters and mutation strategies to individuals with different fitness levels, improving the balance between global exploration and local exploitation. Extensive experiments on the CEC’2014 and CEC’2017 benchmark suites demonstrate the effectiveness of PPOMSDE. The proposed method achieves the lowest average performance ranks of 1.39 on the combined 10-D and 30-D CEC’2014 benchmarks and 1.03 on the 10-D CEC’2017 benchmarks. In agricultural UAV path planning, PPOMSDE generates safer and smoother flight paths while maintaining accurate terrain-following flight, reducing the overall cost by an average of 22.42% compared with ISDE, L-SHADE, SHADE, and ISHACDE.

1. Introduction

UAV has witnessed widespread deployment across a multitude of domains, such as image information acquisition [1], ecological anomaly detection [2], agricultural ground platform navigation [3], and ground cover recognition [4], due to its high flexibility and strong adaptability. Driven by the advancements in modern industry and technological frontiers, computational models in practical UAV applications commonly display characteristics such as high dimensionality, nonlinearity, and complex constraints. Among these problems, UAV path planning is a classic challenging optimization task [5]. UAV path planning aims to find an efficient path from the start point to the destination while avoiding threat regions under physical constraints [6].

Specifically, in agricultural applications, this task is challenging because of complex 3D terrain and strict operational constraints [7]. During practical field operations, agricultural UAVs must safely bypass various hazards in a three-dimensional environment. Meanwhile, they must maintain the overall cost-effectiveness and efficiency of the planned flight trajectory [8]. As agricultural mission profiles become increasingly elaborate and the operational scope broadens, the corresponding search space undergoes an exponential explosion, triggering a substantial escalation in computational overhead [9]. As a result, finding the global optimum in these complex high-dimensional environments remains challenging. This limits the search efficiency and optimization performance of existing algorithms [10].

Traditional methods for agricultural UAV, such as A*, D*, and Rapidly Exploring Random Tree Star (RRT*), show good performance in low-dimensional environments [11]. However, in environments with dense obstacles and complex fitness landscapes, these methods can easily become trapped in local optima. Meanwhile, metaheuristic algorithms, exemplified by DE, Genetic Algorithm (GA), and Particle Swarm Optimization (PSO), mimic natural selection mechanisms or swarm intelligence. They demonstrate strong global search ability and adaptability in solving nonlinear path planning problems with multiple constraints [12]. Among them, the DE algorithm has been extensively studied owing to its uncomplicated design and reliable execution [13].

However, practical agricultural environments are much more complex than idealized optimization scenarios. Varying canopy density, irregular crop distributions, and different obstacles continuously alter the optimization landscape during UAV path planning. These factors produce highly multimodal and non-stationary search spaces, where the balance between exploration and exploitation becomes difficult to maintain. Conventional DE algorithms usually rely on fixed or manually designed mutation strategies. Once the population diversity decreases, they tend to converge prematurely to local optima and cannot adapt their search behavior according to the current evolutionary state. Consequently, their optimization performance deteriorates in complicated agricultural scenarios. These limitations have motivated extensive research on improving the adaptability of DE for UAV path planning. Recent investigations on solving UAV path planning problems based on DE can be summarized into three categories: hybrid mechanisms, parameter adaptation, and reinforcement learning (RL).

Methods based on hybrid mechanisms enhance path search ability by introducing other algorithm frameworks into the mutation strategies of DE. For example, Tu et al. integrate DE into the iterative process of the marine predators algorithm [14]; Huang et al. introduce a challenger mechanism based on DE operators into PSO for path optimization [15]; Yu et al. integrate DE and gray wolf optimizer to harmonize local exploitation and global exploration [16]; and Zhao et al. merge DE and gradient descent into the artificial hummingbird algorithm for route generation [17]. The operator combination and mutation logic of these algorithms are usually based on predefined heuristic rules. Such combinations of heuristic methods can perform well on specific problems. However, when facing extremely complex terrains, the lack of real-time quantitative perception of the population evolutionary state leads to imprecise strategy switching and unstable performance.

Methods based on parameter adaptation use historical information to dynamically tune key control settings. Cao et al. analyze population distribution characteristics and extracts parameter information to guide the optimization process [18]. Zhou et al. adopt a multi-stage parameter adaptation mechanism to capture evolutionary features in different dimensions and apply targeted strategies [19]. Freitas et al. directly generate smooth and differentiable paths using parameterized non-uniform rational B-spline curves [20]. However, these parameter adaptation methods mainly rely on the statistical memory of historical samples. Such simple feedback mechanisms cannot accurately capture fine-grained changes in the fitness distribution, such as skewness and kurtosis. Applying the same parameter adaptation rules to individuals in different states makes it difficult to provide refined support, especially in the later stage of complex optimization. In addition, these methods are typically designed using manually defined heuristic update rules. As a result, their decision process is difficult to generalize to dynamic optimization environments, limiting their ability to adapt to rapidly changing population states.

The third category focuses on intelligent planning frameworks based on RL. Currently, combining RL with evolutionary algorithms has become a popular research direction. Examples include the Q-learning-based multi-strategy cuckoo search algorithm [21], a dendrite net learning multi-objective artificial bee colony algorithm [22] and an online-learning metaheuristic algorithm integrated with a PPO mechanism [23]. For the DE algorithm, researchers have also attempted to introduce RL or hybrid mechanisms to achieve automated algorithm design. Yu et al. use RL to evaluate generated offspring and compare them with their parents, enabling adaptive parameter adjustment [24]. Zhang et al. adopt an RL strategy according to a multi-armed bandit framework to adaptively allocate mutation operators, enhancing solution precision while safeguarding population diversity [25]. Ding et al. proposed a distributed PPO framework for the joint adaptation of mutation strategies and control parameters [26]. In the domain of path planning, Reda et al. utilized Deep Q-Networks (DQN) to dynamically switch DE operators, combining this with external cubic spline interpolation to generate smooth trajectories [27]. However, most existing RL-assisted frameworks mainly regulate population-level information or act directly on specific problem instances. As a result, the evolutionary trajectory of each individual is not clearly captured. This limitation makes it difficult to coordinate population diversity with convergence speed effectively. Furthermore, they often adopt fully coupled online-learning paradigms that lead to unstable training, and lack the fine-grained spatial perception required for highly complex 3D environments.

This paper presents an RL-based multi-strategy differential evolution algorithm, named PPOMSDE. PPOMSDE utilizes PPO from deep reinforcement learning (DRL) to construct a multi-dimensional state pool by extracting the spatial positions and fitness values of the population. This construction effectively resolves the problem of insufficient complex state perception and enhances the capability of the proposed model to perceive the current optimization environment. Multiple improvement strategies are designed to enhance the performance of optimization. Moreover, an independent experience buffer for each individual is introduced to guarantee data isolation. Building upon these foundations, the PPOMSDE algorithm uses a triplet mechanism to complement the automated algorithm design within reinforcement learning, achieving both global dynamic perturbation and localized refinement throughout the optimization procedure. Extensive experiments on benchmarks and agricultural UAV path planning validate PPOMSDE’s higher solution quality and robustness. The main contributions of this work are encapsulated below:

• The PPO algorithm from RL is introduced into DE, enabling a framework for the automated design of algorithmic mechanics. This approach exploits the policy-learning capability of PPO, endowing the DE algorithm with the capacity to autonomously select optimal mutation operators in response to real-time evolutionary states.
• Multi-dimensional state and action pools are designed. Multi-dimensional features, including fitness skewness, kurtosis, and local neighborhood distribution, are introduced to enhance state perception. Additionally, an external archive mechanism is utilized to improve the effectiveness of mutation actions.
• PPOMSDE is compared with seven DE variants and two basic evolutionary algorithms benchmarked against the CEC‘2014, CEC’2017, and agricultural UAV path planning evaluation suites to demonstrate its operational excellence in practical settings.

The remainder of this paper is organized as follows: Section 2 details the problem of agricultural UAV path planning, along with overviews of DE and PPO. Section 3 introduces the proposed PPOMSDE algorithm. Section 4 provides a comprehensive evaluation of PPOMSDE on the CEC’2014 and CEC’2017 benchmark suites, as well as in diverse agricultural UAV path-planning scenarios. Finally, Section 5 concludes the paper.

2. Preliminaries

2.1. Problem Description

2.1.1. Route Representation

In UAV path planning, the spatial locations of every flight node on path are defined as $P_{i,j}=\left(x_{i,j},y_{i,j},z_{i,j}\right)$. For a full flight path containing n path points, the path $P_i$ is mathematically formulated as:

$$P_i=\left[P_{i,1},P_{i,2},\cdots ,P_{i,n}\right]$$

(1)

Within the evolutionary framework, every candidate solution $X_i$ corresponds to an entire flight route. During the initialization phase, these candidate solutions are generated entirely at random within the predefined operational boundaries. Such infeasible paths are naturally penalized through the cost evaluation mechanism during the evolutionary process. Throughout this study, the UAV trajectory is parameterized in a reduced-dimensional form. By using equal intervals along the x-axis, monotonicity in the x-direction is ensured, thereby simplifying the 3D path search to a two-dimensional optimization problem.

Under this encoding scheme, $X_i$ has a clear physical meaning: the first half controls the horizontal shifting of routing, while the second half governs the climbing profile. The representation is defined below:

$$\left\{\begin{array}{c}{X}_i=[y_{i,1},y_{i,2},\cdots ,y_{i,n},z_{i,1},z_{i,2},\cdots ,z_{i,n}]\hfill \\ P_i=\left[\left(x_{i,1},y_{i,1},z_{i,1}\right),\left(x_{i,2},y_{i,2},z_{i,2}\right),\cdots ,\left(x_{i,n},y_{i,n},z_{i,n}\right)\right]\hfill \end{array}\right.$$

(2)

Although the optimization is performed in a two-dimensional search space, each candidate solution is reconstructed into a three-dimensional path during fitness evaluation, ensuring that all constraints are evaluated in the original 3D environment.

2.1.2. Objective Function

Agricultural UAV path planning seeks to identify a minimum-cost route within a given region. Considering the UAV physical limits and complex terrain, restrictions on vertical height limits, minimal interval distance, and navigational orientation are imposed. A composite objective function is adopted to measure the computational performance. Specifically, it includes path length penalty, threat cost penalty, height cost penalty, and path smoothness penalty [28]. The path length cost penalty is defined by:

$$F_1\left(X_i\right)={\displaystyle {\sum }_{j=1}^{n-1}}‖P_{i,j}-P_{i,j+1}‖$$

(3)

where $P_{i,j+1}$ is the ${j+1}^{th}$ waypoint of the $i^{th}$ path, and $‖P_{i,j}-P_{i,j+1}‖$ computes the direct spatial gap separating consecutive coordinates. To evaluate the navigational risk, the hazard cost is established by the following expression:

$$F_2\left(X_i\right)={\displaystyle {\sum }_j^{n-1}}{\displaystyle {\sum }_g^G}{\psi }_g\left(\overrightarrow{P_{i,j}{P}_{i,j+1}}\right)$$

(4)

where $G$ is the total count of threat regions, $\overrightarrow{P_{i,j}{P}_{i,j+1}}$ is the vector two adjacent navigational coordinates $P_{i,j}$ and $P_{i,j+1}$, and ${\psi }_g$ denotes the safety evaluation function, which computes the cost based on the direct spatial separation between the two path points. The formulation is given as follows:

$${\psi }_g\left(\overrightarrow{P_{i,j}{P}_{i,j+1}}\right)=\left\{\begin{array}{c} 0, if {\mathcal{d}}_g>S+d+{\mathcal{R}}_g; \\ {\varsigma }^1, if {\mathcal{d}}_g\le d+{\mathcal{R}}_g; \\ \left(\mathcal{d}+{\mathcal{R}}_g+\mathcal{S}\right)-{\mathcal{d}}_g, if d+{\mathcal{R}}_g<{\mathcal{d}}_g\le d+{\mathcal{R}}_g+S.\end{array}\right.$$

(5)

where ${\mathcal{R}}_g$ is the radius of the collision zone of the $g^{th}$ threat, $\mathcal{S}$ is the radial distance from the collision center to the boundary of the risk zone, and ${\varsigma }^1$ is the collision penalty. For the UAV, the collision zone is defined as a no-fly region that must be strictly avoided, while the risk zone represents a buffer region indicating potential threat exposure. $\mathcal{d}$ is the UAV diameter, which is used to evaluate the positional dynamics between the UAV and the threat margins.

To ensure flight safety and maintain an ideal altitude, three uniformly distributed sampling points are generated along each flight segment j to evaluate terrain variations. The altitude cost of each segment is calculated as the average penalty of these three sampling points. In this study, the use of three sampling points represents a compromise between computational efficiency and evaluation accuracy. The sampling density is a configurable parameter rather than an inherent limitation of the proposed method. For terrains with sharper elevation variations, additional sampling points can be introduced to improve terrain-clearance assessment without changing the PPOMSDE optimization framework. The path height cost is established by:

$$F_3\left(X_i\right)={\displaystyle {\sum }_{j=1}^{n-1}}\left({\displaystyle \frac{1}{3}}{\displaystyle {\sum }_{k=1}^3}{C}_h\left(h_{j,k}\right)\right)$$

(6)

where $C_h\left(h\right)$ is the penalty function, and $h_{j,k}$ represents the actual relative altitude above ground at the $k^{th}$ sampling point of the $j^{th}$ flight segment:

$$h_{j,k}=z_{j,k}-E\left(x_{j,k},y_{j,k}\right)$$

(7)

where $E\left(x,y\right)$ denotes the actual elevation from the underlying Digital Elevation Model (DEM) at that coordinate. To strictly penalize flight operations outside of the safe altitude range, the penalty function $C_h\left(h\right)$ is defined in a dynamic gradient form:

$$C_h\left(h\right)=\left\{\begin{array}{c}{\varsigma }^2+\xi ·\left(h_{min}-h\right), if h<h_{min};\\ {\varsigma }^2+\xi ·\left(h-h_{max}\right), if h>h_{max};\\ \left|h-h_{target}\right|, otherwise.\end{array}\right.$$

(8)

where $h_{min}$ and $h_{max}$ represent the minimum safe altitude and the maximum effective altitude, respectively, while $h_{target}$ denotes the ideal relative altitude. ${\varsigma }^2$ is the base penalty for boundary violations, and $\xi$ is the gradient penalty coefficient. This mechanism guides the UAV to maintain a stable mean altitude inside the allowable limits. The path smoothing cost is calculated using the following expression:

$$F_4\left(X_i\right)={\displaystyle {\sum }_{j=1}^{n-2}}\left|{\theta }_{i,j}^{turn}\right|+\left|{\theta }_{i,j}^{climb}\right|$$

(9)

where ${\theta }_{i,j}^{turn}$ and ${\theta }_{i,j}^{climb}$ are the horizontal heading change and vertical climb angle of the UAV, respectively. The formulations are given as follows:

$${\theta }_{i,j}^{turn}=arctan\left({\displaystyle \frac{‖{\overrightarrow{P_{i,j}{P}_{i,j+1}}}^{,}\times {\overrightarrow{P_{i,j+1}{P}_{i,j+2}}}^{,}‖}{{\overrightarrow{P_{i,j}{P}_{i,j+1}}}^{,}\cdot {\overrightarrow{P_{i,j+1}{P}_{i,j+2}}}^{,}}}\right)$$

(10)

$${\theta }_{i,j}^{climb}=\mathit{arctan}\left[{\displaystyle \frac{z_{i,j+2}-z_{i,j+1}}{\sqrt{{\left(x_{i,j+2}-x_{i,j+1}\right)}^2+{\left(y_{i,j+2}-y_{i,j+1}\right)}^2}}}\right]-\mathit{arctan}\left[{\displaystyle \frac{z_{i,j+1}-z_{i,j}}{\sqrt{{\left(x_{i,j+1}-x_{i,j}\right)}^2+{\left(y_{i,j+1}-y_{i,j}\right)}^2}}}\right]$$

(11)

where ${\overrightarrow{P_{i,j}{P}_{i,j+1}}}^{,}$ and ${\overrightarrow{P_{i,j+1}{P}_{i,j+2}}}^{,}$ denote the projections of two consecutive flight segments onto the horizontal x-y plane, respectively. Thus, by combining the four costs above, the total cost is defined as:

$$F\left(X_i\right)={\displaystyle {\sum }_{\varpi =1}^4}{a}_{\varpi }{F}_{\varpi }\left(X_i\right)$$

(12)

where $a_{\varpi }$ is the weighting coefficient for the ${\varpi }^{th}$ cost terms.

2.2. Differential Evolution

The DE optimizer applies a fixed mutation strategy in each iteration to optimize the problem. Generally, the framework of a standard DE comprises a sequence of four fundamental operations: initialization, followed by iterative cycles of mutation, crossover, and selection [29]. These procedures are elaborated as follows:

(1)

Initialization:

To launch the optimization process, a population is initially generated via a uniform distribution, ensuring that candidate solutions are scattered comprehensively across the predefined search boundaries. Mathematically, the $i^{th}$ candidate solution $X_i$ takes the form of a vector:

$$X_i=\left[X_{i,1},X_{i,2},\dots ,X_{i,D}\right], i=1,2,\dots ,N$$

(13)

where $D$ signifies the problem dimension of the candidate. The specific initialization method is as follows:

$$X_{i,j}=X_{min,j}+rand(0, 1)\cdot (X_{max,j}-X_{min,j})$$

(14)

where $X_{i,j}$ is the $j^{th}$ dimension of $X_i$, $X_{max,j}$ and $X_{min,j}$ are the supremum and infimum limits of the $j^{th}$ positional parameter, and $rand(0, 1)$ signifies a random value drawn uniformly from within $\left[0, 1\right]$.

(2)

Mutation:

In this step, a new mutant vector $V$ is constructed by applying scaled differences between selected individuals to a specific target vector, as dictated by the chosen evolutionary strategy. Frequently employed mutation schemes include the following:

• DE/rand/1

$$V_{i,t}=X_{r_1,t}+F\cdot (X_{r_2,t}-X_{r_3,t})$$

(15)

• DE/best/1

$$V_{i,t}=X_{best,t}+F\cdot (X_{r_1,t}-X_{r_2,t})$$

(16)

• DE/best/2

$$V_{i,t}=X_{best,t}+F\cdot (X_{r_1,t}-X_{r_2,t})+F\cdot (X_{r_3,t}-X_{r_4,t})$$

(17)

• DE/current-to-best/1

$$V_{i,t}=X_{i,t}+F\cdot \left({X_{best,t}-X}_{i,t}\right)+F\cdot (X_{r_1,t}-X_{r_2,t})$$

(18)

• DE/rand-to-best/1

$$V_{i,t}=X_{r_1,t}+F\cdot (X_{best,t}-X_{r_1,t})+F\cdot (X_{r_2,t}-X_{r_3,t})$$

(19)

• DE/current-to-rand/1

$$V_{i,t}=X_{i,t}+F\cdot \left({X_{r_1,t}-X}_{i,t}\right)+F\cdot (X_{r_2,t}-X_{r_3,t})$$

(20)

where $V_{i,t}$ signifies the mutant vector of the $i^{th}$ candidate during generation $t$, $X_{i,t}$ is the $i^{th}$ candidate during generation $t$, and $X_{best,t}$ is the superior candidate during generation $t$. $X_{r_1,t}$, $X_{r_2,t}$, $X_{r_3,t}$, $X_{r_4,t}$ are distinct individuals chosen at random within the current iteration. The parameter $F$ acts as a scaling coefficient, modulating the influence of the differential vectors into the donor generation.

(3)

Crossover:

The trial vector $U$ is synthesized by swapping elements between the corresponding present candidate $X_{i,t}$ and perturbed vector $V_{i,t}$. The crossover procedure is expressed as:

$$U_{j,i,t}=\left\{\begin{array}{c}{V}_{j,i,t}, if rand(0, 1)\le CR or j=j_{rand}\hfill \\ X_{j,i,t}, otherwise\hfill \end{array}\right.$$

(21)

where $U_{j,i,t}$ denotes the $j^{th}$ positional parameter of the $i^{th}$ trial solution during generation $t$, $X_{j,i,t}$ denotes the $j^{th}$ dimension of the $i^{th}$ individual, and $V_{j,i,t}$ is the $i^{th}$ element belonging to the mutant vector. Meanwhile, $j_{rand}$ stands as a stochastic integer chosen from $\left[1, D\right]$, incorporated to guarantee the transfer of a minimum of one component from the mutant solution. The rate at which variables are accepted from the mutated entity is determined by $CR$, a stochastic control parameter defined within $\left[0,1\right]$. After binomial crossover, each trial vector component is clipped to its corresponding lower or upper bound when a boundary violation occurs.

(4)

Selection:

The objective scores of both the target entity and the recently produced trial vector undergo evaluation, whereupon the superior performer is retained for the succeeding iteration. According to the following criterion:

$$X_{i,t+1}=\left\{\begin{array}{c}{U}_{i,t}, if f\left(U_{i,t}\right)<f\left(X_{i,t}\right)\hfill \\ { X}_{i,t}, otherwise\hfill \end{array}\right.$$

(22)

where $X_{i,t+1}$ denotes the $i^{th}$ candidate at the ${t+1}^{th}$ iteration, and $U_{i,t}$ denotes the trial vector of the $i^{th}$ candidate in the $t^{th}$ iteration.

2.3. Proximal Policy Optimization

RL has been integrated into global optimization problems and has demonstrated strong potential for automatically composing metaheuristic algorithms [30]. PPO is a policy-based DRL method. The agent continuously improves its decision-making through policy learning. As a result, the agent gradually converges to the optimal solution in complex optimization tasks. Specifically, PPO combines the advantages of the stochastic policy and the value baseline. The policy function generates actions and traverses the environment, while the value function quantifies the cumulative reward expectation under the current policy. In this framework, the actor network selects actions that maximize the expected return, and the critic network estimates the corresponding state value to provide guidance for policy updates.

Within the Markov decision process (MDP) framework [31], the PPO agent learns through continuous interaction with the environment. At each algorithmic iteration, state, action, reward, and the corresponding log probability of the action are stored in an experience buffer. This buffer records complete interaction trajectories and provides data for subsequent batch updates, thereby improving sample efficiency and reducing training variance.

During policy optimization, PPO introduces a clipping mechanism to constrain the update magnitude, preventing excessive policy changes and improving training stability and convergence. All sampled data are extracted from the experience buffer and aggregated to compute the Return and Advantage, which are used to construct batch training samples, for multiple iterations of updating the policy and value networks.

Finally, the model parameters are updated by optimizing the overall loss function $L\left(\theta \right)$ of PPO, which is prescribed below:

$$L\left(\theta \right)=L^{clip}\left(\theta \right)-{\eta }_1{L}^{vf}\left(\theta \right)+{\eta }_{2}S\left[{\pi }_{\theta }\right]$$

(23)

where $L^{clip}\left(\theta \right)$ denotes the surrogate clipping objective, $L^{vf}\left(\theta \right)$ evaluates the state-value estimation error, and $S\left[{\pi }_{\theta }\right]$ serves as the entropy regularization term. ${\eta }_1$ and ${\eta }_2$ are hyperparameters that regulate the significance of individual loss components. The clipped policy loss $L^{clip}\left(\theta \right)$ is formulated as:

$$L^{clip}\left(\theta \right)=E_t\left[min\left({\rho }_t\right(\theta )A_t,clip({\rho }_t\left(\theta \right),1-\mathcal{E},1+\mathcal{E}\left)A_t\right)\right]$$

(24)

where ${\rho }_t\left(\theta \right)$ is the policy update ratio, $A_t$ is the advantage function, and $\mathcal{E}$ represents a hyperparameter designed to bound the magnitude of the policy updates. The policy update ratio ${\rho }_t\left(\theta \right)$ is defined as follows:

$${\rho }_t(\theta )={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{\theta \_old}\left(a_t\right|s_t)}}$$

(25)

where ${\pi }_{\theta }\left(a_t\right|s_t)$ and ${\pi }_{\theta \_old}\left(a_t\right|s_t)$ signify the conditional probabilities of executing action $a_t$ at state $s_t$ for the new and old policies, respectively. The advantage function $A_t$ is defined as follows:

$$A_t={\mathcal{Q}}_{{\pi }_{\theta \_old}}(s_t,a_t)-{\mathcal{V}}_{{\pi }_{\theta \_old}}(s_t)$$

(26)

where ${\mathcal{Q}}_{{\pi }_{\theta \_old}}(s_t,a_t)$ represents the value estimation for executing action $a_t$ in state $s_t$, and ${\mathcal{V}}_{{\pi }_{\theta \_old}}\left(s_t\right)$ is the value at $s_t$. Consequently, the loss function for the value network $L^{vf}(\theta )$, is formulated as follows:

$$L^{vf}(\theta )=E\left[{\left(\mathcal{V}\right(s)-{\mathcal{V}}_{target})}^2\right]$$

(27)

where $\mathcal{V}\left(s\right)$ is the state value function, and ${\mathcal{V}}_{target}$ is the target value. After computing the total loss function $L\left(\theta \right)$, the parameters of the policy and value networks are updated via backpropagation (BP). This alternating process of experience collection and policy refinement iterates until the maximum training iterations are fulfilled, after which the final optimized network architecture is retained for deployment.

3. The Proposed PPOMSDE

This section introduces the overall framework and core components of the proposed algorithm. Several RL algorithms, such as DQN [32], Advantage Actor–Critic (A2C) [33], and Soft Actor–Critic (SAC) [34], have been successfully applied to sequential decision-making problems. However, their characteristics differ in terms of policy optimization and training stability. DQN is a value-based method designed for discrete action spaces, but its performance may deteriorate in highly non-stationary environments due to Q-value overestimation and unstable learning. A2C improves learning efficiency through an actor–critic architecture, but unconstrained policy updates may lead to oscillatory convergence. SAC is mainly designed for continuous action spaces and introduces entropy regularization to improve exploration, resulting in additional computational overhead.

In contrast, PPO employs a clipped surrogate objective to restrict policy updates, thereby improving training stability while maintaining implementation simplicity. Since the proposed framework formulates mutation strategy selection as a discrete decision problem under continuously changing population states, stable policy optimization is more important than aggressive policy updates. Therefore, PPO provides a suitable balance between stability, learning efficiency, and computational cost, making it an appropriate choice for adaptive mutation strategy selection in the proposed PPOMSDE framework.

Figure 1a presents the overall framework of PPOMSDE. The proposed algorithm starts with the initialization of the neural network architectures and control parameters. During the training phase, the framework iteratively performs trajectory sampling to collect interaction data. These data are utilized to compute the advantage function and the expected return, which subsequently drive the gradient-based updates for the actor and critic networks. Following the update, the experience buffers are cleared to ensure fresh data collection for the next epoch. After the algorithm model has been trained, it can be used in the testing phase to solve actual optimization problems.

Figure 1. The overall framework of PPOMSDE. (a) presents the overall flowchart; (b) details the data sample mechanism of training phase; (c) denotes the test phase.

Figure 1b details the data sample mechanism, which facilitates the interaction between the RL agent and the population evolution process. After initializing the population, a comprehensive state $s_t^i$ is extracted from the states features for each individual $X_{i,t}$ at generation $t$. The critic network processes the state to output a value estimate, representing the expected cumulative reward. Simultaneously, the actor network maps the state to a probability distribution ${\pi }_{\mathrm{t}}^{\mathrm{i}}$ over the actions pool, and outputs a specific action $a_t^i$. The predefined action pool is composed of multiple DE variation operators, and parameter adaptation is also introduced to enhance the performance of actions. Each individual generates a mutant vector when performing its action in the environment module, and then obtains a new population through crossover and selection operators. This process will also return rewards $r_t^i$ and terminal indicator $d_t^i$. All interaction results ${[s}_t^i, a_t^i, r_t^i,{\pi }_{\mathrm{t}}^{\mathrm{i}},{\mathcal{V}}_{\mathrm{t}}^{\mathrm{i}}\left(\mathrm{s}\right), d_t^i]$ are stored in a multi-buffer system, where separate buffers correspond to different individuals, ensuring a diverse set of trajectories for PPOMSDE optimization. The process will continuously cycle and store trajectories, until the maximum number of iterations $T$ is reached. The population will be initialized and the above loop will be repeated. After a predefined number of episodes, the data collection phase will proceed to the network optimization phase.

As shown in Figure 1c, the test phase begins with population initialization, establishing the starting candidate solutions in the search space. At each generation $t$, a state vector $s_t^i$ is extracted for the $i^{th}$ individual through the states features module. The pre-trained actor network processes the state $s_t^i$ to select the most suitable strategy from the actions pool. The actions in the action pool during the testing phase are jointly influenced by the triplet mechanism and parameter adaptation. In the environment module, individuals generate mutant vectors by using the selected actions. Through crossover and selection operators, a new population is generated for the next generation. Once the condition $t<T$ is satisfied, the test process is terminated and the solution of the current optimization problem is output. The main modules of the proposed algorithm are explained as follows.

3.1. Parameter Adaptation Scheme

Based on the effects of the diverse settings of $F$ and $CR$ on the exploitation and exploration of the DE algorithm, PPOMSDE builds on the parameter adaptation mechanism [35]. It restricts the generation range of $CR$ at different stages to balance the progress of exploitation and exploration. The specific procedure proceeds as:

$$F_i=randc(M_{F,R_i}^t,{\delta }_F^2)$$

(28)

$${CR}_i=randn(M_{CR,R_i}^t,{\delta }_{CR}^2)$$

(29)

$${CR}_i=\left\{\begin{array}{c}min\left(max\right({CR}_i, 0.6), 1) if t<⟂\hfill \\ min\left(max\right({CR}_i, 0), 1) otherwise\hfill \end{array}\right.$$

(30)

where $F_{i,t}$ and ${CR}_{i,t}$ denote the mutation scaling factor and crossover rate generated for the $i^{th}$ individual at the current iteration $t$. $M_{F,R_i}^t$ and $M_{CR,R_i}^t$ denote values chosen randomly from the historical memory pools $M_F$ and $M_{CR}$. Both repositories share a uniform capacity $H$ and are designated to record the historical average parameters ($M_F^t$ and $M_{CR}^t$). $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{c}$ and $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{n}$ denote random variables sampled from the Cauchy and normal distributions, respectively. The threshold $⟂$ separates the two evolutionary stages of $CR$. ${\delta }_F^2$ and ${\delta }_{CR}^2$ are the standard deviations.

During the crossover process, $CR$ determines the proportion of genes that the offspring inherits from the mutant individual. In the early stage, $CR$ is restricted within the range of 0.6 to 1. This means that the offspring inherits more features from the mutant individual. It helps the algorithm perform a wide search in the optimization landscape and averts the phenomenon of premature convergence. In the later stage, $CR$ is not restricted. This allows the offspring to retain more advantageous genes from the parent. As a result, the algorithm performs a more refined local search around promising regions and improves convergence accuracy.

In this study, the generation range of F is not as restricted as that of CR. Instead, based on the experimental results reported in [36], the parameters are set as ${\delta }_F^2=0.01$ and ${\delta }_{CR}^2=0.1$. These values have been proven to be effective with support from extensive experimental data. In addition, more constraints are introduced later to match different values and roles of $F$ under different individual positions.

To effectively utilize fitness deviation, the hierarchical weighting mechanism proposed in hierarchically controlled differential evolution [37] is adopted. According to the following expression, the weight of each individual is calculated based on its normalized deviation:

$$w_{k,t}={\displaystyle \frac{∆f\left(X_{k,t}\right)}{{\sum }_{i=1}^N\left|f\left(X_{i,t}\right)-f\left({\overline{X}}_t\right)\right|}}$$

(31)

where $w_{k,t}$ signifies the weight assigned to the $k^{th}$ individual at generation $t$. $∆f\left(X_{k,t}\right)$ corresponds to the objective value of the $k^{th}$ individual, and $f\left({\overline{X}}_t\right)$ is the average objective value of the current population. The specific calculation is expressed as:

$$\left\{\begin{array}{c}∆f\left(X_{k,t}\right)=\left|f\left(X_{k,t}\right)-f({\overline{X}}_t)\right|\hfill \\ f\left({\overline{X}}_t\right)={\displaystyle \frac{1}{N}}{\sum }_{i=1}^{N}f\left(X_{i,t}\right) \end{array}\right.$$

(32)

The parameter adaptation scheme dictates that solutions outperforming the mean fitness receive elevated weighting factors. Conversely, individuals exhibiting minor fitness deviations are assigned diminished importance. All weights are strictly normalized to 1. Therefore, better-performing members have a greater likelihood of being chosen to contribute to the next evolutionary cycle. The detailed generation process is as follows:

$$\left\{\begin{array}{c}{mean}_{WL}(S_F)={\displaystyle \frac{{\sum }_{k=1}^{\left|S_F\right|}{w}_k\cdot {S_F}^2\left(k\right)}{{\sum }_{k=1}^{\left|S_F\right|}{w}_k\cdot S_F\left(k\right)}}\hfill \\ \hfill M_{F,R_i}^{t+1}=\left\{\begin{array}{c}{mean}_{WL}\left(S_F\right) if{ S}_F\ne \varnothing \hfill \\ M_{F,R_i}^t otherwise\end{array}\right.\hfill \end{array}\right.$$

(33)

$$\left\{\begin{array}{c}{mean}_{WL}(S_{CR})={\displaystyle \frac{{\sum }_{k=1}^{\left|S_{CR}\right|}{w}_k\cdot {S_{CR}}^2\left(k\right)}{{\sum }_{k=1}^{\left|S_F\right|}{w}_k\cdot S_{CR}\left(k\right)}}\hfill \\ \hfill M_{CR,R_i}^{t+1}=\left\{\begin{array}{c}{mean}_{WL}\left(S_{CR}\right) if{ S}_{CR}\ne \varnothing \\ M_{CR,R_i}^t otherwise\hfill \end{array}\right.\hfill \end{array}\right.$$

(34)

where ${\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}_{\mathrm{W}\mathrm{L}}$ denotes the weighted Lehmer mean. $S_F^t$ and $S_{CR}^t$ represent the sets of successful scaling factors and crossover rates recorded in the current generation. $\left|S_F^t\right|$ and $\left|S_{CR}^t\right|$ indicate the number of elements within these successful sets, while $S_F^t\left(k\right)$ and $S_{CR}^t\left(k\right)$ correspond to the $k^{th}$ successful parameter value. Lastly, $M_{F,R_i}^{t+1}$ and $M_{CR,R_i}^{t+1}$ denote the updated historical memory values for the $t+1$ step.

3.2. State Representation

The state features formulated in this research are classified into three primary categories: The first category aims to describe the global distribution of the population and the fitness distribution. It captures the dynamic differences during episodes of premature convergence or subsequent evasion of sub-optimal traps. It reflects the global diversity across the swarm. The second category focuses on the search dynamics of individuals and their relative positions in the population. It is used to evaluate the effectiveness of individual improvement. The third category describes the overall evolutionary progress of the algorithm or the duration of stagnation. These three types of features work together. They provide a basis for strategy selection under different evolutionary states. The state composition is as follows:

$$s_t^i=\left[f_1,{ f}_2,\dots , f_{11}\right]$$

(35)

where $s_t^i$ denotes the state of the $i^{th}$ individual at the $t^{th}$ iteration. $f_1, f_2,\dots ,f_{11}$ represent the 11 features designed in this study, which collectively constitute the state vector. The specific feature configurations and their associated parameters are detailed in Table 1 and Table 2. To provide comprehensive state perception, the proposed features are organized into three complementary categories: population distribution and fitness landscape ($f_1$, $f_2$, $f_6$, $f_7$, $f_8$), individual evolutionary status ($f_3$, $f_9$, $f_{10}$, $f_{11}$), and evolution progress and stagnation ($f_4$, $f_5$). These three categories describe the optimization process from the perspectives of the global population, individual behavior, and temporal evolution, enabling the PPO agent to perceive the current search state comprehensively and select appropriate mutation strategies.

Table 1. Designed state feature vector.

Features	Mathematical Expression	Description
$f_1$: global diversity	$\left\{\begin{array}{c}{Ef}_t={\displaystyle \frac{1}{N}}\times {\sum }_{i=1}^N{\mathbb{L}}_{i,t}\hfill \\ {\mathbb{L}}_{i,t}={\displaystyle \frac{1}{N-1}}\times {\sum }_{j\ne i}^N‖X_{j,t}-X_{i,t}‖ \hfill \end{array}\right.$	${Ef}_t$ is the population dispersion within the search space. ${\mathbb{L}}_{i,t}$ is the aggregation degree of an individual relative to the population.
$f_2$: local diversity	${\mathbb{L}}_{i,t}^{near}={\displaystyle \frac{1}{C}}\times {\sum }_{c=1}^C‖X_{c,t}^i-X_{i,t}‖$	${\mathbb{L}}_{i,t}^{near}$ is the local density of the neighborhood.
$f_3$: relative improvement	$\left\{\begin{array}{c}{∆FI}_{i,t}={FI}_{i,t}-{FI}_{i,t-1}\hfill \\ {FI}_{i,t}={\displaystyle \frac{f\left(X_{i,0}\right)-f\left(X_{i,t}\right)}{f\left(X_{i,0}\right)}}\hfill \end{array}\right.$	${∆FI}_{i,t}$ is the change in fitness quality of the current individual relative to the previous generation. ${FI}_{i,t}$ is the fitness quality relative to the initial population.
$f_4$: stop time	${\mathbb{T}}_{i,t}=\left\{\begin{array}{c}{\mathbb{T}}_{i,t-1}+1 if X_{i,t}=X_{i,t-1}\\ 0 otherwise \end{array}\right.$	${\mathbb{T}}_{i,t}$ tracks the consecutive iterations where the candidate solution remains stagnant.
${\mathrm{f}}_5$: search stage	${\mathbb{S}}_t=t/T$	${\mathbb{S}}_t$ is the evolutionary stage of the intelligent agent.
$f_6$: standard deviation	$std\left(f_t\right)=\sqrt{{\displaystyle \frac{1}{N-1}}{\sum }_{i=1}^N{\left(f\left(X_{i,t}\right)-f\left({\overline{X}}_t\right)\right)}^2}$	$std\left(f_t\right)$ captures the statistical dispersion of objective values across the population, as well as the degree of variation in the fitness landscape.
$f_7$: skewness of fitness	${\varrho }_1^t=E\left\{{\left[{\displaystyle \frac{\left(f\left(X_{i,t}\right)-f\left({\overline{X}}_t\right)\right)}{\left(std\left(f\right)\right)}}\right]}^3\right\}$	${\varrho }_1^t$ is the skewness, represents the measurement of fitness landscape asymmetry.
$f_8$: kurtosis of fitness	${\varrho }_2^t={\displaystyle \frac{E\left[{\left(f\left(X_{i,t}\right)-f\left({\overline{X}}_t\right)\right)}^4\right]}{\left({E\left[{\left(f\left(X_{i,t}\right)-f\left({\overline{X}}_t\right)\right)}^2\right]}^2\right)}}$	${\varrho }_2^t$ is the kurtosis, represents the deviation of the fitness distribution relative to a Gaussian baseline.
$f_9$: fitness ${\mathrm{q}}_{1,\mathrm{t}}$ deviation	${\mathbb{Q}}_{i,t}^1=f\left(X_{i,t}\right)-q_{1,t}$	${\mathbb{Q}}_{i,t}^1$ is the fitness discrepancy between the candidate and the lower quartile.
$f_{10}$: fitness ${\mathrm{q}}_{2,\mathrm{t}}$ deviation	${\mathbb{Q}}_{i,t}^2=f\left(X_{i,t}\right)-q_{2,t}$	${\mathbb{Q}}_{i,t}^2$ is the fitness discrepancy between the candidate and the median.
$f_{11}$: fitness ${\mathrm{q}}_{3,\mathrm{t}}$ deviation	${\mathbb{Q}}_{i,t}^3=f\left(X_{i,t}\right)-q_{3,t}$	${\mathbb{Q}}_{i,t}^3$ is the fitness discrepancy between the candidate and the upper quartile.

Table 2. Symbols utilized to formulate the characteristics presented in Table 1.

Symbol	Description
$N$	Total population size.
$C$	The quantity of neighbors that are closest to a given individual.
$X_{k,t}^i$	The $k^{th}$ individual closest to the $i^{th}$ individual.
$X_{i,0}$	The original location of the $i^{th}$ individual.
$t$	Present iteration count.
$T$	Maximum allowable iterations.
$q_{1,t}$	Lower quartile of population fitness at $t^{th}$ generation.
$q_{2,t}$	Median fitness value of the swarm at $t^{th}$ generation.
$q_{3,t}$	Upper quartile of population fitness at $t^{th}$ generation.

The first category describes the global characteristics of the population and the fitness landscape. Global diversity $f_1$ and local diversity $f_2$ indicate whether the population is widely exploring the search space or gradually converging. Standard deviation $f_6$, skewness $f_7$, and kurtosis $f_8$ characterize the statistical properties of the fitness distribution from different perspectives. Together, these features describe the dispersion, asymmetry, and concentration of the population, allowing the agent to better estimate the current search landscape and adjust the exploration intensity accordingly.

The second category focuses on the evolutionary status of each individual. Relative improvement $f_3$ measures the optimization gain obtained after the previous action and provides direct feedback on the effectiveness of the selected mutation strategy. The quartile-based fitness deviations ($f_9$, $f_{10}$ $f_{11}$) describe the relative position of an individual within the current population. They enable the agent to distinguish high-quality, average, and inferior individuals, thereby supporting differentiated strategy selection for individuals with different evolutionary states.

The third category provides temporal information about the optimization process. Stop time $f_4$ records the number of consecutive generations during which an individual has not improved, making it an effective indicator of local stagnation. Search stage $f_5$ represents the current progress of the optimization process. Together, these two features allow the agent to consider both the local search status and the global evolutionary stage. As a result, the policy can encourage stronger exploration for stagnant individuals while gradually shifting from global exploration to local exploitation as the optimization proceeds.

Since the state variables are derived from different cost terms and statistical indicators, their numerical ranges may vary significantly. To eliminate the influence of different scales, all extracted state features are standardized before being input into the PPO network. Specifically, for each individual, the 11 state features are standardized independently using the mean and standard deviation calculated from its own state vector. This individual-wise standardization transforms each state vector to have zero mean and unit variance before it is input into the PPO network. It ensures that each feature has a comparable numerical scale, preventing features with larger magnitudes from dominating the policy learning process and improving the stability of network training. By integrating these three types of features, the agent can fully perceive the evolutionary state from multiple dimensions. It can then accurately identify and select the most appropriate search strategy.

3.3. Action and Triplet Mechanism

The actions are represented as different mutation operations in the proposed algorithm. During the training phase, the mathematical formulas of each action operator are provided in Table 3. Within Table 3, $X_{best,t}^p$ represents a candidate solution drawn from the best-performing $p$ individuals in the present iteration. $X_{r^{*},t}$ identifies a vector randomly drawn from the population. ${\tilde{X}}_{r,t}$ is a vector arbitrarily drawn within the combined collection $P_t\bigcup \mathbb{A}$. $P_t$ represents the swarm at the $t^{th}$ generation, and the individuals eliminated in the ongoing generation are systematically archived within $\mathbb{A}$. $\mathbb{A}$ is designed to counteract intense selection pressure, ensuring that promising individuals are not prematurely discarded from the evolutionary process. Consequently, this mechanism serves to preserve essential population diversity for subsequent evolutionary phases. The capacity of $\mathbb{A}$ is $rac\cdot N$, where $\mathrm{r}\mathrm{a}\mathrm{c}$ denotes a hyperparameter. When the size of $\mathbb{A}$ exceeds its maximum capacity, individuals in $\mathbb{A}$ are randomly removed to keep its size within the limit. This random replacement maintains a diverse collection of historical search directions and prevents the archive from becoming dominated by highly similar elite solutions, thereby improving the diversity of mutation vectors throughout the evolutionary process.

Table 3. Designed mutation operators.

Label	Mathematical Expression
$action 1$	$V_{i,t}=X_{i,t}+F_i\cdot \left({X_{best,t}^p-X}_{i,t}\right)+F_i\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 2$	$V_{i,t}=X_{i,t}+F_i\cdot \left({X_{r_1,t}-X}_{i,t}\right)+F_i\cdot (X_{r_2,t}-X_{r_3,t})$
$action 3$	$V_{i,t}=X_{i,t}+F_i\cdot \left({X_{best,t}-X}_{i,t}\right)+F_i\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 4$	$V_{i,t}=X_{i,t}+F_i\cdot (X_{r_1,t}-{\tilde{X}}_{r,t})$
$action 5$	$V_{i,t}=X_{r_1,t}+F_i\cdot (X_{r_2,t}-X_{r_3,t})$

During the test phase, this study introduces a triplet mechanism inspired by the Triple Competitive DE (TCDE) proposed by Yang et al. [38]. Unlike TCDE, which strictly fixes the mutation strategy for each individual based on its assigned tier, PPOMSDE leverages this grouping mechanism with a distinctly different execution logic. In this study framework, the mutation strategies are dynamically selected by the PPO agent, and the triplet mechanism is applied merely to modify specific components of these chosen strategies. Specifically, it substitutes targeted reference vectors within the selected mutation equation based on the individual’s group to better guide the evolutionary direction. This integration ensures that the algorithm benefits from hierarchical parameter control while fully maintaining the adaptive flexibility of reinforcement learning-based strategy selection.

First, the population is randomly partitioned into $\mathcal{U}$ triplets. If the population is not divided exactly into triplets, the remaining individuals will be directly retained in the next population. Dictated by the fitness evaluations of individuals, they are defined as ${\mathcal{T}}_{best}$, the medium individual ${\mathcal{T}}_{medium}$, and the worst individual ${\mathcal{T}}_{worst}$. Then, according to the number of triplets, three groups of scaling factors $F$ and crossover rates $CR$ values are synthesized iteratively based on the parameter adaptation method in Section 3.1. The parameters within each group are sorted. The smallest values $F_{best}$ are assigned to ${\mathcal{T}}_{best}$, the middle values $F_{medium}$ are assigned to ${\mathcal{T}}_{medium}$, and the largest values $F_{worst}$ are assigned to ${\mathcal{T}}_{worst}$. This hierarchical parameter control mechanism enables differentiated search behaviors. The high-quality individual ${\mathcal{T}}_{best}$ is assigned smaller values of $F$ and $CR$. It performs fine-grained search in its neighborhood and enhances local exploitation. In contrast, the inferior individual ${\mathcal{T}}_{worst}$ is assigned larger parameter values. The detailed triplet mechanism is shown in Figure 2. It uses larger step sizes to expand the search range and improve global exploration. This layered mechanism aims to coordinate exploration and exploitation. It avoids destroying the existing solution structure of high-quality individuals due to excessive perturbation. At the same time, it encourages inferior individuals to evade premature convergence. As a result, it improves the overall search efficiency and convergence performance.

Figure 2. The framework of triplet mechanism.

For the high-quality individual ${\mathcal{T}}_{best}$, a more fine-grained search is required to enhance local exploitation. In addition to generating relatively small values of $F$ and $CR$, a coefficient $W$ is introduced to parameters $F$. This prevents the solution structure from being damaged by excessive perturbation. The specific operations are detailed in Table 4. In Table 4, ${\mathcal{T}}_{best}^{i,t,u}$ is the individual $X_{i,t}$ classified as ${\mathcal{T}}_{\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}}$ within the $u^{th}$ triplet, and $F_{best}^{i,t,u}$ is the parameter $F$ assigned to ${\mathcal{T}}_{best}^{i,t,u}$.

Table 4. Modified ${\mathcal{T}}_{best}$ action pool.

Label	Mathematical Expression
$action 1$	$V_{i,t}={\mathcal{T}}_{best}^{i,t,u}+W\cdot F_{best}^{i,t,u}\cdot (X_{best,t}^p-{\mathcal{T}}_{best}^{i,t,u})+F_{best}^{i,t}\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 2$	$V_{i,t}={\mathcal{T}}_{best}^{i,t,u}+W\cdot F_{best}^{i,t,u}\cdot (X_{r_1,t}-{\mathcal{T}}_{best}^{i,t,u})+F_{best}^{i,t}\cdot (X_{r_2,t}-X_{r_3,t})$
$action 3$	$V_{i,t}={\mathcal{T}}_{best}^{i,t,u}+W\cdot F_{best}^{i,t,u}\cdot (X_{best,t}-{\mathcal{T}}_{best}^{i,t,u})+F_{best}^{i,t}\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 4$	$V_{i,t}={\mathcal{T}}_{best}^{i,t,u}+W\cdot F_{best}^{i,t,u}\cdot (X_{r_1,t}-{\tilde{X}}_{r,t})$
$action 5$	$V_{i,t}=X_{r_1,t}+F_{best}^{i,t,u}\cdot (X_{r_2,t}-X_{r_3,t})$

For the medium individual ${\mathcal{T}}_{medium}$, the definition of $X_{best,t}$ is modified to $X_{best,t}={\mathcal{T}}_{best}$. This aims to guide it toward high-quality regions by learning from better individuals. The specific operations are detailed in Table 5. In Table 5, ${\mathcal{T}}_{medium}^{i,t,u}$ is the individual $X_{i,t}$ classified as ${\mathcal{T}}_{medium}$ within the $u^{th}$ triplet, and $F_{medium}^{i,t,u}$ is the parameter $F$ assigned to ${\mathcal{T}}_{medium}^{i,t,u}$. ${\mathcal{T}}_{best}^{t,u}$ denotes the ${\mathcal{T}}_{best}$ individual relative to $X_{i,t}$ within the $u^{th}$ triplet.

Table 5. Modified ${\mathcal{T}}_{medium}$ action pool.

Label	Mathematical Expression
$action 1$	$V_{i,t}={\mathcal{T}}_{medium}^{i,t,u}+F_{medium}^{i,t,u}\cdot (X_{best,t}^p-{\mathcal{T}}_{medium}^{i,t,u})+F_{medium}^{i,t,u}\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 2$	$V_{i,t}={\mathcal{T}}_{medium}^{i,t,u}+F_{medium}^{i,t,u}\cdot (X_{r_1,t}-{\mathcal{T}}_{medium}^{i,t,u})+F_{medium}^{i,t,u}\cdot (X_{r_2,t}-X_{r_3,t})$
$action 3$	$V_{i,t}={\mathcal{T}}_{medium}^{i,t,u}+F_{medium}^{i,t,u}\cdot ({\mathcal{T}}_{best}^{t,u}-{\mathcal{T}}_{medium}^{i,t,u})+F_{medium}^{i,t,u}\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 4$	$V_{i,t}={\mathcal{T}}_{medium}^{i,t,u}+F_{medium}^{i,t,u}\cdot (X_{r_1,t}-{\tilde{X}}_{r,t})$
$action 5$	$V_{i,t}=X_{r_1,t}+F_{medium}^{i,t,u}\cdot (X_{r_2,t}-X_{r_3,t})$

For the inferior individual ${\mathcal{T}}_{worst}$, based on redefining $X_{best,t}={\mathcal{T}}_{best}$, the definition of $X_{r^{*},t}$ is further modified to $X_{r^{*},t}={\mathcal{T}}_{medium}$. In this way, it learns from better individuals within the triplet and is guided to evolve toward more promising directions. The specific operations are detailed in Table 6. In Table 6, ${\mathcal{T}}_{worst}^{i,t,u}$ is the individual $X_{i,t}$ classified as ${\mathcal{T}}_{worst}$ within the $u^{th}$ triplet, and $F_{worst}^{i,t,u}$ is the parameter $F$ assigned to ${\mathcal{T}}_{worst}^{i,t,u}$. ${\mathcal{T}}_{medium}^{t,u}$ denotes the ${\mathcal{T}}_{medium}$ individual relative to $X_{i,t}$ within the $u^{th}$ triplet.

Table 6. Modified ${\mathcal{T}}_{worst}$ action pool.

Label	Mathematical Expression
$action 1$	$V_{i,t}={\mathcal{T}}_{worst}^{i,t,u}+F_{worst}^{i,t,u}\cdot (X_{best,t}^p-{\mathcal{T}}_{worst}^{i,t,u})+F_{worst}^{i,t}\cdot ({\mathcal{T}}_{medium}^{t,u}-{\tilde{X}}_{r,t})$
$action 2$	$V_{i,t}={\mathcal{T}}_{worst}^{i,t,u}+F_{worst}^{i,t,u}\cdot (X_{r_1,t}-{\mathcal{T}}_{worst}^{i,t,u})+F_{worst}^{i,t}\cdot (X_{r_2,t}-X_{r_3,t})$
$action 3$	$V_{i,t}={\mathcal{T}}_{worst}^{i,t,u}+F_{worst}^{i,t,u}\cdot ({\mathcal{T}}_{best}^{t,u}-{\mathcal{T}}_{worst}^{i,t,u})+F_{worst}^{i,t}\cdot ({\mathcal{T}}_{medium}^{t,u}-{\tilde{X}}_{r,t})$
$action 4$	$V_{i,t}={\mathcal{T}}_{worst}^{i,t,u}+F_{worst}^{i,t,u}\cdot (X_{r_1,t}-{\tilde{X}}_{r,t})$
$action 5$	$V_{i,t}=X_{r_1,t}+F_{worst}^{i,t,u}\cdot (X_{r_2,t}-X_{r_3,t})$

During the evaluation of each triplet, if a trial vector $V_{i,t}$ successfully outperforms its corresponding parent (whether it is classified as ${\mathcal{T}}_{best}$, ${\mathcal{T}}_{medium}$ or ${\mathcal{T}}_{worst}$), the eliminated parent is temporarily recorded in a generational archive list. Once all triplets in the population have completed their mutation, crossover, and selection operations for the current generation $t$, this temporary list is merged into the global external archive $\mathbb{A}$. To maintain computational efficiency and prevent excessive accumulation of outdated solutions, the maximum capacity of $\mathbb{A}$ is restricted to $rac\cdot N$. If the size of the merged archive exceeds this predefined threshold, individuals within $\mathbb{A}$ are randomly removed to meet the constraint. The solutions preserved in $\mathbb{A}$ are subsequently utilized in the mutation actions, which continuously injects historical diversity into the search process and significantly assists the triplets in escaping local optima.

Notably, the triplet mechanism is intentionally excluded from the PPOMSDE training phase to maintain a stationary MDP. If the dynamic grouping of the triplet mechanism were active during training, the environmental dynamics would become highly non-stationary. Specifically, the reward of an individual would be heavily coupled not only with its selected action but also with its randomly assigned group role (${\mathcal{T}}_{best}$, ${\mathcal{T}}_{medium}$, ${\mathcal{T}}_{worst}$) and the corresponding parameter perturbations. This coupling prevents the PPOMSDE agent from accurately evaluating the true value of its policy decisions, leading to high variance and unstable training. To address this, a decoupled paradigm of stable training and dynamic deployment is adopted. During the training phase, a unified and unperturbed environment is utilized to ensure that the agent learns a robust and unbiased baseline mapping from state features to mutation strategies. In the testing phase, the triplet mechanism is introduced as a deterministic heuristic booster, similar to test-time augmentation (TTA). It dynamically refines the selected actions and allocates targeted parameters based on the current population hierarchy.

The triplet mechanism works independently of the PPO policy. It does not change the mutation strategy selected by the PPO agent; rather, it adjusts the control parameters according to the fitness level of each individual. Inferior individuals are encouraged to explore a larger search region, while superior individuals focus on local refinement. As a result, the PPO framework provides adaptive strategy selection, and the triplet mechanism further improves the execution of these strategies. Their combination enhances the balance between exploration and exploitation, helping the algorithm escape local optima and achieve more stable convergence.

3.4. Reward Design

The reward mechanism is vital for the agent to evaluate action efficacy in RL. Following the state and action space definitions, new individuals are generated via evolutionary operations. In the proposed algorithm, the reward is quantified based on the objective value enhancement exhibited by the active candidate relative to the initial fitness ${\mathrm{f}}_{\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{l}}$. Specifically, the reward function captures the incremental improvement over the starting fitness value, formulated as follows:

$${reward}_{i,t}=-clip\left({\displaystyle \frac{f\left(X_{i,t+1}\right)}{f_{inital}}},0,1\right)$$

(36)

To improve the stability of PPO training across optimization problems with significantly different fitness scales, the reward is first normalized using the initial best fitness. This normalization provides a consistent reward scale across different optimization problems and prevents the reward magnitude from varying substantially among benchmark functions. However, since the reward is computed for each individual, many individuals have fitness values much larger than the initial best fitness, especially during the early stage of evolution. As a result, the normalized reward may become an excessively large negative value. PPO is sensitive to such extreme reward signals because they produce large advantage estimates, which can dominate policy updates and increase gradient variance. Therefore, the normalized reward is further clipped to the interval $\left[-1,0\right]$. This clipping prevents abnormal reward values from overwhelming the learning process and improves the stability of policy optimization. Although clipping compresses the reward magnitude, PPO updates are based on the estimated advantage rather than the immediate reward alone. Therefore, the policy can still distinguish different optimization behaviors through accumulated returns while maintaining stable learning performance.

3.5. Multi-Buffer Training Mechanism

In the standard PPO algorithm, a single buffer cannot distinguish the features and rewards of individuals. At each iteration, the environment outputs the features, actions, and rewards of all individuals. When multiple individuals are stored in the buffer within the same time step, they may be incorrectly assigned a sequential temporal order due to the storage process. Therefore, the PPO network is required to allocate and maintain an independent buffer for each individual in the population [39]. This study constructs a multiple buffer set $\left\{B_1,B_2,\dots ,B_N\right\}$ to provide a data-isolated learning environment for each individual. Although crossover combines genetic information from multiple individuals, the generated transition is always assigned to the target individual because the state observation, action selection, reward evaluation, and state transition are all defined with respect to the same decision-making subject. Therefore, crossover only affects the evolutionary search process and does not cause experience mixing among different buffers. The structure of the buffer $B_i$ is defined as $B_{i,t}=\left\{s_t^i, a_t^i, r_t^i,{\pi }_{\mathrm{t}}^{\mathrm{i}},{\mathcal{V}}_{\mathrm{t}}^{\mathrm{i}}\left(\mathrm{s}\right), d_t^i\right\}$. The detailed meanings of these components are shown in Table 7.

Table 7. Buffer structure composition.

Sequence	Description
$s_t^i$	store the state vector extracted for the individual.
$a_t^i$	store the action selected by the policy network at each iteration.
$r_t^i$	store the immediate ${\mathrm{r}\mathrm{e}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}}_{\mathrm{i},\mathrm{t}}$ based on the fitness improvement.
${\pi }_{\mathrm{t}}^{\mathrm{i}}$	store the probability of the selected action.
${\mathcal{V}}_{\mathrm{t}}^{\mathrm{i}}\left(\mathrm{s}\right)$	store the state value estimated by the critic network.
$d_t^i$	record whether the individual reaches a terminal state.

In this study, the complete evolutionary process from population initialization to reaching the predefined evaluation threshold is defined as one training episode. Within each episode, the current policy network with old parameters ${\theta }_{old}$ interacts sequentially with different environments to obtain stable experience data sequences. After completing $\varphi$ training episodes, the algorithm independently extracts the trajectory from each buffer. The network parameters are then sequentially updated for each individual. During the evolutionary progression, interaction data (states, actions, rewards) is recorded independently into $N$ isolated buffers, preventing temporal crossover between different individuals. Once $\varphi$ episodes are completed, the algorithm enters the optimization phase, where it sequentially extracts the temporal trajectory from each buffer to perform gradient descent on the actor and critic networks. Upon completing the sequential updates across all $N$ individuals, a global synchronization is executed: the parameters of the old policy network ${\theta }_{old}$ are updated to match the newly optimized policy network $\theta$. Immediately following this parameter synchronization, all individual buffers are explicitly cleared simultaneously. This update procedure ensures that the subsequent $\varphi$ episodes will strictly collect fresh trajectories driven by the updated policy, thereby maintaining a stable and unbiased advantage estimation.

The model weights undergo successive refinement by driving down the aggregate objective function $L\left(\theta \right)$ sequentially for each individual buffer. Benefiting from the multi-buffer structure, the updating process can extract the experience trajectories of individuals independently. This mechanism enables the algorithm to obtain diverse training samples within a single optimization cycle and effectively avoids over fitting caused by sample correlation. For the $i^{th}$ individual, the raw cumulative discounted return $R_{i,t}$ at time step $t$ is calculated via Monte Carlo estimation based on its independent evolutionary trajectory:

$$R_{i,t}={\displaystyle {\sum }_{\beta =0}^{T_i-t}}{\gamma }^{\beta }\cdot r_{t+\beta }^i$$

(37)

where $T_i$ is the terminal step of the $i^{th}$ individual’s trajectory, and $\gamma$ is the discount factor. Functionally, γ controls the contribution of future rewards to the cumulative return, balancing short-term improvements and long-term optimization. The parameter $\beta$ is a summation index representing the future time-step offset relative to the current step $t$. It dictates the exponential decay factor ${\gamma }^{\beta }$ applied to the future rewards $r_{t+\beta }^i$. This decay mechanism, implemented via Monte Carlo return estimation, ensures that immediate fitness improvements are prioritized over distant future outcomes in the agent’s state-value evaluation. To reduce variance and stabilize the sequential training process, the returns are standardized locally within the individual’s specific trajectory:

$$R_{i,t}^{*}={\displaystyle \frac{R_{i,t}-\mu \left(R_i\right)}{\sigma \left(R_i\right)+ϵ}}$$

(38)

where $R_{i,t}^{*}$ is the standardized cumulative discounted return, $\mu \left(R_i\right)$ and $\sigma \left(R_i\right)$ denote the expected value and statistical dispersion regarding the raw reward signals specific to the $i^{th}$ individual, while $ϵ$ acts as a numerical stabilizer to avoid zero denominators. The advantage function $A_{i,t}$ for the $i^{th}$ individual is then computed as:

$$A_{i,t}=R_{i,t}^{*}-{\mathcal{V}}_{{\pi }_{\theta \_old}}\left(s_t^i\right)$$

(39)

where ${\mathcal{V}}_{{\pi }_{\theta \_old}}\left(s_t^i\right)$ is the value estimated by the critic network at state $s_t^i$. Therefore, during the policy update phase, the formulation of clipped surrogate objective $L^{clip}\left(\theta \right)$ in the total loss $L\left(\theta \right)$, optimized exclusively for the $i^{th}$ individual, is redefined as:

$$L^{clip}\left(\theta \right)={\widehat{E}}_t\left[min\left({\rho }_{i,t}\right(\theta )A_{i,t},clip({\rho }_{i,t}\left(\theta \right),1-\mathcal{E},1+\mathcal{E}\left)A_{i,t}\right)\right]$$

(40)

where ${\widehat{E}}_t$ is the expectation over the temporal trajectory of the single $i^{th}$ individual. The policy update ratio ${\rho }_{i,t}\left(\theta \right)$ in the $i^{th}$ buffer is expressed as:

$${\rho }_{i,t}(\theta )={\displaystyle \frac{{\pi }_{\theta }\left(a_t^i\right|s_t^i)}{{\pi }_{\theta \_old}\left(a_t^i\right|s_t^i)}}$$

(41)

where ${\pi }_{\theta }\left(a_t^i\right|s_t^i)$ and ${\pi }_{\theta \_old}\left(a_t^i\right|s_t^i)$ indicate the likelihoods of executing maneuver $a_t^i$ given the updated and baseline strategies at state $s_t^i$, respectively. Upon deriving the aggregate loss for the $i^{th}$ individual, gradient descent via BP is employed to concurrently update the trainable parameters of both the actor and critic architectures for $K$ optimization epochs. After completing the updates for all individuals sequentially, the old policy network parameters ${\theta }_{old}$ are synchronized to prepare for the next evolutionary cycle.

4. Experimental Results

4.1. Experimental Setup

To evaluate the proposed method, comprehensive simulation studies are conducted to assess the optimization performance of PPOMSDE. To provide a thorough evaluation, PPOMSDE is benchmarked against seven advanced DE derivatives, including TCDE [38], ESADE [40], ISDE [41], ISHACDE [42], HCDE [37], SHADE [43] and L-SHADE [44] as well as the basic DE and PSO algorithms. Two widely used benchmark suites, CEC’2014 [45] and CEC’2017 [46], are employed. The CEC’2014 suite comprises 30 benchmark problems, classified into four structural groups. The CEC’2017 suite includes 29 benchmark problems and follows the same four categories.

In all experiments, the optimization landscape is evaluated on 10-D and 30-D instances of the CEC’2014 suite and on 10-D instances of the CEC’2017 suite. Parameter analysis is performed on seven hyperparameters to determine suitable parameter settings for PPOMSDE. In addition, the action components are analyzed separately, and eight action schemes are designed to verify the effectiveness of the proposed components. Section 4.6 further presents ablation studies to evaluate the contributions of the adaptive parameters, the PPO strategy, and the triplet mechanism. All experiments were implemented in Python 3.9 using PyTorch 2.2.0 with CUDA 11.8, and were conducted on a computer equipped with an Intel Core i9-13900HX CPU and an NVIDIA RTX 4060 Laptop GPU.

To further demonstrate its practical applicability, PPOMSDE and the compared algorithms are applied to agricultural UAV path planning in five complex environments. For a fair comparison, the population size of all algorithms is fixed at 50. For parameter settings, ESADE adopts the six mutation operators introduced in Section 2.2, while the remaining algorithms use the parameter settings recommended in their original publications. The maximum number of fitness evaluations (MaxFE) is set to 50,000 for the UAV path-planning tasks and to Dim×5000 for all benchmark experiments. No fixed random seed was specified in the experiments. Each independent run used a separately generated random initialization and random sequence. Each algorithm is executed independently 30 times. The mean, standard deviation, average rank, and the results of the Wilcoxon rank-sum test at a significance level of p = 0.05 are reported. The detailed parameter settings are listed in Table 8.

Table 8. Parameter settings.

Algorithms	Parameter Settings
PPOMSDE	$H=6,$${ M}_F^t=0.3,$ $M_{CR}^t=0.9,$$p=0.11,$$rac=1.4,$ $W=0.8,$ $⟂=0.2\times MaxFE,$  $C=5$
TCDE	$H=6,$ $M_F^g=0.3,$ $M_{CR}^g=0.9$
ISDE	$FW=0.6,$ $p=0.2$
ISHACDE	${\mu }_{F1}=0.5,$ ${\mu }_{F2}=0.5,$ ${\mu }_{F3}=0.5,$ ${\mu }_{F4}=0.5,$ ${\mu }_{CR1}=0.5,$ ${\mu }_{CR2}=0.5$
HCDE	${rac}_A=1.5,$ ${rac}_B=0.5,$ $p=0.11,$ ${\mu }_F=0.5,$ ${\mu }_{CR}=0.5$
ESADE	$a=0.1,$ $T=0.5,$ $F=0.5,$ $CR=0.9$
L-SHADE	$p=0.11,$ $H=6,$ $M_{CR}=0.5,$ $M_F=0.5$
SHADE	$H=100,$ $M_{CR}=0.5,$ $M_F=0.5$
DE	$F=0.5,$ $CR=0.9$
PSO	$w=0.8,$ $c1=0.5,$ $c2=0.5$

For each task category, a unified PPO model is trained and subsequently used for all test problems within that category. Specifically, separate models are trained for the 10-D CEC’2014 suite, the 30-D CEC’2014 suite, the 10-D CEC’2017 suite, and the UAV path-planning scenarios. After offline training, the corresponding policy is frozen and applied to all functions or scenarios in the same category without further updates. The MaxFE used in each offline training episode is identical to that used in one test run, namely Dim×5000 for the benchmark functions and 50,000 for the UAV path-planning scenarios. The function evaluations consumed during offline training are counted separately and are not included in the test-stage MaxFE budget. When an algorithmic component or parameter setting is changed in the action analysis, ablation study, or parameter analysis, a new model is retrained for the corresponding task category before testing.

4.2. Comparison over CEC’2014

4.2.1. Comparison over 10-D Instances of the CEC’2014

The CEC’2014 benchmark functions are utilized to gauge the search ability, the capability of escaping local optima, and the exploitation performance of algorithms. Figure 3 shows the convergence results of all algorithms on several representative benchmark functions and Table 9 presents the $\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}$, $\mathrm{S}\mathrm{T}\mathrm{D}$, and $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}$ of algorithms over 30 independent runs. Among all 10-D CEC’2014 test functions, PPOMSDE achieves the first rank on all functions. For the unimodal function F1–F3, PPOMSDE, TCDE, ISDE, ISHACDE, SHADE, and L-SHADE all reach the theoretical optimum or achieve results very close to it. In contrast, other advanced algorithms such as HCDE tend to generate excessive perturbations, limiting their convergence accuracy when dealing with extremely narrow global optimum regions. For the multimodal functions F4–F16, algorithms like ISHACDE and TCDE are prone to premature convergence due to historical dependence or predefined static weights. In contrast, PPOMSDE dynamically adjusts mutation strategies based on real-time population state perception. This allows PPO to quickly identify the global optimum direction, achieving high accuracy on functions like F4 and F5. For the highly complex hybrid and composition functions F17–F30, some comparison algorithms such as ISDE attempt to use Double Deep Q-Network (DDQN) for strategy control. However, due to the lack of long-term reward modeling, ISDE is susceptible to premature convergence within sub-optimal basins when navigating highly rugged fitness landscapes such as F27 and F30. PPOMSDE leverages the generalization ability of reinforcement learning to dynamically orchestrate the equilibrium between global spatial traversal and localized refinement. The Wilcoxon rank-sum test further verifies the superiority of PPOMSDE. Compared with SHADE and L-SHADE, PPOMSDE achieves significantly better results on 25 benchmark functions. Against TCDE and ISDE, PPOMSDE outperforms them on 14 functions and performs comparably on the remaining 16, demonstrating its robustness across the 10-D CEC’2014 benchmark suite.

Figure 3. Convergence curves of representative functions on the 10-D CEC’2014.

Table 9. The result of the 10-D CEC’2014.

Func	Index	DE	PSO	SHADE	L-SHADE	TCDE	HCDE	ISDE	ESADE	ISHACDE	PPOMSDE
F1	Mean	1.10 × 105 (+)	2.09 × 106 (+)	1.00 × 102 (=)	1.00 × 102 (=)	1.00 × 102 (=)	6.87 × 102 (+)	1.00 × 102 (=)	4.77 × 102 (+)	1.00 × 102 (=)	1.00 × 102
	Std	6.30 × 104	2.23 × 106	0.00 × 100	0.00 × 100	2.66 × 10−11	4.87 × 102	0.00 × 100	6.63 × 102	1.88 × 10−4	4.19 × 10−13
	Rank	9	10	1	1	1	8	1	7	1	1
F2	Mean	2.12 × 102 (+)	8.64 × 103 (+)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102
	Std	6.48 × 100	4.29 × 103	0.00 × 100	0.00 × 100	0.00 × 100	2.83 × 10−10	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100
	Rank	9	10	1	1	1	1	1	1	1	1
F3	Mean	3.00 × 102 (+)	8.99 × 103 (+)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102
	Std	6.62 × 10−3	6.20 × 103	0.00 × 100	0.00 × 100	0.00 × 100	1.64 × 10−12	0.00 × 100	0.00 × 100	4.68 × 10−4	0.00 × 100
	Rank	1	10	1	1	1	1	1	1	1	1
F4	Mean	4.31 × 102 (+)	4.43 × 102 (+)	4.35 × 102 (+)	4.35 × 102 (+)	4.31 × 102 (+)	4.31 × 102 (+)	4.31 × 102 (+)	4.15 × 102 (+)	4.25 × 102 (+)	4.01 × 102
	Std	1.07 × 101	3.85 × 101	0.00 × 100	0.00 × 100	1.10 × 101	1.08 × 101	1.10 × 101	1.69 × 101	1.64 × 101	1.83 × 100
	Rank	4	10	8	8	4	4	4	2	3	1
F5	Mean	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.18 × 102
	Std	7.56 × 10−2	1.71 × 10−2	1.18 × 10−2	6.98 × 10−3	9.63 × 10−3	1.29 × 10−2	5.65 × 10−2	7.91 × 10−2	5.96 × 10−2	6.32 × 100
	Rank	2	2	2	2	2	2	2	2	2	1
F6	Mean	6.01 × 102 (+)	6.06 × 102 (+)	6.00 × 102 (+)	6.00 × 102 (+)	6.00 × 102 (=)	6.00 × 102 (+)	6.00 × 102 (=)	6.01 × 102 (+)	6.00 × 102 (=)	6.00 × 102
	Std	6.76 × 10−1	1.39 × 100	3.76 × 10−1	3.48 × 10−1	4.16 × 10−1	3.41 × 10−1	0.00 × 100	1.06 × 100	2.82 × 10−1	0.00 × 100
	Rank	8	10	1	1	1	1	1	8	1	1
F7	Mean	7.00 × 102 (+)	7.16 × 102 (+)	7.00 × 102 (+)	7.00 × 102 (+)	7.00 × 102 (=)	7.00 × 102 (+)	7.00 × 102 (=)	7.00 × 102 (+)	7.00 × 102 (+)	7.00 × 102
	Std	7.76 × 10−2	1.53 × 101	5.75 × 10−3	6.13 × 10−3	4.03 × 10−3	6.79 × 10−3	1.36 × 10−2	2.40 × 10−2	1.80 × 10−2	0.00 × 100
	Rank	1	10	1	1	1	1	1	1	1	1
F8	Mean	8.13 × 102 (+)	8.23 × 102 (+)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.07 × 102 (+)	8.00 × 102 (=)	8.00 × 102
	Std	1.63 × 100	1.07 × 101	1.26 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	4.99 × 100	6.29 × 10−1	0.00 × 100
	Rank	9	10	1	1	1	1	1	8	1	1
F9	Mean	9.31 × 102 (+)	9.25 × 102 (+)	9.04 × 102 (+)	9.04 × 102 (+)	9.02 × 102 (=)	9.03 × 102 (+)	9.05 × 102 (+)	9.12 × 102 (+)	9.03 × 102 (+)	9.02 × 102
	Std	5.68 × 100	9.90 × 100	6.70 × 10−1	4.20 × 10−1	6.88 × 10−1	9.12 × 10−1	1.92 × 100	7.51 × 100	1.08 × 100	6.71 × 10−1
	Rank	10	9	5	5	1	3	7	8	3	1
F10	Mean	1.05 × 103 (+)	1.73 × 103 (+)	1.02 × 103 (+)	1.01 × 103 (+)	1.02 × 103 (+)	1.02 × 103 (+)	1.01 × 103 (+)	1.09 × 103 (+)	1.01 × 103 (=)	1.00 × 103
	Std	1.30 × 101	2.49 × 102	8.55 × 100	9.50 × 100	6.90 × 100	1.24 × 101	1.04 × 101	5.76 × 101	3.75 × 101	5.77 × 10−2
	Rank	8	10	5	2	5	5	2	9	2	1
F11	Mean	2.00 × 103 (+)	2.05 × 103 (+)	1.39 × 103 (+)	1.38 × 103 (+)	1.27 × 103 (+)	1.19 × 103 (=)	1.38 × 103 (+)	1.90 × 103 (+)	1.29 × 103 (+)	1.14 × 103
	Std	2.58 × 102	3.07 × 102	5.20 × 101	9.39 × 101	1.35 × 102	7.71 × 101	1.37 × 102	4.89 × 102	7.99 × 101	3.06 × 101
	Rank	9	10	7	5	3	2	5	8	4	1
F12	Mean	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (=)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103
	Std	3.01 × 10−1	1.43 × 10−1	3.55 × 10−2	2.07 × 10−2	3.71 × 10−2	3.31 × 10−2	3.61 × 10−2	4.21 × 10−1	7.90 × 10−2	2.52 × 10−2
	Rank	1	1	1	1	1	1	1	1	1	1
F13	Mean	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103 (=)	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103
	Std	4.94 × 10−2	4.13 × 10−1	1.20 × 10−2	9.38 × 10−3	2.51 × 10−2	2.50 × 10−2	2.27 × 10−2	3.80 × 10−2	2.67 × 10−2	9.14 × 10−3
	Rank	1	1	1	1	1	1	1	1	1	1
F14	Mean	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (=)	1.40 × 103 (+)	1.40 × 103 (=)	1.40 × 103 (+)	1.40 × 103 (=)	1.40 × 103
	Std	4.04 × 10−2	3.33 × 10−1	4.48 × 10−2	2.66 × 10−2	8.50 × 10−2	7.96 × 10−2	8.54 × 10−2	3.56 × 10−2	1.25 × 10−1	5.93 × 10−2
	Rank	1	1	1	1	1	1	1	1	1	1
F15	Mean	1.50 × 103 (+)	1.51 × 103 (+)	1.50 × 103 (+)	1.50 × 103 (+)	1.50 × 103 (+)	1.50 × 103 (+)	1.50 × 103 (+)	1.50 × 103 (+)	1.50 × 103 (+)	1.50 × 103
	Std	5.29 × 10−1	1.64 × 101	8.67 × 10−2	4.28 × 10−2	6.70 × 10−2	1.44 × 10−1	1.55 × 10−1	4.30 × 10−1	1.92 × 10−1	6.66 × 10−2
	Rank	1	10	1	1	1	1	1	1	1	1
F16	Mean	1.60 × 103 (+)	1.60 × 103 (+)	1.60 × 103 (+)	1.60 × 103 (+)	1.60 × 103 (+)	1.60 × 103 (=)	1.60 × 103 (=)	1.60 × 103 (+)	1.60 × 103 (+)	1.60 × 103
	Std	1.47 × 10−1	3.57 × 10−1	8.70 × 10−2	1.34 × 10−1	2.76 × 10−1	3.29 × 10−1	5.41 × 10−1	3.78 × 10−1	5.21 × 10−1	6.15 × 10−1
	Rank	1	1	1	1	1	1	1	1	1	1
F17	Mean	4.66 × 103 (+)	1.44 × 104 (+)	1.86 × 103 (+)	1.95 × 103 (+)	1.70 × 103 (=)	1.81 × 103 (+)	1.73 × 103 (=)	1.89 × 103 (+)	2.01 × 103 (+)	1.70 × 103
	Std	1.03 × 103	9.25 × 103	1.70 × 102	1.37 × 102	8.73 × 10−1	9.93 × 101	1.07 × 102	1.92 × 102	4.20 × 102	4.33 × 10−1
	Rank	9	10	5	7	1	4	3	6	8	1
F18	Mean	1.81 × 103 (+)	3.67 × 104 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (=)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (=)	1.80 × 103 (+)	1.80 × 103
	Std	2.66 × 100	2.73 × 104	3.51 × 10−1	4.23 × 10−1	1.12 × 10−1	5.79 × 100	5.00 × 10−1	7.16 × 10−1	1.23 × 100	1.49 × 10−1
	Rank	9	10	1	1	1	1	1	1	1	1
F19	Mean	1.91 × 103 (+)	1.95 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.90 × 103
	Std	2.50 × 100	5.98 × 101	1.03 × 100	6.22 × 10−1	4.85 × 10−1	1.67 × 100	4.72 × 10−1	2.25 × 100	1.28 × 100	1.62 × 10−1
	Rank	9	10	1	1	1	1	1	1	1	1
F20	Mean	5.03 × 103 (+)	5.57 × 106 (+)	2.00 × 103 (+)	2.00 × 103 (+)	2.00 × 103 (+)	2.09 × 103 (+)	2.00 × 103 (+)	2.00 × 103 (+)	2.00 × 103 (+)	2.00 × 103
	Std	1.95 × 103	1.47 × 107	3.88 × 10−2	4.51 × 10−2	2.38 × 10−1	2.44 × 102	3.60 × 10−1	8.15 × 10−1	2.88 × 100	9.09 × 10−2
	Rank	9	10	1	1	1	8	1	1	1	1
F21	Mean	2.18 × 103 (+)	2.65 × 104 (+)	2.10 × 103 (+)	2.10 × 103 (+)	2.10 × 103 (+)	2.10 × 103 (+)	2.10 × 103 (+)	2.10 × 103 (+)	2.10 × 103 (+)	2.10 × 103
	Std	5.36 × 101	2.30 × 104	2.31 × 10−1	2.37 × 10−1	2.11 × 10−1	1.84 × 100	3.29 × 10−1	9.83 × 10−1	1.14 × 100	1.45 × 10−1
	Rank	9	10	1	1	1	1	1	1	1	1
F22	Mean	2.23 × 103 (+)	2.51 × 103 (+)	2.31 × 103 (+)	2.31 × 103 (+)	2.22 × 103 (+)	2.31 × 103 (+)	2.24 × 103 (=)	2.26 × 103 (+)	2.37 × 103 (+)	2.22 × 103
	Std	1.98 × 100	2.52 × 102	1.10 × 102	1.10 × 102	4.93 × 100	1.10 × 102	7.05 × 101	9.29 × 101	1.10 × 102	8.23 × 100
	Rank	3	10	6	6	1	6	4	5	9	1
F23	Mean	2.54 × 103 (-)	2.55 × 103 (=)	2.54 × 103 (=)	2.54 × 103 (=)	2.54 × 103 (=)	2.54 × 103 (=)	2.54 × 103 (=)	2.54 × 103 (-)	2.54 × 103 (=)	2.54 × 103
	Std	1.71 × 100	2.21 × 101	4.79 × 10−13	4.79 × 10−13	4.79 × 10−13	4.79 × 10−13	2.79 × 100	2.79 × 100	4.79 × 10−13	4.79 × 10−13
	Rank	1	10	1	1	1	1	1	1	1	1
F24	Mean	2.54 × 103 (+)	2.54 × 103 (+)	2.51 × 103 (+)	2.51 × 103 (+)	2.51 × 103 (=)	2.51 × 103 (+)	2.51 × 103 (+)	2.51 × 103 (+)	2.51 × 103 (+)	2.50 × 103
	Std	5.31 × 100	1.33 × 101	5.43 × 10−1	5.09 × 10−1	2.24 × 100	1.55 × 100	2.30 × 100	3.79 × 100	3.18 × 100	3.09 × 100
	Rank	9	9	2	2	2	2	2	2	2	1
F25	Mean	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.69 × 103 (=)	2.70 × 103 (+)	2.70 × 103 (=)	2.70 × 103 (+)	2.70 × 103 (+)	2.69 × 103
	Std	5.88 × 10−7	1.10 × 101	7.46 × 10−3	9.32 × 10−3	2.20 × 101	2.69 × 10−2	4.79 × 10−13	4.76 × 10−1	6.41 × 10−1	2.28 × 101
	Rank	3	3	3	3	1	3	3	3	3	1
F26	Mean	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103 (+)	2.70 × 103
	Std	6.42 × 10−2	2.27 × 10−1	1.26 × 10−2	1.65 × 10−2	1.97 × 10−2	2.24 × 10−2	3.66 × 10−2	6.99 × 10−2	3.24 × 10−2	1.03 × 10−2
	Rank	1	1	1	1	1	1	1	1	1	1
F27	Mean	3.09 × 103 (+)	3.11 × 103 (+)	3.10 × 103 (+)	3.10 × 103 (+)	3.03 × 103 (+)	2.98 × 103 (=)	3.08 × 103 (+)	3.04 × 103 (+)	3.03 × 103 (+)	2.86 × 103
	Std	3.15 × 101	1.52 × 102	1.02 × 10−3	2.35 × 10−3	1.24 × 102	1.53 × 102	4.22 × 101	1.26 × 102	1.24 × 102	2.05 × 102
	Rank	7	10	8	8	3	2	6	5	3	1
F28	Mean	3.18 × 103 (+)	3.30 × 103 (+)	3.27 × 103 (+)	3.21 × 103 (+)	3.19 × 103 (=)	3.23 × 103 (+)	3.19 × 103 (=)	3.28 × 103 (+)	3.19 × 103 (=)	3.17 × 103
	Std	8.78 × 100	8.99 × 101	7.07 × 100	4.80 × 101	3.76 × 101	9.23 × 101	4.56 × 101	1.11 × 102	4.61 × 101	3.67 × 100
	Rank	2	10	8	6	3	7	3	9	3	1
F29	Mean	4.67 × 106 (=)	8.91 × 106 (+)	5.72 × 106 (+)	5.99 × 106 (+)	5.00 × 106 (=)	5.28 × 106 (+)	4.75 × 106 (=)	7.62 × 106 (+)	4.83 × 106 (=)	4.62 × 106
	Std	3.42 × 105	4.48 × 106	6.34 × 105	7.96 × 105	4.76 × 105	7.47 × 105	3.92 × 105	1.97 × 106	4.19 × 105	1.87 × 105
	Rank	2	10	7	8	5	6	3	9	4	1
F30	Mean	2.40 × 104 (+)	4.00 × 106 (+)	6.22 × 104 (+)	4.27 × 104 (+)	1.31 × 104 (=)	3.07 × 105 (+)	2.57 × 104 (+)	2.71 × 104 (+)	1.59 × 104 (+)	5.14 × 103
	Std	2.27 × 104	1.24 × 107	2.09 × 104	1.94 × 104	1.61 × 104	8.91 × 105	2.50 × 104	3.43 × 104	2.01 × 104	2.82 × 102
	Rank	4	10	8	7	2	9	5	6	3	1
Avg.Rank		5.07	7.93	3.03	2.87	1.67	2.87	2.20	3.70	2.20	1.00
Wilcoxon’s rank sum	+/=/-	28/1/1	29/1/0	25/5/0	25/5/0	14/16/0	23/7/0	14/16/0	26/3/1	20/10/0	-/-/-

4.2.2. Comparison over 30-D Instances of the CEC’2014

In the 30-D search space, PPOMSDE demonstrates robust optimization performance across most benchmark functions. It achieves particularly strong results on the complex hybrid functions (F20–F22) and composition functions (F29 and F30), where it consistently outperforms DE, PSO, TCDE, and HCDE. Moreover, PPOMSDE effectively avoids the premature convergence observed in advanced algorithms such as SHADE, L-SHADE, and ISDE, indicating its ability to maintain a better balance between global exploration and local exploitation in complex high-dimensional landscapes. Although PPOMSDE performs competitively on all benchmark functions, it does not achieve the best results on several problems, including the unimodal function F1, the multimodal functions F8–F11, and the hybrid and composition functions F18, F27, and F28. These functions feature narrow global optimum regions, highly irregular landscapes, or numerous local optima, posing significant challenges for evolutionary algorithms. This performance gap may be attributed to the limited diversity of the mutation operators in the current action pool, whose search characteristics are less suitable for these landscapes. In future work, more advanced mutation operators will be incorporated into the action pool to further enhance the optimization capability of PPOMSDE on these challenging problems. The Wilcoxon rank-sum test further supports the effectiveness of PPOMSDE by showing statistically significant improvements over most compared algorithms. Figure 4 shows the convergence results of all algorithms on several representative benchmark functions. These results indicate that the proposed reinforcement learning framework provides robust and reliable optimization performance for high-dimensional optimization problems (Table 10).

Figure 4. Convergence curves of representative functions on the 30-D CEC’2014.

Table 10. The result of the 30-D CEC’2014.

Func	Index	DE	PSO	SHADE	L-SHADE	TCDE	HCDE	ISDE	ESADE	ISHACDE	PPOMSDE
F1	Mean	1.22 × 108 (+)	1.66 × 108 (+)	1.59 × 104 (=)	1.88 × 104 (=)	1.70 × 105 (+)	7.99 × 105 (+)	1.47 × 104 (=)	9.20 × 104 (+)	1.94 × 104 (=)	1.86 × 104
	Std	2.93 × 107	1.11 × 108	4.76 × 103	3.40 × 103	7.26 × 104	4.14 × 105	3.94 × 103	8.09 × 104	7.25 × 103	7.38 × 103
	Rank	9	10	2	4	7	8	1	6	5	3
F2	Mean	2.87 × 107 (+)	1.83 × 1010 (+)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (+)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (+)	2.00 × 102
	Std	7.40 × 106	1.32 × 1010	0.00 × 100	0.00 × 100	0.00 × 100	5.01 × 10−6	0.00 × 100	1.89 × 10−12	1.24 × 10−11	0.00 × 100
	Rank	9	10	1	1	1	1	1	1	1	1
F3	Mean	1.69 × 103 (+)	7.29 × 104 (+)	3.00 × 102 (=)	3.00 × 102 (=)	3.00 × 102 (=)	3.18 × 102 (+)	3.00 × 102 (=)	3.56 × 102 (+)	3.12 × 102 (+)	3.00 × 102
	Std	3.78 × 102	2.21 × 104	0.00 × 100	0.00 × 100	0.00 × 100	4.38 × 101	0.00 × 100	6.03 × 101	2.83 × 101	0.00 × 100
	Rank	9	10	1	1	1	7	1	8	6	1
F4	Mean	5.70 × 102 (+)	1.89 × 103 (+)	4.00 × 102 (+)	4.01 × 102 (+)	4.71 × 102 (+)	5.16 × 102 (+)	4.00 × 102 (=)	4.55 × 102 (+)	4.07 × 102 (+)	4.00 × 102
	Std	7.57 × 100	9.79 × 102	8.72 × 10−1	1.19 × 100	2.21 × 100	3.54 × 101	1.26 × 100	2.69 × 101	2.05 × 101	6.07 × 10−10
	Rank	9	10	1	4	7	8	1	6	5	1
F5	Mean	5.21 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.20 × 102 (+)	5.21 × 102 (+)	5.20 × 102 (+)	5.20 × 102
	Std	3.46 × 10−2	2.18 × 10−1	2.01 × 10−2	8.13 × 10−3	2.02 × 10−2	2.29 × 10−2	4.85 × 10−2	3.05 × 10−2	4.53 × 10−2	9.89 × 10−3
	Rank	9	1	1	1	1	1	1	9	1	1
F6	Mean	6.32 × 102 (+)	6.29 × 102 (+)	6.10 × 102 (+)	6.11 × 102 (+)	6.11 × 102 (+)	6.04 × 102 (+)	6.08 × 102 (+)	6.10 × 102 (+)	6.10 × 102 (+)	6.03 × 102
	Std	7.95 × 100	2.79 × 100	1.33 × 100	1.23 × 100	1.02 × 100	1.19 × 100	2.19 × 100	3.87 × 100	1.70 × 100	1.08 × 100
	Rank	10	9	4	7	7	2	3	4	4	1
F7	Mean	7.01 × 102 (+)	8.51 × 102 (+)	7.00 × 102 (=)	7.00 × 102 (=)	7.00 × 102 (+)	7.00 × 102 (+)	7.00 × 102 (+)	7.00 × 102 (=)	7.00 × 102 (+)	7.00 × 102
	Std	2.74 × 10−2	5.23 × 101	2.34 × 10−3	4.42 × 10−3	1.78 × 10−2	4.73 × 10−2	5.08 × 10−3	2.59 × 10−2	7.68 × 10−3	0.00 × 100
	Rank	9	10	1	1	1	1	1	1	1	1
F8	Mean	9.70 × 102 (+)	9.67 × 102 (+)	8.00 × 102 (-)	8.00 × 102 (-)	8.10 × 102 (-)	8.02 × 102 (-)	8.00 × 102 (-)	8.26 × 102 (+)	8.14 × 102 (=)	8.17 × 102
	Std	1.46 × 101	2.71 × 101	0.00 × 100	0.00 × 100	2.01 × 100	2.61 × 100	0.00 × 100	6.62 × 100	1.45 × 101	3.40 × 100
	Rank	10	9	1	1	5	4	1	8	6	7
F9	Mean	1.15 × 103 (+)	1.08 × 103 (+)	9.27 × 102 (-)	9.29 × 102 (-)	9.45 × 102 (=)	9.17 × 102 (-)	9.30 × 102 (-)	9.35 × 102 (-)	9.49 × 102 (=)	9.49 × 102
	Std	1.16 × 101	2.53 × 101	1.95 × 100	2.94 × 100	4.17 × 100	3.16 × 100	3.65 × 100	1.28 × 101	6.77 × 100	1.23 × 101
	Rank	10	9	2	3	6	1	4	5	7	7
F10	Mean	5.18 × 103 (+)	5.60 × 103 (+)	1.00 × 103 (-)	1.00 × 103 (-)	1.36 × 103 (+)	1.13 × 103 (=)	1.01 × 103 (-)	2.63 × 103 (+)	1.00 × 103 (-)	1.12 × 103
	Std	2.71 × 102	9.98 × 102	2.74 × 100	1.31 × 100	3.51 × 101	1.33 × 102	2.69 × 100	9.80 × 102	1.57 × 100	6.27 × 101
	Rank	9	10	1	1	7	6	4	8	1	5
F11	Mean	8.09 × 103 (+)	5.66 × 103 (+)	3.18 × 103 (=)	3.14 × 103 (=)	3.67 × 103 (+)	2.68 × 103 (-)	3.58 × 103 (+)	7.31 × 103 (+)	3.71 × 103 (+)	3.09 × 103
	Std	2.16 × 102	5.12 × 102	1.22 × 102	1.12 × 102	1.36 × 102	1.71 × 102	1.85 × 102	1.49 × 103	4.79 × 102	1.60 × 102
	Rank	10	8	4	3	6	1	5	9	7	2
F12	Mean	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.20 × 103
	Std	2.29 × 10−1	3.17 × 10−1	8.58 × 10−3	9.91 × 10−3	1.55 × 10−2	3.42 × 10−2	3.90 × 10−2	9.96 × 10−1	4.85 × 10−2	5.27 × 10−2
	Rank	1	1	1	1	1	1	1	1	1	1
F13	Mean	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103 (-)	1.30 × 103 (-)	1.30 × 103 (=)	1.30 × 103 (=)	1.30 × 103 (=)	1.30 × 103 (+)	1.30 × 103 (+)	1.30 × 103
	Std	6.84 × 10−2	1.47 × 100	1.75 × 10−2	1.22 × 10−2	3.64 × 10−2	7.34 × 10−2	3.92 × 10−2	9.50 × 10−2	3.46 × 10−2	3.11 × 10−2
	Rank	1	1	1	1	1	1	1	1	1	1
F14	Mean	1.40 × 103 (+)	1.44 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103
	Std	1.77 × 10−1	3.84 × 101	1.97 × 10−2	5.54 × 10−2	2.23 × 10−2	1.46 × 10−1	2.22 × 10−2	2.20 × 10−1	2.94 × 10−2	2.14 × 10−2
	Rank	1	10	1	1	1	1	1	1	1	1
F15	Mean	1.53 × 103 (+)	1.40 × 105 (+)	1.50 × 103 (+)	1.50 × 103 (-)	1.51 × 103 (+)	1.50 × 103 (=)	1.50 × 103 (+)	1.51 × 103 (+)	1.51 × 103 (+)	1.50 × 103
	Std	3.15 × 100	1.40 × 105	1.31 × 10−1	1.85 × 10−1	7.71 × 10−1	8.26 × 10−1	6.07 × 10−1	5.27 × 100	1.52 × 100	4.09 × 10−1
	Rank	9	10	1	1	6	1	1	6	6	1
F16	Mean	1.61 × 103 (+)	1.61 × 103 (+)	1.61 × 103 (+)	1.61 × 103 (=)	1.61 × 103 (+)	1.61 × 103 (-)	1.61 × 103 (+)	1.61 × 103 (+)	1.61 × 103 (+)	1.61 × 103
	Std	1.89 × 10−1	5.14 × 10−1	1.07 × 10−1	1.16 × 10−1	1.64 × 10−1	3.95 × 10−1	5.41 × 10−1	2.05 × 10−1	4.78 × 10−1	3.33 × 10−1
	Rank	1	1	1	1	1	1	1	1	1	1
F17	Mean	5.65 × 106 (+)	3.95 × 106 (+)	6.91 × 103 (+)	7.66 × 103 (+)	1.30 × 104 (+)	3.79 × 104 (+)	5.55 × 103 (=)	3.18 × 104 (+)	3.00 × 104 (+)	5.40 × 103
	Std	2.89 × 106	4.88 × 106	1.03 × 103	1.18 × 103	2.15 × 103	2.04 × 104	1.08 × 103	1.45 × 104	5.19 × 104	1.21 × 103
	Rank	10	9	3	4	5	8	2	7	6	1
F18	Mean	2.15 × 108 (+)	2.16 × 106 (+)	4.11 × 104 (+)	2.10 × 104 (-)	4.46 × 104 (+)	5.71 × 104 (+)	3.22 × 104 (+)	1.00 × 105 (+)	3.21 × 104 (+)	2.48 × 104
	Std	5.69 × 107	4.12 × 106	3.48 × 103	3.33 × 103	6.43 × 103	1.96 × 104	3.45 × 103	2.19 × 104	1.15 × 104	2.53 × 103
	Rank	10	9	5	1	6	7	4	8	3	2
F19	Mean	3.53 × 106 (+)	5.83 × 105 (+)	1.93 × 103 (+)	1.93 × 103 (+)	1.92 × 103 (+)	1.95 × 103 (+)	1.92 × 103 (+)	1.94 × 103 (+)	2.12 × 103 (+)	1.91 × 103
	Std	1.37 × 106	1.76 × 106	8.89 × 100	7.87 × 100	3.04 × 100	9.99 × 101	5.76 × 100	1.77 × 101	3.32 × 102	1.20 × 100
	Rank	10	9	4	4	2	7	2	6	8	1
F20	Mean	1.21 × 109 (+)	2.21 × 109 (+)	5.62 × 103 (+)	5.60 × 103 (+)	7.54 × 103 (+)	4.80 × 104 (+)	2.88 × 103 (=)	1.64 × 104 (+)	2.07 × 104 (+)	2.56 × 103
	Std	1.39 × 109	3.16 × 109	1.14 × 103	1.99 × 103	4.86 × 103	2.76 × 104	1.18 × 103	1.45 × 104	1.34 × 104	3.05 × 102
	Rank	9	10	4	3	5	8	2	6	7	1
F21	Mean	7.00 × 106 (+)	2.14 × 107 (+)	7.82 × 103 (+)	7.37 × 103 (+)	1.50 × 104 (+)	2.13 × 104 (+)	5.01 × 103 (+)	1.42 × 104 (+)	1.43 × 104 (+)	3.40 × 103
	Std	3.15 × 106	3.34 × 107	2.13 × 103	2.65 × 103	5.96 × 103	1.36 × 104	3.11 × 103	1.09 × 104	8.70 × 103	4.81 × 102
	Rank	9	10	4	3	7	8	2	5	6	1
F22	Mean	1.97 × 106 (+)	8.98 × 103 (+)	2.71 × 103 (+)	2.58 × 103 (+)	2.65 × 103 (+)	2.52 × 103 (+)	2.59 × 103 (+)	2.46 × 103 (+)	2.76 × 103 (+)	2.25 × 103
	Std	2.85 × 106	2.75 × 103	1.04 × 102	1.82 × 102	1.82 × 102	3.14 × 102	1.45 × 102	2.40 × 102	1.09 × 102	6.73 × 101
	Rank	10	9	7	4	6	3	5	2	8	1
F23	Mean	2.52 × 103 (+)	2.61 × 103 (+)	2.52 × 103 (=)	2.52 × 103 (=)	2.52 × 103 (=)	2.52 × 103 (=)	2.52 × 103 (=)	2.52 × 103 (=)	2.52 × 103 (=)	2.52 × 103
	Std	2.61 × 10−1	3.58 × 101	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	3.19 × 10−12	0.00 × 100	0.00 × 100
	Rank	1	10	1	1	1	1	1	1	1	1
F24	Mean	2.62 × 103 (=)	2.70 × 103 (+)	2.64 × 103 (+)	2.64 × 103 (+)	2.64 × 103 (+)	2.64 × 103 (+)	2.64 × 103 (+)	2.64 × 103 (+)	2.64 × 103 (+)	2.62 × 103
	Std	2.27 × 100	4.14 × 101	2.08 × 100	1.46 × 100	3.95 × 100	5.55 × 100	2.22 × 100	7.03 × 100	2.01 × 100	2.81 × 100
	Rank	1	10	3	3	3	3	3	3	3	1
F25	Mean	2.73 × 103 (+)	2.74 × 103 (+)	2.71 × 103 (+)	2.71 × 103 (+)	2.72 × 103 (+)	2.71 × 103 (+)	2.70 × 103 (+)	2.71 × 103 (+)	2.72 × 103 (+)	2.70 × 103
	Std	6.46 × 100	2.55 × 101	2.54 × 100	2.96 × 100	1.90 × 100	6.00 × 100	4.34 × 10−1	2.14 × 100	1.94 × 100	1.45 × 100
	Rank	9	10	3	3	7	3	1	3	7	1
F26	Mean	2.70 × 103 (+)	2.70 × 103 (+)	2.71 × 103 (+)	2.72 × 103 (=)	2.73 × 103 (+)	2.70 × 103 (=)	2.70 × 103 (=)	2.77 × 103 (+)	2.77 × 103 (+)	2.70 × 103
	Std	7.34 × 10−2	1.78 × 100	3.15 × 101	4.20 × 101	4.81 × 101	8.40 × 10−2	3.21 × 10−2	1.07 × 102	4.81 × 101	3.94 × 10−2
	Rank	1	1	6	7	8	1	1	9	9	1
F27	Mean	3.69 × 103 (+)	3.86 × 103 (+)	3.09 × 103 (-)	3.14 × 103 (=)	3.25 × 103 (+)	3.24 × 103 (+)	3.18 × 103 (+)	3.24 × 103 (+)	3.28 × 103 (+)	3.12 × 103
	Std	1.41 × 102	1.00 × 102	4.16 × 101	4.31 × 101	2.66 × 101	7.94 × 101	4.90 × 101	9.47 × 101	6.30 × 101	1.73 × 101
	Rank	9	10	1	3	7	5	4	5	8	2
F28	Mean	3.72 × 103 (=)	4.69 × 103 (+)	3.66 × 103 (=)	3.67 × 103 (=)	3.73 × 103 (+)	3.92 × 103 (+)	3.71 × 103 (=)	4.12 × 103 (+)	3.77 × 103 (+)	3.68 × 103
	Std	5.60 × 101	8.02 × 102	3.56 × 101	1.46 × 101	2.19 × 101	2.75 × 102	1.87 × 101	4.36 × 102	3.43 × 101	7.45 × 101
	Rank	5	10	1	2	6	8	4	9	7	3
F29	Mean	1.00 × 107 (+)	9.25 × 107 (+)	7.00 × 103 (+)	8.21 × 103 (+)	1.01 × 104 (+)	6.60 × 104 (+)	2.28 × 106 (=)	8.91 × 105 (+)	1.14 × 104 (+)	6.04 × 103
	Std	1.32 × 107	6.99 × 107	3.42 × 102	6.29 × 102	8.91 × 102	1.48 × 105	7.20 × 106	2.24 × 106	2.23 × 103	1.05 × 103
	Rank	9	10	2	3	4	6	8	7	5	1
F30	Mean	4.73 × 108 (+)	3.27 × 108 (+)	2.59 × 104 (+)	3.18 × 104 (+)	3.06 × 104 (+)	7.21 × 104 (+)	2.54 × 104 (+)	1.12 × 105 (+)	7.18 × 104 (+)	1.21 × 104
	Std	1.88 × 108	4.99 × 108	1.51 × 104	1.28 × 104	1.54 × 104	5.21 × 104	1.09 × 104	1.84 × 105	9.93 × 104	1.43 × 103
	Rank	10	9	3	5	4	7	2	8	6	1
Avg.Rank		7.30	8.17	2.37	2.60	4.33	4.00	2.30	5.13	4.60	1.77
Wilcoxon’s rank sum	+/=/-	28/2/0	30/0/0	18/7/5	14/10/6	24/5/1	21/5/4	16/11/3	26/3/1	25/4/1	-/-/-

4.3. Comparison over 10-D Instances of the CEC’2017

The present segment further assesses the optimization efficacy of PPOMSDE alongside the compared algorithms using the 10-D CEC’2017 benchmark suite. Table 11 reports the $\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}$, $\mathrm{S}\mathrm{T}\mathrm{D}$, and $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}$ of each algorithm over 30 runs. Figure 5 shows the convergence results of all algorithms on several representative benchmark functions.

Table 11. The result of 10-D CEC’2017.

Func	Index	DE	PSO	SHADE	L-SHADE	TCDE	HCDE	ISDE	ESADE	ISHACDE	PPOMSDE
F1	Mean	6.08 × 102 (+)	1.29 × 109 (+)	1.00 × 102 (=)	1.00 × 102 (=)	1.00 × 102 (=)	1.00 × 102 (+)	1.00 × 102 (=)	1.00 × 102 (=)	1.00 × 102 (=)	1.00 × 102
	Std	3.25 × 102	1.03 × 109	0.00 × 100	0.00 × 100	0.00 × 100	7.13 × 10−10	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100
	Rank	9	10	1	1	1	1	1	1	1	1
F2	Mean	2.00 × 102 (+)	4.59 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102 (=)	2.00 × 102
	Std	1.06 × 10−7	8.19 × 102	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100
	Rank	1	10	1	1	1	1	1	1	1	1
F3	Mean	3.01 × 102 (+)	3.26 × 102 (+)	3.04 × 102 (+)	3.04 × 102 (+)	3.03 × 102 (+)	3.06 × 102 (+)	3.03 × 102 (+)	3.02 × 102 (=)	3.02 × 102 (=)	3.00 × 102
	Std	1.22 × 100	4.25 × 101	0.00 × 100	0.00 × 100	1.93 × 100	1.23 × 100	1.93 × 100	2.10 × 100	2.06 × 100	0.00 × 100
	Rank	2	10	7	7	5	9	5	3	3	1
F4	Mean	4.34 × 102 (+)	1.02 × 103 (+)	4.05 × 102 (+)	4.05 × 102 (+)	4.02 × 102 (=)	4.03 × 102 (=)	4.06 × 102 (+)	4.09 × 102 (+)	4.04 × 102 (+)	4.03 × 102
	Std	4.38 × 100	5.92 × 102	8.46 × 10−1	6.72 × 10−1	4.12 × 10−1	9.87 × 10−1	1.92 × 100	8.30 × 100	1.85 × 100	9.13 × 10−1
	Rank	9	10	5	5	1	2	7	8	4	2
F5	Mean	5.00 × 102 (+)	5.00 × 102 (+)	5.00 × 102 (+)	5.00 × 102 (+)	5.00 × 102 (+)	5.00 × 102 (+)	5.00 × 102 (=)	5.00 × 102 (=)	5.00 × 102 (+)	5.00 × 102
	Std	3.56 × 10−4	9.50 × 10−5	4.18 × 10−8	3.32 × 10−9	7.99 × 10−10	1.02 × 10−9	9.95 × 10−10	2.16 × 10−5	9.30 × 10−7	2.24 × 10−11
	Rank	1	1	1	1	1	1	1	1	1	1
F6	Mean	6.47 × 102 (+)	1.27 × 104 (+)	6.14 × 102 (+)	6.14 × 102 (+)	6.12 × 102 (=)	6.13 × 102 (+)	6.15 × 102 (+)	6.24 × 102 (+)	6.14 × 102 (+)	6.10 × 102
	Std	4.92 × 100	1.48 × 104	3.47 × 10−1	2.98 × 10−1	5.61 × 10−1	7.94 × 10−1	1.61 × 100	1.06 × 101	2.94 × 100	3.59 × 100
	Rank	9	10	4	4	2	3	7	8	4	1
F7	Mean	7.41 × 102 (+)	7.58 × 102 (+)	7.11 × 102 (+)	7.12 × 102 (+)	7.09 × 102 (+)	7.10 × 102 (=)	7.11 × 102 (+)	7.24 × 102 (+)	7.09 × 102 (+)	7.07 × 102
	Std	4.99 × 100	3.25 × 101	9.72 × 10−1	8.42 × 10−1	1.90 × 100	3.92 × 100	3.29 × 100	8.02 × 100	1.98 × 100	1.73 × 100
	Rank	9	10	5	7	2	4	5	8	2	1
F8	Mean	8.00 × 102 (=)	8.01 × 102 (+)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102 (=)	8.00 × 102
	Std	6.96 × 10−13	8.22 × 10−1	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	4.32 × 10−2	0.00 × 100
	Rank	1	10	1	1	1	1	1	1	1	1
F9	Mean	1.64 × 103 (+)	1.75 × 103 (+)	9.85 × 102 (+)	9.80 × 102 (+)	9.37 × 102 (+)	9.14 × 102 (+)	1.13 × 103 (+)	1.47 × 103 (+)	9.78 × 102 (+)	9.06 × 102
	Std	2.87 × 102	3.48 × 102	4.13 × 101	5.53 × 101	4.34 × 101	8.31 × 100	1.07 × 102	4.30 × 102	5.73 × 101	3.31 × 100
	Rank	9	10	6	5	3	2	7	8	4	1
F10	Mean	1.01 × 103 (+)	1.73 × 104 (+)	1.02 × 103 (+)	1.00 × 103 (+)	1.00 × 103 (=)	1.14 × 103 (+)	1.00 × 103 (=)	1.00 × 103 (=)	1.01 × 103 (+)	1.00 × 103
	Std	5.28 × 100	1.85 × 104	2.60 × 101	6.06 × 10−1	5.55 × 100	3.31 × 102	1.07 × 100	1.72 × 100	1.02 × 101	4.51 × 10−1
	Rank	6	10	8	1	1	9	1	1	6	1
F11	Mean	1.43 × 105 (+)	1.54 × 107 (+)	1.36 × 103 (+)	1.33 × 103 (+)	1.14 × 103 (+)	1.35 × 103 (+)	1.21 × 103 (+)	1.33 × 103 (+)	3.36 × 103 (+)	1.10 × 103
	Std	9.15 × 104	2.49 × 107	7.63 × 101	2.50 × 101	9.15 × 101	2.48 × 102	1.15 × 102	9.94 × 101	6.40 × 103	6.58 × 10−2
	Rank	9	10	7	4	2	6	3	4	8	1
F12	Mean	1.25 × 103 (+)	2.19 × 105 (+)	1.20 × 103 (+)	1.20 × 103 (+)	1.21 × 103 (+)	1.22 × 103 (+)	1.20 × 103 (=)	1.21 × 103 (=)	1.21 × 103 (+)	1.20 × 103
	Std	2.36 × 101	2.05 × 105	3.92 × 10−1	3.54 × 10−1	4.45 × 100	1.86 × 101	4.91 × 10−1	1.63 × 101	2.97 × 100	1.40 × 10−1
	Rank	9	10	1	1	5	8	1	5	5	1
F13	Mean	1.35 × 103 (+)	7.80 × 103 (+)	1.32 × 103 (+)	1.32 × 103 (+)	1.32 × 103 (+)	1.32 × 103 (+)	1.32 × 103 (+)	1.32 × 103 (+)	1.32 × 103 (+)	1.30 × 103
	Std	8.86 × 100	9.30 × 103	5.00 × 10−1	6.47 × 10−1	7.83 × 100	5.52 × 100	7.20 × 100	6.85 × 100	7.67 × 100	2.95 × 100
	Rank	9	10	2	2	2	2	2	2	2	1
F14	Mean	1.52 × 103 (+)	6.28 × 105 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.40 × 103 (+)	1.51 × 103 (+)	1.40 × 103 (+)	1.41 × 103 (+)	1.41 × 103 (+)	1.40 × 103
	Std	1.07 × 102	8.55 × 105	2.10 × 10−3	2.59 × 10−1	2.44 × 10−1	1.92 × 102	3.77 × 10−1	1.22 × 101	2.54 × 101	4.29 × 10−2
	Rank	9	10	1	1	1	8	1	6	6	1
F15	Mean	1.50 × 103 (+)	3.52 × 103 (+)	1.76 × 103 (+)	1.61 × 103 (+)	1.52 × 103 (+)	1.57 × 103 (+)	1.59 × 103 (+)	1.63 × 103 (+)	1.57 × 103 (+)	1.50 × 103
	Std	2.78 × 10−1	3.97 × 103	1.99 × 102	1.85 × 102	7.34 × 101	1.12 × 102	1.55 × 102	2.25 × 102	1.48 × 102	1.68 × 10−2
	Rank	1	10	9	7	3	4	6	8	4	1
F16	Mean	1.64 × 103 (+)	2.33 × 103 (+)	1.73 × 103 (+)	1.69 × 103 (+)	1.62 × 103 (+)	1.64 × 103 (+)	1.62 × 103 (+)	1.64 × 103 (=)	1.64 × 103 (=)	1.61 × 103
	Std	8.47 × 100	1.05 × 103	1.14 × 102	1.06 × 102	5.66 × 100	6.23 × 101	1.86 × 100	7.06 × 101	7.09 × 101	8.57 × 100
	Rank	4	10	9	8	2	4	2	4	4	1
F17	Mean	1.74 × 103 (+)	2.81 × 104 (+)	1.72 × 103 (+)	1.72 × 103 (+)	1.71 × 103 (+)	1.72 × 103 (+)	1.70 × 103 (+)	1.72 × 103 (+)	1.72 × 103 (+)	1.70 × 103
	Std	5.48 × 100	1.20 × 104	5.99 × 100	7.84 × 10−2	1.03 × 101	7.74 × 100	6.30 × 100	6.16 × 100	6.76 × 100	2.02 × 10−2
	Rank	9	10	4	4	3	4	1	4	4	1
F18	Mean	1.80 × 103 (+)	1.43 × 104 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103 (+)	1.80 × 103
	Std	4.43 × 10−1	9.44 × 103	1.43 × 10−1	6.82 × 10−2	4.03 × 10−1	1.09 × 10−1	4.63 × 10−1	4.44 × 10−1	3.49 × 10−1	1.28 × 10−2
	Rank	1	10	1	1	1	1	1	1	1	1
F19	Mean	1.92 × 103 (+)	2.05 × 103 (+)	1.91 × 103 (+)	1.90 × 103 (+)	1.90 × 103 (+)	1.91 × 103 (+)	1.91 × 103 (+)	1.97 × 103 (+)	1.90 × 103 (+)	1.90 × 103
	Std	4.61 × 100	9.59 × 101	4.19 × 100	4.40 × 10−1	3.23 × 100	8.80 × 100	5.47 × 100	9.02 × 101	8.36 × 10−1	7.44 × 10−1
	Rank	8	10	5	1	1	5	5	9	1	1
F20	Mean	2.10 × 103 (+)	2.75 × 103 (+)	2.10 × 103 (=)	2.12 × 103 (=)	2.10 × 103 (=)	2.11 × 103 (=)	2.10 × 103 (=)	2.11 × 103 (=)	2.11 × 103 (=)	2.10 × 103
	Std	3.09 × 10−7	2.81 × 102	0.00 × 100	4.35 × 101	0.00 × 100	3.29 × 101	0.00 × 100	3.92 × 101	3.28 × 101	0.00 × 100
	Rank	1	10	1	9	1	6	1	6	6	1
F21	Mean	2.20 × 103 (+)	2.22 × 103 (+)	2.20 × 103 (=)	2.20 × 103 (=)	2.20 × 103 (=)	2.20 × 103 (+)	2.20 × 103 (=)	2.20 × 103 (+)	2.20 × 103 (+)	2.20 × 103
	Std	1.61 × 100	1.36 × 101	7.57 × 10−1	0.00 × 100	0.00 × 100	1.02 × 100	0.00 × 100	2.08 × 100	1.41 × 100	0.00 × 100
	Rank	1	10	1	1	1	1	1	1	1	1
F22	Mean	2.30 × 103 (+)	3.16 × 103 (+)	2.35 × 103 (=)	2.33 × 103 (=)	2.31 × 103 (=)	2.31 × 103 (=)	2.30 × 103 (=)	2.33 × 103 (=)	2.31 × 103 (=)	2.30 × 103
	Std	1.23 × 10−6	5.38 × 102	5.27 × 101	4.83 × 101	3.16 × 101	3.16 × 101	0.00 × 100	4.83 × 101	3.16 × 101	0.00 × 100
	Rank	1	10	9	7	4	4	1	7	4	1
F23	Mean	2.50 × 103 (+)	4.37 × 103 (+)	2.50 × 103 (+)	2.50 × 103 (+)	2.50 × 103 (+)	2.50 × 103 (+)	2.50 × 103 (+)	2.50 × 103 (+)	2.48 × 103 (+)	2.41 × 103
	Std	3.14 × 10−7	7.81 × 102	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	0.00 × 100	4.22 × 101	3.16 × 101
	Rank	3	10	3	3	3	3	3	3	2	1
F24	Mean	2.88 × 103 (+)	2.90 × 103 (+)	2.89 × 103 (+)	2.88 × 103 (+)	2.88 × 103 (=)	2.88 × 103 (+)	2.88 × 103 (=)	2.88 × 103 (+)	2.89 × 103 (+)	2.88 × 103
	Std	1.56 × 10−1	2.96 × 101	1.31 × 101	8.59 × 100	1.23 × 101	9.25 × 100	1.89 × 100	1.57 × 100	1.51 × 101	1.46 × 100
	Rank	1	10	8	1	1	1	1	1	8	1
F25	Mean	2.96 × 103 (+)	3.00 × 103 (+)	2.97 × 103 (+)	2.97 × 103 (+)	2.96 × 103 (+)	2.97 × 103 (+)	2.97 × 103 (+)	2.97 × 103 (+)	2.92 × 103 (+)	2.68 × 103
	Std	4.95 × 100	2.81 × 101	1.12 × 10−1	0.00 × 100	5.23 × 100	3.12 × 100	4.79 × 100	1.45 × 101	1.49 × 102	2.37 × 102
	Rank	3	10	5	5	3	5	5	5	2	1
F26	Mean	2.99 × 103 (=)	3.02 × 103 (+)	2.99 × 103 (=)	2.99 × 103 (=)	2.99 × 103 (=)	3.00 × 103 (+)	2.99 × 103 (=)	2.99 × 103 (=)	3.00 × 103 (+)	2.99 × 103
	Std	4.79 × 10−13	2.79 × 101	3.02 × 100	2.14 × 100	1.72 × 100	3.02 × 100	2.23 × 100	4.82 × 100	2.40 × 100	1.26 × 100
	Rank	1	10	1	1	1	8	1	1	8	1
F27	Mean	2.70 × 103 (+)	3.21 × 103 (+)	2.70 × 103 (=)	2.70 × 103 (=)	2.70 × 103 (=)	2.77 × 103 (=)	2.70 × 103 (=)	2.77 × 103 (=)	2.70 × 103 (=)	2.70 × 103
	Std	1.85 × 10−6	7.93 × 101	0.00 × 100	0.00 × 100	0.00 × 100	1.56 × 102	0.00 × 100	1.15 × 102	0.00 × 100	0.00 × 100
	Rank	1	10	1	1	1	8	1	8	1	1
F28	Mean	3.83 × 103 (+)	2.19 × 104 (+)	3.85 × 103 (+)	4.04 × 103 (+)	3.82 × 103 (+)	4.27 × 103 (+)	3.81 × 103 (+)	4.98 × 103 (+)	4.32 × 103 (+)	3.78 × 103
	Std	1.17 × 101	1.77 × 104	2.68 × 100	5.16 × 102	3.60 × 101	7.99 × 102	3.26 × 101	3.61 × 103	8.61 × 102	2.51 × 100
	Rank	4	10	5	6	3	7	2	9	8	1
F29	Mean	5.33 × 104 (+)	1.19 × 106 (+)	7.52 × 103 (+)	7.52 × 103 (+)	5.21 × 103 (+)	9.93 × 103 (+)	4.83 × 103 (+)	1.01 × 104 (+)	5.42 × 103 (+)	4.28 × 103
	Std	3.22 × 104	2.45 × 106	3.06 × 103	3.06 × 103	8.95 × 102	3.68 × 103	8.47 × 102	6.31 × 103	8.20 × 102	1.04 × 10−2
	Rank	9	10	5	5	3	7	2	8	4	1
Mean Rank		4.83	9.69	4.03	3.48	2.03	4.31	2.62	4.55	3.66	1.03
Wilcoxon’s rank sum	+/=/-	27/2/0	28/1/0	21/8/0	21/8/0	17/12/0	22/7/0	17/12/0	17/12/0	21/8/0	-/-/-

Figure 5. Convergence curves of representative functions on the 10-D CEC’2017.

The results show that PPOMSDE has a clear competitive advantage. It ranks first on all evaluation instances excluding F4. Regarding the unimodal landscapes F1 along with F2, PPOMSDE, TCDE, ISDE, ISHACDE, SHADE and L-SHADE all converge to the theoretical optimum. This verifies their strong local exploitation ability. For the multimodal functions, TCDE performs well and even outperforms PPOMSDE on F4. This is due to its triplet competition mechanism, which effectively maintains population diversity and adapts well to specific fitness landscapes. However, its single strategy mechanism lacks the ability to switch strategies dynamically in changing environments. As a result, it shows weaker search accuracy and robustness compared to PPOMSDE, as observed in F9 and the hybrid test problems. When tackling the composition functions, the merit of PPOMSDE becomes more significant. In particular, for F23, F25, F28, and F29, PPOMSDE shows a remarkable proficiency in evading premature convergence. In contrast, TCDE, ISDE, and ISHACDE achieve similar performance on functions such as F20–F22, but they exhibit a higher susceptibility to local stagnation. Overall, the results from both benchmark suites clearly demonstrate the competitive edge of PPOMSDE in addressing complex problems.

4.4. Parameter Analysis and Implementation Details

4.4.1. Parameter Sensitivity Analysis

To identify appropriate parameter settings for PPOMSDE, a parameter sensitivity analysis is conducted on its seven key hyperparameters. Four representative benchmark functions from the CEC’2014 suite, namely F1, F11, F22, and F29, are selected to represent the four function categories. The experiments are performed in both 10-D and 30-D search spaces. For each parameter setting, the average rank obtained over all test cases is recorded and used to determine the final parameter configuration.

The learning rate of the actor network is first evaluated using four candidate values: 0.0001, 0.0003, 0.0005, and 0.001. According to the results in Table 12, 0.0003 achieves the best average rank of 1.75 and is therefore selected as the final setting.

Table 12. Sensitivity of the actor network.

Func	Dim	0.0001	0.0003	0.0005	0.001
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	2.73 × 104	1.86 × 104	2.83 × 104	1.83 × 104
F11	10	1.23 × 103	1.14 × 103	1.16 × 103	1.21 × 103
	30	3.02 × 103	3.09 × 103	3.04 × 103	3.12 × 103
F22	10	2.21 × 103	2.22 × 103	2.21 × 103	2.22 × 103
	30	2.31 × 103	2.25 × 103	2.28 × 103	2.30 × 103
F29	10	6.66 × 106	4.62 × 106	6.10 × 106	6.09 × 106
	30	7.84 × 103	6.04 × 103	9.53 × 103	7.64 × 103
Avg.rank		2.625	1.75	2.5	2.375

For the critic network, four learning rates (0.0001, 0.0005, 0.001, and 0.003) are investigated. According to the results in Table 13, 0.001 achieves the best average rank of 1.375. As the learning rate deviates from 0.001, the optimization performance gradually declines. Therefore, 0.001 is adopted in the subsequent experiments.

Table 13. Sensitivity of the critic network.

Func	Dim	0.0001	0.0005	0.001	0.003
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	2.55 × 104	3.16 × 104	1.86 × 104	3.04 × 104
F11	10	1.23 × 103	1.17 × 103	1.14 × 103	1.18 × 103
	30	3.27 × 103	3.21 × 103	3.09 × 103	3.32 × 103
F22	10	2.23 × 103	2.21 × 103	2.22 × 103	2.21 × 103
	30	2.23 × 103	2.37 × 103	2.25 × 103	2.28 × 103
F29	10	5.71 × 106	5.45 × 106	4.62 × 106	5.72 × 106
	30	7.57 × 103	8.21 × 103	6.04 × 103	9.20 × 103
Avg.rank		2.5	2.375	1.375	3

The PPO clipping parameter $\mathcal{E}$ is examined using four candidate values (0.1, 0.15, 0.2, and 0.3). According to the results in Table 14, $\mathcal{E}=0.2$ achieves the best average rank of 1.625. Smaller clipping values limit the policy update and reduce learning efficiency, whereas larger values lead to less stable policy updates. As a result, the optimization performance gradually decreases as $\mathcal{E}$ deviates from 0.2. Therefore, $\mathcal{E}=0.2$ is adopted in all subsequent experiments.

Table 14. Sensitivity of the PPO clipping parameter.

Func	Dim	0.1	0.15	0.2	0.3
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	2.94 × 104	2.84 × 104	1.86 × 104	2.77 × 104
F11	10	1.22 × 103	1.23 × 103	1.14 × 103	1.17 × 103
	30	3.05 × 103	2.94 × 103	3.09 × 103	3.10 × 103
F22	10	2.21 × 103	2.22 × 103	2.22 × 103	2.21 × 103
	30	2.37 × 103	2.27 × 103	2.25 × 103	2.33 × 103
F29	10	5.88 × 106	5.73 × 106	4.62 × 106	5.78 × 106
	30	7.01 × 103	8.14 × 103	6.04 × 103	8.15 × 103
Avg.rank		2.625	2.375	1.625	2.625

The discount factor $\gamma$ is evaluated with values of 0.95, 0.99, 0.995, and 0.999. According to the results in Table 15, $\gamma =0.99$ achieves the best average rank (1.50), indicating an effective balance between immediate and long-term rewards.

Table 15. Sensitivity of the discount factor.

Func	Dim	0.95	0.99	0.995	0.999
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	1.87 × 104	1.86 × 104	1.90 × 104	2.19 × 104
F11	10	1.26 × 103	1.14 × 103	1.17 × 103	1.18 × 103
	30	3.12 × 103	3.09 × 103	3.04 × 103	3.11 × 103
F22	10	2.21 × 103	2.22 × 103	2.20 × 103	2.21 × 103
	30	2.38 × 103	2.25 × 103	2.30 × 103	2.30 × 103
F29	10	6.63 × 106	4.62 × 106	6.45 × 106	6.28 × 106
	30	8.20 × 103	6.04 × 103	8.16 × 103	8.83 × 103
Avg.rank		3	1.5	1.875	2.875

The number of PPO epochs $K$ is tested using four values (3, 5, 8, and 10). The best performance in Table 16 is obtained with $K=5$, which achieves an average rank of 1.625 and is therefore selected for network updates.

Table 16. Sensitivity of the PPO epochs.

Func	Dim	3	5	8	10
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	2.59 × 104	1.86 × 104	2.10 × 104	2.72 × 104
F11	10	1.24 × 103	1.14 × 103	1.19 × 103	1.28 × 103
	30	3.14 × 103	3.09 × 103	3.03 × 103	3.09 × 103
F22	10	2.21 × 103	2.22 × 103	2.21 × 103	2.21 × 103
	30	2.33 × 103	2.25 × 103	2.28 × 103	2.33 × 103
F29	10	5.96 × 106	4.62 × 106	5.70 × 106	6.81 × 106
	30	9.43 × 103	6.04 × 103	1.11 × 106	7.99 × 103
Avg.rank		2.75	1.625	2.125	2.75

For the external archive capacity coefficient $rac$, four candidate values (1.2, 1.4, 1.8, and 2.0) are considered. According to the results in Table 17, as the external archive capacity coefficient $rac$ deviates from 1.4, the optimization performance gradually declines. The setting of $rac=1.4$ achieves the best average rank of 1.75 and is adopted as the final archive capacity.

Table 17. Sensitivity of the external archive capacity coefficient $rac$.

Func	Dim	1.2	1.4	1.8	2.0
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	3.74 × 104	1.86 × 104	3.02 × 104	1.73 × 104
F11	10	1.23 × 103	1.14 × 103	1.18 × 103	1.23 × 103
	30	3.34 × 103	3.09 × 103	3.01 × 103	3.22 × 103
F22	10	2.21 × 103	2.22 × 103	2.21 × 103	2.21 × 103
	30	2.31 × 103	2.25 × 103	2.22 × 103	2.28 × 103
F29	10	6.42 × 106	4.62 × 106	5.78 × 106	5.85 × 106
	30	8.02 × 103	6.04 × 103	9.12 × 103	8.73 × 103
Avg.rank		3	1.75	2	2.5

The perturbation weight $W$ in the triplet mechanism is evaluated using four candidate values: 0.6, 0.7, 0.8, and 0.9. According to the results in Table 18, $W=0.8$ achieves the best performance with the lowest average rank of 1.375. When $W$ is smaller than 0.8, the local search is insufficient, leading to reduced optimization performance. In contrast, a larger value introduces stronger perturbations, which may reduce the search stability. Therefore, $W=0.8$ is selected for all subsequent experiments.

Table 18. Sensitivity of the perturbation weight.

Func	Dim	0.6	0.7	0.8	0.9
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	2.29 × 104	2.46 × 104	1.86 × 104	2.59 × 104
F11	10	1.23 × 103	1.23 × 103	1.14 × 103	1.27 × 103
	30	3.08 × 103	3.15 × 103	3.09 × 103	3.14 × 103
F22	10	2.21 × 103	2.21 × 103	2.22 × 103	2.23 × 103
	30	2.31 × 103	2.35 × 103	2.25 × 103	2.31 × 103
F29	10	5.78 × 106	6.33 × 106	4.62 × 106	6.43 × 106
	30	7.22 × 103	9.69 × 103	6.04 × 103	7.49 × 103
Avg.rank		2	2.75	1.375	3.125

According to the above experimental results, the final PPO parameter settings are listed in Table 19. The actor network maps the 11-dimensional state space to a five-dimensional discrete action probability distribution through two hidden layers (64 units each) with Tanh activation functions, followed by a softmax output layer. The critic network shares a similar architecture but maps to a single continuous value estimating the state value. Table 19 summarizes the key training configurations and network architectures utilized in this study.

Table 19. PPO hyperparameter settings.

Parameter	Value/Description
Actor Network Architecture	Linear(11, 64) $\to$ Tanh $\to$ Linear(64, 64) $\to$ Tanh $\to$ Linear(64, 5) $\to$ Softmax
Critic Network Architecture	Linear(11, 64) $\to$ Tanh $\to$ Linear(64, 64) $\to$ Tanh $\to$ Linear(64, 1)
Actor Learning Rate	0.0003
Critic Learning Rate	0.001
PPO Clipping Coefficient	0.2
Discount Factor	0.99
Optimization Epochs	5
Network Update Frequency	updated every 5 episodes
Training Episodes	100 episodes per function
Value Loss Coefficient	${\eta }_1=0.5$
Entropy Coefficient	${\eta }_2=0.01$
Numerical Stabilizer	$ϵ={10}^{-7}$

4.4.2. Implementation Details and Computational Complexity

During the offline training phase, the state feature extraction for each generation requires calculating the Euclidean distances between all individuals to assess local neighborhood density ${\mathbb{L}}_{i,t}^{near}$, which incurs a computational cost of $O\left(N^2·D\right)$. A forward pass of the actor network to select strategies for the entire population requires $O\left(N·W_{nn}\right)$. The fundamental evolutionary operations take $O\left(N·D\right)$. Therefore, the execution cost per generation is bounded by $O\left(N^2·D+N·W_{nn}\right)$.

Periodically, the multi-buffer framework sequentially calculates advantages and performs backpropagation for each individual over its collected trajectory. For $K$ optimization epochs, the backpropagation cost across the entire population is $O\left(N·K·T·W_{nn}\right)$. $W_{nn}$ represents the maximum number of weight parameters in the neural network, which dictates the complexity of a single network inference or backpropagation step. Aggregating these components over $T$ generations and $\varphi$ episodes, the total offline training time complexity is $O\left(\varphi ·T·N·\left(N·D+N·W_{nn}\right)\right)$.

During the testing phase, policy network updates and multi-buffer storage operations are entirely bypassed. The computational overhead primarily stems from state extraction, the triplet sorting mechanism, and actor network forward inference. Sorting the population by fitness to divide them into ${\mathcal{T}}_{best}$, ${\mathcal{T}}_{medium}$, and ${\mathcal{T}}_{worst}$ takes $O\left(N·logN\right)$. Consequently, for a complete optimization run consisting of $\mathrm{T}$ iterations, the testing time complexity is $O\left(T·N·\left(N·D+W_{nn}+logN\right)\right)$.

It is undeniable that the proposed PPOMSDE introduces additional algorithmic overhead compared to standard DE algorithms. However, the practical cost–benefit tradeoff strongly favors the proposed approach. The offline training phase, while computationally intensive, is a one-time sunk cost. Once trained, the RL agent operates purely in inference mode, meaning the online deployment overhead scales linearly with the network width and is easily handled by modern processors.

4.5. Analysis of Action Components

4.5.1. Validation of Mutation Strategies

This section evaluates the effectiveness of the mutation operators used in PPOMSDE. Four representative functions from the CEC’2014 benchmark suite, namely F1, F11, F22, and F29, are selected for evaluation in both the 10-D and 30-D search spaces. First, the five mutation operators introduced in Section 3.3 are removed one at a time to construct Plan 1 to Plan 5. Then, three additional mutation operators, namely DE/best/2, DE/best/1, and DE/rand-to-best, are added to the original action pool to construct Plan 6 to Plan 8. These eight comparative schemes are designed to verify whether the current action combination is appropriate. Table 20 lists all the algorithm components used in this study.

Table 20. Action sets evaluated in the experiments.

Label	Mathematical Expression
$action 1$	$V_{i,t}=X_{i,t}+F_i\cdot \left({X_{best,t}^p-X}_{i,t}\right)+F_i\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 2$	$V_{i,t}=X_{i,t}+F_i\cdot \left({X_{r_1,t}-X}_{i,t}\right)+F_i\cdot (X_{r_2,t}-X_{r_3,t})$
$action 3$	$V_{i,t}=X_{i,t}+F_i\cdot \left({X_{best,t}-X}_{i,t}\right)+F_i\cdot (X_{r^{*},t}-{\tilde{X}}_{r,t})$
$action 4$	$V_{i,t}=X_{i,t}+F_i\cdot (X_{r_1,t}-{\tilde{X}}_{r,t})$
$action 5$	$V_{i,t}=X_{r_1,t}+F_i\cdot (X_{r_2,t}-X_{r_3,t})$
$action 6$	$V_{i,t}=X_{best,t}+F_i\cdot (X_{r_1,t}-X_{r_2,t})+F_i\cdot (X_{r_3,t}-X_{r_4,t})$
$action 7$	$V_{i,t}=X_{best,t}+F_i\cdot (X_{r_1,t}-X_{r_2,t})$
$action 8$	$V_{i,t}=X_{r_1,t}+F_i\cdot (X_{best,t}-X_{r_1,t})+F_i\cdot (X_{r_2,t}-X_{r_3,t})$

Table 21 presents the experimental results of the nine schemes on the representative benchmark functions F1, F11, F22, and F29 in both 10-D and 30-D search spaces. Among all schemes, Plan 9 achieves the best overall performance with the optimal average rank of 1.0. Removing any of the five original mutation operators (Plan 1 to Plan 5) results in performance degradation on most benchmark functions. For example, on the 30-D F1 function, PPOMSDE obtains the best objective value of (1.86 × 10⁴), whereas the results of Plan 1 to Plan 5 range from 4.55 × 10⁴ to 6.32 × 10⁴. These results indicate that each mutation operator contributes to the optimization performance and that the five operators complement each other during the search process. Adding extra mutation operators (Plan 6 to Plan 8) also fails to improve the optimization performance. In particular, Plan 6, which introduces DE/best/2, achieves the worst average rank of 6.625. This suggests that an overly large action pool makes policy learning more difficult and reduces the accuracy of action selection. Overall, the experimental results demonstrate that the current combination of five mutation operators provides the most effective balance between exploration and exploitation, making it the most suitable action pool for PPOMSDE.

Table 21. Experimental results of different action pool configurations.

Func	Dim	Plan1	Plan2	Plan3	Plan4	Plan5	Plan6	Plan7	Plan8	PPOMSDE
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	4.66 × 104	6.32 × 104	6.15 × 104	5.67 × 104	4.55 × 104	4.00 × 104	5.81 × 104	4.87 × 104	1.86 × 104
F11	10	1.36 × 103	1.33 × 103	1.39 × 103	1.39 × 103	1.48 × 103	1.45 × 103	1.53 × 103	1.39 × 103	1.14 × 103
	30	3.63 × 103	3.90 × 103	3.48 × 103	3.90 × 103	3.95 × 103	3.98 × 103	4.25 × 103	3.80 × 103	3.09 × 103
F22	10	2.28 × 103	2.30 × 103	2.27 × 103	2.31 × 103	2.33 × 103	2.35 × 103	2.30 × 103	2.30 × 103	2.22 × 103
	30	2.45 × 103	2.46 × 103	2.49 × 103	2.44 × 103	2.46 × 103	2.50 × 103	2.45 × 103	2.51 × 103	2.25 × 103
F29	10	5.76 × 106	6.06 × 106	5.59 × 106	5.95 × 106	6.48 × 106	7.35 × 106	6.12 × 106	6.35 × 106	4.62 × 106
	30	8.42 × 105	7.31 × 103	9.14 × 103	8.02 × 103	7.77 × 103	1.12 × 106	7.90 × 103	6.80 × 103	6.04 × 103
Avg.rank		3.5	4.25	4.125	4.375	5.5	6.625	5.5	4.5	1

4.5.2. Probability Variations in Action Operators

This study records the probability variations in action selection on the 10-D CEC’2014, 30-D CEC’2014, and 10-D CEC’2017 benchmark suites to illustrate the evolution of action distributions over successive iterations.

As shown in Figure 6, the evolutionary dynamics of the action operators are analyzed across four different types of test functions on the 10-D CEC’2014. During the initial phase of the optimization process, Action 1 and Action 3 dominate the selection probability. This preference enables the algorithm to converge rapidly toward promising regions. In the middle and later stages of the search, the selection probability of Action 4 significantly increases. This strategic shift enhances the exploration capability of the population, effectively facilitating the escape from local optima. Furthermore, the specific probability distributions of operator selection exhibit distinct patterns that adapt dynamically to the varying landscape characteristics of different function types.

Figure 6. Action probability curves on the 10-D CEC’2014 benchmark suite.

Furthermore, Figure 7 illustrates the probability variations in action selection on the more challenging 30-D CEC’2014 benchmark suite. Compared to the 10-D scenarios, the action distributions in high-dimensional landscapes exhibit more complex and task-specific dynamic behaviors. For unimodal functions with relatively straightforward landscapes, PPOMSDE achieves rapid convergence and maintains strong search stability. Consequently, exploitation-oriented operators, particularly Action 1, quickly establish dominance, and the selection probabilities among the various operators remain remarkably stable throughout the optimization process. Conversely, when addressing highly complex or multimodal functions, the algorithm demonstrates a distinctive phased adaptation strategy. During the initial and intermediate stages, operators like Action 3 and Action 5 maintain higher selection probabilities to navigate the expansive search space and preserve population diversity. As the iterations progress into the later stages, the frequency of selecting exploratory operators, notably Action 4, significantly increases to help the population escape from severe high-dimensional local optima.

Figure 7. Action probability curves on the 30-D CEC’2014 benchmark suite.

According to the results in Figure 8, the action selection probabilities on the 10-D CEC’2017 benchmark suite reveal further distinct behaviors. Due to the shifted and highly rotated landscape features of the CEC’2017 functions, the advantageous operators undergo a noticeable shift. Unlike the patterns observed in the CEC’2014 suite, Action 1 gradually starts to dominate the selection probability during the middle stage of the optimization process. This indicates that after the initial global exploration, the population successfully locates promising sub-regions. Consequently, the reinforcement learning agent extensively utilizes the strong exploitation capability of Action 1 to accelerate convergence and perform rapid localized refinement. This adaptive shifting behavior further demonstrates the flexibility of PPOMSDE in dynamically adjusting its search strategies to accommodate entirely different evaluation environments.

Figure 8. Action probability curves on the 10-D CEC’2017 benchmark suite.

4.6. Ablation Study of the PPOMSDE

To verify the effectiveness of each component in PPOMSDE, ablation studies are conducted in this section. Three key components are evaluated separately, including the PPO-based action selection framework, the adaptive parameter mechanism, and the triplet mechanism. The experiments are performed on the 10-D and 30-D CEC’2014 benchmark suite using four representative functions (F1, F11, F22, and F29). The results are summarized in Table 22.

Table 22. Ablation study results.

Func	Dim	w/o Adaptive	w/o Triplet	w/o PPO	PPOMSDE
F1	10	1.00 × 102	1.00 × 102	1.00 × 102	1.00 × 102
	30	1.47 × 105	8.83 × 104	3.86 × 104	1.86 × 104
F11	10	1.69 × 103	2.33 × 103	1.65 × 103	1.14 × 103
	30	4.12 × 103	8.18 × 103	4.75 × 103	3.09 × 103
F22	10	2.41 × 103	2.44 × 103	2.44 × 103	2.22 × 103
	30	2.55 × 103	2.65 × 103	2.70 × 103	2.25 × 103
F29	10	7.49 × 106	6.86 × 106	7.67 × 106	4.62 × 106
	30	1.24 × 104	1.01 × 104	1.01 × 104	6.04 × 103
Avg.rank		2.625	2.75	2.625	1

As shown in Table 22, the complete PPOMSDE achieves the best overall performance among all variants. Removing any component leads to performance degradation, confirming that each module contributes to the optimization performance. Compared with the complete PPOMSDE, the w/o PPO variant achieves worse results on several benchmark functions, especially the composition function F29, demonstrating the effectiveness of the PPO-based action selection framework. Removing the adaptive parameter mechanism (w/o Adaptive) degrades the optimization performance on most benchmark functions, particularly in the 30-D experiments, demonstrating the importance of adaptive parameter adjustment. Similarly, removing the triplet mechanism (w/o Triplet) also leads to worse optimization results, especially on the hybrid and composition functions, confirming its contribution to maintaining population diversity and search effectiveness. Overall, the ablation results verify that the PPO-based action selection framework, the adaptive parameter mechanism, and the triplet mechanism all play important roles in the performance of PPOMSDE.

4.7. Implementation of the PPOMSDE for Agricultural UAV Path Planning

4.7.1. Scenario Settings

To validate the proposed framework in realistic agricultural applications, it is evaluated against a complex agricultural environment with undulating topography, utilizing a high-fidelity DEM representation. Five hillside threat scenarios exhibiting varying degrees of complexity are constructed (the configurations are detailed in Table 23 and Table 24). The purpose is to evaluate the algorithmic proficiency to generate optimal, terrain-adaptive routes across varying agricultural setups. All tests take place within a stationary setting, where in the locations of hazards and contextual parameters are fixed throughout the optimization phase. The total count of trajectory control points is established at $n=15$. It should be emphasized that agricultural UAV experiments in this study are simulation-based. Wind disturbances, sensing and localization uncertainties, communication delays, and detailed vehicle dynamic constraints are not explicitly considered in the experiments. Therefore, the presented results primarily evaluate the optimization capability of PPOMSDE under controlled conditions. According to the encoding method in Section 2.1, the problem dimension for all optimization algorithms, including PPOMSDE, is set to $D=30$. Furthermore, the specific parameters governing the cost functions are summarized in Table 25. Every method undergoes 30 independent trial runs for each scenario, bounded by a computational budget of 50,000 evaluations alongside a consistent swarm size of 50. The findings are summarized utilizing $\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}$, $\mathrm{S}\mathrm{T}\mathrm{D}$, and $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}$, as shown in Table 26.

Table 23. Threat information.

Threat Number	Coordinate	${\mathcal{R}}_{\mathit{g}}$	Threat Number	Coordinate	${\mathcal{R}}_{\mathit{g}}$
1	(400,500,200)	50	9	(500,400,200)	50
2	(500,350,200)	40	10	(300,250,200)	30
3	(600,700,200)	40	11	(650,700,150)	50
4	(800,400,200)	50	12	(500,200,200)	40
5	(400,200,200)	40	13	(800,500,200)	40
6	(700,400,200)	50	14	(700,300,150)	50
7	(400,550,200)	50	15	(600,700,200)	30
8	(600,200,200)	40

Table 24. Threat configuration for the scenario.

Scenario Number	Structural Setup of Scenario Obstacles
1	1,2,3,4
2	1,2,5,6
3	4,7,8,9,10
4	4,7,8,9,10,11
5	7,9,10,12,13,14,15

Table 25. Cost function and simulation parameters.

Parameter	Value
UAV diameter	$\mathcal{d}=10$
Threat buffer width	$\mathcal{S}=20$
Ideal relative altitude	$h_{target}=40$
Minimum altitude	$h_{min}=10$
Maximum altitude	$h_{max}=100$
Weighting coefficients	$a_1=0.2,$ $a_2=0.6,$ $a_3=0.1,$ $a_4=0.1$
Collision penalty	${\varsigma }^1={10}^6$
Boundary penalty	${\varsigma }^2=1000$
Gradient penalty coefficient	$\xi =1000$
Start point	(200,100,200)
Target point	(800,800,200)
Search bounds of y	$\left[100, 800\right]$
Search bounds of z	$\left[0, 400\right]$

Table 26. The result of UAV agricultural path planning.

Func	Index	DE	PSO	SHADE	L-SHADE	TCDE	HCDE	ISDE	ESADE	ISHACDE	PPOMSDE
Scenario 1	Mean	3.65 × 103 (+)	4.99 × 104 (+)	2.25 × 102 (+)	2.87 × 102 (=)	4.45 × 102 (+)	7.00 × 102 (+)	2.20 × 102 (=)	9.09 × 102 (+)	4.80 × 102 (+)	2.14 × 102
	Std	1.86 × 103	1.52 × 105	1.64 × 101	3.55 × 102	5.86 × 102	9.65 × 102	1.14 × 101	1.21 × 103	6.76 × 102	5.51 × 100
	Rank	9	10	3	4	5	7	2	8	6	1
Scenario 2	Mean	2.20 × 103 (+)	7.26 × 104 (+)	3.07 × 102 (+)	2.50 × 102 (+)	7.06 × 102 (+)	7.63 × 102 (+)	2.46 × 102 (+)	4.09 × 104 (+)	3.03 × 102 (=)	2.27 × 102
	Std	1.20 × 103	1.82 × 105	3.69 × 102	4.30 × 101	8.18 × 102	1.08 × 103	3.58 × 101	1.52 × 105	3.50 × 102	8.09 × 100
	Rank	8	10	5	3	6	7	2	9	4	1
Scenario 3	Mean	2.26 × 103 (+)	1.09 × 105 (+)	2.95 × 102 (+)	3.59 × 102 (+)	3.66 × 102 (+)	7.74 × 102 (+)	2.99 × 102 (+)	6.03 × 104 (+)	4.26 × 102 (+)	2.23 × 102
	Std	1.26 × 103	2.25 × 105	3.50 × 102	4.84 × 102	5.02 × 102	8.62 × 102	3.49 × 102	1.83 × 105	5.92 × 102	7.97 × 100
	Rank	8	10	2	4	5	7	3	9	6	1
Scenario 4	Mean	2.40 × 103 (+)	2.75 × 105 (+)	2.96 × 102 (=)	2.30 × 102 (=)	6.31 × 102 (+)	5.29 × 102 (+)	3.64 × 102 (+)	2.10 × 104 (+)	2.98 × 102 (=)	2.26 × 102
	Std	1.41 × 103	3.04 × 105	3.61 × 102	1.40 × 101	8.01 × 102	6.74 × 102	4.84 × 102	1.09 × 105	3.64 × 102	9.41 × 100
	Rank	8	10	3	2	7	6	5	9	4	1
Scenario 5	Mean	1.80 × 103 (+)	2.17 × 105 (+)	3.58 × 102 (=)	3.53 × 102 (=)	4.86 × 102 (+)	2.08 × 104 (+)	2.32 × 102 (=)	2.08 × 104 (+)	3.52 × 102 (=)	2.59 × 102
	Std	8.01 × 102	2.83 × 105	4.85 × 102	4.88 × 102	6.69 × 102	1.09 × 105	1.76 × 101	1.09 × 105	4.93 × 102	1.53 × 102
	Rank	7	10	5	4	6	8	1	8	3	2
Mean Rank		8.00	10.00	3.60	3.40	5.80	7.00	2.60	8.60	4.60	1.20
Wilcoxon’s rank sum	+/=/-	5/0/0	5/0/0	3/2/0	2/3/0	5/0/0	5/0/0	3/2/0	5/0/0	2/3/0	-/-/-

4.7.2. Comparison with Advanced Algorithms

According to Table 26, PPOMSDE achieves the best average fitness values among all compared baseline algorithms in the first four test scenarios. In Scenario 5, although PPOMSDE is slightly outperformed by ISDE but remains highly competitive. Furthermore, the statistical results reveal that PPOMSDE consistently maintains its standard deviation at an extremely low level, ranging between 5.51 × 10⁰ and 1.53 × 10². This highlights its outstanding robustness and search stability over multiple independent runs, sharply contrasting with the severe performance degradation seen in basic algorithms like PSO. Notably, among the compared methods, ISDE and L-SHADE show strong competitiveness on specific tasks, yet PPOMSDE demonstrates the most stable comprehensive performance. In essence, the reinforcement learning framework endows PPOMSDE with a self-organizing search trajectory, ensuring highly reliable optimization performance irrespective of problem complexity.

Unlike general UAV path planning, complex agricultural applications require the flight altitude to maintain a strict, constant distance from the uneven terrain. This terrain-following requirement makes path planning more challenging. As shown in Figure 9 and Figure 10, advanced algorithms can generate feasible trajectories, whereas basic algorithms such as DE and PSO fail to find feasible paths under these conditions. Figure 9 shows that PPOMSDE consistently generates high-quality feasible paths in all five test environments and demonstrates superior performance relative to the compared algorithms. Figure 10 further presents the corresponding flight paths. Compared with the paths generated by the other algorithms, the routes planned by PPOMSDE are smoother and shorter. They also pass safely through narrow corridors between multiple hillside threat regions, resulting in more efficient trajectories. Both the quantitative results and the path planning examples demonstrate the effectiveness of PPOMSDE for complex agricultural UAV path planning. These results confirm that the proposed method has strong potential for 3D path planning in smart agriculture.

Figure 9. Overhead perspective of the generated flight routes.

Figure 10. Comparison plot of generated three-dimensional flight routes.

The reported average cost reduction was calculated using four advanced DE variants: ISDE, L-SHADE, SHADE, and ISHACDE. For each comparison algorithm a, the reduction in scenario s was calculated as $R_{a,s}=\left({\overline{F}}_{a,s}-{\overline{F}}_{PPOMSDE,s}\right)/{\overline{F}}_{a,s}\times 100\%$, where ${\overline{F}}_{a,s}$ denotes the mean cost over 30 independent runs. The reduction percentages were first averaged across the five scenarios for each algorithm and then averaged across the four algorithms. The corresponding reductions were 12.43%, 20.18%, 21.33%, and 35.75%, yielding an overall average reduction of 22.42%.

5. Conclusions

This paper presents a multi-strategy differential evolution algorithm based on a triplet mechanism and PPO, named PPOMSDE. By introducing the PPO reinforcement learning framework, the agent perceives a multi-dimensional state space and guides individuals to select optimal strategies from a designed action pool. An independent multi-buffer is adopted to update each individual’s experience separately. In addition, a parameter adaptation mechanism is integrated, and the triplet mechanism is introduced in the deployment phase. It assigns targeted control parameters to individuals of different quality (best, medium, and worst) and refines mutation actions. This design better guides the evolutionary process and improves the equilibrium coordinating global discovery with localized fine-tuning. On the standard benchmark suites, PPOMSDE achieved the lowest average performance ranks of 1.39 for the combined 10-D and 30-D CEC’2014 instances and 1.03 for the 10-D CEC’2017 instances. For practical agricultural UAV applications, the algorithm demonstrated exceptional capability in generating safe, smooth, and precise terrain-following trajectories across diverse hillside scenarios. Notably, PPOMSDE established new quantitative performance bounds by reducing the overall path planning cost by an average of 22.42% compared with ISDE, L-SHADE, SHADE, and ISHACDE.

Future research will investigate more advanced reinforcement learning algorithms to further improve policy learning stability in complex optimization problems [47]. Furthermore, the proposed framework will be extended toward an online actor–critic reinforcement learning architecture to enable continuous policy updates and adaptive path planning in response to dynamic and non-stationary agricultural environments [48]. More flexible path-encoding strategies based on waypoint-location optimization will be investigated to improve maneuverability in complex terrain. More realistic agricultural operating conditions will be incorporated into the simulation environment, including wind disturbances, sensing and localization uncertainty, communication delays, and practical UAV dynamic constraints. Since the agricultural UAV experiments presented in this study are entirely simulation-based, field experiments using real UAV platforms on farmland remain necessary. These experiments will be conducted to evaluate the robustness, real-time feasibility, and engineering applicability of the proposed method under uncertain environmental conditions. In addition, the deployment of the framework on edge-computing platforms will be investigated to support real-time applications. Reinforcement learning methods integrated with fuzzy logic will be explored to improve decision-making under uncertain agricultural environments.