Q-Learning-Based Multi-Strategy Topology Particle Swarm Optimization Algorithm

Abstract

In response to the issues of premature convergence and insufficient parameter control in Particle Swarm Optimization (PSO) for high-dimensional complex optimization problems, this paper proposes a Multi-Strategy Topological Particle Swarm Optimization algorithm (MSTPSO). The method builds upon a reinforcement learning-driven topological switching framework, where Q-learning dynamically selects among fully informed topology, small-world topology, and exemplar-set topology to achieve an adaptive balance between global exploration and local exploitation. Furthermore, the algorithm integrates differential evolution perturbations and a global optimal restart strategy based on stagnation detection, together with a dual-layer experience replay mechanism to enhance population diversity at multiple levels and strengthen the ability to escape local optima. Experimental results on 29 CEC2017 benchmark functions, compared against various PSO variants and other advanced evolutionary algorithms, show that MSTPSO achieves superior fitness performance and exhibits stronger stability on high-dimensional and complex functions. Ablation studies further validate the critical contribution of the Q-learning-based multi-topology control and stagnation detection mechanisms to performance improvement. Overall, MSTPSO demonstrates significant advantages in convergence accuracy and global search capability.

1. Introduction

Optimization techniques play a central role in fields such as engineering design, machine learning, artificial intelligence, and operations research. From parameter optimization of complex mechanical structures to hyperparameter tuning of deep learning models, efficient optimization algorithms are the key to solving practical problems. Current mainstream optimization algorithms include differential evolution (DE) [1,2], Genetic Algorithm (GA) [3], Bayesian Optimization (BO) [4], and Artificial Bee Colony (ABC) algorithm [5], among which Particle Swarm Optimization (PSO) has been widely applied due to its simple structure, easy parameter adjustment, and convenient implementation [6,7,8]. Since its introduction by Kennedy and Eberhart in 1995, PSO has simulated the social behavior of biological populations to perform optimization search and has spawned numerous improved variants [6]. However, traditional PSO still suffers from significant limitations: most variants are effective only for specific problem types, PSO tends to fall into local optima in high-dimensional multimodal functions, and fixed topology structures limit population diversity [9,10]. To overcome these bottlenecks, researchers have systematically improved PSO in several directions.

First, hybrid strategies integrate the strengths of different algorithms to enhance search performance. Recent studies include GWOPSO, which combines Grey Wolf Optimization (GWO) with PSO by leveraging PSO’s population collaboration mechanism and GWO’s hierarchical hunting strategy [11]; ECCSPSOA, which integrates chaotic crow search with PSO to improve the diversity of initial solutions for feature selection problems [12]; and HGSPSO, which merges gravitational search with PSO to optimize particle interactions via a gravity model [13]. In particular, hybridization of PSO and differential evolution (DE) has become a research hotspot: heterogeneous DE-PSO improves exploitation through local search [14], while Mutual Learning DE-PSO enhances global exploration by knowledge sharing among swarms [15]. These hybrid approaches have demonstrated significant advantages in complex function optimization tasks, as validated on CEC2017 benchmarks [16,17].

Second, learning strategies focus on building efficient guidance mechanisms. Liu et al. proposed simplified PSO models with cognitive, social, and temporal-hierarchical strategies (THSPSO), which dynamically update learning patterns to improve optimization efficiency and significantly shorten convergence time compared with standard PSO in complex function optimization [10]. Huang et al. introduced the LRMODE algorithm, which incorporates ruggedness analysis of the fitness landscape to refine particle learning directions [18]. Lu et al. employed reinforcement learning to guide particle behavior, integrating historical successful experiences into wastewater treatment control optimization [19]. Zhao et al. proposed the mean-hierarchy PSO (MHPSO), which stratifies particles according to population mean fitness and uses stochastic elite neighbor selection to maintain diversity [20]. Wang et al. developed the SCDLPSO algorithm with self-correction and dimension-wise learning ability to strengthen links between individual and global bests for solving complex multimodal problems [21]. Other studies have enhanced adaptability at different stages by introducing adaptive inertia weights and learning factors [22,23]. These methods dynamically adjust learning exemplars to strengthen PSO’s adaptability to complex problems.

Third, neighborhood topology innovations are critical to improving PSO performance. While traditional global topology risks premature convergence and local topology converges slowly [24], dynamic topologies provide balance. For example, SWD-PSO employs small-world networks with dense short-range links and random long-range connections to accelerate global information exchange [25]. DNSPSO adapts neighborhood range via dynamic switching mechanisms [26], while MNPSO applies distance-adaptive neighborhoods to improve nonlinear equation solving [27]. Pyramid PSO (PPSO) enhances population diversity through competition-cooperation strategies and multi-behavior learning [28]. Liang et al. proposed NRLPSO, a neighborhood differential mutation PSO guided by reinforcement learning, with a dynamic oscillatory inertia weight to adapt particle motion [29]. Jiang et al. developed HPSO-TS, a hybrid PSO with time-varying topology and search perturbation, which uses K-medoids clustering to dynamically divide the swarm into heterogeneous subgroups, facilitating intra-group information flow and global-local search transitions [30]. Zeng et al. further proposed a dynamic-neighborhood switching PSO (DNSPSO) with a novel velocity update rule and switching strategy, showing excellent performance on multimodal optimization problems [26]. These topology improvements have greatly enhanced search capability in high-dimensional spaces.

Finally, adaptive parameter control in PSO has evolved from heuristic rules to intelligent learning strategies. Tatsis et al. introduced an online adaptive parameter method that combines performance estimation with gradient search, significantly improving robustness in complex environments [31]. Chen et al. developed MACPSO, which applies multiple Actor-Critic networks to optimize parameters in continuous space, balancing global exploration and local exploitation [16]. Liu et al. proposed QLPSO, where Q-learning trains parameters and Q-tables guide particle actions based on states, improving search efficiency in complex solution spaces [32]. A deep reinforcement learning-based PSO (DRLPSO) [33] employs neural networks to learn state-action mappings and dynamically adjust parameters for high-dimensional tasks. Hamad et al. proposed QLESCA, which introduces Q-learning into the sine cosine algorithm (SCA) to achieve adaptive tuning of key parameters, significantly improving convergence speed and solution accuracy. Subsequent studies applied QLESCA to high-dimensional COVID-19 feature selection, verifying its potential in high-dimensional problems; however, the overall optimization strategy remains primarily parameter-focused [34,35]. RLPSO algorithm employs a reinforcement learning strategy to adaptively control the hierarchical structure of the swarm, adjusting search modes across different levels to balance exploration and exploitation [36]. Most existing RL-PSO methods still concentrate on parameter-level adaptation. Although this improves convergence performance to some extent, little work has addressed adaptive control of the swarm topology. As topology determines the interaction pattern among particles, it plays a crucial role in maintaining population diversity and balancing exploration and exploitation. Therefore, introducing reinforcement learning for dynamic topology selection remains an area requiring further investigation. Based on this background, the main contributions of this paper are as follows:

• A reinforcement learning-based topology switching strategy is proposed, enabling particles to dynamically select among FIPS, small-world, and exemplar-set topologies to balance global exploration and local exploitation.
• A dual-layer Q-learning experience replay mechanism is designed, integrating short-term and long-term memories to stabilize parameter control and improve learning efficiency.
• A stagnation detection mechanism is constructed and combined with differential evolution perturbations and a global restart strategy to enhance population diversity and improve the ability to escape local optima.

Compared with existing RL-PSO methods that mainly focus on parameter adaptation, the proposed MSTPSO applies reinforcement learning to adaptive control of swarm topology, optimizing the information exchange pattern at the structural level and thereby improving convergence efficiency and robustness. The remainder of this paper is organized as follows: Section 2 reviews PSO and reinforcement learning–related theories. Section 3 describes the design and implementation of MSTPSO. Section 4 presents performance validation on the CEC2017 benchmark and ablation analysis of each component. Section 5 concludes the study and outlines future research directions.

2. Background Information

2.1. Particle Swarm Optimization

The Particle Swarm Optimization (PSO) algorithm was proposed by Kennedy and Eberhart in 1995 [7], inspired by the foraging behavior of bird flocks. In PSO, each individual in the population is abstracted as a “particle”, and each particle’s position corresponds to a potential solution to the optimization problem. Through cooperation and information sharing within the swarm, global optimal search can be achieved. In a standard PSO, a particle swarm consisting of N particles is described by two vector quantities: velocity vector v and position vector x. For a D-dimensional search space, the velocity and position of the i-th particle are defined as

$$v_i=(v_{i1},v_{i2},\dots ,v_{iD})\in D,i=1, 2, \dots , N.$$

(1)

$$x_i=(x_{i1},x_{i2},\dots ,x_{iD})\in D,i=1, 2, \dots , N.$$

(2)

Equation (1) represents the velocity vector of the particle in D-dimensional space, while Equation (2) denotes its position in the search space. Each particle has a memory ability and can store the best position it has visited so far, referred to as the individual best position $p{b}_i\left(h\right)$. Among all individual bests $p{b}_i\left(h\right)$, the best one is selected as the global best position $gb\left(h\right)$. The velocity and position of particles are updated iteratively using the following equations:

$$v_i(h+1)={\omega v}_i\left(h\right)+c_1{r}_1(p{b}_i\left(h\right)-x_i\left(h\right))+c_2{r}_2\left(gb\left(h\right)-x_i\left(h\right)\right).$$

(3)

$$x_i(h+1)=x_i\left(h\right)+v_i(h+1).$$

(4)

In the above: $\omega$ is the inertia weight, which maintains the particle’s previous momentum and balances exploration and exploitation; $c_1$ and $c_2$ are cognitive and social acceleration coefficients, representing the particle’s self-learning and swarm-learning abilities, respectively; $r_1,r_2\sim [0, 1]$ are random numbers uniformly distributed within $[0, 1]$, introducing stochasticity to avoid premature convergence; h is the iteration index; $v_i\left(h\right)$ and $x_i\left(h\right)$ represent the velocity and position of the i-th particle at the h-th iteration, respectively. The PSO mechanism is driven by a dual principle of “individual experience + group cooperation”, enabling particles to update their positions by considering both their historical best positions $p{b}_i\left(h\right)$ and the global best position $gb\left(h\right)$, thereby forming a self-organizing search process. This mechanism allows PSO to dynamically balance global exploration and local exploitation through a simple structure and parameter configuration. However, traditional PSO often uses fixed parameter settings, which limits its adaptability. This paper designs an adaptive multi-strategy inertia weight adjustment mechanism based on particle state feedback to achieve dynamic balance. Moreover, coefficients such as $c_1$ and $c_2$ are linearly adjusted to enhance convergence capability and address the parameter sensitivity issue in standard PSO.

2.2. Reinforcement Learning

Reinforcement learning (RL) is a key branch of machine learning that focuses on enabling intelligent agents to make sequential decisions in dynamic environments. Its core objective is to learn optimal or near-optimal policies through interactions with the environment to maximize expected cumulative long-term rewards. This process can be modeled as a Markov Decision Process (MDP), formally defined as a quadruple $(S,A,P,R)$ [37], where S represents the set of states, describing all possible situations the agent may encounter; A denotes the action set available to the agent; $P:S\times A\times S\to [0, 1]$ is the state transition probability function, describing the probability of transitioning from one state to another given an action; $R:S\times A\to R$ is the reward function, quantifying the immediate feedback received by the agent for performing an action in a given state.

Through continuous interaction with the environment, the agent receives a sequence of observations and generates an experience trajectory $\{s,a,r,s^{\prime },\dots \}$, where the cumulative discounted $R_t$ return at time t is defined as

$$R_t=\sum _{i=t}^T{\gamma }^{i-t}{r}_i,\gamma \in [0, 1],$$

(5)

where $\gamma$ is the discount factor, used to balance the importance between immediate and future rewards. The core objective of RL is to find the optimal policy ${\pi }^{*}\left(a\right|s)$ that maximizes the expected cumulative return

$${\pi }^{*}\left(a\right|s)=arg\underset{a}{max}\mathbb{E}\left[R_t\right|s,a].$$

(6)

To achieve this, RL commonly defines a state-value function $V^{*}\left(s\right)$ and an action-value function $Q^{*}(s,a)$, as follows:

$$V^{*}\left(s\right)={\mathbb{E}}_t[r_{t+1}+\gamma V^{*}\left(s_{t+1}\right)|s_t=s].$$

(7)

$$Q^{*}(s,a)={\mathbb{E}}_t[r_{t+1}+\gamma Q^{*}(s_{t+1},a_{t+1})|s_t=s,a_t=a].$$

(8)

When the optimal policy ${\pi }^{*}$ is achieved, the action-value function $Q^{*}(s,a)$ satisfies the Bellman optimality condition, which forms the theoretical foundation of the Q-learning algorithm [38].

2.3. Q-Learning

Q-learning, a representative algorithm in reinforcement learning (RL), aims to learn the optimal action-selection policy in an environment by maximizing the expected cumulative reward. RL typically models the learning process through continuous interactions between the agent and the environment, based on three key elements: state, action, and reward. The theoretical basis of Q-learning was proposed by Watkins in 1989 [39], which focuses on estimating the value (Q-value) of performing an action in a specific state without requiring a model of the environment’s dynamics. Q-learning uses an iterative update rule to approximate the optimal Q-value function, with the update equation given as

$$Q(s_{t+1},a_{t+1},i)=(1-\alpha )Q(s_t,a_t,i)+\alpha \left[R(s_t,a_t)+\gamma \underset{a}{max}Q(s_{t+1},a,i)\right],$$

(9)

where $\alpha \in [0, 1]$ is the learning rate, controlling how much newly acquired information overrides the old one; $\gamma \in [0, 1]$ is the discount factor, controlling the importance of future rewards; $r_{t+1}$ is the immediate reward received after performing action $a_t$ in state $s_t$; ${max}_{a}Q(s_{t+1},a,i)$ represents the maximum estimated future return from the next state $s_{t+1}$, assuming the best action is taken.

This update process is an off-policy method that uses the greedy policy to update Q-values while allowing exploratory behavior during the learning process through strategies such as $ϵ$-greedy selection. The algorithm bootstraps its Q-value estimations, progressively improving policy performance through value iteration. The core architecture of the Q-learning model is illustrated in Figure 1, and its workflow can be summarized as a closed-loop cycle of state perception → action selection → environment interaction → reward acquisition → Q-table update. This mechanism aligns well with the dynamic parameter adjustment requirements of Particle Swarm Optimization (PSO); by defining the search states of PSO (such as particle distribution and fitness variation) as RL states and defining parameter combinations as RL actions, Q-learning enables online optimization of PSO parameters through its adaptive updating capability [40].

3. Multi-Strategy Topology PSO

This paper proposes a multi-topology Particle Swarm Optimization algorithm integrated with reinforcement learning, and its workflow is illustrated in Figure 2. In each iteration, the Q-learning agent selects the topology and parameter configuration according to the current state, after which particles update their velocities and positions and perform fitness evaluation. Stagnation detection is then carried out: if swarm stagnation is detected, a partial restart based on the global best is triggered, while individual stagnation activates a differential evolution–based perturbation. The Q-table is updated in every iteration using immediate feedback, whereas prioritized experience replay from the dual-layer experience pool is triggered periodically to further optimize the strategy.

3.1. State

Under the Q-learning framework, the state space needs to simultaneously reflect both the convergence trend of individual particles and the diversity of the entire swarm, thereby characterizing the dynamic balance between exploration and exploitation. In this paper, the state is designed with two complementary components: the fitness improvement state and the diversity shrinkage state. For particle i at generation t, the change in fitness is defined as

$$\Delta f_i^t=f_i^{t-1}-f_i^t,$$

(10)

where $f_i^t$ denotes the fitness of particle iii at generation t. According to the value of $\Delta f_i^t$, the base state of particle i, denoted as $\mathit{base}\_{\mathit{state}}_i$, is categorized as

$$base\_stat{e}_i=\left\{\begin{array}{cc}1,\hfill & \Delta f_i^t<-{10}^{-8}\hfill \\ 2,\hfill & -{10}^{-8}<\Delta f_i^t<0\hfill \\ 3,\hfill & \Delta f_i^t=0\hfill \\ 4,\hfill & \Delta f_i^t>0\hfill \end{array}\right.,$$

(11)

where the threshold ${10}^{-8}$ is commonly used in swarm intelligence optimization to distinguish between significant and non-significant improvements, ensuring the stability and robustness of numerical judgment. The states correspond respectively to significant degradation, slight degradation, no change, and improvement. To quantify the diversity of the swarm distribution, Shannon entropy is introduced,

$$H_i^t=-\sum _{i=1}^M{p}_i·{log}_2\left(p_i\right),$$

(12)

where M = 10 denotes the number of partitions of the search space, and represents the proportion of particles falling into the j-th interval. According to the entropy value, the diversity level is categorized as

$$s_i=({entropy\_state}_i-1)\times 4+{base\_state}_i.$$

(13)

These correspond to low, medium, and high diversity, respectively. Shannon entropy has been widely applied in PSO to measure the distribution and convergence behavior of the swarm [32].

3.2. Action Design

This study selects three representative topologies as reinforcement learning actions: Fully-Informed Particle Swarm Topology (FIPS) [41], small-world topology [25], and exemplar-set topology [42]. Figure 3 shows the three topological structures adopted in this work. These topologies have been demonstrated to significantly enhance global searchability and convergence efficiency in multimodal and complex optimization problems. Their corresponding velocity update mechanisms are described below.

Under the FIPS topology, the velocity of particle i is influenced by all its neighborhood members with additional weights. Its update equation is

$$v_{i,d}^{t+1}=\omega v_{i,d}+\varphi \sum _{j\in N_i}{r}_{j,d}\left(p_{j,d}^{best}-x_{i,d}^t\right),$$

(14)

where $N_i$ denotes the set of neighbors of particle i, $\varphi$ is an additional weight factor, and $r_{j,d}\sim U(0, 1)$ is a random weight. The inertia weight $\omega$ is linearly decreased,

$$\omega ={\omega }_{max}-({\omega }_{max}-{\omega }_{min}){\displaystyle \frac{t}{T}}$$

(15)

where T is the maximum number of iterations, and t is the current iteration index. This design balances global exploration in early iterations and local exploitation in later stages. Under the small-world topology, particles mainly interact with nearby neighbors while introducing long-distance links with a small probability. Its velocity update is

$$v_{i,d}^{t+1}=\omega v_{i,d}^t+c_1{r}_1\left(p_{i,d}^{\mathrm{best}}-x_{i,d}^t\right)+c_2{r}_2\left(g_{s_i,d}^t-x_{i,d}^t\right),$$

(16)

where $g_{s_i,d}^t$ is the best solution among neighbors, and the learning factors $c_1$, $c_2$ vary linearly,

$$c_1=c_{1,max}-(c_{1,max}-c_{1,min}){\displaystyle \frac{t}{T}}.$$

(17)

$$c_2=c_{2,max}-(c_{2,max}-c_{2,min}){\displaystyle \frac{t}{T}}.$$

(18)

Under the exemplar-set topology, each particle dynamically selects exemplars from the constructed exemplar set as reference points to enhance diversity and effectively avoid premature convergence. Its velocity update is

$$v_{i,d}^{t+1}=\omega v_{i,d}^t+c_1{r}_1(p_{i,d}^{best}-x_{i,d}^t)+c_2{r}_2(p_{sbes{t}_i,d}^{best}-x_{i,d}^t),$$

(19)

where $p_{\mathrm{best},d}^{\mathrm{best}}$ denotes the best individual in the exemplar set. This strategy introduces diverse references, strengthening swarm diversity and improving the ability to escape local optima.

In summary, combining the three topological structures with the dynamic adjustment mechanism of $\omega$, $c_1$ and $c_2$ provides Q-learning with a set of structural actions, achieving a balance between exploration and exploitation. The FIPS topology strengthens fully informed aggregation to improve global convergence; the small-world topology balances information transmission and convergence speed, enhancing search robustness; and the exemplar-set topology maintains swarm diversity through diversified reference individuals, reducing the risk of premature convergence.

3.3. Reward Function

In the MSTPSO algorithm, the reward function is designed to quantify the search performance after executing the current strategy, and it serves as the updating signal for the Q-learning module. The design of the reward comprehensively considers both the improvement of particle fitness and the maintenance of population diversity, thereby guiding the search in the desired direction and avoiding premature convergence and local stagnation. Specifically, the reward of a particle in each generation consists of two parts: (1) the fitness improvement, which reflects the enhancement of a particle’s solution quality, and (2) the diversity maintenance, which encourages exploration behavior. The reward is defined as follows:

$$r_i^t=w_1·\Delta f_i^t+w_2·H_i^t,$$

(20)

where $\Delta f_i^t=f_i^{t-1}-f_i^t$ represents the fitness improvement between the current solution and the historical best of particle i, and $H_i^t$ denotes the diversity contribution of the swarm. In terms of weight setting, this study refers to the related work of Liu et al. [32], in which a reinforcement learning-based PSO parameter control adopts a uniform weighting strategy to balance different optimization objectives. Accordingly, this paper sets $w_1=w_2=0.5$, thereby establishing a stable balance between fitness improvement and diversity maintenance.

The reward is then used to update the Q-value function, following the rule

$$Q(s_i,a_i)\leftarrow (1-\alpha )Q(s_i,a_i)+\alpha \left[r_i+\gamma \underset{a}{max}Q(s_i^{\prime },a)\right],$$

(21)

where $\alpha$ is the learning rate and $\gamma$ is the discount factor.

To balance the rapid response to new information and the long-term stability derived from past experiences, MSTPSO adopts a dual experience replay mechanism. The short-term memory (STM) stores the most recent samples of particle state transitions for rapid adaptation, while the long-term memory (LTM) preserves older and more stable samples to enhance the robustness of strategy learning. Each sample consists of $(s_t,a_t,r_t,s_{t+1})$ and its corresponding priority. The prioritized update rule is defined as

$${\omega }_i={\alpha }_{TD}\left(T{D}_i\right)+{\alpha }_f\Delta f_i^t+{\alpha }_H\Delta H_i^t+\lambda ,$$

(22)

where ${\alpha }_{TD}$, ${\alpha }_f$, and ${\alpha }_H$ are coefficients corresponding to $TD$ error, fitness improvement, and diversity contribution, respectively, $T{D}_i$ denotes the temporal-difference ($TD$) error of the i-th sample, and $\lambda ={10}^{-6}$ is a smoothing term to avoid zero priority. The coefficients are set as $0.7$, $0.5$, and $0.3$, respectively.

During the search, STM and LTM are updated at fixed intervals (every 500 evaluations). A replacement strategy with a ratio of 0.7:0.3 is applied, where 70% of new samples replace those in STM, while 30% are promoted to LTM. In addition, the top 10% of high-priority samples in STM are transferred into LTM to preserve valuable experience, ensuring long-term diversity maintenance in strategy learning.

3.4. Stagnation Detection and Hierarchical Response Mechanism

To address the potential issues of premature convergence and search stagnation during the iterative process of Particle Swarm Optimization, this paper proposes a hierarchical response mechanism based on stagnation detection. The mechanism first determines the level at which stagnation occurs (population or individual) and subsequently triggers either a partial restart strategy based on the global best neighborhood or a differential evolution (DE)-based perturbation mechanism, thereby enhancing global exploration capability while maintaining local exploitation performance.

In each iteration, if the improvement of the global-best fitness over 20 consecutive iterations is less than ${10}^{-6}$, or if the velocity magnitude of all particles falls below the lower bound $v_{min}={10}^{-5}$, the population is deemed stagnant and the restart strategy is executed. If an individual particle’s personal best position remains unchanged for 10 consecutive generations, the particle is considered individually stagnant, and the DE-based perturbation mechanism is activated.

3.4.1. Global-Best-Oriented Restart Strategy

When population stagnation is detected, a partial restart strategy centered on the global-best neighborhood is employed: 20% of the particles are randomly selected for position and velocity reinitialization within an adaptive radius around the global best, thereby restoring search diversity. The new position is generated as

$$R_d=\lambda ·\underset{i}{max}\left|x_{i,d}-g_d\right|,\lambda \in (0, 1),$$

(23)

$$x_{i,d}\leftarrow g_d+\mathrm{rand}(-R_d,R_d),$$

(24)

where $g_d$ denotes the coordinate of the global best in dimension d, and $\mathrm{rand}\left(·\right)$ represents a uniformly distributed random number. This method redistributes particles around the global best within an adaptive radius, helping the swarm escape stagnation caused by local clustering and velocity decay.

3.4.2. DE-Based Perturbation Mechanism

For particles identified as individually stagnant, a DE-based perturbation mechanism is integrated into the stagnation detection framework to enhance diversity and promote escape from local optima. As shown in Figure 4, the specific operation is as follows:

(1)

Global-best region perturbation.

Based on Euclidean distance, the closest $G_{\mathrm{near}}$ and farthest $G_{\mathrm{far}}$ particles from the global best $G_{\mathrm{best}}$ are selected to generate a new candidate,

$$G_{\mathrm{new}}=G_{\mathrm{near}}+F·(G_{\mathrm{far}}-G_{\mathrm{near}}),$$

(25)

where F denotes the differential evolution scaling factor, set to F = 0.6 in our experiments. If the fitness of $G_{\mathrm{new}}$ is better than that of $G_{\mathrm{best}}$, the latter is replaced by the new one.

(2)

Individual-level perturbation.

For each stagnant particle, the nearest and farthest neighbors $p_{\mathrm{near},i}$ and $p_{\mathrm{far},i}$ relative to $p_{\mathrm{best},i}$ are selected,

$$P_{\mathrm{new},i}=P_{\mathrm{near},i}+F·(P_{\mathrm{far},i}-P_{\mathrm{near},i}).$$

(26)

If $P_{\mathrm{new},i}$ is better, it updates $p_{\mathrm{best},i}$; otherwise, it replaces the farthest neighbor to increase local diversity.

3.5. Summary and Complexity Analysis

The MSTPSO algorithm begins with the initialization of particles, PSO parameters, the Q-table, and short-term and long-term experience buffers. In each iteration, particle states are obtained, topology actions are selected through the policy on the Q-table, and particle velocities and positions are updated accordingly. Fitness is then evaluated, personal and global bests are updated, and rewards with transitions are stored into replay buffers. The Q-table is updated regularly with prioritized experience replay. Stagnation detection is performed, and restart or DE-based perturbation strategies are applied when necessary. After completing the iterations, the algorithm returns the global best solution.

As shown in Algorithm 1, the computational complexity of MSTPSO consists of five components: fitness evaluation, Q-learning-based topology selection and particle swarm update, diversity entropy calculation, restart and DE perturbation, and the dual-layer experience replay mechanism.

Fitness evaluation is performed for N particles and costs O($N{C}_f$), where N is the swarm size and $C_f$ is the cost of a single fitness computation. On CEC benchmark functions, $C_f$ = $O\left(D\right)$, with D denoting the problem dimension. In the Q-learning–controlled particle swarm update, each generation each particle selects one of three topologies. For FIPS and small-world topologies, the update is $O\left(ND\right)$. When the exemplar-set topology is chosen, it needs to select from msl = [0.1N] exemplars, incurring an additional $O\left(N^2\right)$ cost. The diversity entropy module relies on Shannon entropy statistics: interval frequency counting for N particles over D dimensions is $O\left(ND\right)$. Restart and DE perturbation are triggered only when stagnation occurs, with amortized complexity $O\left(ND\right)$. The dual-layer experience replay comprises a short-term memory (STM, size 2N) and a long-term memory (LTM, size 5N), with storage complexity $O\left(N\right)$. Each generation writes N transitions into STM and computes priorities in constant time, giving $O\left(N\right)$ per generation. When replay is triggered, B = 3N samples are drawn and the Q-table is updated, costing $O\left(N\right)$. Amortized over all generations, this remains linear and negligible compared with the main terms. Considering the entire evolutionary process, the overall time complexity of MSTPSO is $O\left(N^{2}D\right)$.

Algorithm 1 MSTPSO Algorithm

Input: population size N, dimension D, maximum iterations T Output: global best solution $G_{\mathrm{best}}$ 1: Initialize particle positions, velocities, and the Q-table. 2: Create dual-layer replay buffers. 3: Evaluate initial fitness and determine $G_{\mathrm{best}}$. 4: for $t=1$ to T do 5:   for each particle do 6:      Select topology using Q-learning policy. 7:      Update $x_{i,d}^t$ and $v_{i,d}^t$ according to the selected topology by (15), (17), and (20). 8:   end for 9:   Evaluate fitness and update personal and global bests. 10:   If stagnation is detected, apply perturbation or restart strategy. 11: end for 12: Return $G_{\mathrm{best}}$.

4. Numerical Experiments

4.1. Experimental Settings and Comparison Methods

In this experiment, the performance of the proposed MSTPSO algorithm is evaluated using 30 benchmark functions from the CEC2017 [40] test suite. Since function $f_2$ shows instability in high-dimensional cases, it is excluded from the analysis.

Table 1. The information of the test suite on CEC2017.

No.	Function Name	Range
$f_1$	Shifted and Rotated Bent Cigar Function	[$-100,100$]
$f_3$	Shifted and Rotated Zakharov Function	[$-100,100$]
$f_4$	Shifted and Rotated Rosenbrock’s Function	[$-100,100$]
$f_5$	Shifted and Rotated Rastrigin’s Function	[$-100,100$]
$f_6$	Shifted and Rotated Expanded Scaffer’s Function	[$-100,100$]
$f_7$	Shifted and Rotated Lunacek Bi-Rastrigin’s Function	[$-100,100$]
$f_8$	Shifted and Rotated Non-Continuous Rastrigin’s Function	[$-100,100$]
$f_9$	Shifted and Rotated Levy Function	[$-100,100$]
$f_{10}$	Shifted and Rotated Schwefel’s Function	[$-100,100$]
$f_{11}$	Hybrid Function 1 (N = 3)	[$-100,100$]
$f_{12}$	Hybrid Function 2 (N = 3)	[$-100,100$]
$f_{13}$	Hybrid Function 3 (N = 3)	[$-100,100$]
$f_{14}$	Hybrid Function 4 (N = 4)	[$-100,100$]
$f_{15}$	Hybrid Function 5 (N = 4)	[$-100,100$]
$f_{16}$	Hybrid Function 6 (N = 4)	[$-100,100$]
$f_{17}$	Hybrid Function 6 (N = 5)	[$-100,100$]
$f_{18}$	Hybrid Function 6 (N = 5)	[$-100,100$]
$f_{19}$	Hybrid Function 6 (N = 5)	[$-100,100$]
$f_{20}$	Hybrid Function 6 (N = 6)	[$-100,100$]
$f_{21}$	Composition Function 1 (N = 3)	[$-100,100$]
$f_{22}$	Composition Function 2 (N = 3)	[$-100,100$]
$f_{23}$	Composition Function 3 (N = 4)	[$-100,100$]
$f_{24}$	Composition Function 4 (N = 4)	[$-100,100$]
$f_{25}$	Composition Function 5 (N = 5)	[$-100,100$]
$f_{26}$	Composition Function 6 (N = 5)	[$-100,100$]
$f_{27}$	Composition Function 7 (N = 6)	[$-100,100$]
$f_{28}$	Composition Function 8 (N = 6)	[$-100,100$]
$f_{29}$	Composition Function 9 (N = 3)	[$-100,100$]
$f_{30}$	Composition Function 10 (N = 3)	[$-100,100$]

lists the detailed information of these functions, which are divided into four categories: unimodal functions ($f_1,f_3$), multimodal functions ($f_4$–$f_{10}$), hybrid functions ($f_{11}$–$f_{20}$), and composition functions ($f_{21}$–$f_{30}$). Extensive numerical experiments are conducted to validate the effectiveness of MSTPSO.

For comparative studies, MSTPSO is benchmarked against nine mainstream PSO variants, including DQNPSO [43], APSO_SAC [44], EPSO [45], KGPSO [46], XPSO [22], MSORL [47], DRA [48], RFO [49], and RLACA [50]. The detailed information of these algorithms is given in

Table 2. Parameter settings for the nine compared algorithms.

Algorithms	Year	Parameters Setting
DQNPSO	2024	$c_1=2,c_2=2,\omega =0.5,\gamma =0.001$
APSO_SAC	2024	$\beta =1,\tau =0.005,\gamma =0.001$
EPSO	2023	$c_1=2,c_2=2,\omega =0.5$
KGPSO	2020	$c_1=2.13\sim 1.13,c_2=[0.93, 1.75],c_3=[0.6, 1.3],\omega =0.8\sim 0.35$
XPSO	2023	$\eta =0.2,Sta{g}_{max}=5,p=0.2,\omega =[0.4, 0.9]$
MSORL	2025	${\varphi }_1={\varphi }_2=0.4,\alpha =0.4,\gamma =0.8$
DRA	2025	$N=50,BSP=0.5,RP=0.2$
RFO	2025	$L=100,\beta ={10}^{-4}$
RLACA	2023	$c_1=2.2,c_2=1.8,\omega =[0, 1]$

. All functions are independently executed 51 times according to the CEC standard, with the termination condition set as the maximum number of evaluations equal to $D\times {10}^4$, where D is the problem dimension.

To comprehensively evaluate the effectiveness of MSTPSO, experiments are conducted on the CEC2017 test suite in 10, 30, 50 and 100 dimensions, and the results are compared with the above-mentioned algorithms.

Table 3. The comparative results of MSTPSO and advanced algorithms on 10-D using the CEC2017 benchmark suite.

Function	10-D	MSTPSO	DQNPSO	APSO_SAC	EPSO	KGPSO	XPSO	MSORL	DRA	RFO	RLACA
f1	Mean	6.07 × 102	2.15 × 108	6.67 × 108	1.02 × 103	9.45 × 107	6.76 × 106	7.66 × 107	1.20 × 1010	6.44 × 105	2.77 × 102
	Std	2.65 × 10−2	1.09 × 108	2.58	1.03 × 10−3	1.08 × 106	4.83 × 109	4.09 × 10−2	5.66 × 108	7.01 × 10−5	2.27 × 10−2
f3	Mean	0	1.06	1.15	6.49 × 10−10	1.09 × 102	1.50 × 102	4.07 × 102	1.50 × 104	1.64 × 102	0
	Std	2.10 × 10−1	8.37 × 105	1.16 × 102	5.82 × 10−11	2.54 × 10−13	3.50 × 109	6.31 × 103	1.66 × 103	1.85 × 10−8	0
f4	Mean	7.11 × 10−2	1.95 × 101	3.80 × 101	3.71	1.14 × 101	7.06	2.97 × 101	9.12 × 102	8.60	0
	Std	5.78 × 10−4	8.35 × 101	7.08 × 10−1	1.42 × 10−3	4.87	1.13 × 103	4.85 × 10−8	1.29 × 102	5.77 × 10−11	0
f5	Mean	2.20	1.45 × 101	2.20 × 101	3.06 × 101	9.83	2.48 × 101	4.69 × 101	9.43 × 101	2.04 × 101	1.88 × 101
	Std	3.54 × 10−2	2.39 × 101	1.56	1.54	3.27 × 101	3.87 × 101	2.64 × 101	1.50 × 101	5.02 × 10−11	2.48 × 10−7
f6	Mean	4.78 × 10−7	5.32 × 10−1	8.51 × 10−1	1.25	1.86 × 10−1	2.11	2.41 × 101	5.23 × 101	4.78	8.05 × 10−1
	Std	1.10 × 10−1	1.35 × 10−1	1.64 × 10−2	2.80 × 10−5	8.37 × 10−9	1.27 × 101	3.48 × 101	7.81	2.36 × 10−7	1.27 × 10−9
f7	Mean	1.18 × 101	1.81 × 101	2.39 × 101	4.04 × 101	2.38 × 101	4.83 × 101	2.31 × 101	1.19 × 102	2.99 × 101	2.37 × 101
	Std	3.35 × 10−4	7.71 × 101	3.03	2.71 × 10−1	3.32 × 101	3.58 × 101	3.09	1.22 × 101	3.27 × 10−10	9.71 × 10−14
f8	Mean	1.58	1.16 × 101	2.02 × 101	1.56 × 101	6.90	2.25 × 101	1.92 × 101	6.06 × 101	1.73 × 101	1.91 × 101
	Std	3.88 × 10−1	2.54 × 101	1.28	1.60 × 10−1	3.29 × 101	2.41 × 101	3.10 × 101	8.06	6.25 × 10−11	1.99 × 10−9
f9	Mean	0	1.62 × 10−1	1.65 × 101	5.53 × 101	0	3.16	2.85 × 101	7.33 × 102	2.97 × 101	1.78 × 10−2
	Std	2.96 × 10−2	1.49 × 10−11	8.37 × 10−2	4.37 × 10−1	9.72 × 10−14	4.71 × 102	1.05 × 103	3.00 × 102	2.12 × 10−10	1.38 × 10−14
f10	Mean	1.54 × 102	4.53 × 102	5.64 × 102	8.77 × 102	4.05 × 102	8.28 × 102	1.86 × 103	1.75 × 103	6.96 × 102	6.63 × 102
	Std	7.36 × 10−12	5.44 × 102	6.02 × 101	8.53	3.31 × 102	6.93 × 102	3.45 × 102	2.24 × 102	3.93 × 10−9	1.18
f11	Mean	7.48 × 10−1	2.65 × 101	4.66 × 101	1.24 × 101	9.31	1.63 × 101	4.86 × 101	1.87 × 103	5.65 × 101	1.77 × 101
	Std	1.76 × 10−4	2.23 × 107	1.83	2.81 × 10−1	1.52 × 101	2.98 × 106	3.39 × 10−1	1.33 × 104	2.01 × 10−10	4.66 × 10−7
f12	Mean	1.71 × 104	6.71 × 105	2.75 × 106	7.51 × 103	6.58 × 105	4.65 × 105	1.92 × 104	2.99 × 108	1.91 × 103	8.35 × 103
	Std	3.63 × 10−1	4.72 × 108	2.38 × 105	5.21 × 10−2	3.19 × 107	1.25 × 109	1.97 × 10−3	1.52 × 108	5.73 × 10−7	3.33
f13	Mean	1.89 × 103	9.03 × 103	1.04 × 104	8.01 × 103	6.25 × 103	7.36 × 103	4.36 × 102	5.24 × 104	2.64 × 102	1.73 × 102
	Std	2.48 × 104	5.32 × 108	3.47 × 103	2.44 × 10−1	1.51 × 107	3.18 × 108	2.59 × 104	1.82 × 107	6.71 × 10−8	1.08 × 10−1
f14	Mean	4.00 × 101	1.17 × 102	1.93 × 102	1.13 × 103	5.88 × 101	7.48 × 101	4.49 × 101	2.29 × 102	3.03 × 101	5.56 × 101
	Std	1.59 × 102	1.84 × 108	7.48 × 102	5.55 × 10−1	6.54 × 106	3.65 × 108	4.03 × 104	2.63 × 104	2.90 × 10−9	2.36 × 10−2
f15	Mean	2.36 × 102	9.14 × 102	4.52 × 102	1.38 × 103	6.83 × 102	1.70 × 102	7.61 × 101	5.62 × 103	5.38 × 101	4.18 × 101
	Std	4.75 × 104	1.98 × 109	2.35 × 103	4.38 × 10−1	2.58 × 107	6.49 × 108	2.05 × 105	3.41 × 104	7.83 × 10−8	9.40 × 10−1
f16	Mean	9.10 × 10−1	7.53 × 101	8.93 × 101	1.32 × 102	1.38 × 102	7.68 × 101	3.62 × 102	4.81 × 102	6.40 × 101	1.11 × 102
	Std	1.99 × 10−2	5.48 × 102	1.22 × 101	1.58 × 101	2.08 × 102	6.45 × 102	2.12 × 102	1.46 × 102	7.40 × 10−8	1.76 × 10−1
f17	Mean	1.88 × 101	4.73 × 101	7.12 × 101	4.61 × 101	4.60 × 101	4.21 × 101	1.14 × 102	1.32 × 102	4.48 × 101	5.02 × 101
	Std	1.50	1.05 × 104	7.55	1.12	1.63 × 102	8.43 × 102	1.70 × 102	7.43 × 101	5.22 × 10−9	1.33 × 10−3
f18	Mean	2.82 × 103	1.12 × 104	2.22 × 104	1.17 × 104	1.73 × 104	1.39 × 104	1.24 × 102	8.39 × 106	8.42 × 101	2.56 × 102
	Std	7.89 × 104	1.75 × 109	2.98 × 104	3.11	1.62 × 108	3.15 × 109	1.42 × 10−1	2.82 × 108	7.11 × 10−8	1.22 × 10−2
f19	Mean	8.71 × 101	2.76 × 103	1.25 × 103	5.81 × 103	4.92 × 103	1.40 × 102	2.36 × 101	4.52 × 103	1.98 × 101	2.46 × 101
	Std	4.30 × 104	9.72 × 108	2.47 × 103	4.65 × 10−1	9.36 × 107	1.66 × 109	3.00 × 104	2.97 × 107	6.42 × 10−10	2.12 × 10−3
f20	Mean	1.39 × 101	1.94 × 101	3.87 × 101	3.77 × 101	7.36 × 101	6.04 × 101	2.96 × 102	2.06 × 102	4.56 × 101	7.54 × 101
	Std	2.38 × 10−1	8.52 × 101	2.41	6.29	4.74 × 101	3.47 × 102	1.68 × 102	6.90 × 101	3.83 × 10−8	1.10 × 10−3
f21	Mean	1.55 × 102	2.10 × 102	2.15 × 102	2.16 × 102	1.99 × 102	2.12 × 102	2.35 × 102	2.40 × 102	2.10 × 102	1.57 × 102
	Std	4.29	2.68 × 101	1.25	3.70 × 10−1	3.07 × 101	5.88 × 101	6.00 × 101	2.85 × 101	4.78 × 10−11	8.87 × 10−5
f22	Mean	1.00 × 102	1.32 × 102	1.18 × 102	1.03 × 102	1.14 × 102	1.11 × 102	1.43 × 102	1.09 × 103	1.08 × 102	1.02 × 102
	Std	8.10 × 10−1	1.57 × 101	6.76 × 10−1	3.60 × 10−2	2.98	1.69 × 102	1.25 × 103	9.00 × 101	3.93 × 10−10	3.96 × 10−13
f23	Mean	3.01 × 102	3.23 × 102	3.36 × 102	3.22 × 102	3.24 × 102	3.25 × 102	4.13 × 102	4.45 × 102	3.23 × 102	3.20 × 102
	Std	2.05 × 10−1	1.83 × 101	6.27 × 10−1	1.31	6.75	8.53 × 101	4.01 × 102	2.69 × 101	1.15 × 10−10	1.77 × 10−2
f24	Mean	3.15 × 102	3.38 × 102	3.67 × 102	3.38 × 102	3.21 × 102	3.34 × 102	2.82 × 102	4.39 × 102	3.35 × 102	3.42 × 102
	Std	7.65 × 10−1	2.87 × 101	1.02	8.03 × 10−1	1.66 × 101	8.27 × 101	1.33 × 102	2.47 × 101	9.55 × 10−11	1.00 × 10−2
f25	Mean	4.32 × 102	4.39 × 102	4.50 × 102	4.29 × 102	4.30 × 102	4.33 × 102	4.35 × 102	1.08 × 103	4.34 × 102	4.25 × 102
	Std	1.04 × 10−1	2.74 × 101	7.72 × 10−1	7.40 × 10−3	2.31 × 10−2	7.22 × 102	4.88	6.35 × 101	8.83 × 10−11	1.53 × 10−3
f26	Mean	3.00 × 102	5.16 × 102	5.45 × 102	3.59 × 102	3.66 × 102	2.98 × 102	6.95 × 102	1.50 × 103	5.34 × 102	3.51 × 102
	Std	4.49 × 10−1	1.02 × 101	2.99	5.38 × 10−6	7.33	6.92 × 102	5.69 × 102	1.49 × 102	3.00 × 10−10	4.71 × 10−4
f27	Mean	3.95 × 102	4.08 × 102	4.14 × 102	3.96 × 102	4.16 × 102	4.01 × 102	5.04 × 102	4.81 × 102	4.07 × 102	3.75 × 102
	Std	1.81	2.03 × 101	9.94 × 10−2	2.73 × 10−2	4.05 × 10−2	8.65 × 102	3.11 × 102	2.98 × 101	5.35 × 10−11	6.27 × 10−1
f28	Mean	3.00 × 102	5.71 × 102	5.21 × 102	5.13 × 102	5.12 × 102	5.04 × 102	5.85 × 102	9.46 × 102	5.09 × 102	4.02 × 102
	Std	8.68	1.94 × 101	6.76 × 10−2	8.13 × 10−3	7.84 × 10−13	1.22 × 103	3.79 × 101	5.00 × 101	6.02 × 10−11	2.62 × 101
f29	Mean	2.52 × 102	2.88 × 102	3.13 × 102	3.08 × 102	2.87 × 102	2.92 × 102	3.83 × 102	5.27 × 102	2.75 × 102	2.83 × 102
	Std	4.69 × 10−2	2.10 × 103	9.91	6.91	2.19 × 102	5.59 × 105	2.91 × 103	1.34 × 102	3.73 × 10−9	4.64 × 10−1
f30	Mean	1.00 × 105	4.68 × 105	5.62 × 105	5.39 × 105	2.97 × 105	4.01 × 105	6.26 × 104	8.06 × 106	3.53 × 105	2.13 × 103
	Std	3.29 × 106	2.68 × 108	1.17 × 104	1.20 × 102	1.12 × 107	7.71 × 108	2.40 × 107	1.12 × 107	3.01 × 10−9	1.16 × 105

,

Table 4. The comparative results of MSTPSO and advanced algorithms on 30-D using the CEC2017 benchmark suite.

Function	30-D	MSTPSO	DQNPSO	APSO_SAC	EPSO	KGPSO	XPSO	MSORL	DRA	RFO	RLACA
f1	Mean	1.42 × 103	7.01 × 109	6.54 × 109	3.80 × 103	1.06 × 109	1.34 × 109	1.33 × 1010	6.40 × 1010	2.69 × 1010	1.66 × 10−2
	Std	1.42 × 10−1	1.47 × 108	2.19 × 103	4.46 × 10−3	4.79 × 103	6.91 × 109	6.85 × 10−3	4.06 × 107	4.71 × 10−2	8.71 × 10−8
f3	Mean	1.58 × 103	6.79 × 103	4.57 × 104	5.59 × 103	6.64 × 102	8.48 × 103	3.03 × 104	8.50 × 104	6.48 × 104	0
	Std	3.19 × 107	6.95 × 1013	2.89 × 103	6.29	5.16 × 10−4	2.05 × 1013	2.80 × 10−7	5.61 × 103	9.33 × 10−7	7.15 × 10−14
f4	Mean	8.66 × 101	1.06 × 103	7.13 × 102	7.89 × 101	1.72 × 102	2.59 × 102	2.25 × 103	1.63 × 104	4.18 × 103	5.02 × 101
	Std	2.46 × 102	5.39 × 101	1.13	2.34 × 10−3	6.55	3.01 × 104	1.51 × 10−7	7.43 × 101	1.07 × 10−8	6.33 × 10−1
f5	Mean	1.23 × 101	1.22 × 102	1.66 × 102	1.47 × 102	4.67 × 101	1.87 × 102	2.43 × 102	4.49 × 102	2.55 × 102	1.11 × 102
	Std	6.84 × 10−1	2.76	2.82	1.05	6.62 × 101	5.66 × 101	5.68 × 101	1.13 × 101	1.43 × 10−9	1.02 × 10−4
f6	Mean	1.88 × 10−3	1.18 × 101	1.68 × 101	2.63 × 101	1.01	1.60 × 101	5.69 × 101	9.21 × 101	4.86 × 101	8.75
	Std	1.12 × 10−6	4.69 × 10−1	1.09 × 10−1	1.78 × 10−4	1.46 × 10−8	9.76	2.24 × 101	3.60	5.00 × 10−10	7.64 × 10−8
f7	Mean	3.89 × 101	2.18 × 102	2.32 × 102	2.35 × 102	1.02 × 102	3.10 × 102	3.17 × 102	7.85 × 102	5.63 × 102	1.29 × 102
	Std	6.04	3.46	1.47 × 101	2.12	8.00 × 101	9.00 × 101	2.13 × 10−1	1.19 × 101	5.94 × 10−9	1.48 × 10−13
f8	Mean	1.14 × 101	1.21 × 102	1.38 × 102	1.25 × 102	4.01 × 101	1.83 × 102	1.77 × 102	3.65 × 102	2.17 × 102	1.08 × 102
	Std	3.75	2.02	3.68	1.83	6.54 × 101	5.13 × 101	5.92 × 101	7.89	1.35 × 10−9	3.60 × 10−3
f9	Mean	6.91 × 10−2	1.64 × 103	2.16 × 103	2.98 × 103	6.47 × 101	6.70 × 102	2.65 × 103	1.04 × 104	4.14 × 103	2.23 × 102
	Std	4.64 × 10−6	3.32 × 103	5.82 × 102	3.70 × 101	5.22 × 101	2.07 × 103	1.28 × 103	1.31 × 103	1.84 × 10−7	1.52 × 10−2
f10	Mean	1.77 × 103	3.22 × 103	3.69 × 103	3.63 × 103	2.30 × 103	6.67 × 103	6.78 × 103	8.25 × 103	5.08 × 103	3.37 × 103
	Std	1.20	1.09 × 103	7.65 × 102	4.72	1.74 × 103	1.10 × 103	8.34 × 102	2.07 × 102	3.76 × 10−8	1.81 × 101
f11	Mean	2.77 × 101	3.80 × 102	7.50 × 102	1.21 × 102	9.27 × 101	3.62 × 102	6.48 × 102	7.86 × 103	1.36 × 103	1.25 × 102
	Std	1.91 × 101	4.42 × 104	1.01	6.75 × 10−5	3.23	1.29 × 108	1.15 × 10−5	1.05 × 104	1.73 × 10−8	1.76 × 10−1
f12	Mean	2.33 × 104	3.50 × 108	2.70 × 108	1.52 × 105	6.60 × 107	1.11 × 108	9.22 × 108	1.63 × 1010	9.87 × 108	1.55 × 105
	Std	1.02 × 105	4.09 × 107	1.02 × 106	4.16 × 101	5.21 × 106	2.57 × 109	2.15 × 10−3	6.72 × 107	5.46 × 10−3	1.02 × 104
f13	Mean	5.45 × 103	1.64 × 108	5.68 × 108	1.19 × 104	4.71 × 107	1.21 × 107	2.75 × 107	9.49 × 109	1.63 × 105	1.39 × 104
	Std	1.37 × 101	1.67 × 108	4.36 × 101	2.97 × 10−1	3.08 × 101	6.75 × 109	7.35 × 10−1	3.74 × 108	2.30 × 10−5	2.99 × 10−1
f14	Mean	2.60 × 103	2.65 × 104	6.00 × 104	1.16 × 104	9.28 × 103	2.73 × 104	3.21 × 103	3.80 × 106	3.11 × 102	3.11 × 102
	Std	2.31	3.29 × 108	4.91 × 104	5.56 × 10−1	6.88 × 106	4.60 × 108	3.19 × 10−8	5.38 × 106	6.18 × 10−8	7.95 × 10−2
f15	Mean	3.15 × 103	6.16 × 104	6.42 × 104	2.35 × 103	6.17 × 103	5.24 × 105	1.31 × 104	1.02 × 109	1.91 × 104	3.43 × 103
	Std	8.83	8.60 × 107	2.92 × 104	2.42 × 10−3	6.41 × 105	2.56 × 109	4.04 × 10−7	3.43 × 107	5.06 × 10−6	1.08 × 10−2
f16	Mean	2.12 × 102	8.88 × 102	1.34 × 103	1.14 × 103	6.14 × 102	1.39 × 103	1.86 × 103	4.43 × 103	1.24 × 103	9.41 × 102
	Std	1.95 × 101	4.61 × 102	1.09 × 101	1.86 × 101	3.70 × 102	2.89 × 103	4.03 × 102	3.81 × 102	8.63 × 10−9	1.45
f17	Mean	3.05 × 101	4.64 × 102	7.04 × 102	4.14 × 102	2.65 × 102	3.18 × 102	8.90 × 102	3.64 × 103	4.95 × 102	4.69 × 102
	Std	1.14 × 10−2	9.03 × 103	2.00 × 101	4.40 × 101	1.93 × 102	3.27 × 104	2.31 × 102	3.34 × 102	3.99 × 10−9	2.13
f18	Mean	8.13 × 104	2.42 × 105	1.57 × 106	1.26 × 105	2.35 × 105	8.33 × 105	1.35 × 105	3.51 × 107	4.53 × 104	2.38 × 104
	Std	4.28 × 101	1.02 × 109	2.89 × 105	5.50	9.20 × 107	1.98 × 109	4.79 × 10−7	2.79 × 107	3.24 × 10−6	1.85 × 104
f19	Mean	5.31 × 103	1.94 × 106	6.53 × 107	4.85 × 103	9.62 × 103	1.02 × 106	3.11 × 105	9.23 × 108	2.28 × 105	1.23 × 103
	Std	5.07 × 10−2	9.31 × 106	2.82 × 102	5.34 × 10−4	8.94 × 105	1.48 × 109	5.08 × 101	3.37 × 107	1.70 × 10−5	2.96 × 10−2
f20	Mean	1.38 × 102	3.00 × 102	4.91 × 102	4.02 × 102	2.32 × 102	4.01 × 102	7.35 × 102	1.03 × 103	4.18 × 102	5.16 × 102
	Std	2.78 × 10−2	2.35 × 102	3.69 × 101	1.84 × 101	1.62 × 102	5.72 × 102	3.99 × 102	1.32 × 102	5.76 × 10−9	1.36
f21	Mean	2.12 × 102	3.27 × 102	3.55 × 102	3.16 × 102	2.58 × 102	3.78 × 102	4.52 × 102	6.69 × 102	4.48 × 102	3.11 × 102
	Std	5.96 × 10−1	1.03	4.33	2.75	6.82 × 101	6.36 × 101	7.99 × 101	1.50 × 101	1.32 × 10−9	1.85 × 10−2
f22	Mean	1.00 × 102	2.55 × 103	3.76 × 103	3.57 × 102	8.81 × 102	6.13 × 102	3.45 × 103	7.71 × 103	4.58 × 103	1.47 × 103
	Std	1.34 × 10−3	6.98 × 102	3.59 × 102	1.02 × 10−8	7.62 × 101	1.03 × 103	2.82 × 103	1.40 × 102	2.64 × 10−8	8.03
f23	Mean	3.48 × 102	5.86 × 102	6.79 × 102	4.93 × 102	5.69 × 102	5.43 × 102	9.94 × 102	1.27 × 103	7.15 × 102	5.29 × 102
	Std	2.82	6.21 × 10−1	2.24	2.23	6.94	2.84 × 102	6.45 × 102	5.20 × 101	1.41 × 10−9	1.28 × 10−1
f24	Mean	4.23 × 102	7.15 × 102	7.31 × 102	5.60 × 102	6.58 × 102	6.05 × 102	9.45 × 102	1.40 × 103	7.64 × 102	6.23 × 102
	Std	1.33	1.49 × 101	6.48	3.68	2.17 × 101	2.14 × 102	2.35 × 101	3.12 × 101	1.20 × 10−9	6.97 × 10−2
f25	Mean	3.88 × 102	6.64 × 102	6.70 × 102	3.98 × 102	3.99 × 102	4.95 × 102	8.71 × 102	3.04 × 103	1.49 × 103	3.80 × 102
	Std	8.40	3.67 × 101	8.41 × 10−1	4.81 × 10−13	5.13 × 10−13	2.24 × 103	9.03 × 10−13	2.14 × 101	2.98 × 10−9	9.24 × 10−2
f26	Mean	9.17 × 102	3.47 × 103	3.47 × 103	2.14 × 103	1.65 × 103	2.13 × 103	5.25 × 103	9.73 × 103	5.13 × 103	1.87 × 103
	Std	1.00 × 10−2	2.80 × 101	1.44 × 101	1.10 × 101	7.20 × 101	2.39 × 103	9.83 × 101	7.23 × 101	1.42 × 10−8	6.39 × 10−1
f27	Mean	5.17 × 102	6.04 × 102	6.04 × 102	5.30 × 102	5.88 × 102	5.64 × 102	1.09 × 103	1.71 × 103	6.55 × 102	5.00 × 102
	Std	6.22	5.28 × 10−1	3.68 × 10−1	4.83 × 10−5	1.81 × 10−1	1.49 × 103	4.55 × 102	7.85 × 101	5.51 × 10−10	9.05 × 101
f28	Mean	3.57 × 102	1.19 × 103	1.05 × 103	4.01 × 102	4.97 × 102	5.74 × 102	1.35 × 103	5.23 × 103	2.03 × 103	3.66 × 102
	Std	2.84 × 101	5.91 × 10−4	4.59 × 10−1	2.24 × 10−3	1.70 × 101	5.63 × 103	4.56 × 10−12	7.74	2.88 × 10−9	1.58 × 10−1
f29	Mean	4.43 × 102	1.04 × 103	1.16 × 103	8.33 × 102	6.55 × 102	1.07 × 103	2.32 × 103	6.62 × 103	1.40 × 103	7.72 × 102
	Std	2.54 × 10−1	3.62 × 104	6.94	3.86 × 101	1.20 × 102	6.77 × 105	1.37 × 102	4.12 × 102	5.56 × 10−9	2.41 × 101
f30	Mean	4.16 × 103	4.87 × 106	1.85 × 106	4.96 × 103	9.40 × 104	4.27 × 106	1.32 × 107	1.81 × 109	2.16 × 106	1.17 × 104
	Std	1.58 × 103	1.19 × 106	3.77 × 103	3.21 × 10−2	1.67 × 105	9.84 × 108	8.78	6.46 × 107	4.54 × 10−5	2.85 × 105

,

Table 5. The comparative results of MSTPSO and advanced algorithms on 50-D using the CEC2017 benchmark suite.

Function	50-D	MSTPSO	DQNPSO	APSO_SAC	EPSO	KGPSO	XPSO	MSORL	DRA	RFO	RLACA
f1	Mean	1.97 × 103	3.59 × 1010	2.14 × 1010	1.62 × 103	3.64 × 109	6.26 × 109	4.81 × 1010	1.21 × 1011	8.59 × 1010	3.43 × 103
	Std	2.27 × 10−3	2.97 × 108	1.75 × 106	1.02 × 10−4	4.95 × 101	1.04 × 1010	8.56 × 10−3	4.19 × 106	8.60 × 10−2	4.35 × 10−5
f3	Mean	1.60 × 104	3.14 × 104	1.35 × 105	3.25 × 104	3.69 × 103	2.47 × 104	9.15 × 104	1.81 × 105	1.67 × 105	0
	Std	1.11 × 103	7.63 × 1014	8.05 × 103	8.24	2.00 × 107	6.11 × 1014	1.09 × 105	2.06 × 105	2.27 × 10−6	7.01 × 10−13
f4	Mean	1.84 × 102	4.37 × 103	2.85 × 103	1.16 × 102	3.56 × 102	7.01 × 102	9.18 × 103	4.37 × 104	1.78 × 104	8.65 × 101
	Std	1.59 × 102	1.54 × 102	2.56	7.57 × 10−4	2.19 × 101	1.59 × 104	2.86 × 10−10	6.56	2.77 × 10−8	8.86 × 10−1
f5	Mean	2.45 × 101	2.77 × 102	3.40 × 102	2.70 × 102	8.98 × 101	3.91 × 102	4.55 × 102	7.29 × 102	5.39 × 102	2.48 × 102
	Std	1.06 × 10−2	1.93 × 10−1	2.86	4.56 × 10−3	6.03 × 101	6.10 × 101	5.65 × 101	4.13	2.17 × 10−9	4.24 × 10−9
f6	Mean	3.77 × 10−2	3.09 × 101	3.52 × 101	4.04 × 101	1.88	2.43 × 101	7.02 × 101	1.05 × 102	6.90 × 101	2.11 × 101
	Std	6.42 × 10−2	1.11 × 10−1	2.71 × 10−1	8.75 × 10−5	3.08 × 10−8	8.04	1.83 × 101	2.87	5.57 × 10−10	1.23 × 10−7
f7	Mean	6.70 × 101	8.42 × 102	7.46 × 102	5.36 × 102	1.48 × 102	6.55 × 102	8.06 × 102	1.44 × 103	1.32 × 103	3.25 × 102
	Std	7.84	2.56 × 101	5.71	4.52	1.06 × 102	1.31 × 102	2.98 × 10−10	8.97	8.97 × 10−9	1.77 × 10−13
f8	Mean	2.65 × 101	2.90 × 102	3.57 × 102	2.78 × 102	9.09 × 101	3.87 × 102	4.71 × 102	7.48 × 102	5.57 × 102	2.41 × 102
	Std	1.32 × 10−1	3.35 × 10−1	1.25	1.43	6.28 × 101	6.35 × 101	5.53 × 101	8.06	2.14 × 10−9	1.14 × 10−2
f9	Mean	7.86 × 10−1	7.51 × 103	1.20 × 104	9.69 × 103	8.36 × 102	2.74 × 103	1.42 × 104	3.78 × 104	2.07 × 104	2.19 × 103
	Std	7.19 × 10−10	8.59 × 103	7.00 × 103	3.03 × 101	7.03 × 102	7.86 × 103	4.15 × 103	2.41 × 103	9.71 × 10−7	7.20 × 10−2
f10	Mean	3.35 × 103	6.33 × 103	7.09 × 103	6.85 × 103	4.32 × 103	1.25 × 104	1.20 × 104	1.41 × 104	1.06 × 104	6.55 × 103
	Std	1.00	1.51 × 103	1.28 × 103	1.22 × 102	2.25 × 103	1.42 × 103	1.46 × 103	2.69 × 102	5.34 × 10−8	4.04 × 101
f11	Mean	6.72 × 101	8.17 × 102	1.83 × 103	2.03 × 102	2.01 × 102	9.85 × 102	4.49 × 103	2.62 × 104	1.16 × 104	2.75 × 102
	Std	1.07 × 102	1.91 × 108	7.83 × 101	2.98 × 10−5	1.52	2.29 × 109	3.38 × 10−10	4.86 × 102	1.10 × 10−7	5.25 × 10−2
f12	Mean	1.82 × 105	6.56 × 109	8.12 × 109	9.43 × 105	2.46 × 109	1.22 × 109	1.41 × 1010	1.01 × 1011	2.40 × 1010	2.21 × 106
	Std	4.44 × 105	9.40 × 107	2.34 × 107	5.74 × 101	5.54 × 107	1.19 × 1010	5.73 × 10−3	6.81 × 107	4.94 × 10−2	3.08 × 103
f13	Mean	2.24 × 103	1.77 × 109	2.00 × 109	4.13 × 103	7.36 × 108	1.95 × 108	2.55 × 109	6.29 × 1010	2.75 × 109	1.74 × 104
	Std	1.78 × 102	1.43 × 108	3.06 × 107	8.48 × 10−2	2.34	9.68 × 109	2.59	1.58 × 108	1.65 × 10−2	2.97 × 10−1
f14	Mean	1.68 × 104	2.42 × 105	1.60 × 106	4.24 × 104	2.01 × 105	4.01 × 105	1.04 × 106	9.07 × 107	5.39 × 104	3.68 × 103
	Std	1.19 × 106	2.17 × 108	2.04 × 105	1.91	8.98 × 106	8.12 × 108	2.73 × 10−7	7.65 × 106	2.47 × 10−6	1.62 × 101
f15	Mean	1.06 × 103	7.50 × 107	1.25 × 108	8.53 × 103	1.41 × 106	2.15 × 107	5.53 × 107	1.25 × 1010	7.00 × 105	9.03 × 103
	Std	9.52 × 101	1.68 × 108	1.05 × 101	8.85 × 10−5	1.61 × 101	2.87 × 109	3.72 × 10−4	1.14 × 107	9.66 × 10−5	7.55 × 10−2
f16	Mean	5.49 × 102	2.02 × 103	2.26 × 103	1.92 × 103	1.05 × 103	2.74 × 103	3.83 × 103	9.23 × 103	3.14 × 103	1.74 × 103
	Std	2.96	2.36 × 102	6.42 × 101	4.72 × 101	2.28 × 102	1.99 × 103	4.46 × 102	1.98 × 102	1.60 × 10−8	2.33
f17	Mean	3.02 × 102	1.73 × 103	2.10 × 103	1.49 × 103	8.10 × 102	1.99 × 103	2.00 × 103	2.45 × 104	1.81 × 103	1.28 × 103
	Std	5.47 × 10−7	4.15 × 106	4.11 × 101	3.89 × 101	6.12 × 102	7.09 × 104	1.91 × 102	1.43 × 102	1.14 × 10−8	8.05 × 10−1
f18	Mean	7.53 × 104	1.97 × 106	5.41 × 106	2.00 × 105	8.94 × 105	3.65 × 106	6.00 × 106	2.78 × 108	4.55 × 105	7.15 × 104
	Std	3.60 × 104	8.49 × 108	9.47 × 105	2.32 × 101	4.09 × 107	2.27 × 109	2.87 × 10−4	7.53 × 106	1.01 × 10−5	3.60 × 102
f19	Mean	8.05 × 103	1.09 × 107	5.09 × 107	1.62 × 104	1.53 × 105	1.15 × 107	9.31 × 106	4.97 × 109	2.09 × 107	5.33 × 103
	Std	2.66 × 10−2	1.32 × 107	1.56 × 101	1.59 × 10−3	8.18 × 10−1	1.06 × 109	3.00 × 10−2	1.66 × 107	3.91 × 10−4	3.49 × 10−2
f20	Mean	1.34 × 102	1.02 × 103	1.25 × 103	1.04 × 103	4.18 × 102	1.38 × 103	1.34 × 103	2.53 × 103	1.08 × 103	1.12 × 103
	Std	7.74 × 10−5	6.70 × 102	2.26 × 102	6.07 × 101	7.06 × 102	6.61 × 102	6.27 × 102	1.11 × 102	8.26 × 10−9	1.15
f21	Mean	2.26 × 102	5.24 × 102	5.59 × 102	4.21 × 102	3.33 × 102	5.78 × 102	7.81 × 102	1.19 × 103	7.91 × 102	4.43 × 102
	Std	8.27 × 10−5	4.40 × 10−1	8.11	5.03	4.83 × 101	6.63 × 101	1.70 × 102	1.76 × 101	2.03 × 10−9	3.17 × 10−2
f22	Mean	1.70 × 102	6.63 × 103	7.60 × 103	7.92 × 103	3.84 × 103	9.63 × 103	1.30 × 104	1.50 × 104	1.07 × 104	6.48 × 103
	Std	2.13 × 10−3	1.33 × 103	1.03 × 103	6.09 × 101	1.27 × 103	1.58 × 103	1.34 × 103	2.86 × 102	4.87 × 10−8	2.37 × 101
f23	Mean	4.39 × 102	1.08 × 103	1.13 × 103	7.24 × 102	9.82 × 102	8.31 × 102	1.81 × 103	2.40 × 103	1.36 × 103	8.00 × 102
	Std	1.46 × 10−2	1.55	9.60 × 10−1	5.31	5.33	4.34 × 102	8.50 × 102	5.80 × 101	2.43 × 10−9	1.79 × 10−1
f24	Mean	5.02 × 102	1.20 × 103	1.17 × 103	8.01 × 102	1.06 × 103	8.87 × 102	1.68 × 103	2.41 × 103	1.39 × 103	9.54 × 102
	Std	5.71 × 10−10	4.97	2.72	3.31	3.88 × 101	3.72 × 102	1.71 × 101	3.55 × 101	1.73 × 10−9	1.89 × 10−1
f25	Mean	5.65 × 102	3.11 × 103	1.79 × 103	5.71 × 102	5.74 × 102	1.02 × 103	5.28 × 103	1.45 × 104	9.42 × 103	4.65 × 102
	Std	1.47 × 102	4.30 × 101	2.46	2.81 × 10−4	1.52 × 101	5.20 × 103	9.92 × 10−11	3.16	1.33 × 10−8	8.57 × 10−1
f26	Mean	1.24 × 103	7.65 × 103	7.98 × 103	3.76 × 103	2.62 × 103	3.79 × 103	1.08 × 104	1.55 × 104	1.23 × 104	2.97 × 103
	Std	4.43 × 10−4	3.85 × 101	2.06 × 101	2.91 × 101	1.10 × 102	3.43 × 103	2.91	1.38 × 101	2.03 × 10−8	1.39 × 10−1
f27	Mean	5.68 × 102	1.23 × 103	1.13 × 103	7.79 × 102	1.01 × 103	7.86 × 102	3.33 × 103	4.11 × 103	1.47 × 103	5.00 × 102
	Std	2.07 × 101	3.63	1.07	5.06 × 10−5	2.86 × 10−1	4.92 × 103	9.51 × 102	1.16 × 102	1.49 × 10−9	1.44 × 102
f28	Mean	4.70 × 102	5.80 × 103	4.40 × 103	5.14 × 102	9.21 × 102	7.81 × 102	4.74 × 103	1.24 × 104	7.32 × 103	4.96 × 102
	Std	1.16 × 102	9.28	9.75 × 10−1	7.34 × 10−4	1.30 × 101	4.41 × 103	4.42 × 10−11	5.36	7.14 × 10−9	1.19 × 102
f29	Mean	4.70 × 102	2.51 × 103	2.61 × 103	1.47 × 103	1.03 × 103	2.34 × 103	5.75 × 103	1.83 × 105	4.63 × 103	1.24 × 103
	Std	5.87	1.14 × 104	5.47	2.10 × 101	1.77 × 101	1.16 × 107	1.34 × 101	4.28 × 103	1.48 × 10−8	1.88 × 101
f30	Mean	1.22 × 106	1.14 × 108	3.55 × 107	1.11 × 106	4.66 × 106	1.50 × 108	3.46 × 108	9.48 × 109	2.21 × 108	8.17 × 104
	Std	1.45 × 105	4.47 × 106	1.31 × 105	2.33	2.20 × 106	5.37 × 109	7.35 × 102	3.78 × 107	1.59 × 10−3	1.32 × 106

and

Table 6. The comparative results of MSTPSO and advanced algorithms on 100-D using the CEC2017 benchmark suite.

	100-D	MSTPSO	DQNPSO	APSO_SAC	EPSO	KGPSO	XPSO	MSORL	DRA	RFO	RLACA
f1	Mean	4.16 × 103	1.77 × 1011	9.16 × 1010	5.48 × 103	1.38 × 1010	6.26 × 109	4.81 × 1010	2.81 × 1011	2.47 × 1011	1.72 × 104
	Std	2.54 × 10−3	6.49 × 108	7.57 × 107	3.23 × 10−2	4.06 × 107	1.04 × 1010	8.56 × 10−3	6.35 × 105	1.55 × 10−1	5.74 × 10−3
f3	Mean	1.40 × 105	1.93 × 105	6.58 × 105	1.61 × 105	3.18 × 104	2.47 × 104	9.15 × 104	3.47 × 105	4.17 × 105	2.56 × 10−1
	Std	6.65 × 1011	2.66 × 1016	4.44 × 104	1.52 × 101	1.42 × 1012	6.11 × 1014	1.09 × 105	3.04 × 105	5.09 × 10−6	3.69 × 10−6
f4	Mean	2.97 × 102	3.15 × 104	1.35 × 104	2.71 × 102	1.47 × 103	7.01 × 102	9.18 × 103	1.24 × 105	7.04 × 104	2.37 × 102
	Std	1.84 × 102	3.06 × 101	1.72 × 101	1.10 × 10−3	9.07	1.59 × 104	2.86 × 10−10	4.50	8.30 × 10−8	6.23 × 10−1
f5	Mean	6.77 × 101	9.76 × 102	9.62 × 102	7.09 × 102	2.48 × 102	3.91 × 102	4.55 × 102	1.67 × 103	1.44 × 103	6.66 × 102
	Std	1.56 × 10−5	1.58	7.19	1.81	5.39 × 101	6.10 × 101	5.65 × 101	4.10	3.16 × 10−9	8.89 × 10−10
f6	Mean	3.53 × 10−1	6.06 × 101	5.87 × 101	5.08 × 101	3.84	2.43 × 101	7.02 × 101	1.16 × 102	9.22 × 101	4.43 × 101
	Std	3.32 × 10−9	1.53 × 10−2	3.43 × 10−1	8.74 × 10−5	8.16 × 10−9	8.04	1.83 × 101	7.65 × 10−1	4.52 × 10−10	6.09 × 10−6
f7	Mean	1.53 × 102	3.93 × 103	3.18 × 103	1.64 × 103	3.22 × 102	6.55 × 102	8.06 × 102	3.43 × 103	3.73 × 103	1.13 × 103
	Std	1.64 × 10−12	5.63	2.26 × 101	1.68	1.07 × 102	1.31 × 102	2.98 × 10−10	9.42	1.28 × 10−8	4.44 × 10−13
f8	Mean	6.95 × 101	1.03 × 103	1.05 × 103	7.16 × 102	2.90 × 102	3.87 × 102	4.71 × 102	1.87 × 103	1.57 × 103	7.08 × 102
	Std	8.81 × 10−13	2.59	1.90	1.13 × 101	4.10 × 101	6.35 × 101	5.53 × 101	4.53	3.17 × 10−9	9.84 × 10−10
f9	Mean	1.44 × 101	2.41 × 104	4.35 × 104	2.21 × 104	4.62 × 103	2.74 × 103	1.42 × 104	8.38 × 104	5.88 × 104	1.25 × 104
	Std	1.77 × 10−10	2.64 × 104	1.87 × 104	1.23 × 102	2.36 × 103	7.86 × 103	4.15 × 103	2.29 × 103	2.08 × 10−6	9.58 × 10−8
f10	Mean	8.51 × 103	1.50 × 104	1.58 × 104	1.60 × 104	1.13 × 104	1.25 × 104	1.20 × 104	3.25 × 104	2.59 × 104	1.50 × 104
	Std	1.59 × 10−6	1.78 × 103	2.51 × 103	8.29 × 101	4.59 × 103	1.42 × 103	1.46 × 103	2.18 × 102	7.82 × 10−8	1.17 × 101
f11	Mean	5.18 × 102	1.40 × 104	8.85 × 104	1.50 × 103	3.44 × 103	9.85 × 102	4.49 × 103	2.32 × 105	1.66 × 105	1.40 × 103
	Std	2.16 × 102	2.98 × 109	9.38 × 103	1.81 × 10−1	9.98 × 10−1	2.29 × 109	3.38 × 10−10	4.89 × 105	8.76 × 10−7	6.83 × 10−3
f12	Mean	5.38 × 105	4.56 × 1010	2.74 × 1010	1.98 × 106	8.21 × 109	1.22 × 109	1.41 × 1010	2.22 × 1011	1.24 × 1011	1.42 × 107
	Std	5.78 × 106	2.63 × 108	2.55 × 107	1.21 × 102	1.82 × 108	1.19 × 1010	5.73 × 10−3	2.92 × 106	1.27 × 10−1	2.54 × 104
f13	Mean	1.67 × 103	4.80 × 109	4.70 × 109	6.25 × 103	1.14 × 109	1.95 × 108	2.55 × 109	5.36 × 1010	2.43 × 1010	1.43 × 104
	Std	4.41 × 101	1.62 × 108	3.81 × 106	1.43 × 10−3	1.61 × 108	9.68 × 109	2.59	1.86 × 106	3.66 × 10−2	1.78 × 10−4
f14	Mean	5.39 × 104	1.20 × 107	9.72 × 106	2.45 × 105	4.92 × 105	4.01 × 105	1.04 × 106	1.39 × 108	8.13 × 106	4.39 × 103
	Std	1.98 × 101	1.36 × 107	1.06 × 106	1.61 × 101	2.26 × 107	8.12 × 108	2.73 × 10−7	1.31 × 106	8.34 × 10−5	7.40 × 10−4
f15	Mean	5.34 × 102	1.50 × 109	1.66 × 109	2.25 × 103	3.32 × 108	2.15 × 107	5.53 × 107	2.95 × 1010	5.92 × 109	6.16 × 103
	Std	6.47 × 101	4.33 × 107	1.15 × 107	2.68 × 10−5	3.71 × 107	2.87 × 109	3.72 × 10−4	2.07 × 106	1.73 × 10−2	4.59 × 10−4
f16	Mean	1.83 × 103	6.03 × 103	6.18 × 103	4.37 × 103	2.84 × 103	2.74 × 103	3.83 × 103	2.59 × 104	1.17 × 104	3.78 × 103
	Std	2.26 × 10−6	2.96 × 102	3.76 × 101	5.30 × 101	1.37 × 102	1.99 × 103	4.46 × 102	1.09 × 102	3.33 × 10−8	1.48
f17	Mean	9.62 × 102	2.90 × 104	1.11 × 104	3.52 × 103	3.27 × 103	1.99 × 103	2.00 × 103	1.88 × 107	9.93 × 104	3.58 × 103
	Std	4.45 × 10−6	3.90 × 105	1.38 × 101	2.51 × 101	2.96 × 102	7.09 × 104	1.91 × 102	6.41 × 104	6.47 × 10−7	2.27
f18	Mean	2.02 × 105	1.48 × 107	1.22 × 107	5.09 × 105	1.02 × 106	3.65 × 106	6.00 × 106	3.70 × 108	1.61 × 107	2.47 × 104
	Std	5.01 × 101	1.26 × 107	2.26 × 106	4.74 × 101	1.96 × 107	2.27 × 109	2.87 × 10−4	1.23 × 106	1.25 × 10−4	4.39 × 10−2
f19	Mean	9.17 × 102	1.14 × 109	9.68 × 108	1.65 × 103	3.93 × 108	1.15 × 107	9.31 × 106	2.93 × 1010	6.21 × 109	4.43 × 103
	Std	7.82 × 10−5	3.45 × 107	5.41 × 106	2.16 × 10−2	1.89 × 108	1.06 × 109	3.00 × 10−2	2.62 × 106	1.71 × 10−2	1.12 × 10−3
f20	Mean	9.85 × 102	3.13 × 103	3.65 × 103	3.25 × 103	2.06 × 103	1.38 × 103	1.34 × 103	5.84 × 103	3.63 × 103	2.99 × 103
	Std	1.64 × 10−4	1.55 × 103	6.81 × 102	3.75 × 101	1.63 × 103	6.61 × 102	6.27 × 102	1.80 × 102	1.98 × 10−8	9.64
f21	Mean	2.92 × 102	1.44 × 103	1.34 × 103	8.60 × 102	8.66 × 102	5.78 × 102	7.81 × 102	2.93 × 103	2.17 × 103	9.61 × 102
	Std	3.85 × 10−5	1.25	2.99	1.46 × 101	3.11 × 101	6.63 × 101	1.70 × 102	4.16 × 101	3.30 × 10−9	1.08 × 10−2
f22	Mean	4.46 × 103	1.66 × 104	1.75 × 104	1.85 × 104	1.24 × 104	9.63 × 103	1.30 × 104	3.30 × 104	2.69 × 104	1.59 × 104
	Std	3.85 × 10−5	2.25 × 103	1.86 × 103	1.46 × 101	3.11 × 101	1.58 × 103	1.34 × 103	2.26 × 102	7.59 × 10−8	1.08 × 10−2
f23	Mean	6.49 × 102	2.44 × 103	2.37 × 103	1.31 × 103	2.24 × 103	8.31 × 102	1.81 × 103	4.22 × 103	2.83 × 103	1.55 × 103
	Std	9.13 × 10−1	1.02	1.68	1.44 × 102	3.36 × 103	4.34 × 102	8.50 × 102	6.59 × 101	2.75 × 10−9	2.16 × 101
f24	Mean	8.98 × 102	3.79 × 103	3.62 × 103	1.92 × 103	3.31 × 103	8.87 × 102	1.68 × 103	7.90 × 103	4.65 × 103	2.18 × 103
	Std	4.33 × 10−2	3.57	3.72	8.77 × 10−1	4.20 × 10−2	3.72 × 102	1.71 × 101	6.73 × 101	3.27 × 10−9	2.16 × 10−1
f25	Mean	8.66 × 102	2.14 × 104	7.96 × 103	8.33 × 102	1.07 × 103	1.02 × 103	5.28 × 103	2.91 × 104	2.38 × 104	8.27 × 102
	Std	5.37 × 10−2	1.02 × 102	5.67 × 101	9.43	4.15 × 10−1	5.20 × 103	9.92 × 10−11	3.53 × 10−1	2.58 × 10−8	6.30 × 10−1
f26	Mean	3.46 × 103	2.83 × 104	2.62 × 104	1.18 × 104	1.08 × 104	3.79 × 103	1.08 × 104	5.32 × 104	4.21 × 104	1.17 × 104
	Std	2.31 × 102	3.09 × 101	1.55 × 101	1.32 × 10−3	1.16 × 101	3.43 × 103	2.91	4.09 × 101	3.48 × 10−8	5.42 × 10−1
f27	Mean	7.13 × 102	2.24 × 103	1.67 × 103	9.60 × 102	1.49 × 103	7.86 × 102	3.33 × 103	1.23 × 104	3.92 × 103	5.00 × 102
	Std	6.39 × 10−4	6.72	8.09 × 10−1	3.65 × 10−3	1.98 × 10−5	4.92 × 103	9.51 × 102	9.82 × 101	3.85 × 10−9	8.90 × 10−1
f28	Mean	6.66 × 102	2.15 × 104	1.61 × 104	6.23 × 102	2.19 × 103	7.81 × 102	4.74 × 103	2.96 × 104	2.87 × 104	5.72 × 102
	Std	5.94	4.78 × 101	1.09 × 101	1.17 × 10−4	1.84 × 10−1	4.41 × 103	4.42 × 10−11	1.75 × 101	2.12 × 10−8	2.55 × 102
f29	Mean	1.55 × 103	1.28 × 104	8.68 × 103	4.15 × 103	2.98 × 103	2.34 × 103	5.75 × 103	1.15 × 106	2.26 × 104	3.51 × 103
	Std	1.27 × 102	7.49 × 102	7.44	3.58 × 10−3	3.16	1.16 × 107	1.34 × 101	2.14 × 103	8.80 × 10−8	4.80 × 10−1
f30	Mean	1.22 × 104	3.68 × 109	4.40 × 109	9.88 × 103	1.15 × 109	1.50 × 108	3.46 × 108	4.86 × 1010	1.09 × 1010	1.26 × 105
	Std	8.83 × 105	4.65 × 107	2.61 × 104	6.94 × 10−2	2.15 × 103	5.37 × 109	7.35 × 102	3.05 × 106	2.30 × 10−2	4.87 × 105

present the mean and standard deviation of the solution errors obtained after 51 runs. In addition, as shown in

Table 7. Wilcoxon signed-rank test results of ten algorithms versus MSTPSO on CEC 2017 (counts over functions).

Algorithms	MSTPSO VS	10-D	30-D	50-D
DQNPSO	+	27	28	29
	−	0	0	0
	≈	2	1	0
APSO_SAC	+	26	29	29
	−	1	0	0
	≈	2	0	0
EPSO	+	26	24	25
	−	2	3	3
	≈	1	2	1
KGPSO	+	25	28	28
	−	2	1	1
	≈	2	0	0
XPSO	+	29	29	29
	−	0	0	0
	≈	0	0	0
MSORL	+	26	27	29
	−	2	1	0
	≈	1	1	0
DRA	+	29	29	29
	−	0	0	0
	≈	0	0	0
RFO	+	21	27	28
	−	7	2	0
	≈	0	0	1
RLACA	+	16	20	22
	−	11	7	5
	≈	2	2	2

, the results of the Wilcoxon signed-rank test demonstrate the statistical significance of the algorithm. The data indicate that MSTPSO demonstrates significant advantages over existing PSO variants, confirming its superior performance.

4.2. Performance Comparison Across Different Dimensions

In the 10-dimensional tests, Table 3 reports the mean and standard deviation of solution errors after 51 independent runs. For the unimodal functions $f_1$ and $f_3$, the proposed algorithm ranks first on $f_3$ and second on $f_1$, with only a small gap from the top, reflecting its effectiveness in solving unimodal problems. For the multimodal functions ($f_4$–$f_{10}$), MSTPSO achieves the best results on $f_5$–$f_{10}$, which not only demonstrates its excellent global search ability but also highlights the effectiveness of its reinforcement learning strategy and perturbation mechanism in escaping local optima. In the hybrid ($f_{11}$–$f_{20}$) and composition functions ($f_{21}$–$f_{30}$), MSTPSO also achieves first place on $f_{11}$, $f_{14}$, $f_{16}$, $f_{17}$, $f_{20}$–$f_{23}$, $f_{28}$, and $f_{29}$.

In the 30-dimensional case, Table 4 presents the mean and standard deviation of solution errors after 51 runs. MSTPSO ranks second on the unimodal functions $f_1$ and $f_3$, slightly behind RLACA. For the multimodal functions ($f_4$–$f_{10}$), MSTPSO maintains stable rankings, demonstrating its competitiveness in handling multimodal optimization problems. For the hybrid and composition functions ($f_{11}$–$f_{30}$), MSTPSO shows significant improvement; however, $f_{14}$ experiences a performance drop, indicating its sensitivity to dimensionality, while other functions achieve notable gains. Specifically, MSTPSO ranks first on $f_{12}$, $f_{13}$, $f_{15}$, $f_{26}$ and $f_{30}$, underscoring its strong global search ability and remarkable performance in complex hybrid problems.

In the 50-dimensional case, Table 5 shows the mean and standard deviation of solution errors after 51 runs. MSTPSO achieves a significant improvement on unimodal functions, rising to first place on $f_1$. For the multimodal functions ($f_4$–$f_{10}$), MSTPSO obtains the best results on six functions ($f_5$–$f_{10}$) and ranks third on $f_4$, reflecting its robustness in escaping local optima under complex conditions. For the hybrid and composition functions ($f_{11}$–$f_{30}$), MSTPSO achieves overall improvement, ranking first on 14 out of 20 functions, which demonstrates its powerful global search capability in high-dimensional settings.

In the 100-dimensional case, Table 6 presents the mean and standard deviation of solution errors, as well as the average rankings over 51 independent runs. MSTPSO achieves the best result on the unimodal function $f_1$. For multimodal functions, MSTPSO maintains optimal performance on $f_5$ to $f_{10}$ and ranks second on $f_4$. In hybrid and composition functions, MSTPSO maintains stable rankings, while other RL-PSO variants show significant performance degradation, fully demonstrating the robustness and stability of the topology structure in high-dimensional complex optimization tasks. Overall, compared with several state-of-the-art PSO algorithms, the results indicate that MSTPSO ranks first across the 10-D, 30-D, 50-D, and 100-D CEC 2017 test suites, highlighting its superior performance.

4.3. Convergence Curves and Dynamic Performance Analysis

To further evaluate the convergence performance of MSTPSO, convergence curves are plotted. It should be noted that due to minor differences in initialization strategies, the early-stage convergence behavior of different algorithms may vary. Figure 5 illustrates the convergence processes for functions $f_1$–$f_{17}$ in 50 dimensions. The proposed method achieves the fastest convergence on $f_1$, demonstrating significant advantages in solving unimodal functions through multi-agent reinforcement learning-based parameter training. For multimodal functions, MSTPSO consistently shows the fastest convergence on $f_5$–$f_{10}$. For hybrid functions, MSTPSO also converges quickly on $f_{11}$–$f_{13}$, $f_{15}$–$f_{17}$, with strong capability to escape local optima. These results confirm that MSTPSO possesses strong global search ability and robustness in solving complex hybrid optimization problems.

4.4. Boxplot Analysis and Robustness Evaluation

As shown in Figure 6, the boxplot analysis, indicates that MSTPSO demonstrates varying performance across functions. For unimodal function $f_1$, it shows strong convergence and stability, approaching the optimal solution quickly with almost no outliers. For multimodal functions $f_4$, $f_6$, $f_7$, and $f_{10}$, MSTPSO effectively avoids local optima, although a few outliers appear on $f_{10}$, indicating occasional influence by local traps. For hybrid functions $f_{12}$, $f_{16}$, $f_{19}$, and $f_{20}$, MSTPSO maintains good solution quality and stability, with only minor fluctuations on $f_{19}$ but avoiding many outliers overall. For composition functions $f_{25}$ and $f_{30}$, especially under high-dimensional complex conditions, MSTPSO shows strong global search capability, successfully escaping local optima with fast convergence and fewer outliers. In summary, MSTPSO demonstrates excellent performance across different optimization problems, particularly excelling on unimodal, multimodal, and composition functions.

4.5. Significance Test Analysis (Wilcoxon Rank-Sum Test)

To further verify the statistical superiority of MSTPSO, Table 7 presents the Wilcoxon signed-rank test results of ten algorithms on the CEC2017 benchmark functions under 10-, 30-, and 50-dimensional settings. A significance level of 0.05 was used to determine whether the performance differences are statistically meaningful, and the p-values obtained from the Wilcoxon test indicate the degree of significance. Here, “+” denotes that MSTPSO significantly outperforms the compared algorithm, “−” indicates significant inferiority, and “≈” represents no significant difference. Compared with advanced variants such as DQNPSO, APSO_SAC, XPSO, and DRA, MSTPSO achieved statistically superior results on nearly all test functions, with complete wins across all dimensions against XPSO and DRA. Against EPSO, KGPSO, and MSORL, MSTPSO also maintained clear statistical superiority with only a few non-significant cases. Compared with RFO and RLACA, MSTPSO is slightly better at low dimensions and achieves greater superiority at 30-D and 50-D. Overall, the Wilcoxon results confirm that MSTPSO significantly outperforms most representative PSO variants across the majority of test functions and dimensions, verifying the statistical reliability of its performance gains.

4.6. Ablation Study on Topology Structures

4.6.1. Multi-Topology Ablation

To verify the contribution of each topology in MSTPSO, three ablation versions are tested: MSTPSO1 removes the FIPS topology, MSTPSO2 removes the small-world topology, and MSTPSO3 removes the exemplar topology. Their optimization performance on the 10- and 30-dimensional CEC2017 functions is compared with the full MSTPSO, as shown in

Table 8. Experiments of MSTPSO algorithm and change action algorithm on 10D and 30D.

Optimization	10-D				30-D
	MSTPSO	MSTPSO1	MSTPSO2	MSTPSO3	MSTPSO	MSTPSO1	MSTPSO2	MSTPSO3
Average	2.31	2.82	2.51	2.35	2.22	2.18	3.15	2.43
Unimodal	2.34	2.90	2.36	2.40	1.98	3.81	2.15	2.05
Multimodal	2.30	2.42	2.26	2.31	2.18	3.24	2.38	2.41
Mixing and compound	2.35	2.68	2.60	2.35	2.12	3.32	3.29	2.16

.

The results show that MSTPSO achieves the best average ranking in both dimensions, confirming the effectiveness of the multi-topology integration in balancing global search and local convergence. The average rankings of MSTPSO are 2.31 in 10-D and 2.22 in 30-D, significantly better than the ablation versions. MSTPSO1 shows a clear decline in 10-D, especially on unimodal and multimodal functions, indicating the critical role of the FIPS topology in maintaining diversity and avoiding premature convergence. MSTPSO2 performs worse in multiple functions, particularly in 30-D, with an average rank of 3.81, demonstrating the importance of the small-world topology for global exploration. MSTPSO3 shows performance similar to MSTPSO2 but still inferior to the full version, especially on complex functions. Overall, MSTPSO’s superior performance validates the importance of multi-topology integration.

4.6.2. Ablation of Stagnation Detection and Hierarchical Response Mechanism

As shown in

Table 9. Friedman ranking of four individual stagnation threshold parameters.

Optimization	30-D				50-D
	${\mathit{T}}_{\mathit{I}}=\mathbf{10}$	${\mathit{T}}_{\mathit{I}}=\mathbf{15}$	${\mathit{T}}_{\mathit{I}}=\mathbf{20}$	${\mathit{T}}_{\mathit{I}}=\mathbf{30}$	${\mathit{T}}_{\mathit{I}}=\mathbf{10}$	${\mathit{T}}_{\mathit{I}}=\mathbf{15}$	${\mathit{T}}_{\mathit{I}}=\mathbf{20}$	${\mathit{T}}_{\mathit{I}}=\mathbf{30}$
Average	2.53	2.50	2.48	2.49	2.53	2.49	2.46	2.52
Unimodal	2.45	2.66	2.33	2.56	2.51	2.42	2.36	2.71
Multimodal	2.57	2.33	2.56	2.54	2.46	2.52	2.48	2.54
Mixing and compound	2.52	2.54	2.47	2.47	2.55	2.49	2.46	2.50

, the average ranking results indicate that introducing stagnation detection and the hierarchical response mechanism further improves overall performance in different dimensions. The complete algorithm achieves average ranks of 1.33, 1.43, and 1.47 in 10-D, 30-D, and 50-D, respectively, outperforming the version without this module. The improvement is particularly evident in high-complexity problems. On unimodal functions, both versions converge quickly, but MSTPSO ranks significantly higher in 10-D and 50-D, showing faster convergence and better escape ability. For multimodal functions, stagnation detection enhances global exploration, and in 30-D and 50-D, the average rankings are much better than the baseline, indicating higher solution quality and stability in complex spaces. For hybrid and composition functions, the advantage is most significant, with the complete MSTPSO achieving markedly better average rankings in 10-D and 30-D. These results confirm that stagnation detection and hierarchical response are key modules for improving MSTPSO’s robustness and adaptability in high-dimensional optimization.

To evaluate the impact of stagnation detection thresholds on the stability of the algorithm, a parameter sensitivity study was conducted on the CEC2017 30-D and 50-D test suites, with the Friedman average rankings shown in

Table 10. Experiments of MSTPSO algorithm and stop detection module on 10-D, 30-D and 50-D.

Optimization	10-D		30-D		50-D
	Stagnation	No Stagnation	Stagnation	No Stagnation	Stagnation	No Stagnation
Average	1.33	1.67	1.43	1.57	1.47	1.53
Unimodal	1.73	1.27	1.17	1.32	1.38	1.62
Multimodal	1.33	1.67	1.37	1.77	1.52	1.55
Mixing and compound	1.30	1.70	1.44	1.55	1.49	1.51

,

Table 11. Friedman ranking of the stagnation threshold parameters for four groups.

Optimization	30-D				50-D
	${\mathit{T}}_{\mathit{I}}=\mathbf{5}$	${\mathit{T}}_{\mathit{I}}=\mathbf{10}$	${\mathit{T}}_{\mathit{I}}=\mathbf{15}$	${\mathit{T}}_{\mathit{I}}=\mathbf{20}$	${\mathit{T}}_{\mathit{I}}=\mathbf{5}$	${\mathit{T}}_{\mathit{I}}=\mathbf{10}$	${\mathit{T}}_{\mathit{I}}=\mathbf{15}$	${\mathit{T}}_{\mathit{I}}=\mathbf{20}$
Average	2.53	2.48	2.51	2.48	2.53	2.45	2.49	2.53
Unimodal	2.80	2.33	2.35	2.51	2.55	2.41	2.31	2.72
Multimodal	2.50	2.51	2.53	2.45	2.47	2.48	2.44	2.62
Mixing and compound	2.51	2.49	2.51	2.49	2.48	2.43	2.54	2.55

and

Table 12. Friedman ranking of lower bound parameters at three speeds.

Optimization	30-D			50-D
	${\mathit{V}}_{min}={\mathit{e}}^{-\mathbf{4}}$	${\mathit{V}}_{min}={\mathit{e}}^{-\mathbf{5}}$	${\mathit{V}}_{min}={\mathit{e}}^{-\mathbf{6}}$	${\mathit{V}}_{min}={\mathit{e}}^{-\mathbf{4}}$	${\mathit{V}}_{min}={\mathit{e}}^{-\mathbf{5}}$	${\mathit{V}}_{min}={\mathit{e}}^{-\mathbf{6}}$
Average	2.57	2.33	2.56	2.46	2.52	2.48
Unimodal	2.52	2.54	2.47	2.55	2.49	2.46
Multimodal	2.53	2.50	2.48	2.53	2.49	2.46
Mixing and compound	2.45	2.66	2.33	2.51	2.42	2.36

. The individual stagnation threshold $T_p$, the population stagnation threshold $T_i$, and the lower velocity bound $V_{\min }$ were set within the ranges 10, 15, 20, 30, 5, 10, 15, 20 and $e^{-4}$, $e^{-5}$, $e^{-6}$, respectively. The experimental results show that the performance of MSTPSO remains stable within these parameter ranges, with only slight fluctuations in the average rankings. The configuration $T_p=20$, $T_i=10$ and $V_{\min }=e^{-5}$ achieves the best average rankings in both the 30-D and 50-D cases.

5. Conclusions

This paper proposes a multi-topology Particle Swarm Optimization algorithm controlled by Q-learning (MSTPSO). The method introduces a reinforcement learning–driven adaptive topology selection mechanism into the standard PSO framework, enabling particles to dynamically switch among FIPS, small-world, and exemplar-set topologies to balance global exploration and local exploitation. Meanwhile, a diversity-entropy regulation mechanism is designed to maintain population diversity, and, based on stagnation detection, DE perturbations combined with a global best restart strategy are incorporated to avoid premature convergence and enhance local search capability. Unlike most existing RL-PSO variants, which mainly use reinforcement learning to adapt inertia weight or learning factors, the proposed method focuses on dynamic topology selection and improves performance through the combination of dual-layer experience replay, stagnation detection, and DE perturbations, thus complementing existing approaches in mechanism design. Systematic experiments on the CEC2017 benchmark functions demonstrate that MSTPSO achieves superior performance on unimodal, multimodal, hybrid, and composite functions across four dimensionalities (10, 30, 50, and 100). Compared with various advanced PSO variants, the proposed algorithm attains faster convergence, higher solution accuracy, and better stability, showing particularly strong robustness on high-dimensional complex functions. Further ablation studies reveal that the Q-learning-based topology selection mechanism significantly improves overall convergence quality, while the stagnation-driven DE perturbation and global restart strategy effectively enhance the ability to escape local optima. Nevertheless, some limitations remain. First, the Q-learning parameter settings may require task-specific adjustment and show certain sensitivity. Second, computing diversity entropy and pairwise distance matrices incurs higher computational cost for large populations and high-dimensional problems. Finally, current experiments are limited to benchmark functions, and the applicability of the algorithm to complex constrained optimization and real engineering problems requires further investigation. Future work may proceed in several directions: (1) optimizing the reinforcement learning control strategy to reduce parameter sensitivity and improve efficiency on large-scale, high-dimensional problems; (2) extending the algorithm to more complex scenarios such as constrained, dynamic, and multiobjective optimization to further verify its generality; (3) exploring parallel computing and hardware acceleration methods, such as GPU computing and distributed architectures, to enhance scalability; and (4) applying the algorithm to real engineering tasks—including industrial scheduling, image and vision inspection, and path planning—to validate its practical value.