Enhancing Performance of Evolutionary Strategies with Symmetric Sampling (Furthermore, Weight Decay)

Abstract

Evolutionary Strategies (ESs) are optimization metaheuristics largely adopted in Evolutionary Computation (EC). Since their introduction in the early 70s, researchers in the field have attempted to improve the efficacy of these algorithms. The most advanced ESs, such as the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) and Exponential Natural Evolution Strategies (xNESs), make use of covariance matrices storing relationships between parameters to be optimized, which enable the algorithms to fasten the search in the solution spaces. However, the computational cost of calculating covariance matrices linearly scales with the number of parameters. Recently, the OpenAI Evolutionary Strategy (OpenAI-ES) emerged as an effective ES in different domains, thanks to the parameter information stored in two momentum vectors. Furthermore, OpenAI-ES gains an advantage from the usage of symmetric sampling and weight decay techniques. In this work, I delve into the application of symmetric sampling and weight decay on CMA-ES, xNES and Separable Natural Evolution Strategies (sNESs), with the aim to improve their performance in domains in which they get stuck in local minima outcomes. Specifically, I propose three novel variants for each ES and verify their efficacy with respect to the PyBullet halfcheetah and hopper robot locomotion problems, and two collective tasks (i.e., swarm aggregation and swarm foraging). The findings reveal that symmetric sampling produces performance enhancements in all the domains, whereas the effect of weight decay varies across the considered problems. Furthermore, symmetric sampling allows ESs to keep parameter size limited, which is paramount in these scenarios. This research identifies techniques enhancing the success of modern ESs, proposes several ES variants, and discusses the relationship between algorithmic performance and task properties.

1. Introduction

Evolutionary Strategies (ESs) [1,2,3,4] are a subclass of Evolutionary Algorithms (EAs) [5], which are widespread optimization methods in Evolutionary Computation (EC). As originally defined in [3], ESs make use of mutation, recombination (also termed as “crossover”) and selection operators as Genetic Algorithms (GAs) [6]. Depending on the selection operator used, a canonical ES can be expressed as in either $(a,b)$ or $(a+b)$ (see Equations (1) and (2)).

$$(a,b)-ES:selectBest\left(a\right) fromb$$

(1)

$$(a+b)-ES:selectBest\left(a\right) froma\cup b$$

(2)

Specifically, given $\mu$ parents and $\lambda$ offspring, two selection mechanisms have been defined:

• $(\mu ,\lambda )$: The best $\mu$ offspring are retained for the next iteration of the algorithm;
• $(\mu +\lambda )$: Selection of the best is made through the union of $\mu$ parents and $\lambda$ offspring.

The original ES is a $(1+1)$-ES, i.e., the population is formed by one parent ($\mu =1$) and one offspring ($\lambda =1$). If the performance (also called “fitness”) of the offspring is at least as good as the parent’s fitness, the offspring becomes the new parent. The $(1+\lambda )$-ES algorithm considers $\lambda$ offspring generated by the parent through mutation and/or crossover operators. If one of the offspring is at least as good as the parent, it becomes the new parent. On the other hand, in the $(1,\lambda )$-ES algorithm, the best offspring becomes the new parent regardless of the performance, and the parent is always discarded [7].

Based on this theoretical foundation, several variants have been proposed over time. A tightly related method is the Hill-Climbing (HC) algorithm [8], which considers $\mu$ parents evolving independently of each other through a $(1+1)$-ES procedure. The advantage of considering a pool of independent parents is to increase the chance to discover more effective solutions. For example, in [9] the authors show that HC succeeds in solving the rather complex 5-bit parity problem [10]. However, the convergence speed is generally slow and the resulting performance might be sub-optimal, particularly in domains like pole balancing [9], test function optimization [11] and robot foraging [12]. The Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [13] is a rather sophisticated ES, whose main property lies in the computation of a covariance matrix. In particular, the mutual influence between parameters is encoded in the matrix and allows one to direct the search in the solution space. As a consequence, CMA-ES leverages the covariance matrix to minimize the time spent to discover solutions (exploration) and maximize the performance refinement (exploitation). In the original formulation reported in [13], CMA-ES evolves a single solution, referred to as a “centroid”, thus representing a particular instance of $(1+\lambda )$-ES. At each iteration, the algorithm generate samples from a Gaussian distribution $\mathcal{N}(0,1)$. The samples are modulated by the covariance matrix and added to the centroid to create a set of candidate solutions, which are evaluated and ranked based on their performance. The centroid is then updated based on the recombination of the best ${\displaystyle \frac{\lambda }{2}}$ samples. CMA-ES has been firstly applied to test function optimization [13]. Thanks to its features, it became a popular tool to address other types of problems, such as pole balancing [14,15] and robotics [12,15,16]. Nonetheless, CMA-ES showed poor performance in complex tasks like legged robot locomotion: as shown in [17], CMA-ES performs worse than OpenAI-ES with regard to halfcheetah, hopper and walker2D tasks (see Figure 1 in [17]). Closely related to CMA-ES is the Exponential Natural Evolution Strategy (xNES) algorithm [18,19]: the method is a particular $(1+\lambda )$-ES that calculates a covariance matrix storing the relationship between parameters, which is exploited to accelerate and optimize the search for solutions. According to [19,20], both CMA-ES and xNES explore the solution space based on the natural gradient [21]. Similarly to CMA-ES, at each iteration of xNES, a pool of samples is randomly drawn from a Gaussian distribution $\mathcal{N}(0,1)$. A set of candidate solutions is derived by multiplying samples and covariance matrix, and adding the resulting vectors to the centroid. These candidate solutions are evaluated and ranked based on their performance. However, differently from CMA-ES, xNES uses all the evaluated samples for recombination and centroid updating. A more detailed description is provided in Section 2. The effectiveness of xNES has been proved in test function optimization [18,19], pole balancing [15,19,22], robotics [12,15,23] and chemistry [24]. As for CMA-ES, xNES is uncapable of achieving good performance in robot locomotion tasks [17], in which it obtains remarkably lower performance than OpenAI-ES in the halfcheetah, hopper and walker2D tasks (see Figure 1 in [17]). A notable drawback of CMA-ES and xNES is their computational complexity, which increases non-linearily with the number of evolvable parameters. The Separable Natural Evolution Strategy (sNES) [19,25] is another instance of Natural Evolution Strategies (NESs) that makes use of separable search distributions. Specifically, instead of calculating the full covariance matrix, each parameter variance is computed and stored independently. This is particularly useful in large search spaces. Therefore, the sNES algorithm scales better with the number of evolvable parameters, and reduces the computational burden of CMA-ES and xNES. In fact, depending on the number of parameters to be optimized, computing the covariance matrix might be expensive or even impossible [17,19,25,26,27]. Similarly to CMA-ES and xNES, sNES is a variant of $(1+\lambda )$-ES that extracts a set of samples from a Gaussian distribution $\mathcal{N}(0,1)$ at each iteration. The samples are multiplied by the variances and added to the centroid, thus forming a pool of candidate solutions. These solutions are evaluated and ranked based on their performance. Lastly, the centroid is updated through the recombination of all the samples, similarly to xNES. Examples of the application of sNES can be found in [11,15,17,19,25]. The OpenAI Evolutionary Strategy (OpenAI-ES) algorithm [28] is a modern ES introduced as a viable alternative to Reinforcement Learning (RL) approaches [29], thanks to an efficient parallelization procedure. OpenAI-ES is another instance of $(1+\lambda )$-ES: at each iteration of the algorithm, a pool of samples is extracted from a Gaussian distribution $\mathcal{N}(0,\sigma )$. OpenAI-ES uses symmetric sampling [30] to evaluate both positive and negative parameter modifications. Once all the samples have been evaluated and get a performance value (fitness), the algorithm ranks samples and recombines them to estimate the fitness gradient. Lastly, OpenAI-ES updates two momentum vectors (mean and variance) and the centroid through Adam [31]. A main advantage of OpenAI-ES over CMA-ES and xNES is the limited computational burden, since no covariance matrix has to be computed. This also leads to a better scalability when the number of evolvable parameters increases. A schematic of OpenAI-ES is provided in Figure 1. The authors in [28] showed that OpenAI-ES is capable to achieve good performance in MuJoCo [32] humanoid and Atari games [28]. OpenAI-ES demonstrates a wide capability with respect to MuJoCo and PyBullet [33] robot locomotion [17,34], collective problems [17,35] and many other domains [36,37]. Differently from CMA-ES, sNES and xNES, OpenAI-ES stores two momentum vectors keeping track of how performance varied in order to drive the search in the solution space. Moreover, two main properties affect the performance of OpenAI-ES: (i) symmetric sampling [30] and (ii) weight decay [38]. The former allows the algorithm to understand the direction of parameter variations, thus better coping with the task specificity. The latter enables OpenAI-ES to keep the weights limited in size, which is beneficial for solving PyBullet and MuJoCo robot locomotion problems [17]. Nonetheless, recent studies emphasized that OpenAI-ES underperforms based on the problem characteristics [12]. In fact, the tendency to decrease weight size by OpenAI-ES is detrimental when using a sensory–motor neural network controller, as in [12]. Similarly, the performance of OpenAI-ES is strongly dependent on the hyperparameter settings, as illustrated in [35]. Differently from OpenAI-ES, CMA-ES and xNES suffer from uncontrolled weight growths, which limit their performance in MuJoCo and PyBullet robot locomotion domains [17,35]. Furthermore, the parameter variations are typically applied in one direction only. Therefore, I hypothesize that symmetric sampling and weight decay might be advantageous for enhancing the performance of CMA-ES, sNES and xNES in those scenarios in which they get stuck in sub-optimal results. In fact, analyzing parameter variations in two opposite directions could enable a better understanding of the actual parameters’ influence on the overall performance. Furthermore, keeping weight size limited might allow for a better exploration of the search space, without skipping specific regions corresponding to potentially enhanced performance.

Figure 1. Schematic of OpenAI-ES.

Table 1 provides a summary of the main works utilizing ESs and the related application domains. Moreover, Table 2 provides a thorough comparison of CMA-ES, OpenAI-ES, sNES and xNES in terms of computational complexity, mechanisms to preserve information about parameter–performance relationships and scalability.

Table 1. List of research works on ESs. For each work, I highlighted the publication year, the authors (and the reference), the algorithm used and the application domain(s).

Year	Authors	Algorithm	Domain
1993	Back & Schwefel [39]	($\mu ,\lambda$)-ES	Test function optimization
1998	Papadrakakis et al. [40]	($\mu ,\lambda$)-ES, ($\mu +\lambda$)-ES	Structural optimization
2001	Hansen & Ostermeier [13]	CMA-ES	Test function optimization
2003	Igel [14]	CMA-ES	Pole balancing
2010	Glasmachers et al. [18]	xNES	Test function optimization
2011	Schaul et al. [25]	sNES	Test function optimization
2014	Wierstra et al. [19]	sNES, xNES	Pole balancing, Test function optimization
2017	Salimans et al. [28]	OpenAI-ES	MuJoCo robot locomotion, Atari games
2019	Pagliuca & Nolfi [15]	CMA-ES, sNES, xNES	Pole balancing, Car games, Swarm foraging
2020	Pagliuca, Milano & Nolfi [17]	CMA-ES, OpenAI-ES, sNES, xNES	MuJoCo robot locomotion
2020	Rais Martinez & Aznar Gregori [41]	CMA-ES, OpenAI-ES	Swarm aggregation
2022	Basilio & Goliatt [24]	xNES	Geopolymer self-compacting concrete
2022	Pagliuca & Vitanza [42]	OpenAI-ES	Swarm aggregation
2023	Pagliuca & Vitanza [43]	CMA-ES, xNES	Swarm aggregation
2024	El Saliby et al. [44]	($\mu +\lambda$)-ES, CMA-ES	Test function optimization, 2D navigation, Voxel-based soft robot control
2025	Pagliuca [11]	OpenAI-ES, sNES	Bit-parity, Grid navigation, Pole balancing, Test function optimization
2025	Pagliuca & Vitanza [35]	CMA-ES, OpenAI-ES & xNES	Swarm aggregation

Table 2. Analysis of the characteristics of CMA-ES, OpenAI-ES, sNES and xNES.

	CMA-ES	OpenAI-ES	sNES	xNES
Computational complexity	$O\left(d^2\right)$	$O\left(d\right)$	$O\left(d\right)$	$O\left(d^3\right)$
Exploration strategy	Sampling around centroid’s parameter distribution	Sampling around centroid’s parameter distribution	Sampling around centroid’s parameter distribution	Sampling around centroid’s parameter distribution
Parameters’ importance attribution	Covariance matrix	Momentum vectors	Variance vector	Covariance matrix
Scalability	No	No	Yes	Yes

Motivated by these considerations, this research seeks to fill a gap in the sub-optimality of CMA-ES, sNES and xNES with regard to domains involving hinged robots that have to locomote on flat terrain, or groups of robots that have to cooperate and, possibly, communicate to solve collective tasks. To address this sub-optimality, I propose the incorporation of the techniques positively affecting the performance of OpenAI-ES in such domains. In particular, I delve into the application of symmetric sampling and weight decay into CMA-ES, sNES and xNES, with the aim to verify whether the incorporation of these techniques leads to enhanced performance of the considered ESs in relatively challenging domains. For each ES, I thus propose three variants (denoted for simplicity as “<ES>_s”, “<ES>_wd” and “<ES>_swd”, with <ES> being either CMA-ES, or sNES, or xNES), and I investigate whether or not they provide advantages over the original algorithms in terms of performance. In particular, my analysis focuses on four problems: (i) the halfcheetah PyBullet robot locomotion problem [45], (ii) the hopper PyBullet robot locomotion task [46], (iii) the swarm aggregation problem [35,43] and (iv) the swarm foraging task [15]. Moreover, as a proof-of-concept, I investigate the effects of applying symmetric sampling on the Rosenbrock function [47], a benchmark task for test function optimization. I choose these tasks because CMA-ES, sNES and xNES have already been tested on them [15,17,35,43], which enables a fair comparison of the outcomes with those reported in the literature. The main goal of this research is to provide a theoretical framework showing that ESs may benefit (in terms of enhanced performance) from the incorporation of mechanisms allowing for the assessment of parameter influence, like symmetric sampling and weight decay (when necessary).

The achieved results confirm the hypothesis and demonstrate that the application of symmetric sampling and weight decay corresponds to performance enhancements in CMA-ES, sNES and xNES. This is particularly evident with regard to the halfcheetah and swarm aggregation problems. Specifically, symmetric sampling improves algorithm performance irrespective of both the considered ES and the specific problem. On the other hand, the application of weight decay produces improved results on CMA-ES and sNES, while the effect on xNES depends on the particular task. Moreover, the analysis of the parameter size indicates that keeping parameters limited in amplitude is paramount in the considered domains. Surprisingly, symmetric sampling produces a notable drop of the parameter size.

This research provides the following main contributions:

• I investigate the effectiveness of transferring symmetric sampling and weight decay on three state-of-the-art ESs (i.e., CMA-ES, sNES, xNES), similarly to the operation of OpenAI-ES;
• I propose three variants for CMA-ES, sNES and xNES, in which I apply either symmetric sampling, weight decay, or both;
• I show that the new variants enhance algorithm performance in two PyBullet robot locomotion (i.e., halfcheetah and hopper) and two collective (swarm aggregation and swarm foraging) problems;
• I demonstrate that keeping parameter size limited is paramount to improve performance in the considered domains, aligning this finding with [17,35].

The manuscript is organized as follows: Section 2 illustrates the problems and the ESs. The results are presented in Section 3 and further discussed in Section 4. In Section 5, I, provide my final remarks and future research pathways.

2. Materials and Methods

2.1. Rosenbrock Function Optimization

The first application scenario entails the optimization of the Rosenbrock function [47], which is a non-convex function commonly used to assess algorithmic optimization capabilities [48,49], including ESs [13,19]. Furthermore, it has been recently adopted to assess the efficacy of different algorithms [9,27,50], thus representing a benchmark task for the considered investigation. Given an input vector x of size d, the goal is to minimize the value of function $F_{rosen}$, which is defined in Equation (3).

$$F_{rosen}={\displaystyle \sum _{i=1}^{d-1}}(100{(x_{i+1}-{x_i}^2)}^2+{(x_i-1)}^2)$$

(3)

An illustration of the Rosenbrock function is provided in Figure 2.

Figure 2. Illustration of the Rosenbrock function for a two-dimensional input $x=[x_1,x_2]$. The optimal value $F\left(x\right)=0.0$ is achieved with $x=[1,1]$.

In the experiments reported here, I considered an input vector x of size $d=50$, which was chosen according to [11,27]. Algorithms sought to optimize $F_{rosen}$ in ${10}^6$ time steps. I performed 10 replications of the experiments. To ensure a fair comparison with existing studies, I adopted the same experimental settings employed in [9,11,27,50].

2.2. PyBullet Robot Locomotion Problems

The second and third problems belong to the family of PyBullet robot locomotion tasks [33]. Specifically, I selected the halfcheetah (Figure 3a) [45] and the hopper (Figure 3b) [46] problems. These two tasks are rather challenging for CMA-ES, sNES and xNES, since the behavior is highly affected by the parameter size, which will be shown in Section 3. Therefore, CMA-ES, sNES and xNES might encounter pitfalls in the search for solutions, due to their tendency to grow parameter size (see [12,17]). A similar outcome is reported in [17] with regard to MuJoCo locomotion problems.

Figure 3. PyBullet robot locomotion problems: (a) Halfcheetah; (b) hopper.

Both problems involve a mobile legged robot that has to locomote toward a target location placed approximately at 1 km away from the robot’s starting position. The robot can move only horizontally along the x-axis. The objective is to travel as far as possible. The performance is measured through the fitness functions defined in Equations (4) and (5):

$$F_{ha}={\displaystyle \sum _{i=1}^{N_{steps}}}\left(∆p\left(i\right)-0.1\times {\displaystyle \sum _{j=1}^{N_J}}{l}_j\left(i\right)\right)$$

(4)

$$F_{ho}={\displaystyle \sum _{i=1}^{N_{steps}}}∆p\left(i\right)$$

(5)

where $∆p\left(i\right)$ is the displacement of the robot at step i, $N_{steps}$ denotes the number of steps and $l_j\left(i\right)$ is a flag that activates (i.e., $l_j\left(i\right)=1$) when the joint j reaches its limits at step i. The latter component is adopted to discourage unnatural behaviors that might circumvent the physical limitations of the PyBullet engine.

The robots are controlled by a Multi-Layer-Perceptron (MLP) neural network [51] with one internal layer containing 50 neurons. I used this value since it has been largely employed in previous studies on PyBullet robot locomotion tasks [17,34]. The network has 26 inputs and 6 outputs in the case of the halfcheetah problem (see Table 3), whereas the number of inputs and outputs are, respectively, 15 and 3 for the hopper task (see Table 3).

Table 3. List of sensors (denoted with $s_{<num>}$) and motors (indicated with $m_{<num>}$) of the halfcheetah and hopper robots. The symbols represent the following variables: z is the robot height; $z_{init}$ is the robot height when the episode begins; ${angle}_{to\_target}$ is the relative robot–target angle; $v_x,v_y,v_z$ are the velocity components along the three axes; $\varphi$ and $\theta$ represent the roll and pitch orientation angles of the robot; $(p_j,v_j)$ denotes the position and velocity of the joint j; $c_f$ is set to 1 when the foot f is on the ground, otherwise it is set to 0; and $T_j$ indicates the torque applied to joint j. The number of joints $N_J$ and the number of segments $N_S$ is 6 for the halfcheetah and 3 for the hopper.

Parameter	Value
$s_1$	$z-z_{init}$
$s_2$	$sin({angle}_{to\_target})$
$s_3$	$cos({angle}_{to\_target})$
$s_4$	$0.3\times v_x$
$s_5$	$0.3\times v_y$
$s_6$	$0.3\times v_z$
$s_7$	$\varphi$
$s_8$	$\theta$
$s_9-s_{9+N_J\times 2}$	$(p_j,v_j)forj\in N_J$
$s_{9+N_J\times 2+1}-s_{9+N_J\times 2+1+N_S}$	$c_{i}fori\in N_S$
$m_1-m_{N_M}$	$T_{j}forj\in N_J$

2.3. Collective Problems

The fourth and fifth problems belong to the swarm robotics domain [52,53,54], that is, the scenario in which a group of robots (defined a “swarm”) cooperates with the aim to solve a problem that cannot be tackled by a single individual. More specifically, I delve into the swarm aggregation problem introduced in [35,43], and the swarm foraging task proposed in [15].

The swarm aggregation problem involves a swarm of 10 simulated MarXbot robots [55] living in a square walled arena of size 5 m × 5 m (see Figure 4a). The arena contains a nest (grey circle in Figure 4a), which represents a possible aggregation site. The robots’ initial positions and orientations are randomly set at each episode. Similarly, the location of the nest is randomly initialized. The goal for the swarm is to reach a good level of aggregation in the environment. According to [35,43], I used the fitness function shown in Equation (6).

$$F_{sa}={\displaystyle \sum _{r=1}^{N_R}}{F}_r$$

(6)

Figure 4. Collective problems: (a) Swarm aggregation; (b) swarm foraging.

The fitness $F_{sa}$ of the swarm is the cumulative sum of the fitness $F_r$ achieved by each robot r, which is defined in Equation (7).

$$F_r=nest\left(r\right)+\left(1-{\displaystyle \frac{1}{N_R-1}}{\displaystyle \sum _{o=1}^{N_R-1}}{\displaystyle \frac{d_{ro}}{D}}\right)$$

(7)

In Equation (7), $nest\left(r\right)$ is a bonus activation if the robot r is inside the nest ($nest\left(r\right)=1$), $d_{ro}$ is the distance between the robots r and o, and D is the diagonal of the arena (i.e., the theoretical maximum distance in the environment). I chose this fitness function since it represents the most complex version of the problem described in [35]. In fact, with this function, there is no explicit pressure towards reaching the nest or minimizing the distance with the mates, and ESs are free to discover any solution maximizing the fitness function $F_{sa}$. Clearly, different experimental settings (e.g., different values for $nest\left(r\right)$ and/or using a coefficient $\ge 1$ for the distance component $\left(1-{\displaystyle \frac{1}{N_R-1}}{\sum }_{o=1}^{N_R-1}{\displaystyle \frac{d_{ro}}{D}}\right)$) might affect the algorithmic performance, as illustrared in [35].

A fully recurrent neural network (RNN) [56,57] determines the robot behavior. In particular, the network has one internal layer of 10 neurons, a value derived from the works in [35,42,43,58]. Sensory information is retrieved by 8 InfraRed (IR) units (each one encoding the average value of 4 adiacent sensors), 4 units encoding the information perceived by a ${90}^{\circ }$ Field-Of-View (FOV) frontal linear camera, and 2 units from the ground sensors. The network outputs encode the left and right wheel speeds, and whether or not the frontal-red Light-Emitting Diode (LED) and the rear-blue LED are active. A summary of the sensors and motors is provided in Table 4. The network is shared between the robots.

Table 4. List of sensors (denoted with $s_{<num>}$) and motors (indicated with $m_{<num>}$) used in the collective problems.

	Swarm Aggregation	Swarm Foraging
$s_1-s_8$	IRs	IRs
$s_9-s_{12}$	Camera	Camera
$s_{13}-s_{14}$	Ground	Ground
$s_{15}$	-	Battery
$s_{16}$	-	Bias
$m_1-m_2$	Wheel speeds	Wheel speeds
$m_3-m_4$	LEDs	LEDs

The swarm foraging problem shares the same environmental features of the previous task (number and type of robots in the swarm, size of the arena, presence of the nest). However, the nest represents the source location for the agents, which are initially located randomly inside it. Moreover, the robots have a battery that reduces through time, which has to be periodically recharged in the nest. As in [15], I consider 400 invisible food items in the arena, and the robots must transport the items to the nest in order to forage them. The objective of the swarm is to collect and forage the the highest number of food items in the environment. To this end, I used the fitness function in Equation (8):

$$F_{sf}={\displaystyle \sum _{r=1}^{N_R}}{n}_F\left(r\right)$$

(8)

where $n_F\left(r\right)$ is the number of food items successfully foraged by the robot r.

Similarly to the former problem, a fully recurrent neural network is used to control the robot behaviors. The network architecture is almost identical to the previous problem, except for the addition of one input encoding the battery level and one bias input (see Table 4). Furthermore, in this case, the neural network is shared among the robots.

2.4. Evolutionary Strategies

This study focuses on three state-of-the-art ESs: Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [13], Separable Natural Evolution Strategies (sNESs) [19,25] and Exponential Natural Evolution Strategies (xNESs) [18,19]. I selected these methods since they have already been used to discover solutions for the Rosenbrock function [13,19] and the swarm foraging task [15,17]. Furthermore, they showed poor performance in robot locomotion [17] and obtained sub-optimal results in the swarm aggregation problem [35,43]. The algorithms are illustrated in Algorithms A1–A3 in Appendix A. Despite the availability of existing variants, I considered the original algorithms introduced in [13,18,19,25].

Both CMA-ES and xNES use a covariance matrix to identify relationships between parameters enabling to guide the search in the solution space towards the regions that are likely to correspond to enhanced performance. The two methods are tightly linked, as indicated in [19,20]. In particular, both ESs follow the “natural gradient” [21], which is computed by measuring the Kullback–Leibler (KL) divergence [59] between two parameter distributions and penalizes high variances between the distributions (i.e., uncertainty). A mathematical definition of the natural gradient is provided in Equation (9):

$$\widetilde{{\nabla }_{\theta }}L={F\left(\theta \right)}^{-1}\times {\nabla }_{\theta }L$$

(9)

where $\widetilde{{\nabla }_{\theta }}L$ is the natural gradient of the loss/objective function with respect to parameters $\theta$, ${\nabla }_{\theta }L$ denotes the plain gradient, and ${F\left(\theta \right)}^{-1}$ is the inverse of the Fisher Information Matrix (FIM) defined in Equation (10).

$$\begin{array}{c}\hfill F=\mathbb{E}\left({\nabla }_{\theta }log\pi (x\mid \theta )\times {{\nabla }_{\theta }log\pi (x\mid \theta )}^T\right)\end{array}$$

(10)

In Equation (10), the symbol $\mathbb{E}$ states the expected value, $\pi \left(\right)$ is the policy to be optimized and x indicates a generic sample.

Compared to the plain gradient, which is based on the Euclidean distance, the natural gradient is independent of the distribution parameterization and represents a more reliable update direction [19]. For further details about the natural gradient, the readers are referred to [19,21,60].

The efficacy and versatily of CMA-ES and xNES have been demonstrated in different domains, such as test function optimization [13,16,19], pole balancing applications [15,16] and robotic scenarios [23,35,61]. However, calculating the covariance matrix can be computationally expensive, which prevents CMA-ES and xNES from being used in high-dimensional parameter spaces [19,25,26,27]. Specifically, CMA-ES and xNES do not scale with an increasing number of parameters: as shown in [17,26], CMA-ES and xNES lead to memory allocation errors when tested on high-dimensional problems such as the MuJoCo humanoid task. In fact, Pagliuca and colleagues reported that calculating/updating the covariance matrix on 166,673 parameters, as in the MuJoCo humanoid problem, is computationally intractable [17]. The sNES algorithm allows for the circumvention of these limitations by storing separable parameter distributions [25], hence reducing the computational load of the method. This makes sNES suitable for high-dimensional search space [17]. Moreover, sNES showed good performance with respect to test function optimization [19,25], pole balancing [11,15,19] and car simulation games [15].

Although CMA-ES, sNES and xNES have been applied to a wide range of application domains, previous comparative studies demonstrated that they are bested by the OpenAI Evolutonary Strategy (OpenAI-ES) [28] with respect to MuJoCo robot locomotion tasks [17] and swarm robotics problems [17,35]. As already discussed in Section 1, drawing inspiration from OpenAI-ES, I propose three variants for each considered ES:

• Symmetric variant (termed as “<ES>_s”, where <ES> can be CMA-ES, sNES or xNES): It incorporates symmetric sampling [30] to better estimate the importance of each parameter and, thus, the performance gradient;
• Weight decay variant (termed as “<ES>_wd”): It applies weight decay [38] to keep the parameter size constrained;
• Combined symmetric and weight decay variant (termed as “<ES>_swd”): It applies both previous variants to improve gradient estimation and reduce parameter size.

It is worth noting that the symmetric version of sNES has already been used in [37] to evolve solutions for a collective halfcheetah locomotion task [62].

The implementation of the three variants are integrated in Algorithms A1–A3 through two If-statements (Algorithms A1–A3, lines 7–10 and 15–16). In particular, symmetric sampling is applied on the extracted samples by considered both positive and negative directions (Algorithms A1–A3, lines 7–10). Given the samples extracted from the Gaussian distribution (line 4), symmetric sampling acts by also analyzing the negative samples (lines 8–9). This allows the algorithm to better assess the parameters’ influence on the performance. Instead, weight decay acts directly on the update of the parameters (Algorithms A1–A3, lines 15–16). Specifically, weight decay is applied after the parameter updates (line 14) and is modulated by the centroid’s learning rate (line 15). The decay factor 0.005 was chosen because it does not strongly impact the centroid update and does not interfere with the other mechanisms of the considered ESs. Furthermore, the value was inherited from the OpenAI-ES implementation (see [28,36]).

2.5. Experimental Settings

All the experiments were replicated 10 times by using different seeds. Although this value might theoretically limit the validity of this investigation, it actually represents a good trade-off between computational load, time required to perform simulations and performance evaluation. Furthermore, this value has been used in previous research works [12,17,34,63]. Table 5 reports the experimental settings used. I used the Xavier method [64] to initialize the neural network controller parameters (i.e., connection weights and biases). Moreover, for locomotion problems, I use batch normalization [65], as already done in [17,28].

Table 5. Summary of the experimental settings used for the halfcheetah, hopper, swarm aggregation and swarm foraging problems. These values were taken from [15,17,35,43] and allow us to perform a fair comparison with existing studies. $N_{reps}$ denotes the number of replications; $N_{evals}$ indicates the number of evaluations performed during evolution; $N_R$ represents the number of robots; $N_{eps}$ is the number of evaluation episodes; $N_{steps}$ is the number of steps of an episode; d is the number of evolvable parameters.

	Halfcheetah	Hopper	Swarm Aggregation	Swarm Foraging
$N_{reps}$	10	10	10	10
$N_{evals}$	$5\times {10}^7$	$5\times {10}^7$	$1.5\times {10}^9$	$1.5\times {10}^9$
$N_R$	1	1	10	10
$N_{eps}$	3	3	20	20
$N_{steps}$	1000	1000	1000	1000
d	1656	953	406	434
WeightInit BatchNorm	Yes	Yes	No	No

Concerning CMA-ES, sNES and xNES, the algorithmic parameters are illustrated in Table 6.

Table 6. Parameters used to discover solutions with the CMA-ES, sNES and xNES algorithms. I adopted the default values reported in [13,18,19,25]. The symbols are defined as follows: PopSize defines the population size. $N_{samples}$ is the number of samples. ${\eta }_{\theta }$ and ${\eta }_{\sigma }$ are, respectively, the learning rate for the centroid $\theta$ and the covariance/variance $\sigma$. $n_{evals}$ indicates the number of evaluations performed. The symbol “-” is used when a parameter is not defined for the corresponding ES.

	CMA-ES	sNES	xNES
PopSize	1	1	1
$N_{samples}$	$4+\lfloor 3\times ln\left(d\right)\rfloor$	$4+\lfloor 3\times ln\left(d\right)\rfloor$	$4+\lfloor 3\times ln\left(d\right)\rfloor$
${\eta }_{\theta }$	1.0	1.0	1.0
${\eta }_{\sigma }$	value defined as in [13]	$min({\displaystyle \frac{1}{d}},0.25)$	$min({\displaystyle \frac{1}{d}},0.25)$
StepSize	0.5	${\displaystyle \frac{1}{\frac{4+\lfloor 3\times ln\left(d\right)\rfloor }{2}}}$	-
Termination	$n_{evals}=N_{evals}$	$n_{evals}=N_{evals}$	$n_{evals}=N_{evals}$
Seeds	$1\to 10$	$1\to 10$	$1\to 10$

All the experiments were performed with evorobotpy3 [36], a modern simulation tool that provides a broad list of EAs (including CMA-ES, sNES and xNES) and tasks (including swarm aggregation and swarm foraging). Moreover, it incorporates external libraries such as PyBullet, MuJoCo and Gymnasium [66,67], hence allowing users to run experiments of robot locomotion tasks (including halfcheetah and hopper) and Atari games. Evorobotpy3 has been recently used to create custom environments and run simulations on a wide spectrum of problems [27,63,68]. Furthermore, evorobotpy3 contains the implementation of CMA-ES, sNES and xNES algorithms and their symmetric variants, and I used those versions for the experiments presented in this paper. All the simulations were run on an Intel Xeon-Gold 6252N, 128-gigabyte (GB) Random Access Memory (RAM), and an Ubuntu 20.04 Operating System (OS).

3. Results

This section provides a thorough description of the outcomes achieved in the selected problems. Regarding the statistical analysis, I used the Mann–Whitney U test [69] for the Rosenbroack function optimization, and the Kruskal–Wallis H test [70] combined with the Dunn post hoc test [71] for the other problems. Moreover, to analyze relationships between variables, I employed the Spearman rank correlation coefficient [72]. Statistical significance is reported at a threshold $p<0.05$.

3.1. Rosenbrock Function Optimization

As a proof-of-concept of my hypothesis, I analyzed CMA-ES, sNES, xNES and their symmetric variants with respect to the Rosenbrock function. As shown in Figure 5, CMA-ES and CMA-ES_s converged to an almost optimal solution within the first 5000 steps, while xNES and xNES_s required approximately $1.5\times {10}^5$ steps to discover the optimum. Finally, sNES and sNES_s got stuck in sub-optimal solutions (Figure 5). In statistical terms, both CMA-ES_s and sNES_s significantly outperformed CMA-ES and sNES ($p<0.05$). It is worth noting that I did not apply weight decay given the particular task objective (i.e., parameter minimization).

Figure 5. Fitness convergence of CMA-ES, sNES, xNES and their symmetric variants (denoted with the suffix “_s”) with respect to the Rosenbrock function. Data are averaged over 10 replications.

3.2. PyBullet Robot Locomotion Problems

The performance analysis of the different algorithms and their variants is summarized in Table 7. With regard to the PyBullet robot locomotion problems, the symmetric variants CMA-ES_s, sNES_s and xNES_s significantly outperform the original CMA-ES, sNES and xNES algorithms ($p<0.05$ for each pair). Moreover, CMA-ES_swd is remarkably better than CMA-ES ($p<0.05$). Therefore, the application of symmetric sampling results in enhanced performance. In addition, concerning the hopper problem, the symmetric variants achieve higher performance than the original algorithms (this is particularly evident for the sNES algorithm and its variants), although there is no statistically significant difference ($p>0.05$). Interestingly, the CMA-ES_wd strongly outperforms CMA-ES ($p<0.05$). Overall, these outcomes indicate that symmetric sampling is beneficial for performance enhancement in robot locomotion tasks. The fact that the hopper problem is relatively easier than the halfcheetah task explains why symmetric sampling does not produce significantly better results in the former scenario.

Table 7. Average performance achieved by the different ESs in 10 replications of the experiments. Data are illustrated as $\mu \pm \sigma$ (respectively, mean and standard deviation), and bold values identify the best outcomes.

	Halfcheetah	Hopper	Swarm Aggregation	Swarm Foraging
CMA-ES	$325.245\pm 141.232$	$1428.111\pm 286.928$	$0.896\pm 0.199$	$153.605\pm 13.197$
CMA-ES_s	$1191.764\pm 434.595$	$1532.802\pm 373.663$	$\mathbf{1.147}\pm \mathbf{0.221}$	$\mathbf{179.310}\pm \mathbf{18.482}$
CMA-ES_swd	$1318.588\pm 492.879$	$\mathbf{1862.538}\pm \mathbf{388.316}$	$1.038\pm 0.120$	$160.085\pm 15.476$
CMA-ES_wd	$687.086\pm 375.933$	$1450.183\pm 320.090$	$1.131\pm 0.111$	$169.460\pm 12.858$
sNES	$296.042\pm 58.318$	$826.994\pm 610.100$	$0.726\pm 0.099$	$86.490\pm 13.375$
sNES_s	$1327.817\pm 300.128$	$1158.427\pm 618.993$	$0.787\pm 0.123$	$95.180\pm 4.447$
sNES_swd	$637.021\pm 321.079$	$1313.770\pm 280.792$	$0.948\pm 0.111$	$130.005\pm 8.845$
sNES_wd	$262.203\pm 123.086$	$1394.895\pm 259.452$	$0.823\pm 0.064$	$122.815\pm 8.166$
xNES	$351.127\pm 206.058$	$1618.882\pm 320.384$	$0.699\pm 0.063$	$157.745\pm 13.160$
xNES_s	$\mathbf{1561.019}\pm \mathbf{312.735}$	$1793.127\pm 443.307$	$0.805\pm 0.162$	$161.340\pm 17.163$
xNES_swd	$782.082\pm 161.120$	$1339.842\pm 275.274$	$0.927\pm 0.103$	$151.475\pm 7.011$
xNES_wd	$470.423\pm 309.552$	$1452.774\pm 275.731$	$0.830\pm 0.108$	$156.020\pm 6.542$

The performance distributions achieved in the PyBullet robot locomotion tasks are illustrated in Figure 6a,b. A noteworthy feature is the limited variance in the performance of the CMA-ES, sNES and xNES algorithms (red, green and blue violins) with respect to the halfcheetah problem (see Figure 6a). This implies that these ESs converge towards similar solutions in the different experimental replications. Conversely, the variants display performance differences, with higher median values (white horizontal lines within the violins). Concerning the hopper task, all the algorithms show more pronounced performance variances (see Figure 6b). This is especially evident for sNES and sNES_s (Figure 6b, green and neon green violins).

Figure 6. Performance distribution of the original methods (i.e., CMA-ES, sNES and xNES) and their variants in the considered problems: (a) Halfcheetah; (b) hopper; (c) swarm aggregation; (d) swarm foraging.

3.3. Collective Problems

By looking at the results achieved in the swarm aggregation problem, all the proposed variants outperform the original CMA-ES, sNES and xNES algorithms. More specifically, CMA-ES_s is remarkably better than CMA-ES ($p<0.05$). In addition, sNES_swd and xNES_swd are superior to sNES and xNES ($p<0.05$), respectively. As can be observed in Table 7, both symmetric sampling and weight decay lead to higher fitness values on average. However, the combination of the two techniques generates the best outcomes for sNES and xNES. On the other hand, I observe a significant performance enhancement for CMA-ES with the application of symmetric sampling only.

Concerning the swarm foraging problem, CMA-ES_s is significantly better than CMA-ES ($p<0.05$). Furthermore, sNES_swd and sNES_wd strongly outperform sNES ($p<0.05$). Instead, there are no significant differences between xNES and its variants ($p>0.05$). Therefore, similarly to the former case, I observe that symmetric sampling is beneficial for all the algorithms, particularly for CMA-ES. Moreover, the combination of symmetric sampling and weight decay produces better performance, especially for sNES.

As pointed out in [17,35], a noteworthy feature affecting performance of the original CMA-ES, sNES and xNES algorithms in the considered domains is the parameter size $\left|w\right|$. As observed in Table 8, all the variants evolve solutions characterized by remarkably lower values of $\left|w\right|$ ($p<0.05$ for each pair of comparison). Interestingly, CMA-ES_s produces solutions with lower $\left|w\right|$ than CMA-ES_wd (see Table 8). This depends on the implementation of the weight decay technique in CMA-ES (see Algorithm A1).

Table 8. Average parameter size of the individuals evolved by the different ESs in 10 replications of the experiments. Data are illustrated as $\mu \pm \sigma$ (respectively, mean and standard deviation).

	Halfcheetah	Hopper	Swarm Aggregation	Swarm Foraging
CMA-ES	$8.861\pm 2.124$	$4.942\pm 2.061$	$3.359\pm 2.517$	$5.003\pm 1.173$
CMA-ES_s	$1.132\pm 0.043$	$1.031\pm 0.045$	$0.687\pm 0.161$	$0.761\pm 0.045$
CMA-ES_swd	$0.176\pm 0.049$	$0.216\pm 0.103$	$0.295\pm 0.134$	$0.301\pm 0.061$
CMA-ES_wd	$1.526\pm 0.605$	$1.096\pm 0.107$	$0.785\pm 0.268$	$0.548\pm 0.140$
sNES	$21.672\pm 4.524$	$32.675\pm 24.908$	$4.684\pm 5.916$	$18.791\pm 7.384$
sNES_s	$5.843\pm 0.712$	$3.956\pm 2.190$	$4.688\pm 3.383$	$5.785\pm 1.144$
sNES_swd	$1.602\pm 0.033$	$1.594\pm 0.094$	$1.740\pm 0.063$	$1.404\pm 0.081$
sNES_wd	$2.478\pm 0.048$	$2.648\pm 0.049$	$2.739\pm 0.298$	$2.851\pm 0.092$
xNES	$26.672\pm 9.848$	$12.088\pm 2.747$	$5.717\pm 4.919$	$29.654\pm 6.738$
xNES_s	$7.734\pm 0.623$	$6.584\pm 1.302$	$5.030\pm 5.184$	$11.029\pm 2.250$
xNES_swd	$2.042\pm 0.083$	$1.938\pm 0.101$	$2.112\pm 0.494$	$2.520\pm 0.236$
xNES_wd	$3.690\pm 0.533$	$2.958\pm 0.272$	$4.136\pm 0.581$	$4.869\pm 0.893$

I also analyzed the runtime execution of the different ESs, which is summarized in Table 9. As can be seen, the usage of symmetric sampling reduces the time required to perform simulations of CMA-ES and xNES. Concerning sNES, I observe similar runtime values for the original algorithm and its variants. Clearly, because sNES does not store any covariance matrix, it is remarkably faster than CMA-ES and xNES (see Table 9).

Table 9. Runtime execution (in seconds) of the different ESs in 10 replications of the experiments. Data are shown as $\mu +\sigma$ (respectively, mean and standard deviation).

	Halfcheetah	Hopper	Swarm Aggregation	Swarm Foraging
CMA-ES	$2,403,262.600\pm 346,527.141$	$587,685.000\pm 64,178.968$	$227,857.700\pm 16,119.858$	$602,310.400\pm 30,751.172$
CMA-ES_s	$895,595.200\pm 138,957.561$	$431,028.900\pm 170,173.171$	$221,279.400\pm 23,526.027$	$495,266.300\pm 55,198.447$
CMA-ES_swd	$1,165,787.800\pm 447,464.025$	$486,938.000\pm 178,398.972$	$191,814.900\pm 12,953.183$	$443,185.100\pm 12,211.343$
CMA-ES_wd	$2,605,455.600\pm 847,680.235$	$585,831.000\pm 36,673.750$	$245,264.000\pm 6957.128$	$661,917.000\pm 15,266.188$
sNES	$403,481.300\pm 5340.709$	$117,324.200\pm 4455.388$	$75,741.600\pm 1742.606$	$693,070.300\pm 5420.562$
sNES_s	$392,439.600\pm 4915.888$	$116,163.700\pm 2784.170$	$80,960.600\pm 2036.227$	$686,855.700\pm 10,435.205$
sNES_swd	$399,412.100\pm 4441.495$	$114,529.900\pm 1072.164$	$89,258.500\pm 1991.245$	$685,831.600\pm 8647.254$
sNES_wd	$402,771.300\pm 3047.876$	$114,956.200\pm 668.527$	$85,327.100\pm 2094.511$	$695,024.800\pm 7660.745$
xNES	$3,085,422.600\pm 704,472.062$	$761,742.000\pm 8755.846$	$447,330.400\pm 2642.274$	$707,339.500\pm 7728.604$
xNES_s	$1,014,073.000\pm 47,048.866$	$640,952.100\pm 12,914.478$	$337,717.800\pm 5760.611$	$662,463.200\pm 14,796.472$
xNES_swd	$1,578,066.100\pm 292,822.148$	$646,775.100\pm 7294.765$	$348,927.700\pm 9513.006$	$677,261.000\pm 6785.183$
xNES_wd	$3,377,565.200\pm 616,647.901$	$777,373.3\pm 17,405.381$	$452,076.000\pm 1642.544$	$709,013.400\pm 3205.601$

The analysis of correlation between the parameter size $\left|w\right|$ and the performance F is shown in Table 10. Specifically, there exists a significantly negative correlation between the two variables in the halfcheetah, swarm aggregation and swarm foraging problems (Table 10, last row). This means that reducing $\left|w\right|$ produces higher values of F. This is in line with the outcomes reported in [35]. However, there are significant differences between the considered ESs. In fact, the performance of CMA-ES and its variants increases when the parameter size $\left|w\right|$ decreases regardless of the task. This is particularly evident for the halfcheetah and hopper problems (see Table 10). Regarding sNES, there is a strong negative correlation between $\left|w\right|$ and F in the swarm foraging problem (see Table 10). Instead, the performance of xNES positively correlates with parameter size in the hopper problem (see Table 10).

Table 10. Spearman correlation coefficient $\rho$ between parameter size $\left|w\right|$ and performance F for each evolutionary problem. Statistical significance is highlighted in bold.

	Halfcheetah	Hopper	Swarm Aggregation	Swarm Foraging
CMA-ES	−0.607	−0.444	$-0.094$	$-0.135$
sNES	$-0.031$	$0.052$	$-0.108$	−0.825
xNES	$-0.160$	0.525	$0.281$	$0.303$
Overall	−0.347	$-0.034$	−0.310	−0.327

Figure 7 shows how performance F varies based on $\left|w\right|$ with respect to the considered problems, where I take into account all the solutions discovered by CMA-ES, sNES, xNES and their variants. As can be observed, the evolved solutions are generally characterized by a broad range of $\left|w\right|$, and the best performance is obtained by solutions with limited $\left|w\right|$ (see Figure 7, left side of each subplot). A worthwhile exception is represented by the hopper task, for which all the algorithms typically evolve solutions with moderate values of $\left|w\right|$ (Figure 7b, see also Table 10).

Figure 7. Correlation between the average absolute value of weights $\left|w\right|$ and the performance F: (a) Halfcheetah; (b) hopper; (c) swarm aggregation; (d) swarm foraging.

4. Discussion

Overall, the results illustrated in Section 3 produce useful considerations. The analysis involving the Rosenbrock function, which represents the proof-of-concept of this study, demonstrates that symmetric sampling is beneficial for sNES and xNES in terms of convergence speed. Moreover, it significantly enhances the performance of both CMA-ES and sNES. These outcomes prove that symmetric sampling positively influences the considered ESs even in this relatively simple scenario. Future studies might clarify whether these outcomes hold as the input size d scales.

With respect to the remaining problems, the outcomes emphasize that symmetric sampling is advantageous (in terms of performance enhancements) for CMA-ES, sNES and xNES. In detail, CMA-ES_s is significantly better than CMA-ES in the halfcheetah, swarm aggregation and swarm foraging problems. Furthermore, with regard to halfcheetah, sNES_s and xNES_s are remarkably superior to sNES and xNES, respectively. The addition of weight decay is beneficial to CMA-ES with regard to all the problems and produces a performance enhancement for sNES in three problems (hopper, swarm aggregation and swarm foraging), whereas the benefits are limited to two tasks (halfcheetah and swarm aggregation) for xNES. Finally, the combination of symmetric sampling and weight decay is advantageous for CMA-ES and sNES with respect to all the problems: CMA-ES_swd significantly outperforms CMA-ES in the halfcheetah task, whereas sNES_swd is better than sNES in the two collective problems. In addition, xNES_swd exhibits a performance enhancement over xNES in two problems (halfcheetah and swarm aggregation), with a significant difference between the methods in the swarm aggregation task.

To provide a more comprehensive analysis, I report the results of an additional series of experiments on the HC algorithm, by using the parameters provided in Table 11. As can be seen in Table 12, HC poorly performs on the halfcheetah and hopper tasks compared to the other considered ESs. Interestingly, HC is better than other methods in the swarm aggregation task, and is also the second-best performing algorithm in the swarm foraging problem. Lastly, it achieves better performance than sNES and sNES_s in the Rosenbrock function. Therefore, these outcomes indicate that the complexification of ESs through techniques storing the parameter importance, like covariance matrix or variance vector, works depending on the considered problem and is not a general feature. These findings align with the results presented in recent literature studies [11,12,27,44].

Table 11. Parameters used to run experiments with the HC algorithm. I adopted the same settings as in [12].

Parameter	Value
$N_{reps}$	10
$PopSize$	50
$MutRate$	0.01

Table 12. Performance of the HC algorithm on the considered tasks in 10 replications of the experiments. Data are shown as $\mu +\sigma$ (respectively, mean and standard deviation).

Rosenbrock	Halfcheetah	Hopper	Swarm Aggregation	Swarm Foraging
$0.240\pm 0.021$	$67.337\pm 22.855$	$301.997\pm 138.418$	$1.399\pm 0.130$	$176.422\pm 64.989$

Another relevant outcome of the presented analysis lies in the effect of symmetric sampling on parameter size $\left|w\right|$, whose impact on the final performance is remarkable in the presented domains (see Table 10). Indeed, as shown in Table 8, considering both positive and negative samples result in a remarkable drop of $\left|w\right|$. A similar trend has been observed for OpenAI-ES in [11]. Surprisingly, CMA-ES_s discovers solutions that have a smaller $\left|w\right|$ value than CMA-ES_wd in three problems (halfcheetah, hopper and swarm aggregation, see Table 8).

Given that the application of weight decay facilitates the reduction of $\left|w\right|$ by definition, the combination of symmetric sampling and weight decay leads to a notable drop in parameter size (see Table 8). In particular, the decrease of $\left|w\right|$ ranges from $11.386$ (swarm aggregation) to $50.347$ (halfcheetah) for CMA-ES, from $2.692$ (swarm aggregation) to $20.499$ (hopper) for sNES, and from $2.707$ (swarm aggregation) to $13.062$ for xNES (halfcheetah). Therefore, the swarm aggregation problem is less affected by the reduction of $\left|w\right|$. This is confirmed by the fact that the solutions discovered by CMA-ES, sNES and xNES for this task are characterized by smaller $\left|w\right|$ than for the other problems (see Table 8). In particular, concerning sNES, there is a significant difference in the parameter size between the solutions evolved for the swarm aggregation task and those discovered for the other problems ($p<0.05$). A similar trend is observed for xNES with regard to halfcheetah and swarm foraging ($p<0.05$), and for CMA-ES with respect to halfcheetah ($p<0.05$). Furthermore, the combined effect of symmetric sampling and weight decay is more pronounced for the halfcheetah and hopper problems. In fact, in these tasks, keeping $\left|w\right|$ small is paramount to finding effective solutions.

In general, the parameter size of the solutions discovered by CMA-ES is smaller than sNES and xNES, especially with regard to halfcheetah, hopper and swarm foraging tasks ($p<0.05$, see Table 8). The same outcome is observed by looking at the CMA-ES, sNES and xNES variants, irrespective of the specific problem ($p<0.05$). A worthwhile difference between CMA-ES and the other ESs lies in the usage of the best half of samples to update the solution (Algorithm A1, line 12), which narrows the search for influential parameter relationships. Instead, sNES and xNES attribute importance to all samples, albeit proportional to their efficacy. Another dissimilarity regards the covariance matrix update: CMA-ES exploits cumulative paths (Algorithm A1, line 13) and keeps the matrix update linear (Algorithm A1, line 14). Conversely, xNES updates the covariance matrix through a matrix exponential operator (Algorithm A3, line 14), which produces bigger $\left|w\right|$ values in the long term.

It is worth underscoring that these observations hold for the PyBullet robot locomotion and collective problems considered here. In fact, there is a tight relationship between the efficacy of symmetric sampling and weight decay and the properties of the problem. For example, when considering a robot controlled by a sensory–motor neural network, keeping the parameter size limited is likely to produce poor performance. Similarly, the analysis in [12] demonstrates that there is no guarantee that symmetric sampling and weight decay are advantageous in collective problems involving heterogeneous agents. Future studies should clarify the domains in which symmetric sampling and weight decay might play a role, hence providing guidelines for researchers.

5. Conclusions

Evolutionary Strategies (ESs) are optimization metaheuristics that are inspired by evolutionary principles and apply evolutionary operators (i.e., mutation, recombination and selection). Since their introduction in the early 70s, several modifications of the original formulation have been proposed, and the most advanced versions apply rather sophisticated techniques. For example, ESs like the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) and Exponential Natural Evolution Strategies (xNESs) calculate a covariance matrix storing the mutual relationships between the parameters to be optimized, which is computationally demanding and scales linearly with the number of parameters. On the other hand, the Separable Natural Evolution Strategy algorithm (sNES) reduces this burden by storing separate parameter variances, which is suitable for large search spaces. Moreover, the OpenAI Evolutionary Strategy (OpenAI-ES) keeps historical information of how performance varies by means of two momentum vectors. Furthermore, it applies symmetric sampling and weight decay, which allow the algorithm to both have a better understanding of how parameter variations affect performance and keep parameter size limited. The latter property is pivotal in domains such as PyBullet and MuJoCo robot locomotion problems. Because CMA-ES, sNES and xNES achieve sub-optimal performance in these domains, applying mechanisms like symmetric sampling and weight decay to these ESs might positively impact their efficacy.

This research investigates the effects of transferring symmetric sampling and weight decay to CMA-ES, sNES and xNES. Specifically, I propose three variants for CMA-ES, sNES and xNES, termed as “<ES>_s”, “<ES>_wd” and “<ES>_swd” (with <ES> corresponding to CMA-ES, sNES or xNES), which apply, respectively, symmetric sampling, weight decay, or both. To showcase the effects of these variants, I perform a comparison on four problems: (i) halfcheetah robot locomotion; (ii) hopper robot locomotion; (iii) swarm aggregation and (iv) swarm foraging. Furthermore, as a proof-of-concept, I provide evidence of the effects of symmetric sampling on the Rosenbrock function optimization. Experiments in these domains demonstrate that the proposed variants are in general more effective than the original algorithms. In particular, symmetric sampling and weight decay improve the performance of CMA-ES irrespective of the considered problem. The positive impact of symmetric sampling on CMA-ES is statistically significant with regard to halfcheetah, swarm aggregation and swarm foraging tasks. Similarly, symmetric sampling produces a performance enhancement in sNES, especially for the halfcheetah task, while the simultaneous application of symmetric sampling and weight decay to sNES is remarkably advantageous in the collective problems. As concerns xNES, symmetric sampling is particularly beneficial for the halfcheetah problem, whereas the combined symmetric sampling and weight decay variant is more effective in the swarm aggregation task. These outcomes are motivated by the fact that all the proposed variants produce a drop in the parameter size $\left|w\right|$, which is crucial to finding effective solutions in these domains [17,35]. However, I observed different trends depending on the specific ES and task: CMA-ES keeps $\left|w\right|$ smaller than sNES and xNES, thanks to its operational mechanisms. In addition, the solutions for the swarm aggregation task are typically characterized by smaller $\left|w\right|$ than those found for the other problems. Lastly, the application of symmetric sampling to CMA-ES and sNES is significantly advantageous for optimizing the Rosenbrock function. Overall, the achieved results demonstrate that, in the considered domains, symmetric sampling and weight decay are useful techniques to enhance the performance of ESs.

As discussed in Section 4, the efficacy of symmetric sampling and weight decay is related to the task properties. Therefore, a straightforward pathway for future research is to extend this analysis to a broad spectrum of domains and identify the scenarios in which symmetric sampling and weight decay produce performance enhancements. In particular, analyzing the effectiveness of these techniques on benchmark tasks, like the Traveling Salesman Problem (TSP) or Vehicle Routing Problem (VRP), and on other test functions will be the object of future studies. This in turn will enable researchers to consciously choose the most suitable ESs. Furthermore, future works could investigate alternative techniques, further enhancing the performance of CMA-ES, sNES and xNES. In this respect, the incorporation of local refinement techniques [27,50] could be object of future studies.