Combining Parallel Stochastic Methods and Mixed Termination Rules in Optimization

Abstract

Parallel optimization enables faster and more efficient problem-solving by reducing computational resource consumption and time. By simultaneously combining multiple methods, such as evolutionary algorithms and swarm-based optimization, effective exploration of the search space and achievement of optimal solutions in shorter time frames are realized. In this study, a combination of termination criteria is proposed, utilizing three different criteria to end the algorithmic process. These criteria include measuring the difference between optimal values in successive iterations, calculating the mean value of the cost function in each iteration, and the so-called “DoubleBox” criterion, which is based on the relative variance of the best value of the objective cost function over a specific number of iterations. The problem is addressed through the parallel execution of three different optimization methods (PSO, Differential Evolution, and Multistart). Each method operates independently on separate computational units with the goal of faster discovery of the optimal solution and more efficient use of computational resources. The optimal solution identified in each iteration is transferred to the other computational units. The proposed enhancements were tested on a series of well-known optimization problems from the relevant literature, demonstrating significant improvements in convergence speed and solution quality compared to traditional approaches.

1. Introduction

The problem of finding the global minimum of a multidimensional function occurs in various scientific areas and has wide applications. In this context, the programmer seeks the absolute minimum of a function subject to some assumptions or constraints. The objective function is defined as $f:S\to R,S\subset R^n$ and the mathematical formulation of the global optimization problem is as follows:

$$x^{\ast }=\arg \underset{x\in S}{min}f\left(x\right).$$

(1)

where the set S is as follows:

$$S=\left[a_1,b_1\right]\otimes \left[a_2,b_2\right]\otimes \dots \left[a_n,b_n\right]$$

Such problems frequently arise in sciences like physics, where genetic algorithms prove effective in locating positions in magnetic plasmas and in developing optimization tools [1]. Additionally, the combined use of various optimization techniques enhances performance and stability in complex problems [2]. Furthermore, the hybrid approach that combines artificial physics techniques with particle swarm optimization has proven to be particularly effective in solving energy allocation problems [3]. In the field of chemistry, minimizing potential energy functions is crucial for understanding the ground states of molecular clusters and proteins [4]. Optimizing these functions has been used for accurate protein structure prediction, with results closely matching crystal structures [5]. Moreover, a new computational framework combining variance-based methods with artificial neural networks has significantly improved accuracy and reduced computational cost in sensitivity analyses [6]. Even in the field of economics, optimization plays a crucial role. The particle swarm optimization (PSO) method is shown to be highly effective in solving the economic dispatch problem in power systems, providing higher quality solutions with greater efficiency compared to genetic algorithms [7]. Additionally, a multi-objective optimization approach for economic load dispatch and emission reduction in hydroelectric and thermal plants achieves optimal solutions that align with the user’s preferences and goals [8]. Finally, optimization is also applied in biological process models [9] and medical data classification, achieving high accuracy in diagnosing diseases and analyzing genomes [10]. A series of methods have been proposed in the recent literature to handle problems described by Equation (1). Methods used to handle problems described by Equation (1) are usually divided into two categories: deterministic and stochastic methods. In the first category, the most common method is the interval method [11,12], where the set S is divided through a series of steps into subregions and some subregions that do not contain the global solution can be removed using some pre-defined criteria. The second category contains stochastic techniques that do not guarantee finding the total minimum, and they are easier to program since they do not rely on some assumptions about the objective function. In addition, they also constitute the vast majority of global optimization methods. Among them, there are methods based on considerations derived from Physics, such as the Simulated Annealing method [13], the Henry’s Gas Solubility Optimization (HGSO) [14], the Gravitational Search Algorithm (GSA) [15], the Small World Optimization Algorithm (SWOA) [16], etc. Also, a series of evolutionary-based techniques have also been suggested for Global Optimization problems, such as Genetic Algorithms [17], the Differential Evolution method [18,19], Particle Swarm Optimization (PSO) [20,21], Ant Colony Optimization (ACO) [22], Bat Algorithm (BA) [23], Whale Optimization Algorithm (WOA) [24], Grasshopper Optimization Algorithm (GOA) [25], etc.

However, the above optimization methods require significant computational power and time. Therefore, parallel processing of these methods is essential. Recently, various methods have been proposed to leverage parallel processing. Specifically, one study details the parallel implementation of the Particle Swarm Optimization (PSO) algorithm, which improves performance for load-balanced problems but shows reduced efficiency for load-imbalanced ones. The study suggests using larger particle populations and implementing asynchronous processing to enhance performance [26]. Another review provides a comprehensive overview of derivative-free optimization methods, useful for problems where objective and constraint functions come only from a black-box or simulator interface. The review focuses on recent developments and categorizes methods based on assumptions about functions and their characteristics [27]. Lastly, the PDoublePop version 1.0 software is presented, which implements parallel genetic algorithms with advanced features, such as an enhanced stopping rule and advanced mutation schemes. The software allows for coding the objective function in C++ or Fortran77 and has been tested on well-known benchmark functions, showing promising results [28]. Moreover, there are methods that utilize GPU architectures for solving complex optimization problems. In the case of the Traveling Salesman Problem (TSP), a parallel GPU implementation has been developed to achieve high performance on large scales, solving problems of up to 6000 cities with great efficiency [29]. Additionally, the use of the multi-start model in local search algorithms has proven to be particularly effective when combined with GPUs, reducing computation time and improving solution quality for large and time-intensive problems [30]. Finally, a parallel algorithm using Peano curves for dimension reduction has demonstrated significant improvements in speed and efficiency when run on GPUs, compared to CPU-only implementations, for multidimensional problems with multiple local minima [31]. Parallel optimization represents a significant approach in the field of optimization problem-solving and is applied across a broad range of applications, including the optimization of machine learning model parameters. In the case of GraphLab, a platform is presented that enhances existing abstract methods, such as MapReduce, for parallel execution of machine learning algorithms with high performance and accuracy [32]. Additionally, the new MARSAOP method for hyperparameter optimization uses multidimensional spirals as surrogates and dynamic coordinate search to achieve high-quality solutions with limited computational costs [33]. Lastly, the study proposes a processing time-estimation system using machine learning models to adapt to complex distributions and improve production scheduling, achieving significant reduction in completion time [34].

The use of parallel optimization has brought significant advancements in solving complex control and design problems [35]. In one case, the parallel application of the Genetic Algorithm with a Fair Competitive Strategy (HFCGA) has been used for the design of an optimized cascade controller for ball and beam systems. This approach allows for the automation of controller parameter tuning, improving performance compared to traditional methods, with parallel computations helping to avoid premature convergence to suboptimal solutions [36]. In another case, the advanced Teaching–Learning-Based Optimization (TLBO) method has been applied to optimize the design of propulsion control systems with hydrogen peroxide. This method, which includes improvements in the teaching and learning phases as well as in the search process, has proven highly effective in addressing uncertainties in real-world design problems, enhancing both accuracy and convergence speed of the optimization process [37].

Similarly, parallel optimization has significant applications in energy and resource management. In the case of pump and cooling system management, a two-stage method based on an innovative multidimensional optimization algorithm has been proposed, which reduces overall energy consumption and discrepancies between pump and cooling system flow rates, achieving significant improvements in energy efficiency [38]. In energy network management, a coordinated scheduled optimization method has been proposed, which simultaneously considers various energy sources and uncertainty conditions. This method, using the Competitive Swarm Optimization (CSO) algorithm and adjusted based on chaos theory, has proven 50% faster than other methods and effective in managing uncertainties and energy [39]. In topological optimization, parallel processing through CPUs and GPUs has drastically improved speed and energy consumption for complex problems, achieving up to 25-times faster processing and reducing energy consumption by up to 93% [40]. Lastly, in building energy retrofitting, a multi-objective platform was used for evaluating and optimizing upgrade strategies, demonstrating that removing subsidies and providing incentives for energy upgrades yield promising results [41].

In conclusion, parallel optimization has provided significant solutions to problems related to sustainable development and enhancing sustainability. In the field of increasing the resilience of energy systems, the development of strategies for managing severe risks, such as extreme weather conditions and supplier disruptions, has been improved through parallel optimization, enabling safer and more cost-effective energy system operations [42]. In addressing groundwater contamination, the use of parallel optimization has highlighted the potential for faster and more effective design of remediation methods, significantly reducing computational budgets and ensuring better results compared to other methods [43]. Finally, in the field of sustainable production, metaheuristic optimization algorithms have proven useful for improving production scheduling, promoting resource efficiency, and reducing environmental impacts, while contributing to the achievement of sustainable development goals [44].

By harnessing multiple computational resources, parallel optimization allows for the simultaneous execution of multiple algorithms, leading to faster convergence and improved performance. These computational resources can communicate with each other to exchange information and synchronize processes, thereby contributing to faster convergence towards common solutions. Additionally, leveraging multiple resources enables more effective handling of exceptions and errors, while increased computational power for conducting more trials or utilizing more complex models leads to enhanced performance [45]. Of course, this process requires the development of suitable algorithms and techniques to effectively manage and exploit available resources. Each parallel optimization algorithm requires a coherent strategy for workload distribution among the available resources, as well as an efficient method for collecting and evaluating results.

Various researchers have developed parallel techniques, such as the development and evaluation of five different parallel Simulated Annealing (SA) algorithms for solving global optimization problems. The paper compares the performance of these algorithms across an extensive set of tests, focusing on various synchronization and information exchange approaches, and highlights their particularly noteworthy performance on functions with large search spaces where other methods have failed [46]. Additionally, other research focuses on the parallel application of the Particle Swarm Optimization (PSO) method, analyzing the performance of the parallel PSO algorithm on two categories of problems: analytical problems with low computational costs and industrial problems with high computational costs [47]. This research demonstrates that the parallel PSO method performs exceptionally well on problems with evenly distributed loads, while in cases of uneven loads, the use of asynchronous approaches and larger particle populations proves more effective. Another study develops a parallel implementation of a stochastic Radial Basis Function (RBF) algorithm for global optimization, which does not require derivatives and is suitable for computationally intensive functions. The paper compares the performance of the proposed method with other parallel optimization methods, noting that the RBF method achieves good results with one, four, or eight processors across various optimization problems [48]. Parallel techniques are also applied in real-time image processing, which is critical for applications such as video surveillance, diagnostic medicine, and autonomous vehicles. A related article examines parallel architectures and algorithms, identifying effective applications and evaluating the challenges and limitations of their practical implementation, aiming to develop more efficient solutions [49]. Another study reviews parallel computing strategies for computational fluid dynamics (CFD), focusing on tools like OpenMP, MPI, and CUDA to reduce computational time [50]. Finally, a significant study explores parallel programming technologies for processing genetic sequences, presenting three main parallel computing models and analyzing applications such as sequence alignment, single nucleotide polymorphism calling, sequence preprocessing, and pattern detection [51].

Genetic Algorithms (GAs) are methods that can be easily parallelized, and several researchers have thoroughly examined them in the literature. For example, a universal parallel execution method for Genetic Algorithms, known as IIP (Parallel, Independent, and Identical Processing), is presented. This method achieves acceleration with the use of m processors. The technique calculates execution speed and compares results on small-size problems and the non-parametric Inverse Fractal Problem [52]. Similarly, another paper presents a distributed mechanism for improving resource protection in a digital ecosystem. This mechanism can be used not only for secure and reliable transactions but also to enhance collaboration among digital ecosystem community members to secure the environment, and also employs Public Key Infrastructure to provide strong protection for access workflows [53]. Genetic Algorithms have also been used as a strategy for optimizing large and complex problems, specifically in wavelength selection for multi-component analysis. The study examines the effectiveness of the genetic algorithm in finding acceptable solutions in a reasonable time and notes that the algorithm incorporates prior information to improve performance, based on mathematically grounded frameworks such as the schema theorem [54]. Additionally, another paper develops two genetic algorithm-based algorithms to improve the lifetime and energy consumption in mobile wireless sensor networks (MWSNs). The first algorithm is an improvement of the Unequal Clustering Genetic Algorithm, while the second algorithm combines the K-means Clustering Algorithm with Genetic Algorithms. These algorithms aim to better adapt to dynamic changes in network topology, thereby enhancing network lifetime and energy consumption [55]. Lastly, the use of Genetic Algorithms has promising applications across various medical specialties, including radiology, radiotherapy, oncology, pediatrics, cardiology, endocrinology, surgery, obstetrics and gynecology, pulmonology, infectious diseases, orthopedics, rehabilitation medicine, neurology, pharmacotherapy, and health care management. A related paper reviews and introduces applications of Genetic Algorithms in disease screening, diagnosis, treatment planning, pharmacovigilance, prognosis, and health care management, enabling physicians to envision potential applications of this metaheuristic method in their medical careers [56].

In this paper, a new optimization method is proposed which is a mixture of existing global optimization techniques running in parallel on a number of available computing units. Each technique is executed independently of the others and periodically the optimal values retrieved from them are distributed to the rest of the computing units using the propagation techniques presented here. In addition, for the most efficient termination of the overall algorithm, intelligent termination techniques based on stochastic observations are used, which are suitably modified to adapt to the parallel computing environment. The contributions of this work as summarized as follows:

• Periodic local search was integrated into the Differential Evolution (DE) and Particle Swarm Optimization (PSO) methods.
• A mechanism for disseminating the optimal solution to all PUs was added in each iteration of the methods.
• The overall algorithm terminates based on the proposed termination criterion.

The following sections are organized as follows: In Section 2, the three main algorithms participating in the overall algorithm are described. In Section 3 the algorithm for parallelizing the three methods is described, along with the proposed mechanism for propagating the optimal solution to the remaining methods. In Section 4, experimental models and experimental results are described. Finally, in Section 5, the conclusions from the application of the current work are discussed.

2. Adopted Algorithms

The methods executed in each processing unit are fully described in this section.

2.1. The Method PSO

PSO is an optimization method inspired by the behavior of swarms in nature. In PSO, a set of particles moves in the search space seeking the optimal solution. Each particle has a current position and velocity, and it moves based on its past performance and that of its neighboring particles. PSO continuously adjusts the movement of particles, aiming for convergence to the optimal solution [57,58,59]. This method can be programmed easily and the set of the parameters to be set is limited. Hence, it has been used in a series of practical problems, such as problems that arise in physics [60,61], chemistry [62,63], medicine [64,65], economics [66], etc. The main steps of the PSO method are presented in Algorithm 1.

Algorithm 1 The main steps of the PSO method

Initialization. (a) Set $k=0$ (iteration counter). (b) Set the number of particles $N_p$. (c) Set the maximum number of iterations allowed $k_{\max }$ (d) Set the local search rate $p_l\in [0,1]$. (e) Initialize randomly the positions of the m particles $x_1,x_2,\dots ,x_m$, with $x_i\in S\subset R^n$ (f) Initialize randomly the velocities of the m particles  $u_1,u_2,\dots ,u_m$, with $u_i\in S\subset R^n$ (g) For $i=1\dots N_p$ do $p_i=x_i$. The $p_i$ vector are the best located values for every particle i. (h) Set $p_{\mathrm{best}}=arg{min}_{i\in 1\dots m}f\left(x_i\right)$ Termination Check. Check for termination. If termination criteria are met then stop. For $i=1\dots N_p$ Do (a) Update the velocity: $$u_i=\omega u_i+r_1{c}_1\left(p_i-x_i\right)+r_2{c}_2\left(p_{\mathrm{best}}-x_i\right)$$ The parameters $r_1,r_2$ are random numbers with  $r_1\in [0,1]$ and $r_2\in [0,1]$. The constant number $c_1,c_2$ are in the range  $[1,2]$. The variable $\omega$ is called inertia, with  $\omega \in [0,1]$. (b) Update the position $$x_i=x_i+u_i$$ (c) Set $r\in [0,1]$ a random number. If $r\le p_m$ then $x_i=LS\left(x_i\right)$, where $LS\left(x\right)$ is a local search procedure. (d) Evaluate the fitness of the particle i, $f\left(x_i\right)$ (e) If $f\left(x_i\right)\le f\left(p_i\right)$ then $p_i=x_i$ End For Set $p_{\mathrm{best}}=arg{min}_{i\in 1\dots m}f\left(x_i\right)$ Set $k=k+1$. Goto Step 2

2.2. The Method Differential Evolution

The DE method relies on differential operators and is particularly effective in optimization problems that involve searching through a continuous search space. By employing differential operators, DE generates new solutions that are then evaluated and adjusted until the optimal solution is achieved [67,68]. The method was used in a series of problems, such as electromagnetics [69], energy consumption problems [70], job shop scheduling [71], image segmentation [72], etc. The Differential Evolution operates through the following steps: Initially, initialization takes place with a random population of solutions in the search space. Then, each solution is evaluated based on the objective function. Subsequently, a process is iterated involving the generation of new solutions through modifications, evaluation of these new solutions, and selection of the best ones for the next generation. The algorithm terminates when a termination criterion is met, such as achieving sufficient improvement in performance or exhausting the number of iterations. The main steps of the DE method are presented in Algorithm 2.

Algorithm 2 The main steps of the DE method

INPUT: (a) The population size $N_d\ge$ 4. The members of this population are also called agents. (b) The crossover probability $CR\in [0,1]$. (c) The differential weight $F\in [0,2].$ OUTPUT: (a) The agent $x_{\mathrm{best}}$ with the lowest function value $f\left(x_{\mathrm{best}}\right)$. Initialize all agents in S. While termination criteria are not met do (a) For $i=1\dots N_d$ do Select as x the agent i. Select randomly three agents $a,b,c$ with the property $a\ne b,b\ne c,c\ne a$. Select a random position $R\in \left\{1,\dots ,n\right\}$ Create the vector $y=\left[y_{1,}{y}_2,\dots ,y_n\right]$ with the following procedure For $j=1,\dots ,n$ do Set $r_i\in [0,1]$ a random number. If $r_j<\mathrm{CR}$ or $j=R$ then $y_j=a_j+F\times \left(b_j-c_j\right)$ else $y_j=x_j$. If $y\in S\mathrm{AND}f\left(y\right)\le f\left(x\right)$ then $x=y$. EndFor (b) EndFor End While

2.3. The Method Multistart

In contrast, the Multistart method follows a different approach than previous methods. Instead of focusing on a single initial point, it repeats the search from various initial points in the search space. This allows for a broader exploration of the search space and increases the chances of finding the optimal solution. The Multistart method is particularly useful when optimization algorithms may get trapped in local minima [73,74,75]. The approach of multiple starts belongs to the simplest techniques for global optimization. At the outset of the process, an initial distribution of points is made in Euclidean space. Subsequently, local optimization begins simultaneously from these points. The discovered minima are compared, and the best one is retained as the global minimum. Local optimizations rely on the Broyden Fletcher Goldfarb Shanno (BFGS) method [76]. The main steps of the multistart method are presented in Algorithm 3.

Algorithm 3 The main steps of the Multistart method

Initialization step. (a) Set $N_m$ as the total number of samples. (b) Set$\left(x^{\ast },y^{\ast }\right)$ as the global minimum. Initialize $y^{\ast }$ to a very large value. Sampling step. (a) For $i=1\dots N_m$Do Sample a point $x_i\in S$ $y_i=\mathrm{LS}\left(x_i\right)$. Where LS(x) is a local search procedure. If $y_i\le y^{\ast }$ then $x^{\ast }=x_i,y^{\ast }=y_i$ (b) EndFor

3. The Proposed Method

In the present work, a mixture of termination rules is proposed as a novel technique, which can be used without partitioning in any stochastic global optimization technique. Also, a new parallel global optimization technique is proposed that utilizes the new termination rule.

The existing method that was presented in Section 2, employs Particle Swarm Optimization (PSO), Differential Evolution (DE), and Multistart techniques, each with its own independent termination criteria. In contrast, the proposed method that will be presented in this section integrates an innovative combination of termination rules, including Best-Fitness, Mean-Fitness, and DoubleBox, enhancing the termination process of the search. By incorporating these advanced termination strategies into a parallel algorithm, the new method achieves faster convergence and improved solution quality. This approach combines the efficiency of parallel execution with the flexibility of advanced termination criteria, representing a significant advancement over traditional methods. In the following subsections, the new termination rule is fully described, followed by the new optimization method.

3.1. The New Termination Rule

The proposed termination rule is a mixture of several stochastic termination rules proposed in the recent literature. The first algorithm will be called best—fitness and in this termination technique, at each iteration k, the difference between the current best value $f_{min}^{\left(k\right)}$ and the previous best value $f_{min}^{(k+1)}$ is calculated [77], i.e., the absolute difference:

$$\left|f_{\min }^{\left(k\right)}-f_{\min }^{(k-1)}\right|$$

(2)

If this difference is zero for a series of predefined consecutive iterations $N_k$, then the method terminates. In the second termination rule, which will be called mean—fitness, the average function value for each iteration is computed. If this value remains relatively unchanged over a certain number of consecutive iterations, it indicates that the method may not be making significant progress towards discovering a new global minimum. Therefore, termination is warranted [19]. Hence, in every iteration k, we compute:

$${\delta }^{\left(k\right)}=\left|\sum _{i=1}^{\mathrm{NP}}\left|f_i^{\left(k\right)}\right|-\sum _{i=1}^{\mathrm{NP}}\left|f_i^{(k-1)}\right|\right|$$

(3)

The value $NP$ denotes the number of particles or chromosomes that participate in the algorithm. The termination rule is defined as follows: terminate if ${\delta }^{\left(k\right)}\le ϵ$ for a predefined number $N_k$ of iterations. The third stopping rule is the so-called DoubleBox stopping rule, which was initially proposed in the work of Tsoulos [78]. In this criterion, the search process is terminated once sufficient coverage of the search space has been achieved. The estimation of the coverage is based on the asymptotic approximation of the relative proportion of points that lead to local minima. Since the exact coverage cannot be directly calculated, sampling is conducted over a larger region. The search is stopped when the variance of the sample distribution falls below a predefined threshold, which is adjusted based on the most recent discovery of a new local minimum. According to this criterion, the algorithm terminates when one of the following conditions is met:

• The iteration k count exceeds a predefined limit of iterations $k_{max}$;
• The relative variance ${\sigma }^{\left(k\right)}$ falls below half of the variance ${\sigma }^{\left(k_{last}\right)}$ of the last iteration $k_{\left(last\right)}$ where a new optimal functional value was found.

  $$k\ge N_{k}or{\sigma }^{\left(k\right)}\le {\displaystyle \frac{{\sigma }^{\left(klast\right)}}{2}}$$

  (4)
  This proposed termination criterion is the combination of the aforementioned similarity types. Optimization termination is achieved when any homogeneity condition is satisfied, resulting in improved speed. Hence, by using Equations (2)–(4), the following termination rule is derived:

$$\left|f_{\min }^{\left(k\right)}-f_{\min }^{(k-1)}\right|or$$

$${\delta }^{\left(k\right)}=\left|\sum _{i=1}^{\mathrm{NP}}\left|f_i^{\left(k\right)}\right|-\sum _{i=1}^{\mathrm{NP}}\left|f_i^{(k-1)}\right|\right|or$$

$${\sigma }^{\left(iteration\right)}\le {\displaystyle \frac{{\sigma }^{\left(klast\right)}}{2}}or$$

$$k\ge N_k$$

3.2. The Proposed Algorithm

In the scientific literature, parallelization typically involves distributing the population among parallel computing units to save computational power and time [77,79]. In the present study, we concurrently compute the optimal solutions of three different stochastic optimization methods originating from different classification categories. The proposed algorithm is shown in Algorithm 4.

Algorithm 4 The proposed overall algorithm

Set as $N_I$ the total number of parallel processing units. Set as $N_k$ the total number of allowed iterations. Set $k=0$ the iteration number. For $j=1,\dots ,N_I$ do in parallel (a) Execute an iteration of any stochastic algorithm mentioned in Section 2. (b) Find the best element from all optimization methods and propagate it to the rest of processing units. End For Update  $k=k+1$ Check the proposed termination rule. If the termination rule is valid, then goto step 7a else goto step 4. (a) Terminate and report the best value from all processing units.

The proposed algorithm is graphically outlined in Figure 1. The methods that are used in each processing unit may include Particle Swarm Optimization, the Multistart method, and Differential Evolution (DE). These techniques have been adopted since they have been widely used in the relevant literature and provide the possibility to be parallelized relatively easily. Initially the algorithm distributes the three methodologies across $N_I$ threads, where $N_I>3$. The first thread executes the DE method, the next executes PSO, and the third runs the Multistart method. As shown in Figure 1, at each iteration, every computational unit that uses this specific optimization method calculates the best solution and replaces the worst solutions of the other computational units. This process is repeated to ensure an equivalent distribution of the methods across all threads, optimizing the utilization of available computational resources. When the number of threads is equal to or a multiple of the number of methods, the distribution is 100% equivalent. In other cases, the degree of equivalence increases with the number of threads.

Figure 1. Flowchart of the overall process. The DE acronym stands for the Differential Evolution method, the PU acronym represents the Parallel Unit that executes the algorithm.

As presented in this subsection, the proposed algorithm exploits the parallel execution of three distinct optimization methods, with its efficiency in termination procedures incorporated through the criteria described in Section 3.1. The algorithm uses three termination criteria: the difference between the best values in successive iterations, the stability of the mean fitness value, and the relative variance of the cost function values. Each criterion contributes to the evaluation of the algorithm’s progress, dynamically switching between these criteria based on the current state of the search. Specifically, when any of the criteria meets its predefined conditions, the algorithm terminates the search, thus adapting the process to the continuous search conditions. This strategy ensures the flexibility and efficiency of the algorithm, minimizing computational resources and time wastage.

4. Experiments

All experiments conducted were repeated 30 times to ensure the reliability of the algorithm producing the results. In experimental research, the number of repetitions required to ensure reliable results depends on the nature of the experiment, the variability of the data, and the precision requirements. There is no universal answer that applies to all experiments, but general guidelines and best practices have been established in the literature. In many scientific studies, at least 30 repetitions are recommended to achieve satisfactory statistical power. This is based on statistical theories such as the Law of Large Numbers and theorems for estimating sample distributions. The initial sample distribution is the same across all optimization methods, and the other parameters have been kept constant, as shown in Table 1. Specifically, in the parallel algorithm, the total number of initial samples remains as constant as possible, regardless of the number of threads used. The parallelization was achieved using the OpenMP library [80], while the implementation of the method was done in ANSI C++ within the optimization package OPTIMUS, available at https://github.com/itsoulos/OPTIMUS (accessed on 24 July 2024). The experiments were executed on a Debian Linux system that runs on an AMD Ryzen 5950X processor with 128 GB of RAM. The parameters used here are either suggested in the original publications of the methods (e.g., Differential Evolution, Particle Swarm Optimization) or have been chosen so that there is a compromise between the execution time of the method and its efficiency.

Table 1. The following parameters were considered for conducting the experiments.

Parameter	Value	Explanation
$N_p$	120	Total particles for PSO
$N_d$	120	Total elements for DE
$N_m$	120	Total elements for Multistart
$N_k$	200	Maximum number of iterations
$N_s$	15	Similarity max count
F	0.8	Differential weight for DE
$CR$	0.9	Crossover Probability for DE
$P_m$	0.005	Local search rate for DE and PSO
$C_1,C_2$	0.5	Parameters of PSO

4.1. Test Functions

The test functions [81,82] presented below exhibit varying levels of difficulty in solving them; hence, a periodic local optimization mechanism has been incorporated. Periodic local optimization plays a crucial role in increasing the success rate in locating the minimum of functions. This addition appears to lead to a success rate approaching 100% for all functions, regardless of their characteristics such as dimensionality, minima, scalability, and symmetry. A study by Z.-M. Gao and colleagues [83] specifically examines the issue of symmetry and asymmetry in the test functions. The test functions used in the conducted experiments are shown in Table 2. Generally, the region of attraction $A\left(x^{\ast }\right)$ of a local minimum $x^{\ast }$ is defined as:

$$A\left(x^{\ast }\right)=\left\{x:x\in S,L\left(x\right)=x^{\ast }\right\}$$

(5)

where $L\left(x\right)$ is a local search procedure, such as BFGS. Global optimization methods usually find points which are in the region of attraction $A\left(x^{\ast }\right)$ of a local minimum $x^{\ast },$ but not necessarily the local minimum itself. For this reason, at the end of their execution, a local minimization method is applied in order to ensure the exact finding of a local minimum. Of course, this does not imply that the minimum that will be located will be the total minimum of the function, since a function can contain tens or even hundreds of local minima.

Table 2. The test functions used in the conducted experiments.

Name	Formula	Dimension
Cigar10	$f\left(x\right)=x_1^2+{10}^6{\sum }_{i=2}^n{x}_i^2$	10
BF1	$f\left(x\right)=x_1^2+2x_2^2-{\displaystyle \frac{3}{10}}cos\left(3\pi x_1\right)-{\displaystyle \frac{4}{10}}cos\left(4\pi x_2\right)+{\displaystyle \frac{7}{10}}$	2
BF2	$f\left(x\right)=x_1^2+2x_2^2-{\displaystyle \frac{3}{10}}cos\left(3\pi x_1\right)cos\left(4\pi x_2\right)+{\displaystyle \frac{3}{10}}$	2
Branin	$f\left(x\right)={\left(x_2-{\displaystyle \frac{5.1}{4{\pi }^2}}{x}_1^2+{\displaystyle \frac{5}{\pi }}{x}_1-6\right)}^2+10\left(1-{\displaystyle \frac{1}{8\pi }}\right)cos\left(x_1\right)+10$	2
CM	$f\left(x\right)={\sum }_{i=1}^n{x}_i^2-{\displaystyle \frac{1}{10}}{\sum }_{i=1}^{n}cos\left(5\pi x_i\right)$	4
Discus	$f\left(x\right)={10}^6{x}_1^2+{\sum }_{i=2}^n{x}_i^2$	10
Easom	$f\left(x\right)=-cos\left(x_1\right)cos\left(x_2\right)exp\left({\left(x_2-\pi \right)}^2-{\left(x_1-\pi \right)}^2\right)$	2
Exp	$f\left(x\right)=-exp\left(-0.5{\sum }_{i=1}^n{x}_i^2\right),-1\le x_i\le 1$	$n=4,16$
Griewank2	$f\left(x\right)=1+{\displaystyle \frac{1}{200}}{\sum }_{i=1}^2{x}_i^2-{\prod }_{i=1}^2{\displaystyle \frac{cos\left(x_i\right)}{\sqrt{\left(i\right)}}}$	2
Griewank10	f$\left(x\right)=1+{\displaystyle \frac{1}{200}}{\sum }_{i=1}^{10}{x}_i^2-{\prod }_{i=1}^{10}{\displaystyle \frac{cos\left(x_i\right)}{\sqrt{\left(i\right)}}}$	10
Gkls [84]	$f\left(x\right)=\mathrm{Gkls}(x,n,w)$	$n=2,3w=50,100$
Hansen	$f\left(x\right)={\sum }_{i=1}^{5}icos\left[(i-1)x_1+i\right]{\sum }_{j=1}^{5}jcos\left[(j+1)x_2+j\right]$	2
Hartman3	$f\left(x\right)=-{\sum }_{i=1}^4{c}_{i}exp\left(-{\sum }_{j=1}^3{a}_{ij}{\left(x_j-p_{ij}\right)}^2\right)$	3
Hartman6	$f\left(x\right)=-{\sum }_{i=1}^4{c}_{i}exp\left(-{\sum }_{j=1}^6{a}_{ij}{\left(x_j-p_{ij}\right)}^2\right)$	6
Elp	$f\left(x\right)={\sum }_{i=1}^n{\left({10}^6\right)}^{{\displaystyle \frac{i-1}{n-1}}}{x}_i^2$	10
Potential [85]	$V_{LJ}\left(r\right)=4ϵ\left[{\left({\displaystyle \frac{\sigma }{r}}\right)}^{12}-{\left({\displaystyle \frac{\sigma }{r}}\right)}^6\right]$	$n=9,15,21,30$
Rastrigin	$f\left(x\right)=x_1^2+x_2^2-cos\left(18x_1\right)-cos\left(18x_2\right)$	2
Rosenbrcok	$f\left(x\right)={\sum }_{i=1}^{n-1}\left(100{\left(x_{i+1}-x_i^2\right)}^2+{\left(x_i-1\right)}^2\right),-30\le x_i\le 30$	$n=4,8,16$
Shekel5	$f\left(x\right)=-{\sum }_{i=1}^5{\displaystyle \frac{1}{(x-a_i){(x-a_i)}^T+c_i}}$	4
Shekel7	$f\left(x\right)=-{\sum }_{i=1}^7{\displaystyle \frac{1}{(x-a_i){(x-a_i)}^T+c_i}}$	4
Shekel10	$f\left(x\right)=-{\sum }_{i=1}^{10}{\displaystyle \frac{1}{(x-a_i){(x-a_i)}^T+c_i}}$	4
Sinusoidal [86]	$f\left(x\right)=-\left(2.5{\prod }_{i=1}^{n}sin\left(x_i-z\right)+{\prod }_{i=1}^{n}sin\left(5\left(x_i-z\right)\right)\right),0\le x_i\le \pi$	$n=4,8$
Test2N	$f\left(x\right)={\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{x}_i^4-16x_i^2+5x_i$	$n=4,9$
Test30N	${\displaystyle \frac{1}{10}}{sin}^2\left(3\pi x_1\right){\sum }_{i=2}^{n-1}\left({\left(x_i-1\right)}^2\left(1+{sin}^2\left(3\pi x_{i+1}\right)\right)\right)+{\left(x_n-1\right)}^2\left(1+{sin}^2\left(2\pi x_n\right)\right)$	$n=3,4$

4.2. Experimental Results

All tables presented here demonstrate the average function calls obtained for each test problem. The average function can be considered as a measure of the speed of each method, since each function has a different complexity and therefore, the average execution time would not be a good indication of the computational time required. In the first set of experiments we examined if the presence of the suggested stopping rule, which is the combination of a series of stopping rules, affects the optimization techniques without parallelization. During parallel execution, the total sum of calls is computed across all computing units nodes, ensuring efficient load distribution and resource utilization. However, the algorithm terminates at the computing unit that identifies the optimal solution, signaling the achievement of the desired convergence criterion. In Table 3, we observe the performance of all termination rules for the DE method. From these, both statistical and quantitative comparisons arise, depicted in Figure 2.

Table 3. Average function calls for the Differential evolution method, using different termination rules.

Problem	Best	Mean	DoubleBox	All
BF1	6761	10,511	10,369	6114
BF2	7693	9222	11,524	7560
BRANIN	4639	10,982	4907	4289
CAMEL	6116	13,290	9905	5940
CIGAR10	7647	4348	21,116	3111
CM4	7913	4175	11,851	4079
DISCUS10	5191	2042	13,614	2034
EASOM	1917	15,103	1721	1721
ELP10	6046	24,675	12,519	6031
EXP4	5216	18,151	6385	5040
EXP16	5588	27,387	7552	5339
GKLS250	5227	2641	8379	2641
GKLS350	5624	3298	17,437	3206
GRIEWANK2	8027	10,458	14,756	6915
GRIEWANK10	9664	37,839	17,312	9539
POTENTIAL3	5278	24,823	13,871	5256
POTENTIAL5	8225	32,439	20,828	7742
POTENTIAL6	8467	34,946	19,169	8197
POTENTIAL10	10,330	41,300	26,308	10,643
HANSEN	5219	23,050	14,245	4263
HARTMAN3	4613	17,966	7333	4566
HARTMAN6	5734	17,109	9354	5550
RASTRIGIN	5912	17,240	8987	5999
ROSENBROCK8	8503	34,429	20,980	8051
ROSENBROCK16	10,052	41,335	26,156	8307
SHEKEL5	6064	25,800	19,877	5933
SHEKEL7	5597	26,168	20,274	5677
SHEKEL10	6037	26,439	22,960	5100
SINU4	5751	24,620	12,511	4776
SINU8	6340	25,477	18,741	4980
TEST2N4	5848	24,947	11,640	5727
TEST2N9	7303	26,571	16,165	6288
TEST30N3	3169	8395	8134	2869
TEST30N4	2841	10,381	7853	2714
Total	214,552	677,557	474,733	186,197

Figure 2. Statistical comparison of function calls with different termination rules of differential evolutions.

As it is evident, the combination of the stopping rules reduces the required number of function calls that are required to obtain the global minimum by the DE method. The proposed termination scheme outperforms by a percentage that varies from 13% to 73% the other termination rules used here. In addition, in order to ascertain which of the three termination rules has the greatest effect on the termination with the mixed rule, the rule responsible for the termination of the DE optimization technique was recorded for all functions of the test set. The results from this recording are shown in Table 4. The two techniques that seem to be the most decisive for terminating each function are the Best and the DoubleBox methods, and in fact the Best method appears to significantly outperform the DoubleBox termination scheme on average.

Table 4. Percentage of the rule which is responsible for termination in the mixed stopping rule.

Problem	Mean	Best	DoubleBox
BF1	0.00%	36.67%	63.33%
BF2	6.67%	33.33%	60.00%
BRANIN	0.00%	6.67%	93.33%
CAMEL	10.00%	76.67%	13.33%
CIGAR10	0.00%	20.00%	80.00%
CM4	0.00%	90.00%	10.00%
DISCUS10	0.00%	6.67%	93.33%
EASOM	0.00%	0.00%	100.00%
ELP10	0.00%	0.00%	100.00%
EXP4	0.00%	86.67%	13.33%
EXP16	0.00%	76.67%	23.33%
GKLS250	6.67%	73.33%	20.00%
GKLS350	20.00%	80.00%	0.00%
GRIEWANK2	3.33%	26.67%	70.00%
GRIEWANK10	0.00%	10.00%	90.00%
POTENTIAL3	3.33%	66.67%	30.00%
POTENTIAL5	10.00%	86.67%	3.33%
POTENTIAL6	0.00%	90.00%	10.00%
PONTENTIAL10	13.33%	86.67%	0.00%
HANSEN	16.67%	50.00%	33.33%
HARTMAN3	0.00%	70.00%	30.00%
HARTMAN6	3.33%	76.67%	20.00%
RASTRIGIN	0.00%	83.33%	16.67%
ROSENBROCK8	0.00%	13.33%	86.67%
ROSENBROCK16	0.00%	13.33%	86.67%
SHEKEL5	3.33%	93.33%	3.33%
SHEKEL7	10.00%	90.00%	0.00%
SHEKEL10	6.67%	93.33%	0.00%
SINU4	6.67%	80.00%	13.33%
SINU8	0.00%	100.00%	0.00%
TEST2N4	0.00%	76.67%	23.33%
TEST2N9	0.00%	100.00%	0.00%
TEST30N3	0.00%	33.33%	66.67%
TEST30N4	0.00%	40.00%	60.00%
AVERAGE	3.53%	57.84%	38.63%

Subsequently, the same experiment was conducted for the PSO method and the results are shown in Table 5 and the statistical comparison is outlined graphically in Figure 3.

Table 5. Average function calls for the PSO method, with different termination rules.

Problem	Best	Mean	DoubleBox	All
BF1	3418	3127	3122	3005
BF2	3265	2995	2919	2889
BRANIN	2601	2549	2431	2417
CAMEL	2801	2675	2547	2547
CIGAR10	3987	3674	3638	3638
CM4	3863	3299	3144	3144
DISCUS10	2611	2409	2405	2405
EASOM	2420	2232	2232	2232
ELP10	1705	1741	1586	1586
EXP4	2751	2558	2558	2558
EXP16	2898	2676	2676	2676
GKLS250	2796	2553	2422	2422
GKLS350	2947	2645	3833	2558
GRIEWANK2	4405	2839	4282	2820
GRIEWANK10	5327	4561	4797	4248
POTENTIAL3	3289	3231	3222	3170
POTENTIAL5	4926	4881	7282	4730
POTENTIAL6	6699	5537	7534	5077
POTENTIAL10	10,582	6737	29,865	6539
HANSEN	3788	2687	3062	2587
HARTMAN3	2807	2743	2550	2550
HARTMAN6	3081	2915	2809	2809
RASTRIGIN	3558	2990	2829	2829
ROSENBROCK8	4279	4263	4036	3969
ROSENBROCK16	5761	5432	5170	5170
SHEKEL5	3132	2823	12,476	2816
SHEKEL7	3064	2855	11,520	2856
SHEKEL10	3128	2942	14,786	2866
SINU4	3047	2740	2673	2657
SINU8	3177	2824	4113	2828
TEST2N4	3037	2839	2681	2681
TEST2N9	4188	3106	3105	3067
TEST30N3	2943	3130	2762	2762
TEST30N4	3202	3185	2915	2915
Total	125,483	110,393	169,982	106,023

Figure 3. Statistical comparison of function calls with different termination rule of PSO method.

Once more, the proposed termination scheme outperforms the other stopping rules in almost every test function. Likewise, the same experiment was carried out for the Multistart method and the results are depicted in Table 6 and the statistical comparison is shown in Figure 4.

Table 6. Multistart function calls with different termination rules.

Problem	Best	Mean	DoubleBox	All
BF1	53,873	479,374	50,762	50,762
BF2	37,670	239,220	35,711	35,711
BRANIN	11,063	11,046	10,521	10,521
CAMEL	16,083	153,559	15,255	15,255
CIGAR10	14,937	14,937	14,697	14,697
CM4	62,236	62,192	58,581	58,581
DISCUS10	6296	6296	6056	6056
EASOM	5412	55,130	51,132	5412
ELP10	14,757	14,981	14,517	14,517
EXP4	10,022	54,174	9774	9774
EXP16	10,528	54,680	10,280	10,280
GKLS250	6202	47,948	5908	5908
GKLS350	7560	42,145	7087	7087
GRIEWANK2	19,179	188,429	17,877	17,877
GRIEWANK10	65,206	567,153	61,818	61,818
POTENTIAL3	17,773	126,632	17,161	17,161
POTENTIAL5	31,347	226,873	30,249	30,249
POTENTIAL6	35,457	247,144	33,883	33,883
POTENTIAL10	44,957	283,230	43,618	43,618
HANSEN	18,568	201,543	17,541	17,541
HARTMAN3	17,395	162,934	16,562	16,562
HARTMAN6	22,010	179,073	21,015	21,015
RASTRIGIN	24,696	275,610	23,015	23,015
ROSENBROCK8	17,850	17,850	17,610	17,610
ROSENBROCK16	25,300	25,300	25,060	25,060
SHEKEL5	17,725	151,648	16,972	16,972
SHEKEL7	17,894	154,373	17,127	17,127
SHEKEL10	17,931	159,825	17,135	17,135
SINU4	16,429	155,634	15,647	15,647
SINU8	19,519	170,167	18,674	18,674
TEST2N4	16,486	154,092	15,806	15,806
TEST2N9	22,035	160,651	20,716	20,716
TEST30N3	16,934	161,628	16,145	16,145
TEST30N4	16,686	159,753	15,907	15,907
Total	758,016	5,165,224	769,819	724,099

Figure 4. Statistical comparison of function calls with different termination rule of multistart.

Observing the tables with the corresponding statistical and quantitative comparisons, it is evident that the calls to the objective function for the proposed termination rule are fewer than any other rule in any method. Also, the proposed termination rule was applied to a recent global optimization method named Grey Wolf Optimizer [87] and the results are shown in Table 7 and the corresponding statistical comparison in Figure 5.

Table 7. Experiments using the variety of termination rules and the GWO optimization method.

Problem	Best	Mean	DoubleBox	All
BF1	3080	4675	6719	3080
BF2	3083	4747	6923	3083
BRANIN	3217	3217	5105	2661
CAMEL	2750	3411	3780	2724
CIGAR10	14,718	23,003	24,134	14,718
CM4	3187	5638	8802	3187
DISCUS10	12,946	22,194	24,134	12,946
EASOM	2701	2965	10,201	2135
ELP10	6440	22,449	24,123	6440
EXP4	2796	5198	7830	2796
EXP16	4991	10,842	22,025	4991
GKLS250	3072	3072	4840	2940
GKLS350	4041	4041	8620	4041
GRIEWANK2	3155	4818	7258	3155
GRIEWANK10	8568	10,196	22,841	8568
POTENTIAL3	3766	3766	13,063	3701
POTENTIAL5	4082	4082	19,826	3645
POTENTIAL6	3754	3754	19,477	3380
POTENTIAL10	3502	3502	13,129	2538
HANSEN	3407	3407	8345	2795
HARTMAN3	3251	3251	7587	2911
HARTMAN6	3639	3639	13,342	3408
RASTRIGIN	2878	4499	6463	2806
ROSENBROCK8	5859	22,384	24,123	5859
ROSENBROCK16	8528	23,513	24,132	8528
SHEKEL5	3381	3381	16,781	2778
SHEKEL7	3600	3600	17,989	3324
SHEKEL10	3258	3258	19,227	2866
SINU4	2923	2923	9767	2231
SINU8	2934	2934	10,714	2502
TEST2N4	3623	3623	9619	3603
TEST2N9	4327	4327	21,381	4207
TEST30N3	3225	3237	3513	2769
TEST30N4	3324	3324	6124	3144
Total	152,006	236,870	451,937	144,460

Figure 5. Statistical comparison for the GWO optimizer using different termination rules.

Once again the combinatorial termination technique achieves the lowest number of function calls of all available techniques. Each optimization method and termination criterion function differently and yield varying results. This variation is further influenced by the differing nature of the cost functions involved. When integrated into the overall algorithm, each method, in combination with the termination criterion that best suits it, can converge to optimal solutions more quickly, thereby enhancing the algorithm’s overall performance.

The proposed termination rule was also applied to the current parallel optimization technique for the test functions described previously. The experimental results for the so-called DoubleBox stopping rule and the given parallel method are shown in Table 8. The columns in the table stand for the following:

Table 8. Experiments using the proposed optimization technique and the DoubleBox stopping rule. The experiment was performed on the test functions described previously. Numbers in cells denote average function calls.

Problem	1 × 500	2 × 250	5 × 100	10 × 50
BF1	36,018	28,282	18,419	10,859
BF2	42,191	37,555	16,568	10,710
BRANIN	45,165	45,726	45,757	34,381
CAMEL	54,786	54,782	54,253	53,839
CIGAR10	18,387	18,300	18,307	11,950
CM4	60,796	60,501	60,287	60,200
DISCUS10	25,115	24,586	14,565	8957
EASOM	40,376	40,211	40,100	40,099
ELP10	20,193	14,745	11,211	7691
EXP4	54,779	54,676	54,416	54,412
EXP16	52,768	52,205	52,563	52,560
GKLS250	51,980	51,240	51,233	51,231
GKLS350	48,577	48,761	48,702	4100
GRIEWANK2	56,767	56,690	51,170	26,076
GRIEWANK10	20,941	20,721	14,870	11,765
POTENTIAL3	49,418	49,800	49,109	49,002
POTENTIAL5	59,827	59,656	59,367	59,322
POTENTIAL6	62,720	62,398	62,294	62,280
POTENTIAL10	72,899	72,745	72,943	72,578
HANSEN	45,064	45,689	45,510	45,424
HARTMAN3	47,307	47,112	47,111	47,100
HARTMAN6	49,370	49,302	49,312	49,222
RASTRIGIN	57,617	56,577	56,513	56,400
ROSENBROCK8	22,997	22,903	14,139	14,001
ROSENBROCK16	42,117	34,856	21,392	13,722
SHEKEL5	50,631	50,198	50,233	44,662
SHEKEL7	51,212	51,200	51,200	45,610
SHEKEL10	51,741	51,607	51,587	47,761
SINU4	48,140	48,100	48,111	48,099
SINU8	48,761	48,434	48,423	48,204
TEST2N4	48,904	48,307	48,354	48,354
TEST2N9	48,838	48,555	48,600	48,335
TEST30N3	51,640	51,307	51,194	51,143
TEST30N4	48,992	48,549	48,540	48,487
Total	1,587,034	1,556,276	1,476,353	1,338,536

• Column Problem denotes the test function used.
• Column 1 × 500 denotes the application of the proposed technique with one processing unit and 500 particles.
• Column 2 × 250 stands for the usage of two processing units. In each unit, the number of particles was set to 250.
• Column 5 × 100 denotes the incorporation of 5 processing units into the proposed method. In each processing unit the number of particles was set to 100.
• Column 10 × 50 stands for the usage of 10 processing units. In each processing unit, the number of particles was set to 50.

In this experiment, the total number of particles was set to 500, in order to have credibility in the values recorded. The results indicate that the increase in processing units may reduce the required number of function calls. The same series of experiments was also conducted using the so-called mean fitness termination rule, which has been described in Equations (2) and (3). The obtained results are presented in Table 9 and Table 10.

Table 9. Experiments using the proposed optimization technique and the mean—fitness stopping rule. Numbers in cells denote average function calls.

Problems	1 × 500	2 × 250	5 × 100	10 × 50
BF1	49,714	41,284	41,089	19,596
BF2	58,272	46,270	41,324	15,878
BRANIN	45,165	45,665	44,894	7268
CAMEL	54,786	54,989	54,651	10,295
CIGAR10	63,026	62,206	60,277	27,217
CM4	60,796	60,333	40,072	12,977
DISCUS10	50,988	50,690	36,151	14,040
EASOM	40,376	34,178	21,355	7377
ELP10	50,397	16,787	9334	7809
EXP4	54,749	54,445	29,812	11,024
EXP16	52,768	51,233	24,782	10,722
GKLS250	51,980	50,333	37,696	9641
GKLS350	48,577	48,442	28,420	9337
GRIEWANK2	56,767	40,471	30,632	12,356
GRIEWANK10	68,439	58,410	53,818	30,317
POTENTIAL3	49,418	49,328	33,657	10,140
POTENTIAL5	59,827	58,259	34,436	11,515
POTENTIAL6	62,720	62,715	34,158	14,417
POTENTIAL10	72,899	72,800	64,964	16,780
HANSEN	45,064	45,582	32,013	9318
HARTMAN3	47,307	47,983	31,033	8899
HARTMAN6	49,370	47,983	27,884	9602
RASTRIGIN	54,637	53,551	33,095	10,169
ROSENBROCK8	67,885	57,314	46,961	25,383
ROSENBROCK16	77,207	66,172	59,098	41,461
SHEKEL5	50,631	49,800	32,025	10,389
SHEKEL7	51,212	50,630	29,136	11,193
SHEKEL10	51,741	50,321	33,462	11,200
SINU4	48,140	52,788	32,412	9721
SINU8	48,761	55,864	32,685	10,410
TEST2N4	48,904	48,800	28,959	8643
TEST2N9	48,838	47,001	36,281	8085
TEST30N3	51,514	50,100	39,828	11,699
TEST30N4	48,992	46,875	42,564	11,954
Total	1,841,867	1,729,602	1,258,958	456,832

Table 10. Experiments using the proposed optimization technique and the best—fitness stopping rule. Numbers in cells denote average function calls.

Problems	1 × 500	2 × 250	5 × 100	10 × 50
BF1	9270	6253	6497	6089
BF2	9838	6249	6078	5853
BRANIN	5787	5022	4991	4989
CAMEL	12,204	5882	5765	5399
CIGAR10	7631	6482	6350	6420
CM4	16,222	5037	7422	6366
DISCUS10	5997	7364	4906	5037
EASOM	4701	4752	4653	4668
ELP10	6230	5223	5407	5314
EXP4	10,277	5675	5638	5526
EXP16	9577	5575	5471	5470
GKLS250	13,328	5864	5323	4993
GKLS350	10,131	4998	5442	4919
GRIEWANK2	8504	7345	6108	5876
GRIEWANK10	8535	7704	7771	7126
POTENTIAL3	11,338	5726	5471	5699
PONTENTIAL5	12,414	7192	6977	7070
PONTENTIAL6	14,760	8480	7210	7511
PONTENTIAL10	17,729	12,218	10,217	9439
HANSEN	8379	5945	5023	4972
HARTMAN3	10,674	5613	5703	5163
HARTMAN6	10,841	5541	5485	5536
RASTRIGIN	13,657	6636	6129	5399
ROSENBROCK8	9456	6979	7165	7049
ROSENBROCK16	9853	8620	8110	8379
SHEKEL5	10,959	5437	5684	5616
SHEKEL7	10,651	5715	5739	5486
SHEKEL10	1533	6064	5772	5516
SINU4	10,291	6249	5426	5188
SINU8	9993	6847	6080	5582
TEST2N4	10,242	6131	5837	5480
TEST2N9	12,998	7402	6455	5702
TEST30N3	6228	5712	5723	5321
TEST30N4	5540	5437	5490	5484
Total	335,768	217,369	207,518	199,637

And, in this series of experiments, the columns of the table retain the same meaning as in Table 8. Furthermore, once again, it is observed that the increase in the number of computing units significantly reduces the required number of function calls to find the global minimum. Additionally, there is a significant reduction in the number of function calls required to be compared to the previous termination rule. Finally, the proposed termination rule is utilized in the parallel optimization technique and the experimental results are outlined in Table 11.

Table 11. Experiments using the proposed optimization technique and proposed stopping rule. Numbers in cells denote average function calls.

Problems	1 × 500	2 × 250	5 × 100	10 × 50
BF1	10,839	6543	6375	5933
BF2	9838	6317	5978	5716
BRANIN	5787	5003	5076	4936
CAMEL	12,204	5889	5703	5434
CIGAR10	7631	6575	6582	6500
CM4	16,222	8935	6837	6392
DISCUS10	5997	5039	5001	5252
EASOM	4701	4791	4772	4751
ELP10	6230	5363	5471	5205
EXP4	10,277	5542	5620	5533
EXP16	9577	5518	5544	5532
GKLS250	13,328	5993	5205	5011
GKLS350	10,131	7373	5094	4869
GRIEWANK2	8504	8284	6371	5761
GRIEWANK10	8535	7585	7229	6922
POTENTIAL3	11,338	5666	5830	5695
POTENTIAL5	12,414	7310	7126	6996
POTENTIAL6	14,760	8450	7477	7561
POTENTIAL10	17,729	11,928	10,431	9338
HANSEN	8379	6862	4883	4970
HARTMAN3	10,674	5395	5219	5078
HARTMAN6	10,841	5486	5649	5348
RASTRIGIN	13,657	8283	5850	5292
ROSENBROCK8	9456	6751	7232	6979
ROSENBROCK16	9853	7993	8377	8086
SHEKEL5	10,959	5749	5810	5884
SHEKEL7	10,651	5828	5843	5698
SHEKEL10	1533	5863	6112	5588
SINU4	10,291	6427	5371	5251
SINU8	9993	7139	6274	5583
TEST2N4	10,242	7516	5573	5129
TEST2N9	12,998	10,137	6112	5566
TEST30N3	6228	5512	5402	5485
TEST30N4	5540	5749	5750	5350
Total	337,337	228,794	207,179	198,624

And, in this case, it is observed that the increase in parallel processing units significantly reduces the required number of function calls, as in the two previous termination rules. However, the proposed termination rule dramatically reduces the number of required function calls and consequently the computation time compared to previous termination rules. This reduction in fact intensifies as the number of computing nodes increases. Also, for the same set of experiments, a statistical test (Wilcoxon test) was conducted and it is graphically presented in Figure 6.

Figure 6. Wilcoxon rank–sum test results for the comparison of different termination rules as used in the proposed parallel optimization technique and for five processing units.

A p-value of less than 0.05 (two-tailed) was used to determine statistical significance and is indicated in bold.

An additional experiment was performed to assess the capabilities of the parallel technique using the CM function for various values of the dimension. This function is defined as:

$$f\left(x\right)=\sum _{i=1}^n{x}_i^2-{\displaystyle \frac{1}{10}}\sum _{i=1}^{n}cos\left(5\pi x_i\right)$$

The proposed method was tested on this function for different values of the dimension n and for a variety of processing units and the results are graphically illustrated in Figure 7.

Figure 7. Experiments using the proposed parallel technique and the CM function. The dimension of the function n varies from 1 to 20 and the number of processing units changes from 1 to 20.

From the experimental results, it is obvious that the required number of function calls is drastically reduced with the increase in computing units from 1 to 2 or to 5.

Also, the proposed method was applied on the GKLS function for different versions of the function. The dimension of the function was in the range $[2\dots 5]$. In each dimension, 10 different versions of the functions were used and average execution times and function calls were recorded. Also, the proposed method was applied on the GKLS function using different numbers of processing units. The average function calls from this experiment are show graphically in Figure 8 and the average execution times in Figure 9.

Figure 8. Average function calls for the proposed method and different dimensions of the GKLS function.

Figure 9. Average execution time for the GKLS function and the proposed method.

The graphs above show once again that adding processing units to the proposed technique can reduce not only the execution time, as expected, but also the required number of function calls. In the specific scheme, the execution time increases slightly for a certain number of execution units and above, as this time also includes the time spent by the execution engine for the synchronization of the processing units as well as for the exchange of information between them.

Furthermore, in Figure 10, the behavior of speedup and efficiency is depicted as a function of the number of PUs. The execution of the algorithm with a single processing unit, where the speedup is 1, serves as the reference point. As the number of PUs increases, a corresponding increase in speedup is observed, though it is often less than the ideal (equal to the number of PUs) due to communication delays and overhead. Conversely, efficiency decreases with the increase in PUs, a common phenomenon in parallel computing, caused by non-linear scaling and uneven workload distribution.

Figure 10. Speedup and efficiency of the parallel approach.

5. Conclusions

In this paper, the use of mixed termination rules for global optimization methods was thoroughly presented, and a new global optimization method that takes full advantage of parallel computing structures was afterward proposed. The use of mixed termination rules leads a series of computational global optimization techniques to find the global minimum of the objective function faster, as was also shown by the experimental results. The termination rules exploited are based on asymptotic criteria and are general enough to be applicable to any global optimization problem and stochastic global optimization technique.

Furthermore, the new stopping rule is utilized in a novel stochastic global optimization technique that involves some basic global optimization techniques. This method is designed to be executed in parallel computation environments. At each step of the method, a different global optimization method is also executed on each parallel computing unit, and these different techniques exchange optimal solutions with each other. In current work, the global optimization techniques of Differential Evolution, Multistart and PSO were used in the processing units, but the method can be generalized to use other stochastic optimization techniques, such as genetic algorithms or simulated annealing. Also, the overall method terminates with the combined termination rule proposed here, and the experimental results performed on a series of well-known test problems from the relevant literature seem to be very promising.

However, the present work is limited to a specific set of optimization methods and termination criteria. Therefore, future extensions of this work may include the integration of additional and more advanced termination criteria, as well as the implementation of alternative global optimization methods in the processing units, such as genetic algorithms or variations of simulated annealing.