Utilization of Upper Confidence Bound Algorithms for Effective Subproblem Selection in Cooperative Coevolution Frameworks

Abstract

In cooperative coevolution (CC) frameworks, it is essential to identify the subproblems that can significantly contribute to finding the optimal solutions of the objective function. In traditional CC frameworks, subproblems are selected either sequentially or based on the degree of improvement in the fitness of the optimal solution. However, these classical methods have limitations in balancing between exploration and exploitation when selecting the subproblems. To overcome these weaknesses, we propose upper confidence bound (UCB)-based new subproblem selection methods for the CC frameworks. Our proposed methods utilize UCB algorithms to strike a balance between exploration and exploitation in subproblem selection, while also incorporating a non-stationary mechanism to account for the convergence of evolutionary algorithms. These strategies possess novel characteristics that distinguish our methods from existing approaches. In comprehensive experiments, the CC frameworks using our proposed subproblem selectors achieved remarkable optimization results when solving most benchmark functions comprised of 1000 interdependent variables. Thus, we found that our UCB-based subproblem selectors can significantly contribute to searching for optimal solutions in CC frameworks by elaborately balancing exploration and exploitation when selecting subproblems.

1. Introduction

Since the development of evolutionary algorithms, many studies have been conducted to address the challenge of complex large-scale global optimization (LSGO) problems by utilizing evolutionary algorithms [1,2,3,4,5]. In particular, cooperative coevolutionary (CC) frameworks [6] have achieved notable performance in addressing many LSGO problems. Consequently, numerous studies have aimed to enhance the optimization abilities of CC frameworks to effectively addressing complex LSGO problems, particularly large-dimensional black-box functions [7,8,9,10,11].

The CC framework employs the divide-and-conquer strategy to effectively address LSGO problems [6,7]. In detail, the CC framework first divides the solution space, which is the domain space of the given objective function, into one or more subspaces, i.e., subproblems, with smaller dimensions than the original one. The CC framework then selects one subproblem to be explored in the subsequent evolutionary step. Thereafter, the evolutionary algorithms, such as genetic algorithms (GA) [12,13], differential evolution (DE) [14,15], and particle swarm optimization (PSO) [16,17], search for the optimal solutions locally within the subspace related to the selected subproblem by utilizing one or more individuals. In this process, the individuals are evolved through evolutionary operations such as crossover, mutation, and selection. Finally, the CC framework evaluates the fitness values of the evolved individuals and updates an optimal solution, i.e., an individual with the best fitness. These processes are repeated until either the optimal solution converges or the maximum number of fitness evaluations is exhausted.

One of the essential issues within the CC framework is the selection of the subproblem to be searched in the next step, which is referred to as the subproblem selection task [7]. The CC framework designates the subspace to be explored by the evolutionary algorithm in every evolutionary step. Because the decision variables contribute differently to fitness computation, their corresponding subproblems also play distinct roles in searching for optimal solutions. In this case, many CC frameworks attempt to intensively select and evolve the subproblems that make the most significant contribution to improving the fitness of the best individuals, thereby facilitating a rapid search for the optimal solution. Nevertheless, such fast solution searches often cause premature convergence, seriously impairing the efficacy and accuracy of solution searches [18,19].

To effectively prevent this phenomenon, CC frameworks should sometimes choose subproblems with relatively low contributions and search the subspaces they span. That is, the subproblem selection task in CC frameworks is inherently related to the exploration-and-exploitation trade-off problem [20,21,22]. In exploration-based subproblem selection, a broad search of various solutions in the solution space is performed to identify a diverse range of potential solutions. On the other hand, the exploitation-based mechanism focuses on selecting subproblems that have yielded the best solution search results to intensively search for candidate optimal solutions near the current optimal solution, thereby enabling faster convergence. If exploration-based subproblem selection is excessively performed, individuals will slowly converge, eventually failing to find the best solution within limited computational resources. Conversely, excessive exploitation-based subproblem selection can result in premature convergence by missing the opportunity to discover diverse candidate solutions. Thus, CC frameworks aim to balance the exploration- and exploitation-based subproblem selection mechanisms to effectively find the optimal solution.

In order to achieve this goal successfully, the multi-armed bandit (MAB) algorithms [23], such as $\epsilon$-greedy algorithm [24], the $\epsilon$-first and decreasing algorithms [25,26], and upper confidence bound (UCB) algorithms [27,28], have been widely utilized. Among them, the UCB algorithms have shown remarkable abilities in numerous applications requiring sophisticated control of exploration and exploitation. Therefore, it is reasonable to utilize the MAB algorithms as fundamental techniques to address the subproblem selection task in the CC framework.

Accordingly, in this paper, we propose MAB algorithms, specifically UCB algorithm-based new subproblem selection methods for the CC frameworks, to effectively control the balance between exploration and exploitation when selecting the subproblem to be explored in the next step. Our proposed subproblem selectors utilize the UCB algorithms as the base algorithm to identify the promising subproblems that can significantly contribute to improving the optimization results in each evolutionary step. Moreover, our subproblem selectors employ non-stationary mechanisms [29,30,31] to effectively address the characteristics of evolutionary algorithms in which the convergence status of individuals dynamically changes. In the empirical experiments with 1000-dimensional benchmark functions [32,33], we found that the CC frameworks with our proposed subproblem selectors could search for optimal solutions more effectively than traditional CC frameworks. In particular, when solving the benchmark functions with numerous non-separable variables, the CC frameworks with our subproblem selectors achieved better optimization results than those of the classical CC ones. These experimental results indicate that our proposed UCB-based subproblem selection methods are effective in addressing the LSGO functions with complex interdependencies.

This paper comprises five sections. In Section 2, we explain the background knowledge about evolutionary algorithms required to understand our study. In Section 3, we provide several preliminaries that are needed to study the subproblem selection methods used in the CC frameworks. In Section 4, we propose four new UCB-based subproblem selection methods and show their detailed implementations. In Section 5, we present the experiments conducted to evaluate the performance of the proposed subproblem selectors in practical CC frameworks, along with their results. Finally, we summarize our study results and explain future study plans in Section 6.

2. Related Works

An optimization problem aims to find the minimum (or maximum) solution to an objective function. If the domain of the function consists of large-scale dimensions, we primarily refer to such problem as an LSGO problem. In general, the objective functions addressed in the LSGO have a complicated surface in the solution space. Particularly, if there exist strong interdependencies among the variables of the objective function, they span relatively further complex solution spaces, having many local minima, saddle points, and multimodal peaks, which make the search for optimal solutions extremely challenging. In this case, traditional analytic or numerical methods are inadequate to address these complicated objective functions because they require excessive computational costs to find their optimal solutions.

Accordingly, various methods to solve LSGO problems effectively using evolutionary algorithms have been widely studied [1,2,3,4,5]. Evolutionary algorithms aim to find approximately optimal solutions for complex optimization problems by leveraging evolutionary processes such as natural selection, survival of the fittest, and reproduction [4]. Evolutionary algorithms commonly involve three core elements, i.e., individuals, fitness, and solution search mechanisms. Firstly, an individual represents a possible candidate solution of the objective function addressed by the evolutionary algorithm. To address the objective function with an n-dimensional domain, an individual is composed of n elements which are mapped to the variables spanning the solution space one by one. Secondly, an individual has a unique fitness, a numerical value evaluated by the objective function, to present its quality. Thirdly, the solution search mechanism defines detailed methods for finding an optimal solution to the objective function, i.e., an individual with the best fitness in the solution space. In the evolutionary algorithm, all individuals evolve iteratively through operations such as mutation, crossover, and selection in every evolutionary step until they no longer show significant evolution. Through these repeated processes, the evolutionary algorithm searches for the approximately optimal solution of the objective function in the solution space.

Evolutionary algorithms have shown a notable ability to solve various LSGO problems. For example, X. Huang et al. proposed a surrogate-assisted gray prediction evolutionary algorithm to efficiently solve large-dimensional optimization problems [5]. They demonstrated that the surrogate model and inferior offspring learning strategies can further enhance the solution search ability of the gray prediction evolutionary algorithm. M. Song et al. developed a learning-driven algorithm with dual evolution patterns to effectively tackle large-scale multi-objective optimization problems [2]. To this end, they proposed a method that learns dual evolution patterns in the evolutionary process to efficiently generate promising solutions. Moreover, Z.-J. Wang et al. studied effective methods using the distributed particle swarm optimization algorithm to efficiently address large-dimensional optimization problems [34]. They demonstrated that a dynamic group learning strategy and an adaptive renumbering strategy can significantly contribute to solving large-dimensional cloud workflow scheduling problems using the distributed PSO algorithms.

Nevertheless, evolutionary algorithms still face many challenges in effectively addressing the problems with complex interdependencies among their variables. In particular, the prohibitive computational cost required to solve LSGO problems makes searching for their optimal solutions extremely challenging. To overcome this difficulty, a divide-and-conquer-based evolutionary algorithm framework, known as the CC framework, has been widely utilized [6,7], which is demonstrated in the next section.

3. Preliminaries

3.1. CC Frameworks to Solve LSGO Problems

The CC framework is one of the most effective evolutionary algorithms for solving a LSGO problem. It is formulated as

$${\mathbf{x}}^{*}={arg\; min}_{\mathbf{x}}f\left(\mathbf{x}\right)$$

(1)

where $f\left(\mathbf{x}\right)$ is an n-dimensional scalar function such that $f:{\mathbb{R}}^n\to \mathbb{R}$ and $\mathbf{x}$ is an n-dimensional decision vector in the domain of the function f. Before explaining the CC framework in detail, we first define a concept of subproblem as follows.

Definition 1. 

For an objective function $f:{\mathbb{R}}^n\to \mathbb{R}$, a problem is a set of the decision variables spanning an n-dimensional domain space of f. That is, $V^{\left(f\right)}=\left\{x\right|x\mathit{spans}{\mathbb{R}}^n\mathit{of}f\}.$

In the CC framework, a problem for the given objective function f, i.e., $V^{\left(f\right)}$ is decomposed into K subsets by the problem decomposers such as DG [8], FII [35], RDG [36], EVIID [37], and ERDG [38]. To this end, the modern problem decomposers identify interdependencies among all the variables and group them into K disjoint subsets based on their interdependencies. The variable interdependency is defined as follows [8].

Definition 2. 

For an objective function $f:{\mathbb{R}}^n\to \mathbb{R}$, if any two variables $x_i$ and $x_j$ in $V^{\left(f\right)}$ satisfy

$$\begin{array}{cc}\hfill f(\dots ,x_i+{\sigma }_i,& \dots ,x_j,\dots )-f(\dots ,x_i,\dots ,x_j,\dots )\ne \hfill \\ \hfill & f(\dots ,x_i+{\sigma }_i,\dots ,x_j+{\sigma }_j,\dots )-f(\dots ,x_i,\dots ,x_j+{\sigma }_j,\dots )\hfill \end{array}$$

(2)

they are interdependent. Otherwise, they are independent.

In Definition 2, ${\sigma }_i$ and ${\sigma }_j$ are constants used for perturbations of $x_i$ and $x_j$, respectively. If two variables $x_i$ and $x_j$ are interdependent, it is notated as “$x_i\leftrightarrow x_j$”. Otherwise, two variables are independent and it is notated as “$x_i\nleftrightarrow x_j$”.

According to the variable interdependency identification rule shown in Equation (2), all the variables in $V^{\left(f\right)}$ are clustered into one or more disjoint subsets by grouping the interdependent variables into one set. Accordingly, the problem $V^{\left(f\right)}$ is decomposed into K disjoint subsets, called subproblems, as follows.

Definition 3. 

For an objective function $f:{\mathbb{R}}^n\to \mathbb{R}$, a subproblem is a disjoint subset of $V^{\left(f\right)}$ and the variables in the set are interdependent. If $V_i^{\left(f\right)}$ is the ith subproblem of f, i.e., $V_i^{\left(f\right)}\subseteq V^{\left(f\right)}$, it satisfies that $\forall x_j,x_k\in V_i^{\left(f\right)},x_j\leftrightarrow x_k$ and $\forall r\ne i,V_r^{\left(f\right)}\cap V_i^{\left(f\right)}=\varphi$.

That is, the subproblems of $V^{\left(f\right)}$ are generated by grouping all the variables in $V^{\left(f\right)}$ based on their interdependencies. The variables in the same subproblem are interdependent. Based on Definitions 1 and 2, the CC framework efficiently searches for a global optimal solution of the objective function with large-dimensional domain space in a divide and conquer manner.

Algorithm 1 shows a pseudocode of the traditional CC framework [6]. Algorithm 1 takes two inputs f and m where f is an objective function $f:{\mathbb{R}}^n\to \mathbb{R}$ and m is the number of individuals used in the evolutionary algorithm. The parameter $maxFEs$ is the maximum number of allowable fitness evaluations. In the initial phase, the CC framework divides the objective function into K disjoint subproblems depending on the interdependencies among the variables that constitute the domain of the objective function. To this end, problem decomposers, such as DG [8], FII [35], RDG [36], EVIID [37], and ERDG [38], are utilized. Then, in the evolutionary phase, the CC framework chooses one of the K subproblems using the subproblem selector to determine the subproblem that will be evolved in the next step. Afterward, the CC framework evolves the individuals in the subpopulation related to the selected subproblem by using evolutionary algorithms, such as DE and PSO. Subsequently, the evolved individuals in the subpopulation are evaluated by the objective function, and the individual with the best fitness is identified as the optimal individual in the current step (to evaluate the fitness values of the individuals, a function “Feval” is utilized in the CC frameworks. This function is described in Appendix A). The evolutionary phase is repeated until the number of fitness evaluations (i.e., FEs) exceeds the maximum number of fitness evaluations (i.e., maxFEs) [7]. Finally, the CC framework returns the individual with the best fitness in the evolved population as the optimal solution for the objective function.

Algorithm 1 Traditional algorithm: BasicCC

Require:  f, m; maxFEs   1:   n = dimensions of the domain space of f   2:   P = make_matrix(m, n, “randomly”)   3:   fit_list = make_vector(m, 0)   4:   fit_list = Feval(f, P)   5:   FEs = m   6:   cv = P$[{arg\; min}_i($Feval(f, P)), : ]   7:   $\{S_1,\dots ,S_K\}$ = problemDecomposer(f, cv)   8:   while FEs ≤ maxFEs do   9:       for $i=1$; $i\le K$; $i=i+1$ do   10:         subP = P$[:,S_i]$   11:         subP = Evolution(subP, P, cv, ...)   12:         P$[:,S_i]$ = subP   13:         sol = P$[{arg\; min}_i$(Feval(f, P)) , : ]   14:         FEs = FEs + n   15:         cv = sol   16:     end for   17: end while   return  sol

3.2. Subproblem Selection Task in CC Frameworks

One of the essential tasks in the CC framework is choosing the subproblem that will be evolved by the evolutionary algorithm in the next step [7]. A subproblem is composed of one or more variables that comprise the domain of the objective function. Then, selecting a subproblem restricts the solution search range to the specific area corresponding to the selected subproblem in the solution space. Because the variables involved in the subproblems have distinct influences on the function in the CC framework, the selection of subproblems significantly affects the search for an optimal solution.

Figure 1 describes how the subproblem selection affects the ability to search for an optimal solution for the objective function $f\left(x\right)=2x_1^4-x_2^2$. This objective function is decomposed into two subproblems $\left\{x_1\right\}$ and $\left\{x_2\right\}$ by the problem decomposers. At the current optimal solution, the gradient of $x_1$ is relatively sharper than that of $x_2$. If the first subproblem $\left\{x_1\right\}$ is selected based on the exploitation mechanism, the solution search will be intensively conducted on $x_1$, accelerating its convergence. However, this rapid process may lead to premature convergence into local minima or saddle points, thereby degrading the quality of the final solution. On the other hand, if the second subproblem $\left\{x_2\right\}$ is chosen instead of $\left\{x_1\right\}$ based on the exploration, its convergence becomes relatively slower because the axis of $x_2$ has a gentler slope than $x_1$. However, the solution search on a smooth slope, like $x_2$, can lead to discovering more optimal solutions by bypassing the local minima and saddle points on the solution surface.

The example in Figure 1 illustrates that CC frameworks should search for candidate solutions across various scopes in the solution space, maintaining a balance between exploration and exploitation to discover the optimal solution successfully. Through such a solution search, we can avoid local minima or saddle points within the large-dimensional solution space. To perform these effective solution searches, the subproblem selectors in the CC frameworks must effectively address the exploration–exploitation trade-off when selecting the subproblem to be evolved in the next step. Accordingly, in the following section, we present novel subproblem selection methods to effectively identify the most promising subproblem that can most effectively balance exploration and exploitation during solution search.

4. Proposed Methods

In this section, we propose four new UCB-based subproblem selectors for generic CC frameworks. Figure 2 illustrates the overall architecture of our novel subproblem selectors. Our proposed subproblem selectors consist of three fundamental components: a reward evaluation function (REF), a contribution score computation function (CSF), and a subproblem selection function (SSF).

The proposed subproblem selector is utilized in the CC framework as follows: first, the CC framework extracts the subpopulation corresponding to the selected subproblem from the original population and evolves it using the evolutionary algorithm (Step 1). The evolved subpopulation is then re-merged into the original population (Step 2), and all individuals in the updated population are evaluated by the objective function (Step 3). Next, the REF in the subproblem selector computes a reward score for the evolved subproblem by referencing the individuals’ fitness values and indexes its computation results for incremental computation (Step 4). The CSF then calculates the contribution scores of all the subproblems by aggregating various information about them and their individuals, such as the reward scores, the number of evolutions of each subproblem, and the total number of steps (Step 5). After the CSF updates the contribution scores of the subproblems, the SSF selects a subproblem to be evolved in the next step based on the contribution scores (Step 6).

4.1. Component 1: Reward Evaluation Function (REF)

After the evolved subpopulation is merged into the population, the objective function evaluates all the individuals in the updated population to identify the new best individual. In this process, it is necessary to evaluate the degree of improvement in the best fitness after the subpopulation has evolved in order to measure how much the subproblem contributed to finding the best individual, i.e., the optimal solution. Let $f_p$ and $f_c^{\left(i\right)}$ be the non-negative fitness values of the best individuals observed before and after the ith subproblem evolves, respectively. We can then formulate the REF to evaluate the degree of improvement in the best fitness obtained after evolving the ith subproblem as

$$REF(f_p,f_c^{\left(i\right)};\gamma )={\displaystyle \frac{\left|f_p-f_c^{\left(i\right)}\right|}{\left|f_p\right|+\gamma }}$$

(3)

where $\gamma$ is a positive smoothing factor to prevent that the denominator becomes zero (i.e., $0<\gamma <1$). Accordingly, the reward score for each subproblem is always adjusted to a value between zero and one, regardless of when the subproblem has evolved and the individuals’ convergence status. Thus, the REF computes the reward score for the evolved subproblem using Equation (3) and passes its result to the CSF in the subproblem selector.

4.2. Component 2: Contribution Score Computation Function (CSF)

After the REF calculates the reward score of the evolved subproblem, the CSF computes the contribution scores of all the subproblems to evaluate how much each subproblem contributed to improving the fitness of the best individual. The simplest method to measure the contribution of the subproblem is to average all of its reward scores. However, this method causes several subproblems with relatively higher reward scores to be repeatedly selected. In this case, the specific scopes spanned by the selected subproblems are biasedly explored, thereby disturbing the exploration of various candidate solutions. Thus, subproblems with relatively small reward scores should sometimes be selected to explore various scopes in the solution space. It is a typical exploration–exploitation trade-off issue addressed by MAB algorithms.

Accordingly, the CSF utilizes various arm selection policies of the UCB algorithms to compute the contribution scores for the subproblems. The CSF computes the contribution score of the ith subproblem using

$$CS\left(i\right)=\mu (i;n_i)+\sqrt{\xi (i;n,n_i)}$$

(4)

where $\mu (i;n_i)$ is an average function and $\sqrt{\xi (i;n,n_i)}$ is a padding function, which are computed from the reward scores obtained after evolving the ith subproblem. n is the total number of evolutionary steps and $n_i$ is the number of times the ith subproblem is selected and evolved. In Equation (4), $\mu (i;n_i)$ computes the empirical average of the reward scores and $\sqrt{\xi (i;n,n_i)}$ controls the trade-off between exploration and exploitation when selecting the subproblems. By varying these components, we can derive four new UCB-based contribution score computation methods. The detailed methods to calculate the contribution scores are explained in the following sections.

4.2.1. UCB1-Based Contribution Score Computation Method

Our first method utilizes the arm selection strategy of the UCB1 algorithm. In this method, the contribution score of the ith subproblem is determined by averaging all its reward scores and calculating the padding function. That is, the contribution score for the ith subproblem is computed by

$$C{S}_{U1}\left(i\right)={\displaystyle \frac{{\sum }_{j=1}^{n_i}{r}_{ij}}{n_i}}+\sqrt{{\displaystyle \frac{2lnn}{n_i}}}.$$

(5)

where $r_{ij}$ is the jth reward score obtained after evolving the ith subproblem. In Equation (5), the total number of evolutionary steps (i.e., n) follows a logarithmic scale, and the number of selections of the ith subproblem (i.e., $n_i$) follows a linear scale. Accordingly, in the initial evolutionary steps, subproblems with relatively low reward scores can often be selected by the padding function. It is the typical exploration mechanism that enables exploring various scopes in the solution space. Meanwhile, as the evolutionary process progresses, the subproblems with higher reward scores are selected more often than others based on the empirical average of the reward scores, which is the typical exploitation mechanism. Thus, we can more carefully control the balance between exploration and exploitation when selecting the subproblems in the CC framework.

4.2.2. UCB1-Tuned-Based Contribution Score Computation Method

The second method for computing the contribution scores in the CSF involves improving the padding function shown in Equation (5) by utilizing the UCB1-tuned mechanism. Because the padding function involved in Equation (5) does not consider the variance in the reward scores, the contribution scores computed by Equation (5) may be sensitive to the distribution of the reward scores, i.e., their variance. To overcome this weakness, it is necessary to consider the variance of all reward scores when computing the contribution scores. That is, we can modify the padding function by introducing the variance in the reward scores. Then, the contribution score for the ith subproblem is computed by

$$C{S}_{UT}\left(i\right)={\displaystyle \frac{{\sum }_{j=1}^{n_i}{r}_{ij}}{n_i}}+\sqrt{{\displaystyle \frac{lnn}{n_i}}min\left({\displaystyle \frac{1}{4}},V_i+\sqrt{{\displaystyle \frac{2lnn}{n_i}}}\right)}$$

(6)

where $1/4$ indicates the minimum variance used to prevent the result of the padding function from becoming zero, and $V_i$ is the variance of reward scores obtained after evolving the ith subproblem, which is computed by

$$V_i={\displaystyle \frac{{\sum }_{j=1}^{n_i}{r}_{ij}^2}{n_i}}-{\left({\displaystyle \frac{{\sum }_{j=1}^{n_i}{r}_{ij}}{n_i}}\right)}^2.$$

(7)

The variance in the reward scores can significantly contribute to enhancing the variety in the solution search by improving the possibility that the subproblems with relatively lower reward scores are selected for evolution. In detail, the padding function returns a larger value as the reward scores have a greater variance. As a result, the diversity of the solution search can be further enhanced by selecting various subproblems. On the other hand, as the evolutionary process progresses, the variance in the reward scores decreases because they converge toward their empirical average. Accordingly, the influence of their average is more strengthened than the padding function, and thus, the possibility of selecting subproblems with higher reward scores becomes further improved. Simultaneously, the contribution score computed by Equation (6) is relatively less sensitive to the distribution of the reward scores compared to the UCB1-based contribution score evaluated by Equation (5). These merits can help the CC framework perform a more stable solution search, even though the individuals’ convergence status is drastically changed.

4.2.3. Non-Stationary UCB1 and UCB1-Tuned-Based Contribution Score Computation Methods

The third and fourth methods apply the non-stationary mechanism to compute the contribution scores of the subproblems. In the CC framework with evolutionary algorithms, the distribution of reward scores is dynamically changed as the individuals in the subpopulation corresponding to the selected subproblem converge to the optimal solution. Accordingly, the average and variance of the reward scores also vary dynamically over time as the evolutionary process progresses. Thus, it is reasonable to consider the dynamic characteristics of the reward score distributions when computing the contribution scores of the subproblems.

To this end, we apply the non-stationary mechanism when computing the average and variance of the reward scores used to measure the contribution scores for the subproblems. Because the average and variance of the reward scores converge rapidly over time, these values are more strongly influenced by recent reward scores than by previous ones. In other words, as the evolutionary process progresses, the weights of past reward scores should be reduced to strengthen the influence of recent reward scores when calculating their average and variance. This mechanism can be implemented by introducing a decay factor, which is exponentially reduced over time, into the formulae to compute the average and variance of the reward scores. Accordingly, an exponentially weighted average of reward scores for the ith subproblem is formulated as

$${\widehat{\mu }}_i^{\left(\alpha \right)}={\displaystyle \frac{1}{N_{\alpha }\left(i\right)}}\sum _{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}$$

(8)

where $\alpha$ is a decay factor to control the ratio of decreasing the past reward scores and $N_{\alpha }\left(i\right)$ is the sum of the weights, which is calculated by

$$N_{\alpha }\left(i\right)=\sum _{j=1}^{n_i}{\alpha }^{n_i-j}.$$

(9)

In other words, the average of the previous reward scores decreases exponentially over time each time a new reward score is calculated. Accordingly, the reward scores that have been recently evaluated carry more significance than those from the past. Simultaneously, the influence of the past reward scores exponentially diminishes as the evolutionary process progresses. Thus, the empirical average of the reward scores is strongly affected by more recent reward scores rather than past ones.

Similar to Equation (8), an exponentially weighted variance of the reward scores for the ith subproblem is computed by applying the non-stationary mechanism as

$${\hat{V}}_i^{*}={\displaystyle \frac{{\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}^2}{N_{\alpha }\left(i\right)}}-{\left({\displaystyle \frac{{\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}}{N_{\alpha }\left(i\right)}}\right)}^2.$$

(10)

Thus, we can derive the non-stationary UCB1-based contribution score computation method for the ith subproblem by applying Equations (8) and (9) into Equation (5) as

$$C{S}_{NSU}\left(i\right)={\widehat{\mu }}_i^{\left(\alpha \right)}+\sqrt{{\displaystyle \frac{2ln{\sum }_{j=1}^K{N}_{\alpha }\left(j\right)}{N_{\alpha }\left(i\right)}}}.$$

(11)

When compared to the UCB1-based contribution score presented in Equation (5), Equation (11) utilizes the weighted sum function, i.e., $N_{\alpha }\left(i\right)$, instead of $n_i$ in calculating the average reward scores. Accordingly, the padding function is also computed non-stationary as the empirical average. Furthermore, we can also derive the non-stationary UCB1-tuned-based contribution score computation method by applying Equations (8)–(10) to Equation (6) as

$$C{S}_{NSUT}\left(i\right)={\widehat{\mu }}_i^{\left(\alpha \right)}+\sqrt{{\displaystyle \frac{ln{\sum }_{j=1}^K{N}_{\alpha }\left(j\right)}{N_{\alpha }\left(i\right)}}\times V_M(i;\alpha )}$$

(12)

where $V_M(i;\alpha )$ is the minimum variance used in the padding function, which is calculated by

$$V_M(i;\alpha )=min\left({\displaystyle \frac{1}{4}},{\widehat{V}}_i^{*}+\sqrt{{\displaystyle \frac{2ln{\sum }_{j=1}^K{N}_{\alpha }\left(j\right)}{N_{\alpha }\left(i\right)}}}\right).$$

(13)

When compared to the UCB1 and UCB1-tuned-based contribution scores, this computation method can more carefully adjust the trade-off between exploration and exploitation when selecting the subproblem to be evolved in the CC framework by considering their variance calculated based on the non-stationary mechanism. Accordingly, the non-stationary-based contribution score computation methods shown in Equations (11) and (12) can effectively evaluate the degree of contribution for each subproblem, even though the distributions of the reward scores vary dynamically over time. Therefore, the non-stationary-based contribution score computation methods can significantly contribute to searching for optimal solutions in the large-dimensional solution space when compared to existing UCB algorithm-based subproblem selection methods.

4.2.4. The CSF Algorithm

The CSF can be implemented by combining the four contribution score computation methods into one. Algorithm 2 describes the detailed pseudocode for the CSF. To calculate the contribution scores for the subproblems, the CSF takes four inputs: $[r_1,\dots ,r_K]$, $[n_1,\dots ,n_K]$, n, and K. $[r_1,\dots ,r_K]$ represents the list of all the reward scores for the K subproblems where $r_i$ is a set of all reward scores recorded after evolving the ith subproblem (i.e., ${\forall }_{1\le j\le |r_i|},r_{ij}\in r_i$). $[n_1,\dots ,n_K]$ denotes the times each subproblem is selected and evolves. n is the total number of steps, i.e., the sum of times that all subproblems are selected and evolved. K is the number of subproblems. Meanwhile, $\alpha$ and sp_name indicate the parameters of the CSF. $\alpha$ is a decay factor used in the non-stationary UCB1 and UCB1-tuned-based contribution score computation methods shown in Equations (11) and (12). sp_name determines the base UCB algorithm for calculating the contribution scores.

Algorithm 2 Sub-algorithm: CSF
Require: [$r_1,\dots ,r_K$], [$n_1,\dots ,n_K$], n, K; $\alpha$, sp_name
1:   conts = make_vector(K, 0)
2:   for $i=1$; $i\le K$; $i=i+1$do
3:      if sp_name == UCB then	▹ Equation (5)
4:          $\mu =\left({\sum }_{j=1}^{n_i}{r}_{ij}\right)/n_i$
5:          $\xi =(2lnn)/n_i$
6:      else if sp_name == UCBT then	▹ Equation (6)
7:          $\mu =\left({\sum }_{j=1}^{n_i}{r}_{ij}\right)/n_i$
8:          $\nu =\left({\sum }_{j=1}^{n_i}{r}_{ij}^2\right)/n_i-{\mu }^2$
9:          $\xi =((lnn)/n_i)\times min(1/4,\nu +\sqrt{(2lnn)/n_i})$
10:    else if sp_name == NSU then	▹ Equation (11)
11:        $N_{\alpha }\left(i\right)={\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}$
12:        $\mu =\left({\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}\right)/N_{\alpha }\left(i\right)$
13:        $\xi =(2ln{\sum }_{j=1}^K{N}_{\alpha }\left(j\right))/N_{\alpha }\left(i\right)$
14:    else if sp_name == NSUT then	▹ Equation (12)
15:        $N_{\alpha }\left(i\right)={\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}$
16:        $\mu =\left({\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}\right)/N_{\alpha }\left(i\right)$
17:        $\nu =\left({\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}^2\right)/N_{\alpha }\left(i\right)-{\mu }^2$
18:        $\xi =\left(ln{\sum }_{j=1}^K{N}_{\alpha }\left(j\right))/N_{\alpha }\left(i\right)\right)\times min\left(1/4,\nu +\sqrt{(2ln{\sum }_{j=1}^K{N}_{\alpha }\left(j\right))/N_{\alpha }\left(i\right)}\right)$
19:    else
20:        error(“Incorrect subproblem selector name”)
21:        return null
22:    end if
23:    conts$\left[i\right]=\mu +\sqrt{\xi }$
24: end for
return conts

In Algorithm 2, the contribution scores for all subproblems are computed by one of the four UCB-based contribution score computation methods. After calculating the contribution scores, the CSF returns the list of evaluated contribution scores. The returned contribution scores are used to determine a subproblem to be evolved in the following step in the subproblem selection function (SSF).

4.3. Component 3: Subproblem Selection Function (SSF)

The SSF chooses a subproblem to be explored in the next step based on the contribution scores computed by the CSF. Similar to the policies that the MAB algorithms select an arm to be pulled in the following step, the SSF also preferentially selects the subproblem with the highest contribution score as

$$\mathit{i}={arg\; max}_{1\le j\le K}\Psi \left(j\right)$$

(14)

where K is the number of subproblems and $\Psi \left(j\right)$ is the contribution score of jth subproblem. Algorithm 3 describes the SSF algorithm. In Algorithm 3, is_init is a boolean variable that indicates the current evolutionary phase is initial or not. prev_idx is an index of the previously chosen subproblem. conts[] is a list variable containing the contribution scores of all the subproblems. Finally, K is the number of subproblems. In the initial evolutionary phase (i.e., is_init == true), each subproblem is chosen sequentially from the first to the last in a round-robin manner. This process is performed because each subproblem should be evolved at least once to obtain one or more reward scores for each of them. After the initial evolutionary phase is completed (i.e., is_init == false), the SSF determines the subproblem that will be evolved in the next step based on the contribution scores for the subproblems. In other words, the subproblem with the highest contribution score is preferentially chosen among all the subproblems, as shown in Equation (14). Finally, the SSF returns an index of the selected subproblem to the subproblem selector.

Algorithm 3 Sub-algorithm: SSF
Require: is_init, prev_idx, conts[], K
1: if is_init == true then	▹ round-robin-based selection
2:    new_idx = prev_idx + 1
3: else	▹ contribution-based selection
4:    new_idx = ${arg\; max}_i$ (conts[1], ..., conts[K])
5: end if
return new_idx

4.4. Implementation of the UCB-Based Subproblem Selector and Utilization in the CC Frameworks

By combining the three core functions, we can implement an integrated UCB-based subproblem selector.

Table 1. Names and abbreviations of the four proposed UCB-based subproblem selectors. The “CC frameworks” column indicates the names of CC frameworks that use the proposed subproblem selector. The “sp_name” column lists the valid values of the parameter “sp_name” used in Algorithm 2.

Base Algorithms	Names (Abbreviations)	CC Frameworks	sp_Name
UCB1	UCBSP	UCBSPCC	UCB
UCB1-tuned	UCBTSP	UCBTSPCC	UCBT
Non-stationary UCB1	NSUSP	NSUSPCC	NSU
Non-stationary UCB1-tuned	NSUTSP	NSUTSPCC	NSUT

describes the names and abbreviations of the proposed UCB-based subproblem selectors. The first and second columns present the complete names and abbreviations of the four subproblem selectors. The third column lists the names of the CC frameworks that utilize our subproblem selector. Finally, the final column shows the values of the parameter sp_name, used to determine the base subproblem selector in Algorithm 2.

Meanwhile, Algorithm 4 describes a main algorithm of our proposed UCB-based subproblem selector. Our subproblem selector has six inputs as follows: is_init is a boolean variable to indicate whether the current evolutionary phase is initial or not; $f_{prev}$ and $f_{new}$ are the fitness values of the previous and current best optimal solutions, respectively; prev_idx and idx are indices of the previously and currently selected subproblems, respectively; and K is the number of subproblems. In addition, it takes three parameters: $\alpha$, $\lambda$, and sp_name. $\alpha$ is the decay factor used for the non-stationary UCB1 and UCB1-tuned-based contribution score computations; $\lambda$ is the smoothing factor used in the REF; and finally, sp_name indicates the name of the base UCB algorithm.

Algorithm 4 Main algorithm: subproblemSelector

Require:  is_init, $f_{prev}$, $f_{new}$, prev_idx, idx, K; $\alpha$, $\gamma$, sp_name   1:   i = idx   2:   if the first execution then   3:       for $j=1$; $j\le K$; $j=j+1$ do   4:           $r_j=n_j=0$   5:       end for   6:   end if   7:   reward = REF($f_{prev}$, $f_{new}$; $\gamma$)   8:   $r_i$ = append($r_i$, reward)   9:   $n_i=n_i+1$   10: $n=n+1$   11: conts = CSF($[r_1,\dots ,r_K]$, $[n_1,\dots n_K]$, n, K; $\alpha$, sp_name)   12: new_idx = SSF(is_init,prev_idx, conts, K)   return  new_idx

Meanwhile, Algorithm 5 illustrates how our proposed subproblem selector is utilized in the CC framework. Algorithm 5 takes two inputs and four parameters. Regarding its inputs, $f:{\mathbb{R}}^n\to \mathbb{R}$ is n-dimensional objective function and m is a population size, i.e., the number of individuals used to make the population. Meanwhile, $\alpha$ and $\gamma$ are the parameters used in the CSF and REF, respectively. sp_name indicates the names of base UCB algorithm used in CSF. In Algorithm 5, the CC framework calls the proposed subproblem selector shown in Algorithm 4 to choose the subproblem to be searched in the next step. Simultaneously, the CC framework delivers the indices of the previously and currently selected subproblems (i.e., prev_i and i), as well as the past and latest best fitness values (i.e., $f_{prev}$ and $f_{new}$), to the subproblem selector.

In the subproblem selector of Algorithm 4, the REF evaluates the reward score of the evolved subproblem as shown in Equation (3). Then, the CSF computes the contribution scores of all the subproblems based on their reward scores. After all the contribution scores have been updated, the SSF chooses a subproblem to be evolved in the next step by selecting the subproblems with the highest contribution scores among the K ones. After the subproblem selection is completed, the CC framework in Algorithm 5 evolves the selected subproblem using the evolutionary algorithm and evaluates the fitness values of the evolved individuals using the “Feval” function presented in Appendix A. Then, the new optimal solution with the best fitness is found from the evolved population. These processes are repeated until the number of fitness evaluations (FEs) exceeds the maximum number of FEs (maxFEs). After all the evolutionary processes are completed, the CC framework returns the individual with the best fitness as its final optimal solution.

As demonstrated in Algorithms 1 and 5, the CC frameworks generally restrict the number of FEs performed to evolve the population to maxFEs. In this case, it is important to minimize unnecessary evolutions for unpromising subproblems that contribute little to finding the optimal solution. The subproblem selector described in Algorithm 4 can not only prevent the unnecessary evolution of subproblems with low contribution to improving optimization performance but also maintain various solution searches by sophisticatedly controlling the trade-off between control exploration and exploitation. Accordingly, the CC framework, combined with our subproblem selector, can significantly improve optimization performance.

Algorithm 5 Example algorithm: CC with UCB-based subproblem selector

Require:  $f:{\mathbb{R}}^n\to \mathbb{R}$, m; maxFEs, $\alpha$, $\gamma$, sp_name   1:   ${\Omega }_E$ = parameter set used in evolutionary algorithm   2:   P = make_matrix(m, n, “randomly”)   3:   fit_list = make_vector(m, 0)   4:   fit_list = Feval(f, P)   5:   FEs = m   6:   cv = P$[{arg\; min}_i$(fit_list), :]   7:   $f_{new}=min$(fit_list)   8:   $\{S_1,\dots ,S_K\}$ = problemDecomposer(f, cv)   9:   is_init = true   10: explorer_num = 0; idx= 1;   11: while FEs ≤ maxFEs do   12:     prev_idx = idx   13:     i = subproblemSelector(is_init, $f_{prev}$, $f_{new}$,   14:      prev_idx, idx, K; $\alpha$, $\gamma$, sp_name)   15:     if is_init == true then   16:         explorer_num = explorer_num + 1   17:         if explorer_num == K then   18:            is_init = false   19:         end if   20:     end if   21:     $f_{prev}=f_{new}$   22:     subP = P$[:,S_i]$   23:     subP_new = Evolution(subP, P, cv; ${\Omega }_E$)   24:     P$[:,S_i]$ = subP_new   25:     fit_list = Feval(f, P, $S_i$, cv)   26:     sol = P$[{arg\; min}_i$(fit_list), :]   27:     $f_{new}=min$(fit_list)   28:     cv = sol   29:     FEs = FEs + 1   30: end while   return  sol

4.5. Theoretical Analysis

4.5.1. Computational Complexity Analysis

Among Algorithms 1–4, the main function called to choose a subproblem in the CC framework is the subproblem selector function shown in Algorithm 4. Accordingly, we analyze a time complexity required to perform Algorithm 4 in the CC framework as follows.

Theorem 1. 

When an objective function $f:{\mathbb{R}}^n\to \mathbb{R}$ is given, its domain space is decomposed to K disjoint subproblems, the time complexity needed to perform our proposed subproblem selectors in the CC framework is $\Theta \left(K\right)$.

Proof of Theorem 1. 

When Algorithm 4 is first executed, the variables used for incremental computations for each of the K subproblems are initialized in lines 2–6, which requires an $\Theta \left(K\right)$ time complexity. This work is performed only once at the first execution. After the initialization work is completed, lines 7–10 are executed within constant time, i.e., $\Theta \left(1\right)$.

Afterward, the CSF is called to compute the contribution scores of all the subproblems at line 11. In the CSF, the average function $\mu$ and the padding function $\sqrt{\xi }$ are calculated for each of the K subproblems to measure their contribution scores. These can be efficiently computed by using the incremental computation method. In detail, when calculating the contribution scores based on the UCB or UCB-tuned algorithm (i.e., sp_name == “UCB” or “UCBT”), the CSF can incrementally compute two terms ${\sum }_{j=1}^{n_i}{r}_{ij}$ and ${\sum }_{j=1}^{n_i}{r}_{ij}^2$ by maintaining the previous computation results for each subproblem to avoid unnecessary repeating of the calculations. Similarly, when the non-stationary UCB and UCB-tuned algorithms were used (i.e., sp_name == “NSU” or “NSUT”), three exponentially decayed terms, i.e., ${\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}$, ${\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}$, and ${\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}^2$, can also be incrementally computed while preserving their computation results for K subproblems. In this case, only $\Theta \left(1\right)$ complexity is needed to update each summation term. Because the contribution scores for K subproblems are updated whenever the CSF is executed, the total time complexity to execute the CSF once is $\Theta \left(K\right)$.

After the CSF computes the contribution scores, the SSF selects a subproblem to be evolved among K subproblems. When “is_init == true”, all the subproblems are sequentially chosen by the SSF one by one in a round-robin manner. On the other hand, when “is_init == false”, the SSF chooses the subproblem with the best contribution score. Thus, the time complexity required to conduct the SSF once is $\Theta \left(1\right)$.

Therefore, when the number of subproblems is K, the total time complexity needed to perform the subproblem selector of Algorithm 4 is derived as

$$c\Theta \left(K\right)+\Theta \left(1\right)+\Theta \left(K\right)+\Theta \left(1\right)=\Theta \left(K\right)$$

(15)

where $c=1$ if Algorithm 4 is firstly performed, and 0 otherwise.    □

In other words, our proposed subproblem selector has a linear time complexity that is proportional to the number of subproblems. Meanwhile, we can derive the space complexity needed to conduct our subproblem selector as follows.

Theorem 2. 

When the number of subproblems is K, the space complexity required to perform our subproblem selector of Algorithm 4 is $\Theta \left(K\right)$.

Proof of Theorem 2. 

As explained in the proof of Theorem 1, the summation terms ${\sum }_{j=1}^{n_i}{r}_{ij}$, ${\sum }_{j=1}^{n_i}{r}_{ij}^2$, ${\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}$, ${\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}$, and ${\sum }_{j=1}^{n_i}{\alpha }^{n_i-j}{r}_{ij}^2$ should be retained for each of K subproblems to calculate them incrementally. To do this, K variables are needed to index them. Thus, $\Theta \left(K\right)$ space complexity is required to perform the subproblem selector shown in Algorithm 4.    □

4.5.2. Theoretical Analysis of the Effects of the Decay Factor $\alpha$

In both NSUSP and NSUTSP algorithms, the decay factor $\alpha$ controls how many previous reward scores are used to compute the contribution score of the ith subproblem. The following theorem demonstrates the exploration and exploitation effects in selecting the subproblems in our proposed methods according to the variations of $\alpha$.

Theorem 3. 

As α becomes close to one, the exploitation effect is further enhanced in the subproblem selection task. On the contrary, the influence of the exploration is further strengthened in the subproblem selection as α decreases toward zero.

Proof of Theorem 3. 

To analyze the effect of $\alpha$ in the NSUSP and NSUTSP algorithms, we first represent $N_{\alpha }\left(i\right)$ of Equation (9) as a sum of geometric sequence as

$$N_{\alpha }\left(i\right)={\displaystyle \frac{1-{\alpha }^{n_i}}{1-\alpha }}.$$

(16)

Assume the number of times the ith subproblem is selected converges to infinite (i.e., $n_i\to \infty$). Then, by taking the limit with respect to $n_i\to \infty$ in Equation (16), we can derive

$$\underset{n_i\to \infty }{lim}{N}_{\alpha }\left(i\right)={\displaystyle \frac{1}{1-\alpha }}.$$

(17)

Equation (17) indicates that $N_{\alpha }\left(i\right)$ converges to $1/(1-\alpha )$ if the ith subproblem is extremely often selected (i.e., $n_i\to \infty$). Thus, we can analyze the influence of $\alpha$ when calculating the contribution score of the ith subproblem by examining the padding function as $n_i\to \infty$. For convenient analysis, the padding functions used in NSUSP and NSUTSP, as shown in Equations (11) and (12), are denoted as

$$\sqrt{{\xi }_i}=\sqrt{{\displaystyle \frac{q}{N_{\alpha }\left(i\right)}}ln\left(\sum _{j=1}^K{N}_{\alpha }\left(j\right)\right)V_{i,\alpha }}$$

(18)

where q is a constant and $V_{i,\alpha }$ is a function that returns 1 if $\sqrt{{\xi }_i}$ is used in the NSUSP, $V_M(i;\alpha )$ of Equation (13), otherwise. Then, we can take the limit of Equation (18) as

$$\underset{n_i\to \infty }{lim}\sqrt{{\xi }_i}\approx \sqrt{q(1-\alpha )ln(·){\tilde{V}}_{i,\alpha }}$$

(19)

where $ln(·)$ and ${\tilde{V}}_{i,\alpha }$ are approximated results of $ln\left({\sum }_j{N}_{\alpha }\left(j\right)\right)$ and $V_{i,\alpha }$ when $n_i\to \infty$.

In Equation (19), the padding function $\sqrt{q(1-\alpha )ln(·){\tilde{V}}_{i,\alpha }}$ becomes close to zero as $\alpha$ approaches one. That is, the influence of the padding function is reduced and thus the exploitation-based subproblem selection is further enhanced. On the other hand, as $\alpha$ approaches zero, the padding function increases progressively. Consequently, the exploration-based subproblem selection is further enhanced.    □

Theorem 3 indicates that the parameter $\alpha$ plays a role in manually controlling the degree of exploration and exploitation in the subproblem selection task. Moreover, from the proof of Theorem 3, we also find that the effective range of $\alpha$ for stable convergence is $0<\alpha <1$ because Equation (16) diverges if $\alpha$ is greater than one.

4.5.3. Theoretical Analysis of the Effects of the Smoothing Factor $\gamma$

The smoothing factor $\gamma$ in Equation (3) is used to prevent the denominator from becoming zero. To analyze the effect of $\gamma$ when computing the contribution scores of the subproblems, we denote $r_i^{\left(t\right)}$ as the reward score observed after evolving the ith subproblem at time t. Then, $r_i^{\left(t\right)}$ is expressed as

$$r_i^{\left(t\right)}={\displaystyle \frac{|f_{p,t}-f_t^{\left(i\right)}|}{|f_{p,t}|+\gamma }}$$

(20)

where $f_{p,t}$ is a fitness of the previous optimal solution and $f_t^{\left(i\right)}$ is a fitness of the optimal solution found after evolving the ith subproblem at time t. Accordingly, we can analyze the effect of $\gamma$ in terms of the magnitude of the reward scores computed by the REF as follows.

Theorem 4. 

As γ increases, the magnitude of the reward scores calculated by the REF decreases.

Proof of Theorem 4. 

To analyze the effects of $\gamma$ in the REF, we take the partial derivative of $r_i^{\left(t\right)}$ with respect to $\gamma$ as

$${\displaystyle \frac{\partial r_i^{\left(t\right)}}{\partial \gamma }}=-{\displaystyle \frac{|f_{p,t}-f_t^{\left(i\right)}|}{\left(\right|f_{p,t}{|+\gamma )}^2}}$$

(21)

where the nominator and denominator on the right-hand side are clearly positive. In other words, Equation (21) is always less than or equal to zero. Thus, the increasing $\gamma$ continuously decreases the magnitude of the reward scores computed by the REF.    □

Meanwhile, $\gamma$ can influence the average and variance of the reward scores. Accordingly, it can affect the exploration and exploitation effects in selecting the subproblems in our proposed methods. The following theorems describe how the variations of $\gamma$ influence the average and variance of the reward scores.

Theorem 5. 

As γ increases, the average of the reward scores computed by the REF decreases.

Proof of Theorem 5. 

In general, the average of the reward scores for the ith subproblem is calculated by

$${\mu }_i^{\left(\alpha \right)}={\displaystyle \frac{{\sum }_t{\alpha }_{i,t}{r}_i^{\left(t\right)}}{N_{\alpha }\left(i\right)}}={\displaystyle \frac{1}{N_{\alpha }\left(i\right)}}\sum _t{\alpha }_{i,t}\left({\displaystyle \frac{|f_{p,t}-f_t^{\left(i\right)}|}{|f_{p,t}|+\gamma }}\right)$$

(22)

where ${\alpha }_{i,t}$ is the decay factor of $r_i^{\left(t\right)}$ at time t and $N_{\alpha }\left(i\right)$ is the sum of the decay factors shown in Equation (9). If Equation (22) is used in the UCBSP or UCBTSP algorithms, all the decay factors are set to one. In contrast, it is utilized in the NSUSP or NSUTSPCC algorithms, the decay factors can be set to any positive values less than one. Then, we can derive the partial derivative of Equation (22) with respect to $\gamma$ as

$${\displaystyle \frac{\partial {\mu }_i^{\left(\alpha \right)}}{\partial \gamma }}=-{\displaystyle \frac{1}{N_{\alpha }\left(i\right)}}\sum _t{\alpha }_{i,t}{\displaystyle \frac{|f_{p,t}-f_t^{\left(i\right)}|}{\left(\right|f_{p,t}{|+\gamma )}^2}}$$

(23)

where $N_{\alpha }\left(i\right)$, ${\alpha }_{i,t}$, $|f_{p,t}-f_t^{\left(i\right)}|$, and $\left(\right|f_{p,t}{|+\gamma )}^2$ are non-negative for all time t. Thus, $\partial {\mu }_i^{\left(\alpha \right)}/\partial \gamma$ is less than zero. Therefore, increasing $\gamma$ lowers the average ${\mu }_i^{\left(\alpha \right)}$.    □

Theorem 6. 

As γ increases, an upper bound of the variance of the reward scores decreases.

Proof of Theorem 6. 

The reward score for the ith subproblem at time t, i.e., $r_i^{\left(t\right)}$ satisfies the following inequality

$$0\le r_i^{\left(t\right)}={\displaystyle \frac{|f_{p,t}-f_t^{\left(i\right)}|}{|f_{p,t}|+\gamma }}\le {\displaystyle \frac{|f_{p,t}|}{|f_{p,t}|+\gamma }}$$

(24)

because $|f_{p,t}-f_t^{\left(i\right)}|\ge 0$. Then, the reward score $r_i^{\left(t\right)}$ always lies within an interval $\left[0,|f_{p,t}|/(|f_{p,t}|+\gamma )\right]$, i.e.,

$$r_i^{\left(t\right)}\in \left[0,{\displaystyle \frac{|f_{p,t}|}{|f_{p,t}|+\gamma }}\right].$$

(25)

Meanwhile, if a random variable X is defined on an interval $[0,A]$, its variance $Var\left(X\right)$ has an upper bound $A^2/4$ (i.e., $Var\left(X\right)\le A^2/4$). Accordingly, we can derive the upper bound of the variance of the reward scores for the ith subproblem as

$$Var\left(r_i^{\left(t\right)}\right)\le {\displaystyle \frac{1}{4}}{\left({\displaystyle \frac{|f_{p,t}|}{|f_{p,t}|+\gamma }}\right)}^2.$$

(26)

Equation (26) indicates that as $\gamma$ increases, the upper bound of the variance $Var\left(r_i^{\left(t\right)}\right)$ becomes lower quadratically. That is, the upper bound of the variance of the reward scores for the ith subproblem decreases as $\gamma$ increases.    □

Theorems 4–6 describe how variations in $\gamma$ can affect the exploration and exploitation effects in selecting the subproblem in our proposed methods. In the UCBSP and NSUSP algorithms, their padding functions do not account for the variance of the reward scores, as shown in Equations (5) and (11). In this case, the variation of $\gamma$ only affects their average functions. Accordingly, as $\gamma$ approaches one, the average of the reward scores decreases, and thus the exploration effects can be further enhanced in the subproblem selection task.

Meanwhile, the padding functions in the UCBTSP and NSUTSP algorithms use the variance of the reward scores, as described in Equations (6) and (12). In this case, the variation of $\gamma$ influences their average and padding functions simultaneously. That is, as $\gamma$ becomes large, both the average and padding functions become small. Accordingly, the exploration and exploitation effects can be reduced when performing the subproblem selection task in the UCBTSP and NSUTSP.

5. Experiments

It is essential to evaluate how significantly our proposed subproblem selectors contributed to finding optimal solutions in practical CC frameworks. Accordingly, in this section, we present the detailed results of experiments conducted to evaluate the solution search abilities of CC frameworks that utilize practical subproblem selectors, including our proposed ones.

5.1. Configurations for Experiments

First, we used the high-dimensional global optimization functions involved in the CEC’2010 [32] and CEC’2013 [33] benchmark suites to evaluate the optimization performance of the CC frameworks with the subproblem selectors. The CEC’2010 and CEC’2013 benchmark suites are official benchmark problems that have been widely used to evaluate the optimization capabilities of various CC frameworks. The benchmark functions are 1000-dimensional scalar functions in which 1000 decision variables are wholly or partially interdependent. Because the functions involve intricately interdependent variables, these solution spaces are also considerably complicated. Therefore, the CEC’2010 and CEC’2013 benchmark functions are suitable for evaluating how much the subproblem selectors could contribute to improving the optimization performance in the CC frameworks.

Second, we adopted the SaNSDE [39] algorithm as a base optimizer used in the CC frameworks. The SaNSDE, one of the advanced DE algorithms, employs self-adaptive control mechanisms to enhance the searchability of individuals in the population. As a result, the SaNSDE has shown better results than traditional DE algorithms for various optimization problems in diverse LSGO studies [39]. Moreover, we adopted ERDG [38] as the base problem decomposer. ERDG is a state-of-the-art problem decomposer that splits a given LSGO function into diverse subproblems based on the interdependencies among the variables that constitute the function. As shown in

Table 2. The number of variable groups generated after the ERDG decomposes each of the benchmark functions. Each variable group is addressed as an independent subproblem in the CC frameworks.

Benchmarks	Functions	Separable Variables	Non-Separable Variables	Separable Variable Groups *	Non-Separable Variable Groups	Used?
CEC’2010	$f_1$, $f_2$	1000	0	50	0	✔
	$f_3$	0	1000	0	1	✘
	$f_4$, $f_5$, $f_7$, $f_8$	950	50	48	1	✔
	$f_6$	0	1000	0	2	✔
	$f_9$, $f_{10}$, $f_{12}$, $f_{13}$	500	500	25	10	✔
	$f_{11}$	0	1000	0	11	✔
	$f_{14}$, $f_{15}$, $f_{16}$, $f_{17}$, $f_{18}$	0	1000	0	20	✔
	$f_{19}$, $f_{20}$	0	1000	0	1	✘
CEC’2013	$f_1$, $f_2$	1000	0	50	0	✔
	$f_3$	0	1000	0	1	✘
	$f_4$, $f_5$, $f_7$	700	300	35	7	✔
	$f_6$	0	1000	0	7	✔
	$f_8$	200	800	10	17	✔
	$f_9$, $f_{10}$, $f_{11}$	0	1000	0	20	✔
	$f_{12}$, $f_{15}$	0	1000	0	1	✘
	$f_{13}$	0	905	0	2	✔
	$f_{14}$	0	905	0	1	✘

* The separable variable groups are generated by grouping 20 separable variables for more efficient optimization in the CC frameworks according to [40]. Meanwhile, the symbols ✔ and ✘ in the final column indicate that the corresponding benchmark function was used or not used in the experiment, respectively.

, we decomposed 20 benchmark functions included in the CEC’2010 suite using the ERDG and found that $f_3$, $f_{19}$, and $f_{20}$ were decomposed into single subproblems. Similarly, among 15 CEC’2013 benchmark functions, ERDG decomposed $f_3$, $f_{12}$, $f_{14}$, and $f_{15}$ into single subproblems. Accordingly, we used the remaining functions as benchmark problems, excluding these seven functions.

Third, we adopted five CC frameworks that utilize classical subproblem selectors: BasicCC, RandomCC, CBCC1, CBCC2, and BBCC, as models for comparison. BasicCC [6], the earliest CC framework shown in Algorithm 1, selects the subproblems in a round-robin manner. RandomCC, which is a variation of BasicCC [6], chooses the subproblem randomly at each evolutionary step according to the uniform distribution. CBCC1 [9] and CBCC2 [9] are typical contribution-based CC frameworks. Unlike BasicCC and RandomCC, CBCC1 and CBCC2 select a subproblem to be evolved based on the contribution scores of all the subproblems, i.e., the degree of fitness improvement of the best individual. Meanwhile, BBCC [10] utilizes the $\epsilon$-greedy algorithm [25] to identify the subproblem to be evolved in the next step.

Table 3. The detailed configurations of the parameters used in our experiments. The omitted parameters were set equivalent to the values shown in their research papers.

Parameter Settings	Descriptions
$\alpha =0.5$	The decay factor used in NSUSP.
$\alpha =0.3$	The decay factor used in NSUTSP.
$\gamma ={10}^{-8}$	The smoothing factor to prevent that the denominator becomes zero.
maxFEs = $3\times {10}^6$	The allowable maximum number of FEs.
$M=100$	The number of individuals in a population.
$\iota =20$	The maximum number of separable variables in a separable variable group [40].
$C{R}_{\mu }=0.5$	The initial mean value of the Gaussian distribution used to adaptively control the crossover rate in the SaNSDE optimizer.
$ϵ=0.1$	The exploration-exploitation control factor used in BBCC.

lists the parameter configurations of all the subproblem selectors evaluated in our experiments.

Fourth, we implemented nine CC frameworks, each incorporating one of nine subproblem selectors, to evaluate the optimization performance of the CC frameworks with the subproblem selectors including our proposed methods. We then search for the optimal solution of each benchmark function by utilizing each of the nine CC frameworks 25 times independently, based on the guidelines of CEC’2010 and CEC’2013. Afterward, we calculated the average fitness of the 25 optimal solutions the CC framework found for each benchmark function.

Fifth, we performed Wilcoxon rank-sum one-way ANOVA tests [41] to pairwise compare the fitness values achieved by the CC frameworks that utilize our proposed subproblem selectors and existing ones for each benchmark function. In the ANOVA tests, we analyzed whether the average fitness values found by the CC frameworks with our subproblem selectors were better, worse than, or equivalent to the results of the CC frameworks with the traditional subproblem selectors for each benchmark function. We then counted the number of benchmark functions for which the CC frameworks using our subproblem selectors achieved “win,” “lose,” and “tie” when compared to the CC frameworks using traditional subproblem selection methods. To this end, we used a significance level of p = 0.05 and conducted Holm’s p-correction method [41] for more accurate comparisons.

Finally, our experiments were conducted using MATLAB R2023b in a system environment with the following specifications: Intel CoreTM i7-14700K 3.40GHz CPU, 128GB RAM, and Windows 11 Professional operating system. The statistical ANOVA tests were performed using R 4.2.1.

5.2. Ablation Studies

5.2.1. Ablation Studies for the Decay Factor $\alpha$

As explained in Section 4.2, NSUSP and NSUTSP commonly use the decay factor $\alpha$ to adjust the weights of the past reward scores in computing the average reward score. In other words, the decay factor $\alpha$ can significantly impact the optimization performance of CC frameworks that utilize NSUSP and NSUTSP as subproblem selectors.

To evaluate the influence of the decay factor $\alpha$ in NSUSP and NSUTSP, it is reasonable to measure the improvement in optimization results when the non-stationary mechanism is applied to UCBSP and UCBTSP. Accordingly, for each benchmark function, we compared the average fitness values of the final optimal solutions achieved by NSUSPCC and NSUTSPCC with those of UCBSPCC and UCBTSPCC in a pairwise manner using the Wilcoxon rank-sum ANOVA tests. To this end, we implemented 10 NSUSPCC and NSUTSPCC models by setting their decay factors to the five values: 0.1, 0.3, 0.5, 0.7, and 0.9. Afterward, we counted the benchmark functions for which NSUSPCC and NSUTSPCC models achieved better, worse, or equivalent results compared to UCBSPCC and UCBTSPCC, respectively, when the decay factor $\alpha$ was set from $\{0.1,0.3,0.5,0.7,0.9\}$.

Table 4. The comparison results for the optimization results performed by NSUTSPCC with $\alpha \in \{0.1,0.3,0.5,0.7,0.9\}$ and UCBTSPCC. All detailed experimental results are shown in Tables S1 and S2 in the Supplementary File.

Benchmark Suite	CEC’2010 (17 Benchmark Functions)					CEC’2013 (11 Benchmark Functions)
Decay factor $\mathsf{\alpha }$ of NSUTSP	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9
Improved benchmark functions	10	10	9	4	3	7	7	8	6	2
Equivalent benchmark functions	2	3	3	9	12	2	2	1	3	7
Worse benchmark functions	5	4	5	4	2	2	2	2	2	2

presents the comparison results between the optimization results obtained by NSUTSPCC with each of the five decay factor values and those of UCBTSPCC for the CEC’2010 and 2013 benchmark functions. In Table 4, the row labeled “Improved benchmark functions” indicates the number of benchmark functions for which NSUTSPCC achieved better optimization results than those of UCBTSPCC. On the other hand, the row labeled “Worse benchmark functions” describes the number of benchmark functions for which NSUTSPCC made worse results than those of UCBTSPCC. If NSUTSPCC and UCBTSPCC show statistically equivalent results, the number of its benchmark functions is written in the row labeled “Equivalent benchmark functions.”

As shown in Table 4, we found that NSUTSPCC achieved the best optimization results compared to those of UCBSPCC when $\alpha$ = 0.3. In detail, NSUTSPCC with $\alpha$ = 0.3 yielded better optimization results for 10 CEC’2010 benchmark functions out of a total of 17. Similar to the results of NSUTSPCC with $\alpha$ = 0.3, NSUTSPCC with $\alpha$ = 0.1 also showed better results than those of UCBTSPCC for 10 CEC’2010 benchmark functions. However, the number of benchmark functions for which NSUTSPCC with $\alpha$ = 0.3 showed worse results was less than that of NSUTSPCC with $\alpha$ = 0.1. In the comparison tests with the CEC’2013 benchmark functions, NSUTSPCC showed the best optimization results when $\alpha$ = 0.5. In detail, NSUTSPCC achieved better results for eight benchmark functions, and equivalent ones for one function when compared to the results of UCBTSPCC. NSUTSPCC with $\alpha$ = 0.3 yielded better results than those of UCBTSPCC for seven benchmark functions. The number of benchmark functions for which NSUTSPCC showed equivalent results to those of UCBSPCC was two. When $\alpha$ = 0.1, the results were equivalent to those when $\alpha$ = 0.3.

Meanwhile, the comparison results between UCBSPCC and NSUSPCC with five decay factor values are described in

Table 5. The comparison results for the optimization results performed by NSUSPCC with $\alpha \in \{0.1,0.3,0.5,0.7,0.9\}$ and UCBSPCC. All detailed experimental results are presented in Tables S3 and S4 in the Supplementary File.

Benchmark Suite	CEC’2010 (17 Benchmark Functions)					CEC’2013 (11 Benchmark Functions)
Decay factor $\mathsf{\alpha }$ of NSUSP	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9
Improved benchmark functions	10	9	10	5	1	7	6	7	5	0
Equivalent benchmark functions	2	3	3	10	15	1	2	2	4	10
Worse benchmark functions	5	5	4	2	1	3	3	2	2	1

. Unlike the results presented in Table 4, NSUSPCC yielded the best optimization results when $\alpha$ = 0.5 for the CEC’2010 and 2013 benchmark functions. In detail, NSUSPCC with $\alpha$ = 0.5 achieved better optimization results than those of UCBSPCC for 10 CEC’2010 benchmark functions. Similarly, when $\alpha$ = 0.5, NSUSPCC outperformed UCBSPCC in seven out of the 13 CEC’2013 benchmark functions. Meanwhile, NSUSPCC with $\alpha$ = 0.1 showed the second-best results for both the CEC’2010 and 2013 benchmark functions.

Finally, Figure 3 summarizes the comprehensive results of Table 4 and Table 5. In detail, Figure 3a describes that NSUTSPCC achieved better results than those of UCBTSPCC for 17 benchmark functions when $\alpha$ was set to 0.1, 0.3, or 0.5, respectively. Among these results, we found that NSUTSPCC with $\alpha$ = 0.3 produced equivalent results to those of UCBTSPCC for five benchmark functions. Figure 3b shows that NSUSPCC achieved better results than UCBSPCC for 17 benchmark functions when $\alpha$ was set to 0.1 and 0.5. However, NSUSPCC with $\alpha$ = 0.5 yielded equivalent results to those of UCBSPCC for five benchmark functions, which were superior to those of NSUSPCC with $\alpha$ = 0.1. Meanwhile, both NSUTSPCC and NSUSPCC showed significantly lower performance improvements compared to those of UCBTSPCC and UCBSPCC when $\alpha$ was set to 0.7 or 0.9, respectively. These results are evident because the weights of the reward scores approach one, which is equivalent to the traditional average reward shown in Equations (5) and (6). Thus, as the decay factor $\alpha$ converges to one, the results of NSUTSPCC and NSUSPCC become close to those of UCBTSPCC and UCBSPCC, respectively.

According to the results of these ablation studies, we determined the optimal values of the decay factor $\alpha$ in NSUTSP and NSUSP as 0.3 and 0.5, respectively.

5.2.2. Ablation Studies for the Population Size m and Smoothing Factor $\gamma$

To analyze the influence of other parameters used in the base optimizer and reward evaluation function (REF), we conducted several additional parameter sensitivity tests using the NSUTSPCC. To this end, we adopted two significant control parameters, m and $\gamma$. The parameter m represents the number of individuals used in the base optimizer, i.e., the SaNSDE algorithm. That is, m determines the size of the population utilized to search for an optimal solution of the given objective function. The parameter $\gamma$ is a smoothing factor used to prevent the denominator from becoming zero in Equation (3). To maintain a consistent experimental environment, we typically set the decay factor $\alpha$ of the NSUTSPCC to 0.3 in both tests for m and $\gamma$.

Table 6. The mean and median of the fitness values achieved by NSUTSPCC after solving the CEC’2010 and CEC’2013 benchmark functions according to the settings of the population size m. The detailed results are provided in Tables S5 and S6 of the Supplementary File.

Benchmarks	Measures	$\mathit{m}=50$	$\mathit{m}=75$	$\mathit{m}=100$	$\mathit{m}=125$	$\mathit{m}=150$
CEC’2010	Mean	9.11 × 108	1.07 × 109	2.84 × 109	2.82 × 109	4.76 × 109
	Median	3.37 × 102	4.20 × 102	5.22 × 102	5.89 × 102	1.08 × 103
CEC’2013	Mean	4.81 × 109	5.08 × 109	1.60 × 1010	1.85 × 109	8.58 × 108
	Median	2.37 × 107	3.91 × 107	3.72 × 107	3.29 × 107	2.63 × 107

presents the means and medians of the average fitness values achieved by the NSUTSPCC for each of the CEC’2010 and 2013 benchmark functions, with m set to 50, 75, 100, 125, and 150, respectively. As shown in Table 6, the NSUTSPCC yielded the best results on average, in terms of both the mean and median, when m was set to 50. Similarly, when m was set to 75, the NSUTSPCC still showed relatively better results compared to larger population sizes, such as 125 and 150. These experimental results demonstrate that the population size, i.e., the number of individuals, affects the search ability of the base optimizer in the CC framework to find optimal solutions.

In fact, the number of optimal individuals can be differentiated considerably depending on the characteristics of the base optimizers and the objective functions being tackled. If the number of individuals is too large, the search performance may actually deteriorate due to severe mutual interference while the base optimizer searches for optimal solutions. On the other hand, if the number of individuals is too small, there is a risk that the diversity of the optimal search may decrease, potentially leading to a decline in search performance. Thus, it is required to carefully determine the population size by fully considering the features of the base optimizer algorithm and the LSGO problems within CC frameworks.

Meanwhile,

Table 7. The mean and median of the fitness values achieved by NSUTSPCC after solving the CEC’2010 and CEC’2013 benchmark functions according to the settings of the smoothing factor $\gamma$. The detailed results are provided in Tables S7 and S8 of the Supplementary File.

Benchmarks	Measures	$\mathit{\gamma }={10}^{-2}$	$\mathit{\gamma }={10}^{-4}$	$\mathit{\gamma }={10}^{-6}$	$\mathit{\gamma }={10}^{-8}$	$\mathit{\gamma }={10}^{-10}$
CEC’2010	Mean	1.40 × 109	1.33 × 109	1.21 × 109	2.84 × 109	1.31 × 109
	Median	4.85 × 102	5.04 × 102	4.79 × 102	5.22 × 102	4.90 × 102
CEC’2013	Mean	2.44 × 109	4.17 × 109	3.71 × 109	1.60 × 1010	9.64 × 109
	Median	4.08 × 107	4.35 × 107	4.87 × 107	3.72 × 107	3.13 × 107

presents the mean and median of the average fitness values evaluated by the NSUTSPCC for the CEC’2010 and 2013 benchmark functions when the smoothing factor $\gamma$ was set to five distinct values: ${10}^{-2}$, ${10}^{-4}$, ${10}^{-6}$, ${10}^{-8}$, and ${10}^{-10}$. As explained previously, $\gamma$ plays a role in preventing the denominator of Equation (3) from becoming zero. Accordingly, it scales the reward scores within a range of zero to one.

From the ablation study results shown in Table 7, we found that the overall optimization results were similarly evaluated in the five experiments. In particular, the median values were almost identical across the five distinct settings of $\gamma$ for both the CEC’2010 and 2013 benchmark sets. These results indicate that if the smoothing factor $\gamma$ in Equation (3) is set to sufficiently small values close to zero, it does not significantly affect the evaluation of the reward scores of the subproblems in the REF. Nevertheless, as discussed in Section 4.5.2, the smoothing factor $\gamma$ can influence the evaluation of the reward scores of the subproblems if it is set to large values. Therefore, gamma must be set to minimal values close to zero to prevent unexpected effects caused by $\gamma$ when computing the reward scores in the REF.

5.3. Optimization Test Results with Wilcoxon Rank-Sum ANOVA Tests

5.3.1. Optimization Test Results for the CEC’2010 Benchmark Functions

To evaluate how much our proposed subproblem selectors can contribute to improving the optimization performance in practical CC frameworks, we performed the optimization tests for the CC frameworks with nine subproblem selectors for the CEC’2010 and CEC’2013 benchmark functions.

Table 8. The Wilcoxon rank-sum one-way ANOVA test results performed to compare the optimization results performed by NSUTSPCC and the CC frameworks with other subproblem selectors to solve the CEC’2010 benchmark functions.

Func.	Measures	NSUTSPCC	NSUSPCC	UCBSPCC	UCBTSPCC	BasicCC	RandomCC	BBCC	CBCC1	CBCC2
$f_1$	Mean	9.88 × 10−3	1.27 × 10−4	1.53 × 10−5	2.22 × 10−5	3.36 × 10−2	7.21 × 106	1.66 × 107	3.39 × 10−2	3.43 × 10−2
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	L	L	L	W	W	W	W	W
$f_2$	Mean	2.85 × 102	1.39 × 102	1.17 × 102	1.19 × 102	1.18 × 102	6.76 × 102	5.05 × 102	1.57 × 102	3.17 × 102
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	3.44 × 10−4
	W/T/L	-	L	L	L	L	W	W	L	W
$f_4$	Mean	4.81 × 1010	3.06 × 1011	3.45 × 1012	1.59 × 1012	1.01 × 1013	1.06 × 1013	1.04 × 1010	3.86 × 1012	7.51 × 109
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	L	W	L
$f_5$	Mean	1.82 × 108	3.11 × 108	3.57 × 108	3.18 × 108	3.96 × 108	4.14 × 108	1.08 × 108	3.13 × 108	1.79 × 108
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	6.62 × 10−1
	W/T/L	-	W	W	W	W	W	L	W	T
$f_6$	Mean	1.00 × 101	1.16 × 101	1.44 × 101	1.39 × 101	1.53 × 101	1.53 × 101	1.79 × 101	1.55 × 101	1.25 × 101
	p-value	-	3.53 × 10−2	1.23 × 10−5	4.74 × 10−5	1.21 × 10−7	1.21 × 10−7	1.07 × 10−8	2.11 × 10−7	1.88 × 10−3
	W/T/L	-	W	W	W	W	W	W	W	W
$f_7$	Mean	1.35 × 10−4	3.57 × 10−5	3.63 × 107	1.60 × 10−5	1.99 × 109	8.29 × 109	4.95 × 103	1.70 × 108	5.30 × 10−3
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	L	W	L	W	W	W	W	W
$f_8$	Mean	6.53 × 105	3.95 × 107	7.02 × 107	3.90 × 107	2.53 × 108	1.41 × 1011	4.84 × 105	7.06 × 107	1.59 × 105
	p-value	-	3.45 × 10−5	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	8.53 × 10−5	1.07 × 10−8	2.78 × 10−5
	W/T/L	-	W	W	W	W	W	L	W	L
$f_9$	Mean	9.18 × 106	1.92 × 107	3.11 × 107	2.54 × 107	3.77 × 107	5.81 × 107	1.05 × 107	3.55 × 107	1.89 × 109
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	2.72 × 10−3	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	W	W	W
$f_{10}$	Mean	3.51 × 103	3.66 × 103	3.61 × 103	3.69 × 103	4.20 × 103	4.56 × 103	3.40 × 103	4.10 × 103	3.99 × 103
	p-value	-	5.95 × 10−4	8.42 × 10−3	5.95 × 10−4	1.07 × 10−8	1.07 × 10−8	5.60 × 10−2	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	T	W	W
$f_{11}$	Mean	1.02 × 101	1.03 × 101	1.18 × 101	1.16 × 101	1.17 × 101	1.17 × 101	1.13 × 101	1.17 × 101	1.23 × 101
	p-value	-	8.08 × 10−1	1.38 × 10−5	1.44 × 10−4	2.83 × 10−5	5.52 × 10−6	2.28 × 10−1	2.03 × 10−5	1.36 × 10−6
	W/T/L	-	T	W	W	W	W	T	W	W
$f_{12}$	Mean	1.54 × 100	2.24 × 101	2.03 × 103	5.64 × 102	5.30 × 103	2.23 × 104	4.78 × 103	4.47 × 103	1.61 × 104
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	W	W	W
$f_{13}$	Mean	5.22 × 102	6.75 × 102	8.92 × 102	7.51 × 102	1.30 × 103	1.13 × 107	3.79 × 103	1.21 × 103	2.08 × 103
	p-value	-	3.37 × 10−6	2.95 × 10−8	5.17 × 10−7	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	2.21 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	W	W	W
$f_{14}$	Mean	2.96 × 107	2.90 × 107	3.00 × 107	3.07 × 107	3.03 × 107	3.70 × 107	4.84 × 107	3.27 × 107	7.36 × 109
	p-value	-	1.00 × 100	1.00 × 100	7.88 × 10−1	1.00 × 100	7.45 × 10−7	2.12 × 10−7	3.55 × 10−3	1.07 × 10−8
	W/T/L	-	T	T	T	T	W	W	W	W
$f_{15}$	Mean	5.41 × 103	5.31 × 103	5.25 × 103	5.40 × 103	5.35 × 103	5.93 × 103	5.46 × 103	5.40 × 103	5.91 × 103
	p-value	-	5.69 × 10−1	6.44 × 10−2	1.00 × 100	1.00 × 100	2.69 × 10−5	1.00 × 100	1.00 × 100	8.49 × 10−5
	W/T/L	-	T	T	T	T	W	T	T	W
$f_{16}$	Mean	7.62 × 10−2	1.58 × 10−1	1.17 × 10−1	3.44 × 10−1	4.05 × 10−1	2.69 × 10−1	8.40 × 10−1	3.04 × 10−1	4.51 × 10−1
	p-value	-	6.11 × 10−2	1.51 × 10−6	2.34 × 10−2	6.11 × 10−2	8.69 × 10−7	1.02 × 10−6	6.11 × 10−2	5.22 × 10−7
	W/T/L	-	T	W	W	T	W	W	T	W
$f_{17}$	Mean	1.93 × 102	1.61 × 102	1.32 × 102	1.31 × 102	1.32 × 102	9.25 × 102	1.95 × 104	1.96 × 102	1.56 × 103
	p-value	-	8.42 × 10−3	2.61 × 10−6	1.29 × 10−6	1.29 × 10−6	1.07 × 10−8	1.07 × 10−8	3.99 × 10−1	1.07 × 10−8
	W/T/L	-	L	L	L	L	W	W	T	W
$f_{18}$	Mean	1.08 × 103	1.07 × 103	1.17 × 103	1.12 × 103	1.20 × 103	1.22 × 103	3.41 × 103	1.13 × 103	1.42 × 103
	p-value	-	8.39 × 10−1	4.29 × 10−2	4.79 × 10−1	1.28 × 10−2	5.26 × 10−3	1.94 × 10−8	4.79 × 10−1	7.81 × 10−7
	W/T/L	-	T	W	T	W	W	W	T	W
Total	Win (W)	-	8	12	10	12	17	11	12	14
	Tie (T)	-	5	2	3	3	0	3	4	1
	Lose (L)	-	4	3	4	2	0	3	1	2

describes the optimization test results of the CC frameworks when solving the CEC’2010 benchmark functions. Moreover, we compared the average fitness values evaluated by NSUTSPCC and other CC frameworks in a pairwise manner by conducting Wilcoxon rank-sum one-way ANOVA tests. These comparison results are shown in Table 8 as “W/T/L.” If the p-value is less than 0.05, the average fitness values evaluated by NSUTSPCC and the other CC framework are significantly different. In this case, if the average fitness value achieved by NSUTSPCC is better than the other, we determine its result as “W (win)”; conversely, if its value is worse than the other, we determine the result as “L (lose).” Meanwhile, if the p-value is greater than or equal to 0.05, we determine that the two results are statistically equivalent, i.e., “T (tie),” because they do not have significant differences. By performing these pairwise comparisons for all the benchmark functions, we evaluated the number of benchmark functions for which NSUTSPCC achieved better results than other CC frameworks.

From the experimental results in Table 8, we found that NSUTSPCC attained the “win” and “tie” results for eight and five benchmark functions, respectively, compared to those of NSUSPCC. Moreover, NSUTSPCC also achieved remarkable optimization results compared to UCBSPCC and UCBTSPCC. In detail, NSUTSPCC showed better results than UCBSPCC and UCBTSPCC by achieving the “win” for 12 and 10 benchmark functions, respectively. These experimental results indicate that our proposed NSUTSP contributes most to finding optimal solutions of objective functions with complicated interdependencies in practical CC frameworks among the four proposed UCB-based subproblem selectors.

In comparisons to the traditional CC frameworks, our NSUTSPCC also showed the most outperformed optimization results, i.e., achieved the “win” for most benchmark functions. On the other hand, the CC frameworks with traditional subproblem selectors exhibited relatively worse optimization results than those of NSUTSPCC. In detail, when compared to the results of BasicCC and RandomCC, NSUTSPCC showed superior results, i.e., the “win” for most benchmark functions, i.e., 12 and 17 benchmark functions, respectively. These results indicate that the subproblem selection task in the CC framework considerably influences the optimization performance when solving practical LSGO problems. Meanwhile, BBCC, which uses the $\epsilon$-greedy strategy for the subproblem selection, presented better optimization results for only three benchmark functions. Similarly, CBCC1 and CBCC2 only achieved better results than NSUTSPCC for two and three benchmark functions, respectively. These comparison results show that our UCB-based subproblem selection strategies significantly help the CC frameworks solve the large-dimensional optimization problem by carefully identifying promising subproblems that have the most significant influence on searching for optimal solutions in the large-dimensional solution space.

5.3.2. Optimization Test Results for the CEC’2013 Benchmark Functions

Table 9. The Wilcoxon rank-sum one-way ANOVA test results performed to compare the optimization results performed by NSUTSPCC and the CC frameworks with other subproblem selectors to solve the CEC’2013 benchmark functions.

Func.	Measures	NSUTSPCC	NSUSPCC	UCBSPCC	UCBTSPCC	BasicCC	RandomCC	BBCC	CBCC1	CBCC2
$f_1$	Mean	1.05 × 10−2	1.67 × 10−4	1.94 × 10−5	3.36 × 10−5	4.37 × 10−2	1.72 × 108	1.91 × 107	3.94 × 10−2	4.28 × 10−2
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	L	L	L	W	W	W	W	W
$f_2$	Mean	3.50 × 102	2.05 × 102	1.95 × 102	1.88 × 102	1.97 × 102	7.96 × 102	2.48 × 103	2.25 × 102	3.78 × 102
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.11 × 10−5
	W/T/L	-	L	L	L	L	W	W	L	W
$f_4$	Mean	5.24 × 107	2.75 × 108	1.88 × 109	7.53 × 108	4.53 × 109	1.80 × 1010	1.59 × 109	2.36 × 109	1.45 × 1010
	p-value	-	2.77 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	2.66 × 10−5	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	W	W	W
$f_5$	Mean	4.58 × 106	6.95 × 106	7.77 × 106	6.82 × 106	9.30 × 106	9.81 × 106	2.92 × 106	7.56 × 106	5.00 × 106
	p-value	-	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	2.77 × 10−8	1.07 × 10−8	3.72 × 10−3
	W/T/L	-	W	W	W	W	W	L	W	W
$f_6$	Mean	1.06 × 106	1.06 × 106	1.06 × 106	1.06 × 106	1.06 × 106	1.06 × 106	1.05 × 106	1.06 × 106	1.06 × 106
	p-value	-	1.34 × 10−1	9.20 × 10−2	6.62 × 10−1	4.01 × 10−1	9.20 × 10−2	1.20 × 10−8	2.53 × 10−2	3.76 × 10−5
	W/T/L	-	T	T	T	T	T	L	L	L
$f_7$	Mean	3.15 × 104	6.20 × 105	8.71 × 107	1.90 × 107	1.21 × 108	2.13 × 108	1.52 × 108	1.19 × 108	2.51 × 108
	p-value	-	4.86 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	1.07 × 10−8	2.10 × 10−7	1.07 × 10−8	1.07 × 10−8
	W/T/L	-	W	W	W	W	W	W	W	W
$f_8$	Mean	1.74 × 1011	6.85 × 1011	4.97 × 1013	2.04 × 1013	1.32 × 1014	1.10 × 1014	2.88 × 1011	3.50 × 1013	7.38 × 1011
	p-value	-	1.52 × 10−3	1.07 × 10−8	1.09 × 10−8	1.07 × 10−8	1.07 × 10−8	1.28 × 10−1	1.07 × 10−8	6.99 × 10−7
	W/T/L	-	W	W	W	W	W	T	W	W
$f_9$	Mean	1.42 × 108	1.73 × 108	2.74 × 108	2.35 × 108	3.22 × 108	3.53 × 108	1.60 × 108	2.28 × 108	1.93 × 108
	p-value	-	3.69 × 10−3	2.90 × 10−4	5.08 × 10−5	3.12 × 10−5	5.22 × 10−7	5.35 × 10−2	2.90 × 10−4	1.45 × 10−4
	W/T/L	-	W	W	W	W	W	T	W	W
$f_{10}$	Mean	9.42 × 107	9.45 × 107	9.44 × 107	9.44 × 107	9.45 × 107	9.46 × 107	9.32 × 107	9.43 × 107	9.43 × 107
	p-value	-	1.45 × 10−1	6.32 × 10−1	4.04 × 10−1	1.10 × 10−1	8.65 × 10−3	7.02 × 10−8	1.00 × 100	1.00 × 100
	W/T/L	-	T	T	T	T	W	L	T	T
$f_{11}$	Mean	1.79 × 109	4.12 × 109	1.94 × 1010	1.44 × 1010	2.11 × 1010	4.72 × 1010	3.18 × 108	1.52 × 1010	9.57 × 109
	p-value	-	3.36 × 10−2	5.67 × 10−7	5.67 × 10−7	5.67 × 10−7	1.55 × 10−7	1.43 × 10−6	5.67 × 10−7	5.67 × 10−7
	W/T/L	-	W	W	W	W	W	L	W	W
$f_{13}$	Mean	3.72 × 107	2.90 × 107	1.08 × 108	5.00 × 108	4.14 × 108	1.23 × 108	5.37 × 107	6.16 × 107	3.62 × 1011
	p-value	-	1.38 × 10−1	2.92 × 10−6	2.92 × 10−6	1.36 × 10−7	2.03 × 10−7	5.00 × 10−2	3.81 × 10−4	1.07 × 10−8
	W/T/L	-	T	W	W	W	W	T	W	W
Total	Win (W)	-	6	7	7	8	10	4	8	9
	Tie (T)	-	3	2	2	2	1	3	1	1
	Lose (L)	-	2	2	2	1	0	4	2	1

lists the average fitness values achieved by the CC frameworks with nine subproblem selectors when solving the CEC’2013 benchmark functions and their comparison results. As described in Table 9, NSUTSPCC still showed the best optimization results among the nine CC frameworks, even though the base benchmark functions were changed from the CEC’2010 to the CEC’2013 ones. In detail, NSUTSPCC achieved the “win” for seven benchmark functions compared to the results of both UCBSPCC and UCBTSPCC. Moreover, NSUTSPCC attained better or equivalent results for nine benchmark functions compared to those of NSUSPCC. These results indicate that our non-stationary mechanisms are more effective in identifying a promising subproblem that can contribute to further enhancing optimization performance in the CC frameworks, regardless of the base optimizers.

Moreover, NSUTSPCC also showed the best optimization results for most benchmark functions when compared to the traditional CC frameworks. In detail, NSUTSPCC showed better results, i.e., the “win” for an average of 7.8 benchmark functions when compared to five traditional CC methods. On the other hand, the average number of benchmark functions for which the traditional CC frameworks achieved better results than NSUTSPCC was only 1.6. Exceptionally, BBCC, utilizing $\epsilon$-greedy, showed better optimization results than other traditional CC frameworks. Nevertheless, its results still were lower than those of our NSUTSPCC.

These results indicate that our subproblem selectors can significantly contribute to solving any complicated LSGO problems, such as the CEC’2013 benchmark functions, in the CC frameworks. Thus, we found that the UCB, particularly the non-stationary UCB mechanisms, could significantly contribute to identifying a promising subproblem in practical CC frameworks, as demonstrated by the experimental results.

5.3.3. Total Result Analysis and Discussion

Finally,

Table 10. The total comparison results between the CC frameworks using the proposed four UCB-based subproblem selectors and the CC frameworks with traditional subproblem selection mechanisms when solving the CEC’2010 benchmark functions.

CCs with Proposed Methods	Results	CC Frameworks with Other Subproblem Selection Methods					Total	Average
		BasicCC	RandomCC	BBCC	CBCC1	CBCC2
NSUTSPCC	Win	12	17	11	12	14	66	13.2
	Tie	3	0	3	4	1	11	2.2
	Lose	2	0	3	1	2	8	1.6
NSUSPCC	Win	13	17	10	13	13	66	13.2
	Tie	2	0	3	4	1	10	2
	Lose	2	0	4	0	3	9	1.8
UCBTSPCC	Win	9	13	10	10	12	54	10.8
	Tie	8	3	1	6	2	20	4
	Lose	0	1	6	1	3	11	2.2
UCBSPCC	Win	10	14	9	10	11	54	10.8
	Tie	7	3	1	6	1	18	3.6
	Lose	0	0	7	1	5	13	2.6

and

Table 11. The total comparison results between the CC frameworks using the proposed four UCB-based subproblem selectors and the CC frameworks with traditional subproblem selection mechanisms when solving the CEC’2013 benchmark functions.

CCs with Proposed Methods	Results	CC frameworks with Other Subproblem Selection Methods					Total	Average
		BasicCC	RandomCC	BBCC	CBCC1	CBCC2
NSUTSPCC	Win	8	10	4	8	9	39	7.8
	Tie	2	1	3	1	1	8	1.6
	Lose	1	0	4	2	1	8	1.6
NSUSPCC	Win	8	9	3	8	6	34	6.8
	Tie	2	2	4	3	3	14	2.8
	Lose	1	0	4	0	2	7	1.4
UCBTSPCC	Win	8	8	3	7	5	31	6.2
	Tie	2	2	1	2	2	9	1.8
	Lose	1	1	7	2	4	15	3
UCBSPCC	Win	6	9	3	3	5	26	5.2
	Tie	5	2	1	6	1	15	3
	Lose	0	0	7	2	5	14	2.8

present the total and average numbers of benchmark functions for which the CC frameworks with our four subproblem selectors outperformed, tied, and underperformed compared to the existing CC frameworks for the CEC’2010 and the CEC’2013 benchmark suites, respectively. (The detailed ANOVA test results of NSUSPCC, UCBTSPCC, and UCBSPCC are presented in Tables S9–S14 in the Supplementary File.). These tables list the summarized comparison results between the CC frameworks equipped with the proposed four UCB-based subproblem selectors and the existing five CC frameworks when the CEC’2010 and CEC’2013 benchmark functions were used as the target problems, respectively.

As described in Table 10 and Table 11, all the CC frameworks utilizing our proposed subproblem selectors outperformed the traditional ones, achieving the “win” for most benchmark functions. In detail, NSUTSPCC attained the “win” for 13.2 and 7.8 functions on average for the CEC’2010 and CEC’2013 benchmark suites, respectively. Similarly, the average number of functions for which NSUSPCC achieved the “win” was 13.2 for the CEC’2010 benchmark suite, and 6.8 for the CEC’2013 ones. Meanwhile, UCBTSPCC also achieved remarkable optimization results that were almost equivalent to those of NSUTSPCC and NSUSPCC when compared to traditional CC frameworks. The average numbers of functions for which UCBTSPCC attained the “win” were 10.8 and 6.2 for the CEC’2010 and CEC’2013 benchmark suites, respectively. UCBSPCC also demonstrated significant optimization results compared to traditional CC frameworks; on average, UCBSPCC outperformed traditional CC methods for 10.8 and 5.2 functions when the CEC’2010 and CEC’2013 benchmark suites were used, respectively.

Based on the experimental results in Table 10 and Table 11, we found that our CC frameworks, particularly NSUTSPCC and NSUSPCC, achieved the best optimization results among all the compared CC frameworks. Simultaneously, UCBSP and UCBTSP also could remarkably help the CC frameworks search for optimal solutions for large-dimensional black-box objective functions. Therefore, we can conclude that our proposed UCB-based subproblem selectors, especially non-stationary UCB-based methods, can significantly improve the optimization performance in practical CC frameworks.

5.4. Convergence Curve Analysis

Figure 4 and Figure 5 illustrate the convergence curves of nine CC frameworks for 12 selected CEC’2010 benchmark functions. We found that most of the CC frameworks with our subproblem selectors achieved more stable and faster convergence than others that use classical subproblem selection mechanisms. In detail, UCBSPCC and UCBTSPCC also achieved faster or nearly equivalent convergence than the other CC frameworks for $f_1$, $f_2$, and $f_{17}$. Meanwhile, NSUTSPCC and NSUSPCC demonstrated more stable and rapid convergence than the CC frameworks with traditional subproblem selection methods when solving $f_9$–$f_{18}$, which comprised more than half of the variables. For example, for $f_9$–$f_{13}$, which involve 500 separable and 500 non-separable variables, the four CC frameworks with our subproblem selectors resulted in more stable and faster convergence than those of the other CC frameworks. Likewise, the CC frameworks utilizing our UCB-based subproblem selectors showed the best convergence abilities for $f_{14}$–$f_{18}$, which comprise all non-separable variables and are decomposed into 20 subproblems. Meanwhile, the CC frameworks with classical subproblem selection mechanisms exhibited slower or worse convergence performance compared to our proposed ones. These results indicate that our four subproblem selectors have notable abilities as base subproblem selectors, contributing to improved optimization performance in practical CC frameworks.

Figure 6 describes the convergence curve plots of the CC frameworks with nine subproblem selectors for the six chosen CEC’2013 benchmark functions. Unlike the CEC’2010 functions, the CEC’2013 ones are decomposed into several imbalance-sized subproblems, making it hard to find their optimal solutions. Nevertheless, we found that the CC frameworks using our proposed subproblem selectors generated more stable and faster convergence curves than those of other CC frameworks for many benchmark functions. In detail, UCBTSPCC and UCBSPCC showed better or almost similar convergence results when compared to those of NSUTSPCC and NSUSPCC for the benchmark functions that were composed of 1000 separable variables, such as $f_1$ and $f_2$. Meanwhile, for the benchmark functions that constitute more than half of the variables, NSUTSPCC and NSUSPCC made better convergence curves than those of UCBSPCC and UCBTSPCC. For example, NSUTSPCC and NSUSPCC showed relatively better convergence performance than the others for $f_4$, $f_7$, $f_9$, and $f_{13}$. Notably, NSUTSPCC achieved superior fast convergence for $f_4$ and $f_7$, which were composed of 700 separable and 300 non-separable variables. For $f_9$, which involves 1000 non-separable variables decomposed into 20 subproblems, BBCC demonstrated notable convergence results. Nevertheless, our NSUTSPCC presented the better convergence curve than BBCC. Finally, for $f_{13}$, which is composed of 905 non-separable variables and decomposed into two subproblems by ERDG, our NSUTSPCC and NSUSPCC achieved the best convergence results.

From the analysis of the convergence curves, we discovered that our CC frameworks have significantly greater convergence abilities for most benchmark functions, especially for the functions with many non-separable variables, when compared to other existing CC frameworks. These experimental results indicate that our proposed UCB-based subproblem selectors can contribute significantly to improving the optimization performance when solving objective functions with complicated variable interdependencies. In other words, our strategies using the UCB and its variation algorithms to identify promising subproblems can significantly help the CC frameworks improve convergence performance by carefully controlling exploration and exploitation in the solution search process.

5.5. Discussions

5.5.1. Discussion About the Experimental Results

In the experiments, the CC frameworks with our proposed subproblem selectors showed overall better optimization results across most benchmark functions. In particular, they achieved outperformed results for the benchmark functions in which more than half of the variables are non-separable, i.e., $f_9$–$f_{18}$ (CEC’2010) and $f_8$–$f_{13}$ (CEC’2013). These experimental results can be explained in terms of the characteristics of the benchmark functions and our UCB-based subproblem selectors. In general, non-separable variables are strongly coupled based on their interdependencies. Accordingly, the non-separable variable groups have a significant influence on finding the optimal solutions of the objective function. On the other hand, the separable variable groups have relatively weaker influences than the non-separable ones because there are no interdependencies among the variables in the separable variable group. As explained, the CC frameworks with our subproblem selectors achieved further outperformed results for $f_9$–$f_{18}$ (CEC’2010) and $f_8$–$f_{13}$ (CEC’2013). Because these benchmark functions are composed of more than half non-separable variable groups, each of them has a strong effect on searching for the optimal solution. In this case, it is essential to carefully control the trade-off between exploration and exploitation to choose subproblems. Thus, the CC frameworks with our subproblem selectors, which utilize the UCB algorithms—especially the non-stationary UCB and UCB-tuned algorithms—can exhibit better solution search ability for these benchmark functions than other CC frameworks with traditional subproblem selection methods.

Meanwhile, the NSUTSPCC and NSUSPCC showed relatively less overwhelming optimization performance on benchmark functions composed of many separable variables, i.e., $f_4$–$f_8$ (CEC’2010) and $f_4$–$f_7$ (CEC’2013), compared to their performance on functions composed of many non-separable variables. As presented in Table 2, they consist of a few non-separable variable groups and many separable variable groups. In this case, by intensively selecting the subproblems with the non-separable variable group, its ability to search for optimal solutions can be further enhanced because the non-separable variable group has more decisive influence on finding the optimal solution of the function than the separable variable groups. That is, the existing subproblem selection strategies that use a fixed ratio of exploration and exploitation-based selections, such as BBCC or CBCC, can achieve slightly better results than the CC frameworks with other subproblem selectors. On the other hand, our subproblem selectors, which adaptively control the ratio of exploration and exploitation-based subproblem selections, can yield relatively less satisfactory solution search results for several functions that require intensive exploitation-based subproblem selection. Nevertheless, when considering it is more difficult to address the LSGO function with many complicated non-separable variables, our proposed methods that exhibit further strong solution search ability for such complicated functions.

In summary, these experimental results are attributed to the features of our UCB-based subproblem selectors, which aim to prevent excessive exploitation in subproblem selection and maintain the exploration-exploitation trade-off. Therefore, our proposed subproblem selectors, especially NSUTSP and NSUSP, have strengths that are even more specialized in solving any complicated functions with many non-separable variables.

5.5.2. Utilization Method of the CC Framework with Proposed Subproblem Selectors to Address Actual Engineering Problems

In general, the CC framework addresses high-dimensional, complex functions composed of many dependent and independent variables with numerous non-separable variable groups. Then, many real-world problems, which consist of numerous separable or dependent elements, can be effectively solved using the CC framework. For example, elements with strong interactions in any system can be modeled as mutually dependent decision variables and grouped. To implement these strong interdependencies between variables, a rotation operation using matrix multiplication can be applied. Accordingly, highly interdependent variables can be formulated as a non-separable group of variables. Conversely, elements with weak interactions can be modeled as independently separable variables. This can be easily implemented by summing their values. Finally, the objective function is implemented by integrating these into a single function, which can be utilized as an optimization model for designing systems with numerous complex interdependencies. That is, we can find the optimal values for the elements (or factors) constituting the system by searching for optimal solutions that minimize the fitness of the function using the CC framework.

Figure 7 illustrates an example method for modeling various factors used in a battery manufacturing system as an objective function. In this example, we assume that there are 1000 factors that can cause negative effects in the battery manufacturing process, and they are mutually dependent or independent. Among the factors, 500 ones—including temperature, pressure, process time, and gas flow rate—should be simultaneously maintained within 10 independent modules. That is, since these factors have strong interdependencies, they can be modeled as 10 non-separable variable groups, with each group involving 50 interdependent variables. To made the interdependencies of the variables involved in a group, 50 variables related to each factor are combined by multiplying a rotation matrix M with the variables, as shown in Figure 7b. Through this method, we can model 10 non-separable groups, where one group contains 50 interdependent variables. On the other hand, the remaining 500 factors for sensor bias and fine calibration shown in Figure 7a have relatively weak interdependencies. These factors are modeled as separable variables by taking their weighted sum with shift operations, as shown in Figure 7c. Accordingly, we can design a complete objective function for this system by combining the separable variable term (a) and the non-separable variable group terms (b) into one function $F\left(\mathbf{x}\right)$. Consequently, we can efficiently find the optimal values for 1000 factors affecting this system by searching for the global optimum of this objective function using the CC frameworks with our proposed subproblem selectors.

The example illustrated in Figure 7 shows that various real-world engineering and scientific problems can be modeled as objective functions and solved using the CC frameworks with our proposed methods. At this point, we can apply our subproblem selector to the CC framework to effectively explore the global optimum of LSGO, which involves numerous variables with complex interdependencies. This enables us to effectively explore the global optimum of the given objective function while maintaining a sophisticated balance between exploration and exploitation.

6. Conclusions

In this paper, we propose four new UCB-based subproblem selectors for the CC frameworks—UCBSP, UCBTSP, NSUSP, and NSUTSP—to accurately identify subproblems that significantly contribute to finding optimal solutions of the objective function while maintaining a trade-off between exploration and exploitation. Our proposed subproblem selectors utilize the UCB algorithms and non-stationary mechanisms to measure the contribution of each subproblem in the CC frameworks, while simultaneously considering the dynamic characteristics of the evolutionary algorithms. In the experiments, the CC frameworks with our proposed subproblem selectors yielded better optimization results than those with classical subproblem selection methods when solving most benchmark functions. These experimental results indicate that utilizing our proposed strategies to select the subproblems in the CC frameworks can significantly contribute to finding optimal solutions to any LSGO problems with complex interdependencies.

Nevertheless, our study still has several limitations. We conducted the performance evaluations using the limited evolutionary algorithm and problem decomposer, such as the SaNSDE optimizer and ERDG algorithm, respectively. Accordingly, in future studies, we will conduct more in-depth experiments using various base evolutionary algorithms, such as the DE and PSO algorithms, as well as other problem decomposers, for example, DG2 [11] and EVIID [37]. Furthermore, we plan to investigate various performance evaluation methods [42] to enhance the practicality of this study and conduct diverse experiments based on these findings.