Clustering-Based Binarization Methods Applied to the Crow Search Algorithm for 0/1 Combinatorial Problems

Abstract

Metaheuristics are smart problem solvers devoted to tackling particularly large optimization problems. During the last 20 years, they have largely been used to solve different problems from the academic as well as from the real-world. However, most of them have originally been designed for operating over real domain variables, being necessary to tailor its internal core, for instance, to be effective in a binary space of solutions. Various works have demonstrated that this internal modification, known as binarization, is not a simple task, since the several existing binarization ways may lead to very different results. This of course forces the user to implement and analyze a large list of binarization schemas for reaching good results. In this paper, we explore two efficient clustering methods, namely KMeans and DBscan to alter a metaheuristic in order to improve it, and thus do not require on the knowledge of an expert user for identifying which binarization strategy works better during the run. Both techniques have widely been applied to solve clustering problems, allowing us to exploit useful information gathered during the search to efficiently control and improve the binarization process. We integrate those techniques to a recent metaheuristic called Crow Search, and we conduct experiments where KMeans and DBscan are contrasted to 32 different binarization methods. The results show that the proposed approaches outperform most of the binarization strategies for a large list of well-known optimization instances.

1. Introduction

Optimization problems can be seen in different areas of the modern world, such as bridge reinforcement [1], load dispatch [2], location of emergency facilities [3], marketing [4], and social networks [5], among others. To solve them, we must identify and understand to which model the problem belongs. There are three types of models, the first has discrete variables, the second has continuous variables, and the third has both types of variables. Discrete optimization has widely been used to define a large number of problems belonging to the NP-hard class [6]. It is necessary to emphasize that the time to solve this type of problem increases exponentially according to its size. For this reason, we consider it totally prudent to solve these problems through metaheuristics [7,8], which deliver acceptable results in a limited period of time.

There are many metaheuristics and we can see a complete collection of all nature-inspired algorithms in [9] but most work on continuous space. Some researchers have been working in depth on developing binary versions that make these metaheuristics operate in binary spaces. A study of putting continuous metaheuristics to work in binary search spaces [10] clearly describes the two-step technique. These techniques require a large number of combinations and tuning of parameters to binarize and thus be able to obtain the best of all, which is a great waste of time.

Despite successful published work, binary techniques require offline analysis and an expert user in the field to adjust the metaheuristic to the problem domain in addition to working with the various available combinations of the two-step technique. Using gained expertise, our work is focused on two binarization clustering methods to smartly modify a set of real solutions to binary solutions by grouping decision variables, without the need for combinations as in the two-step technique, saving time and algorithm manipulation. The first one proposes to use KMeans [11,12] clustering algorithm, the most common unsupervised machine learning algorithm, to assign each variable in defined groups by the user. It is necessary to assign values of transition to each cluster in order to binarize each variable. The second one integrates DBscan [13,14] as a powerful clustering method for arraying values in a nonapriori determined number of clusters. Unlike KMeans, this technique creates the clusters by itself just knowing the number of variables required to be included in a cluster and the distance between each point in the data set. Furthermore, the transition value to binarize is calculated online and depends on each cluster. To test our approaches, we include these clustering methods in the Crow Search Algorithm (CSA) [15] which is a population-based metaheuristic inspired by the behavior of crows (easy implementation, it works very fast, and there are few parameter settings). In this method, each individual follows others trying to discover the places where they hide the food in order to steal it.

Based on [16,17], both works were used as gold standards, and we perform an experimental design led by quantitative methodology. We implement different criteria such as quality of solutions, robustness in terms of the instances, solving large-scale of well known binary optimization problems: Set Covering Problem (SCP) [18] and 0/1 Knapsack Problem (KP) [19]. These two problems are very popular benchmarks to address and try new strategies with because they have known results with which to compare.

The remainder of this manuscript is organized as follows: Section 2 presents the related work. Section 3 exposes binarization strategies. CSA is described in Section 4. Section 5 explains how to integrate binarizations on CSA. Next, experimental results including a research methodology are explained in Section 6. Discussions are shown in Section 7. At the end of the manuscript, conclusions are presented in Section 8.

2. Related Work

As previously mentioned, binarization is mandatory for continuous metaheuristics when the problem ranges in a binary domain. This task is not simple since experimentation with different binarization schemes is needed to reach good results. The transfer function is the most used binarization method and it was introduced in [20]. The transfer function is a very cheap operator, his range provides probabilities values and tries to model the transition of the particle positions. This function is responsible for the first step of the binarization method which corresponds to map the ${\mathbb{R}}^n$ solutions in ${[0,1]}^n$ solutions. Two types of functions have been used in the literature, the S-shaped [21] and V-shaped [22]. The second step is to apply a binarization rule to the result of the transfer function. Examples of binarization rules are a complement, roulette, static probability, and elitist [22]. “Two-step” binarization is employed to refer to the use of transfer functions and binarization rule together. A detailed description of binarization techniques can be seen in [10].

Our goal here is to alleviate the user involvement in binarization testing and to provide a unique alternative that works better on average when contrasted to a large set of “Two-step” binarization techniques. In this context, the use of machine learning techniques appears as a good candidate to support the binarization process and as a consequence to obtain better search processes. Machine learning has already been used to improve search in metaheuristics [23]. For instance, the first trend is to employ machine learning for parameter tuning, as in [24] the authors use machine learning to parameter control in metaheuristics. The techniques reported in this context can be organized in four groups: Parameter control strategies [25], parameter tuning strategies [26], instance-specific parameter tuning strategies [27], and reactive search [28]. The general idea is to exploit search information in order to find good parameter configurations; this may lead to better quality solutions. An example of this is in [29], where the author uses a metaheuristic to determine the optimal number of hidden nodes and their proper initial locations on neural networks. Machine learning has also been used to build approximation models when objective function and constraints are expensive computationally [30]. Population management can also be improved via machine learning algorithms, obtaining information from already visited solutions, and using it to build new ones [31].

Machine learning has also been used to reduce the search space, for instance by using clustering [32] and neural networks [33]. In these works, two main areas of machine learning were used: forecasting and classify. It is important to mention the uses of machine learning with metaheuristics in these areas. It has been used to forecast different kinds of problems, like optimizing the low-carbon flexible job Shop Scheduling problem [34], electric load [35,36] use to forecast economic recessions in Italy, and other industrial problems as in [37]. As we mentioned, machine learning also can be used to classify some different groups of data sets; in this scenario, this technique has also been used in different problems as the image classification [38], electronic noise classification of bacterial foodbones pathogens [39], urban management [40], to mention some recent studies.

Although the participation of machine learning in the metaheuristic field is large, the specific use in binarization is indeed very limited. To the best of our knowledge, only a few experiments with the clustering techniques DBscan and KMeans have been reported [13,41]. In this way, we believe that binarization has a lot of room for exploration beyond the improvement of metaheuristics, the use of just one machine learning technique to binarization, compared to the 32 combinations of two-step technique that let the user gain simplicity, speediness, and less effort to use the binarization on metaheuristics, which is the aim pursued in this work.

3. Binarization

A study of putting continuous metaheuristics to work in binary search spaces [10] exposes two main groups of binarization techniques. The first group called two-step binarization allows one to put in work the continuous metaheuristics without operator modifications and the second group called continuous-binary operator transformation redefines the algebra of the search space, thereby reformulating the operators.

In this work, we used the first group to test our experiments. These two steps refer to transfer functions, which are responsible for mapping solutions in the real domain to a binary domain. Then we are going to use a machine learning strategy of clustering called KMeans in order to cluster the column of solutions and another strategy called DBscan that will cluster all the solutions (matrix). Both strategies will be used to map solutions between 0 and 1.

3.1. Two Step Binarization

Two-step binarization consists of applying, as its name says, two stages in order to bring the variables of the real search space to the world of binary variables. The first step uses eight transfer functions and the second step uses four discretization functions, therefore generating 32 possible strategies to apply.

3.1.1. First Step—Transfer Functions

These operators are very cheap to implement in the algorithm and their range provides probability values and attempts to model the transition of the particle positions; in other words, these functions transform a real solution that belongs to a real domain, to another real solution but restricted to a domain between 0 and 1. For that, there exist two classic types in the literature [42,43] called S-Shape and V-Shape, which are used.

3.1.2. Second Step—Binarization

A set of rules can be applied to transform a solution in the real domain to a solution in the binary domain. We use Standard, Complement, Static probability, and Elitist. In particle swarm optimization, this approach was first used in [20].

3.2. KMeans

KMeans is one of the simplest and most popular unsupervised machine learning algorithms for clustering. The main goal of KMeans [44,45] is to group similar data points given certain similarities. To achieve this objective, a number of clusters in a data set must be set. Then, every data point is allocated to each of the clusters, the arithmetic representation is as follows, see Equation (1):

$$J=\sum _{j=1}^k\sum _{i=1}^n{\parallel x_i^j-c_l\parallel }^2$$

(1)

where k represents the number of clusters, n depicts the number of cases, $x_i^j$ defines jth case of ith solution, and $c_l$ exposes the lth centroid (see pseudo-code in Algorithm 1).    

Algorithm 1: KMeans pseudo-code

The mechanism initially requires the k value as the number of the clusters and the best solution vector of dimension d composed by the decision variables $\widehat{X}=\langle x_1^1,\dots ,x_j^1,\dots ,x_k^1,\dots ,x_l^1,\dots ,x_d^1\rangle$. This vector is passed as input data for being grouped (see the left side of Figure 1). Iteratively, KMeans assigns each decision variable to one group based on the similarity of their features and these are allocated to the closest centroid, minimizing Euclidean distances. After, KMeans recalculates centroids by taking the average of all decision variables allocated to the cluster of that centroid, thus decreasing the complete intracluster variance over the past phase.

The right side of Figure 1 illustrates the computed groups. Positions are adjusted in each iteration of the process until the algorithm converges.

3.3. DBscan

Density-Based Spatial Clustering of Applications with Noise (DBscan) [46] is a well-known data clustering algorithm which is commonly used in data mining and machine learning. It can be used to identify clusters of any shape in a data set containing noise and outliers. Clusters are thick data space areas separated by areas with reduced points density.

The goal is to define thick areas that can be measured by the amount of near-point objects. It is indispensable to know that DBscan has two important parameters that are required for work.

• Epsilon ($ϵ$): Sets how close points should be regarded as part of a cluster to each other.
• Minimum points (MinPts): The minimum number of points to form a dense region.

The main idea is that there must be at least a minimum number of points for each group in the neighborhood of a given radius. The $ϵ$ parameter defines the neighborhood radius around a point x, any point x in the data set with a neighbor above or equal to MinPts is marked as a center point, and x is a limit point whether the number of its neighbors is less than MinPts. Finally, if a point is not a core or a point of the edge, then it is called a point of noise or an outer point.

As a summary, this technique helps when you do not know much information about data, see pseudo-code in Algorithm 2.

The right side of Figure 2 illustrates the computed groups. Positions are adjusted in each iteration of the process until the algorithm converges.

Algorithm 2: DBscan pseudo-code

4. Crow Search Algorithm

Crows (crow family or corvids) are considered the most intelligent birds. They contain the largest brain relative to their body size. Based on a brain-to-body ratio, their brain is slightly lower than a human brain. Evidence of the cleverness of crows is plentiful. They can remember faces and warn each other when an unfriendly one approaches. Moreover, they can use tools, communicate in sophisticated ways, and recall their food hiding place up to several months later.

CSA tries to simulate the intelligent behavior of the crows to find the solution of optimization problems [15]. Optimization point of view: the crows are searchers, the environment is the search space, each position of the crow corresponds to a feasible solution, the quality of food source is the objective function, and the best food source of the environment is the global solution of the problem. Below, parameters for CSA are shown.

• N: Population.
• AP: Awareness probability.
• $fl$: Flight length.
• $iter$: Maximum number of iterations.

When CSA is working, two things can happen. For this, let Crow 1 and Crow 2 be different possible solutions belonging to the search space. Firstly, Crow 1 does not know that Crow 2 is following it. As a result, Crow 2 will approach the hiding place of Crow 1. This action is called State 1. On the other hand, Crow 1 knows that Crow 2 is following it. Finally, to protect its hidden place from being stolen, Crow 1 will fool Crow 2 by going to another position of the search space. This action is called state 2. The representation of these states is represented for Equation (2).

$$x_{i,j}^{iter+1}=\left\{\begin{array}{cc}{x}_{i,j}^{iter}+r_{1}fl(m_{i,j}^{iter}-x_{i,j}^{iter})& r_2⩾AP\\ \mathrm{a}\mathrm{random}\mathrm{position}& \mathrm{otherwise}\end{array}\right.$$

(2)

where:

• $r_1$ and $r_2$ are random numbers with uniform distribution between 0 and 1.
• $fl$ denotes the flight length of crows.
• $AP$ denotes the awareness probability of crows (intensification and diversification).

Below, pseudo-code of CSA is shown in Algorithm 3.

Algorithm 3: CSA pseudo-code

5. Integration: Binarizations on CSA

As we shown in Section 3, to work in a binary search space CSA [15], one must apply the binarization techniques. Firstly, we will solve the problem using Two-steps technique, then we will apply KMeans technique.

To have a high level explanation, the steps are described below:

When CSA evaluate a $x_i$, a real value is generated, then:

• Two-steps: $x_i$ in t + 1 is binarized column by column applying the first step getting a continuous solution that enters as an input to the second step making the binarization evaluate the solution in the Objective function.
• KMeans: In this case, $x_i$ in t + 1 is clustered. Then each cluster is assigned to a cluster transition value already defined as a parameter. Small centroids get the small cluster transition value and a high centroid value gets a high value of transition. It is necessary to evaluate every point in the clusters, asking if random values between 0 an 1 are equal or greater than a cluster transition value. If we get the position $x_{i,j}$ and apply the complement to the $x_i$ in t, then the objective function is evaluated.

For theses two techniques we take the same strategy that is illustrated in Algorithm 4.

Algorithm 4: CSA + Two-steps and Kmeans binarization techniques pseudo-code

A graphic representation can be seen in Figure 3. A continuous solution from the metaheuristic is transformed into a solution that belongs to real domain to a solution in a real domain but restricted to a domain between 0 and 1, applying the first step. Then, the solution again is transformed into a solution in a binary domain using step 2. Figure 4 shows how the real solution is clusterized and then transformed to a binary solution applying cluster transition value using KMeans.

Figure 4 shows how the real solution is clusterized and then is transformed to a binary solution applying cluster transition value using KMeans.

The third technique used is DBscan. This strategy is slightly different from previous ones because what we do is binarize the complete matrix instead of binarizing the solution vector. For that, we will create a matrix replica in its initial state ($x^{\prime }$), then the metaheuristic will normally work by performing its movements in the real search space. Once each iteration is complete, the entire matrix will be clustered. As we have seen before, DBscan receives a distance ($ϵ$) and a minimum of points (minPts) as input parameter, with this data the algorithm will only deliver a number of clusters depending on the data behavior. The following steps are performed after the clustering has been completed:

• Calculate the mean of the points to every cluster.
• Sort the clusters from least to greatest taking into account the average value of each cluster.
• Evaluate each point in the clusters by checking if a random value between 0 and 1 is equal to or greater than the average of each cluster; if it is, then we get the position $x_{i,j}$ and we apply the complement to the replica of the matrix ($x^{\prime }$). Then the objective function is evaluated.

For this technique, we take the strategy that is illustrated in Algorithm 5. Finally, Figure 5 shows how the matrix x is clusterized and then, it is transformed to a binary matrix applying Dbscan.

Algorithm 5: CSA + DBscan binarization technique pseudo-code

6. Experimental Results

We propose to test our approach by solving the instances of the SCP [47] and the 0/1 Knapsack Problem [48].

6.1. Methodology

A performance analysis is totally necessary to properly evaluate the performance of metaheuristics [49]. In this study, we use the well known benchmark taken from OR-Library [50] in order to compare the given best solution of the CSA against the best known result of the benchmark. Figure 6 illustrates the steps for making an exhaustive analysis of the improved metaheuristic. For experimental design, we design aims and guidelines to demonstrate the proposed approach is a real alternative to binarization systems. Next, we establish the best value as the critical metric for evaluating future results. In this context, we apply an ordinal analysis and we employ statistical tests to determine which approach is significantly better. Finally, we describe the used hardware and software features to reproduce computational experiments, and then we report all results in tables and distribution charts.

Moreover, it is very important to show how significant of a difference the three strategies have in this type of problem. For this reason we perform a contrast statistical test for each instance through the Kolmogorov-Smirnov-Lilliefors test to determine the independence of samples [51] and the Mann–Whitney–Wilcoxon test [52] to compare the results statistically (see Figure 7).

Kolmogorov-Smirnov-Lilliefors test allows us to analyze the independence of the samples by determining the $Z_{MIN}$ or $Z_{MAX}$ (depending if the problem is minimization or maximization) obtained from the 31 executions of each instance.

The results are evaluated using the relative percentage deviation (RPD). The RPD value quantifies the deviation of the objective value $Z_{min}$ from $Z_{opt}$, which is the minimal best-known value for each instance in our experiment, and it is calculated as follows:

$$RDP=\left({\displaystyle \frac{Z_{min}-Z_{opt}}{Z_{opt}}}\right)$$

(3)

6.2. Set Covering Problem (SCP)

The Set Covering Problem [47] is a classic combinatorial optimization problem, belonging to the NP-hard class [53], that has been applied in several real-world problems like the location of emergency facilities [3], airline and bus crew scheduling [54], vehicle scheduling [55], steel production [56], among others.

Formally, the SCP is defined as follows: let $A=\left(a_{ij}\right)$ be a binary matrix with M-rows ($\forall i\in I=\{1,\dots ,M\}$) and N-columns ($\forall j\in J=\{1,\dots ,N\}$), and let $C=\left(c_j\right)$ be a vector representing the cost of each column j, assuming that $c_j>0,\forall j=\{1,\dots ,N\}$. Then, it observes that a column j covers a row i if $a_{ij}=1$. Therefore, it has:

$$a_{ij}=\left\{\begin{array}{c}1,\mathrm{if}\mathrm{row}i\mathrm{can}\mathrm{be}\mathrm{covered}\mathrm{by}\mathrm{column}j\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(4)

The SCP consists of finding a set of elements that cover a range of needs at the lowest cost. In its matrix form, a feasible solution corresponds to a subset of columns and the needs are associated with the rows and treated as constraints. The problem aims at selecting the columns that optimally cover all the rows.

The Set Covering Problem finds a minimum cost subset S of columns such that each row is covered by at least one column of S. An integer programming formulation of the SCP is as follows:

$$\begin{array}{c}{\displaystyle \mathtt{minimize}\sum _{j=1}^n{c}_j{x}_j}\\ \mathrm{subject} \mathrm{to}:\\ {\displaystyle \sum _{j=1}^n{a}_{ij}{x}_j\ge 1\forall i\in I}\\ x_j\in \{0,1\}\forall j\in J\end{array}$$

(5)

Instances: In order to evaluate the algorithm performance when solving the SCP, we use 65 instances taken from the Beasley’s OR-library [50], which are organized in 11 sets.

Table 1. Instances taken from the Beasley’s OR-Library.

Instance	M	N	Cost	Density	Best
Group			Range	(%)	Known
4	200	1000	[1,100]	2	Known
5	200	2000	[1,100]	2	Known
6	200	1000	[1,100]	5	Known
A	300	3000	[1,100]	2	Known
B	300	3000	[1,100]	5	Known
C	400	4000	[1,100]	2	Known
D	400	4000	[1,100]	5	Known
NRE	500	5000	[1,100]	10	Unknown
					(except NRE.1)
NRF	500	5000	[1,100]	20	Unknown
					(except NRF.1)
NRG	1000	10,000	[1,100]	2	Unknown
					(except NRG.1)
NRH	1000	10,000	[1,100]	5	Unknown

describes instance group, number of rows M, number of columns N, range of costs, density (percentage of nonzeroes in the matrix).

Reducing the instance size of SCP: In [57] different preprocessing methods have particularly been proposed to reduce the size of the SCP, two of them have been taken as the most effective ones, which are Column Domination and Column Inclusion; these methods are used to accelerate the processing of the algorithm.

Column Domination: It consists of eliminating the redundant columns of the problem in such a way that it does not affect the final solution.

Steps:

• All the columns are ordered according to their cost in ascending order.
• If there are equal cost columns, these are sorted in descending order by the number of rows that the column j covers.
• Verify if the column j whose rows can be covered by a set of other columns with a cost less than $c_j$ (cost of the column j).
• It is said that column j is dominated and can be eliminated from the problem.

Column Inclusion: If a row is covered only by one column after the domination process, it means that there is no better column to cover those rows; consequently, this column must be included in the optimal solution.

This reduction will be used as input data of instances.

6.3. Knapsack (KP)

0–1 Knapsack Problem is still an actual issue [58] which is a typical NP-hard problem in operations research. There are many studies that use this problem to prove their algorithm, such as in [59,60]. The problem may be defined as follows: given a set of objects from N, each o object has a certain weight and value. Furthermore, the number of items to be included in the collection must be determined to ensure that the full weight is less than or equal to the limit and that the full value is as large as possible.

$$\begin{array}{c}{\chi }_j=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{the}\mathrm{item}\mathrm{is}\mathrm{included}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\\ \forall j\in J=\{1,2,\dots ,N\}.\end{array}$$

(6)

The problem is to optimize the objective function:

$$max\sum _{j=1}^n{p}_j{x}_j$$

(7)

Subject to:

$$\begin{array}{c}\sum _{j=1}^n{w}_j{x}_j\le b\\ x_j\in \{0,1\},\forall j\in J\end{array}$$

(8)

Binary decision variables $x_j$ are used to indicate if item j is included in the knapsack or not. It can be assumed that all benefits and weights are positive and that all weights are smaller than capacity b.

Instances: In order to evaluate the algorithm performance of solving the KP, we used 10 instances [61].

Table 2. 0/1 Knapsack Problem instances.

Instance	Dimension	Parameters
f1	10	w = {95, 4, 60, 32, 23, 72, 80, 62, 65, 46}  p = {55, 10, 47,5, 4, 50, 8, 61, 85, 87}
		b = 269
f2	20	w = {92, 4, 43, 83, 84, 68, 92, 82, 6, 44, 32, 18, 56, 83,25, 96, 70, 48, 14, 58}
		p = {44, 46, 90, 72, 91, 40, 75, 35,8, 54, 78, 40, 77, 15, 61, 17, 75, 29, 75, 63}  b = 878
f3	4	w = {6, 5, 9, 7}  p = {9, 11, 13, 15}  b = 20
f4	4	w = {2, 4, 6, 7}  p = {6, 10, 12, 13}  b = 11
f5	15	w = {56.358531, 80.87405, 47.987304, 89.59624,74.660482, 85.894345, 51.353496,1.498459,36.445204, 16.589862, 44.569231, 0.466933,37.788018, 57.118442, 60.716575}
		p = {0.125126,19.330424, 58.500931, 35.029145, 82.284005,17.41081, 71.050142,30.399487, 9.140294,14.731285, 98.852504, 11.908322, 0.89114,53.166295, 60.176397}
		b = 75
f6	6	w = {30, 25, 20, 18, 17, 11, 5, 2, 1, 1}  p = {20, 18, 17, 15,15, 10, 5, 3, 1, 1}  b = 60
f7	7	w = {31, 10, 20, 19, 4, 3, 6}  p = {70, 20, 39, 37, 7, 5,10}  b = 50
f8	23	w = {983, 982, 981, 980, 979, 978, 488, 976, 972, 486, 486, 972, 972, 485, 485, 969,966, 483, 964, 963, 961,958, 959}
		p = {81,980, 979, 978, 977, 976, 487, 974,970, 85, 485, 970, 970, 484, 484, 976, 974,482, 962,961, 959, 958, 857}  b = 10,000
f9	5	w = {15, 20, 17, 8, 31}  p = {33, 24, 36, 37, 12}  b = 80
f10	20	w = {84, 83, 43, 4, 44, 6, 82, 92, 25, 83, 56, 18, 58, 14,48, 70, 96, 32, 68, 92}
		p = {91, 72, 90, 46, 55, 8, 35, 75,61, 15, 77, 40, 63, 75, 29, 75, 17, 78, 40, 44}  b = 879

describes instance, dimension, and parameters: weight w, profit p, and capacity of the bag b.

As we can see in the instance Table 2, distinct sorts of problems are solved, each with specific sizes and values to get a wide concept of the algorithm’s behavior.

Software and Hardware: CSA was implemented in Java 1.8. The characteristics of the Personal Computer (PC) were: Windows 10 with an Intel Core i7-6700 CPU (3.4Ghz), 16 GB of RAM.

Parameter settings: Below,

Table 3. Crow Search Algorithm (CSA) Parameters for Set Covering Problem (SCP) and Knapsack Problem (KP)—Two-steps.

Population	Awareness Probability	Fligth Length	Iterations
30	0.1	2	3000

shows the setup for Two-steps strategy,

Table 4. CSA Parameters for SCP and KP—KMeans.

Population	Awareness Probability	Flight Length	Number of Clusters	Cluster Transition Value	Iterations
30	0.3	2	3	0.04, 0.06, 0.08	3000

shows the setup for KMeans strategy, and finally

Table 5. CSA Parameters for SCP and KP—DBscan.

Population	Awareness Probability	Flight Length	Minimum Number of Points	Maximum Distance	Iterations
30	0.1	2	2	0.009	1000

shows setup for DBscan strategy.

Sixty-five instances of the SCP and ten instances of the 0/1 Knapsack Problem have been considered, which were executed 31 times each.

6.4. Comparison Results and Strategies

In [62] and in Appendix section (Appendix A.1 and Appendix A.2), we can see the results obtained; the information is detailed, also indicating each of the transfer functions and the binarization methods that were used. The data is grouped as follows: instance: name of the instance, MIN: the minimum reached value, MAX: the maximum value reached, AVG: the average value, BKS: the best known solution, RPD it is defined by the Equation (3) and finally TIME: time of execution in seconds.

As we mentioned, the first strategy implemented was Two-steps, the second strategy implemented was KMeans, and the last strategy implemented was DBscan.

Set Covering Problem Results

Comparison strategy:

Table 6. CSA/SCP—Two-steps vs. KMeans vs. DBscan resume summary (t.o = time out).

Instance	BKS	Stand.		Comp.		Static		Elitist		KMeans		DBscan
		MIN	AVG	MIN	AVG	MIN	AVG	MIN	AVG	MIN	AVG	MIN	AVG
S1 G1	336.08	344.92	356.75	373.96	382.11	420.88	491.21	346.92	360.91	336.80	339.43	335.84	346.80
S1 G2	151.55	156.85	164.18	174	177.14	544.45	719.47	157.25	163.91	155.70	157.81	156.55	164
S1 G3	67.10	81.25	91.63	83.70	85.27	3673.70	4385.35	73.50	78.78	75.75	75.80	t.o.	t.o.
AVG	184.91	194.34	204.18	210.55	214.84	1546.34	1865.34	192.55	201.20	189.41	191.01	t.o.	t.o.
S2 G1	336.08	341.56	354.92	372.64	381.55	716	842.13	348.24	367.78	336.80	339.43	335.84	338.25
S2 G2	151.55	156.45	262.97	173.70	177.05	1222.70	1383.50	157.70	164.15	155.70	157.81	156.55	164
S2 G3	67.10	366.80	423.83	83.50	85.19	5619.10	5986.45	74.05	79.27	75.45	75.50	t.o.	t.o.
AVG	184.91	288.27	347.24	209.94	214.59	2519.26	2737.36	193.33	203.73	189.31	190.91	t.o.	t.o.
S3 G1	336.08	340.80	353.72	373.20	380.81	854.32	809.68	347.28	362.93	336.80	339.44	335.84	338.25
S3 G2	151.55	126.86	133.19	142.20	144.55	1243	1417.11	127.26	132.79	126.46	128.65	127.73	136.57
S3 G3	67.10	625.10	712.35	83.80	85.31	5291.90	5656.64	84.15	96.71	75.75	75.80	t.o.	t.o.
AVG	184.91	364.25	399.75	199.73	203.55	2463.07	2627.81	186.23	197.47	179.67	181.29	t.o.	t.o.
S4 G1	336.08	341.64	353.17	372.04	380.58	364.56	393.54	336.12	339.52	336.80	339.43	335.84	338.25
S4 G2	151.55	156.55	163.31	173.80	176.94	309.35	390.49	151.70	155.96	155.70	157.81	156.55	164
S4 G3	67.10	973.95	1071.68	84.05	85.49	2168.80	2546.21	69.80	73.84	75.75	75.80	t.o.	t.o.
AVG	184.91	490.71	529.38	209.96	214.33	947.57	1110.08	185.87	189.77	189.41	191.01	t.o.	t.o.
V1 G1	336.08	738.40	849.97	385.88	507.63	742.76	883.61	580.48	659.84	336.80	339.44	335.84	338.25
V1 G2	151.55	1290.05	1435.96	172.75	177.36	1287.55	1467.41	809.90	964.76	155.70	157.82	156.55	164
V1 G3	67.10	5741.30	6160.88	83.45	85.41	5732.75	6209.07	3679.25	4089.29	75.75	75.80	t.o.	t.o.
AVG	184.91	2589.91	2815.60	214.02	256.80	2587.68	2853.36	1689.87	1904.63	189.41	191.02	t.o.	t.o.
V2 G1	336.08	726.88	827.56	376	387.02	710.48	834.56	565.16	653.04	336.80	339.43	335.84	338.25
V2 G2	151.55	1229.95	1392.4395	174.70	179.786	1232	1431.86	782.50	914.29	155.70	157.82	156.55	164
V2 G3	67.10	5590.55	5979.80	83.85	85.37	5595.90	6012.68	3532.40	3891.67	75.75	75.80	t.o.	t.o.
AVG	184.91	2515.79	3403.68	211.51	217.39	2512.79	2759.70	1626.68	1819.66	189.41	191.01	t.o.	t.o.
V3 G1	336.08	698.84	787.76	372.80	380.01	697	800.87	530.80	606.17	336.80	339.44	335.84	338.25
V3 G2	151.55	1144.75	1303.63	174.05	176.59	1174.15	1342.70	665.60	785.03	155.70	157.82	156.55	164
V3 G3	67.10	5255.20	5610.33	84.10	85.67	5215.45	5633.99	3056.45	3403.96	75.75	75.80	t.o.	t.o.
AVG	V3 184.91	2366.26	2567.24	210.31	214.09	2362.20	2592.52	1417.61	1598.38	189.41	191.02	t.o.	t.o.
V4 G1	336.08	685.24	776.24	370	394.93	656.96	763.27	501.60	573.14	336.80	430	335.84	338.25
V4 G2	151.55	1165.20	1347.83	173.50	176.32	1189.12	1198.33	589	697	155.70	157.82	156.55	164
V4 G3	67.10	5241.35	5615.42	97.90	109.59	4802.40	5148.97	2548.55	2937.04	77.93	75.80	t.o.	t.o.
AVG	184.91	2363.93	2579.83	213.8	226.94	2216.16	2370.19	1213.05	1402.39	190.14	221.20	t.o.	t.o.

shows a summary of the instances grouped by their complexity; we define three groups: G1 (easy), G2 (medium), and G3 (hard). The first group includes instances 4.1 to 6.5, the second group includes instances a.1 to d.5, and the third group includes instances nre.1 to nrh.5.

To get more information of the comparison between strategies, see [62]. It will easily realize how complex it is to develop all the combinations of two steps in order to get good results vs. clustering techniques.

Summary: Next, we rank the strategies in order of BKS obtained. In addition, the instances that obtained BKS in the 3 strategies are shown.

• Score
  • Two-steps got 51/65 BKS.
  • DBscan got 25/65 BKS.
  • KMeans got 16/65 BKS.
• The three strategies got BKS in the same instances: 4.1-4.3-4.6-4.7-5.1-5.4-5.5-5.6-5.7-5.8-5.9-6.2-6.4-a.5

Instance distribution: Now that we have all the results of all the strategies that solved SCP, we compare the distribution of the samples of each instance through a box plot that shows the full distribution of the data. In order to make a summary of all the instances below, we present and describe the hardest instances of each group (4.10, 5.10, 6.5, a.5, b.5, c.5, d.5, nre.5, nrf.5, nrg.5 and nrh.5):

We can see that Figure 8 and Figure 9 are alike. The Two-steps and DBscan strategies obtained BKS but KMeans did not. However, the distribution is quite homogeneous. If we see the instance 4.10 illustrated in Figure 8, DBscan’s behavior is remarkable.

Figure 10 shows again that the Two-steps and DBscan strategies obtained BKS and the KMeans strategy did not. However, it can be appreciated again that DBscan is far more compact than Two-steps, which indicates that clustering processing is accurate in this instance. Then, Figure 11 shows us for the first time a very nice and solidly compact distribution. It is necessary to emphasize that the three strategies obtained BKS. Finally, Figure 12 shows how the Two-steps and KMeans strategies obtained BKS while DBscan did not. However, if it was very close, despite this the most compact strategies in its distribution were KMeans and DBscan.

If we continue analyzing the samples, we can realize that Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 are very similar, the two-step strategies and KMeans far exceeded DBscan. This gives us an indication that DBscan works better by solving small instances with this metaheuristic, perhaps due to a large number of calculations made by grouping the complete set of solutions versus KMeans that groups the solution and Two-steps that perform binarization instantly on each vector column solution.

Finally, Figure 18 show us a null contribution of DBscan, who could not complete the executions due to the excessive delay of the strategy when clustering the most difficult instances; later we can observe KMeans absolutely trapped in an optimal local. However, the Two-steps approach continues to show good results, being the most robust at the BKS level as mentioned earlier in the strategy comparison.

6.5. Knapsack Problem Results

Unlike the previous problem (SCP), the Knapsack problem has fewer instances, so it was not necessary to group by complexity.

Comparison strategy:

Table 7. CSA/KP—Two-steps vs. KMeans vs. DBscan resume summary.

Instance	BKS	Stand.		Comp.		Static		Elitist		KMeans		DBscan
		MIN	AVG	MIN	AVG	MIN	AVG	MIN	AVG	MIN	AVG	MIN	AVG
S1	1293.91	1145.52	1049.61	1136.45	958.18	1176.22	1124.34	946	789.78	825.81	832.05	813.88	789.28
S2	1293.91	1076.23	1015.91	853.49	800.82	928.31	896.84	931.6	908.27	825.81	832.05	813.88	789.28
S3	1293.91	1164.51	1128.06	1169.75	1139.02	1153.52	1136.82	1176.37	1158.26	825.81	832.05	813.88	789.28
S4	1293.91	1069.53	1035.32	1071.46	1005.99	1117.15	1063.21	1203.77	1163.96	825.81	832.05	813.88	789.28
V1	1293.91	1070.59	1059.06	1199.57	1165.4	1165.53	1112.61	1235.02	1201.62	825.81	832.05	813.88	789.28
V2	1293.91	940.25	903.22	895.97	854.73	789.72	748.06	320.69	1617.22	825.81	832.05	813.88	789.28
V3	1293.91	1230.63	1220.57	1187.74	1153.86	1235.54	1208.77	1237.23	1203.28	825.81	832.05	813.88	789.28
V4	1293.91	1230.63	1220.57	1187.74	1153.86	1235.54	1208.77	1237.23	1203.28	825.81	832.05	813.88	789.28
AVG	1293.91	1115.98	1079.04	1087.77	1028.98	1100.19	1062.43	1036.012	1155.71	825.81	832.05	813.88	789.28

shows a summary of the instances.

To get more information of the comparison between strategies, see [62].

Summary: Next, we rank the strategies in order of BKS obtained. In addition, the instances that obtained BKS in the three strategies are shown.

• Score

  -

  Two-steps got 6/10 BKS.

  -

  DBscan got 5/10 BKS.

  -

  KMeans got 4/10 BKS.

• The three strategies got BKS in the same instances: f3-f4-f7-f9

Instance distribution: Now that we have all the results of all the strategies that KP solved, we compare the distribution of the samples of each instance through a box diagram that shows the complete distribution of the data. To summarize all the instances below, we present and describe instances where the optimum was not reached because instances f3, f4, f7, and f9 always obtain BKS in each execution with the three strategies, so it is not necessary to graph these instances.

Figure 19, Figure 20, Figure 21, Figure 22, Figure 23 and Figure 24 shows us that the Two-steps strategy achieved better results but DBscan strategy is far more compact than KMeans strategy. Figure 24 shows the only instance where DBscan did not win to KMeans.

7. Statistical Test

As we mentioned, in order to determine independence, we propose the following hypotheses:

-

$H_0$: states that $Z_{min}$/$Z_{max}$ follows a normal distribution.

-

$H_1$: states the opposite.

The test performed has yielded $p\_value$ lower than $0.05$; therefore, $H_0$ cannot be assumed. Now that we know that the samples are independent and it cannot be assumed that they follow a normal distribution, it is not feasible to use the central limit theorem. Therefore, for evaluating the heterogeneity of samples we use a nonparametric evaluation called Mann–Whitney–Wilcoxon test. To compare all the results of the hardest instances, we propose the following hypotheses:

-

$H_0$: Two-steps is better than KMeans

-

$H_1$: states the opposite.

-

$H_2$: Two-steps is better than DBscan

-

$H_3$: states the opposite.

-

$H_4$: KMeans is better than DBscan

-

$H_5$: states the opposite.

Finally, the contrast statistical test demonstrates which technique is significantly better.

Table 8. p-values for instance 4.10.

	Two-Steps	KMeans	DBscan
Two-steps	×	6.47 × 10−4	SWS
KMeans	SWS	×	SWS
DBscan	1.02 × 10−2	2.91 × 10−7	×

,

Table 9. p-values for instance 5.10.

	Two-Steps	KMeans	DBscan
Two-steps	×	3.34 × 10−2	SWS
KMeans	SWS	×	SWS
DBscan	SWS	3.09 × 10−2	×

,

Table 10. p-values for instance 6.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	1.84 × 10−3	×	6.42 × 10−4
DBscan	SWS	SWS	×

,

Table 11. p-values for instance a.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	1.85 × 10−2	×	4.25 × 10−2
DBscan	SWS	SWS	×

,

Table 12. p-values for instance b.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	SWS	×	SWS
DBscan	1.14 × 10−4	9.54 × 10−4	×

,

Table 13. p-values for instance c.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	4.17 × 10−12
KMeans	SWS	×	4.69 × 10−12
DBscan	SWS	SWS	×

,

Table 14. p-values for instance d.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	2.29 × 10−9	9.60 × 10−12
KMeans	SWS	×	2.35 × 10−13
DBscan	SWS	SWS	×

,

Table 15. p-values for instance nre.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	5.59 × 10−12
KMeans	SWS	×	2.34 × 10−13
DBscan	SWS	SWS	×

,

Table 16. p-values for instance nrf.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	9.16 × 10−12	1.87 × 10−12
KMeans	SWS	×	2.37 × 10−13
DBscan	SWS	SWS	×

,

Table 17. p-values for instance nrg.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	3.26 × 10−7	6.49 × 10−12
KMeans	SWS	×	2.37 × 10−13
DBscan	SWS	SWS	×

,

Table 18. p-values for instance nrh.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	D/A
KMeans	5.24 × 10−4	×	D/A
DBscan	D/A	D/A	×

,

Table 19. p-values for instance f.1.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	1.68 × 10−10	×	8.92 × 10−12
DBscan	SWS	SWS	×

,

Table 20. p-values for instance f.2.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	9.66 × 10−9	×	3.06 × 10−2
DBscan	7.23 × 10−8	SWS	×

,

Table 21. p-values for instance f.5.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	6.50 × 10−12	×	6.68 × 10−12
DBscan	3.49 × 10−3	SWS	×

,

Table 22. p-values for instance f.6.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	1.32 × 10−13	×	1.32 × 10−13
DBscan	SWS	SWS	×

,

Table 23. p-values for instance f.8.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	6.53 × 10−12	×	2.77 × 10−3
DBscan	6.54 × 10−12	SWS	×

and

Table 24. p-values for instance f.10.

	Two-Steps	KMeans	DBscan
Two-steps	×	SWS	SWS
KMeans	SWS	×	SWS
DBscan	6.53 × 10−12	6.49 × 10−12	×

compare SCP and 0/1 KP binariztion strategies for the hardest instances tested via the Wilcoxon’s signed rank test. As the significance level is also established to $0.05$, smaller values than $0.05$ defines that $H_0$, $H_2$, and $H_4$ cannot be assumed.

We have a method belonging to the PISA system to conduct the test run that supports the study. To this method, we indicate all data distributions (each in a file and each data in a line) and the algorithm will give us a p-value for the hypotheses.

The following tables show the result of the Mann-Whitney-Wilcoxon test. To understand them, it is necessary to have knowledge of the following acronyms:

• SWS = Statistically without significance.
• D/A = Does not apply.

SCP—p-value

KP—p-value

All reported p-values are less than 0.05; as indicated above, SWS indicates that it has no statistical significance, and D/A indicates that the comparison cannot be done because one of the strategies does not contain any data. So, with this information, we can clearly see which strategy was better than the three in each instance reported. As a resume for results for the SCP p-values lower than 0.05 were 10 for Two-steps, 10 for KMeans, and 5 for DBscan. On the other hand, results for the 0/1 KP p-values lower than 0.05 were 0 for Two steps, 10 for KMeans, and 5 for DBscan.

8. Conclusions

After implementing and analyzing binarization strategies through clusters, we can realize that DBscan achieved better results than KMeans at the BKS level. However, the execution times of DBscan were longer than KMeans; we believe that this has a clear explanation. The way in which the experiment was developed with clustering is crucial; KMeans cluster a vector while the BDscan clusters the complete matrix, which makes this technique take longer. Moreover, we do not report the solving time results due to the fact that it is known that this measure basically depends of the characteristics of the used machine (computer). Finally, our work is focused on enhancing the binariazation process for metahuerisics in order to find better results. In this way, we certainly have sacrificed the solving time but we think that DBscan is very promising because with these experiments we can realize that although we cluster the complete matrix with this technique, which took more time to execute, it still obtained good results.

When solving SCP, we have tested 65 nonunique instances of the Beasley OR Library and 10 well-known instances to solve KP, where classical technique obtained better results than clustering techniques in both problems. We believe that due to the number of combinations made (992 experiments for Two-steps), it allows a better performance on this metahuristic and also that it is necessary to make adjustments, perhaps of autonomous search on the clustering techniques to obtain a better performance.

The best results were obtained with the S-Shape transfer functions (S-Shape 4 predominates over the others) and Elitist binarization method. The worst results were the combination of the V-Shape transfer function with the binarization complement. It is very relevant to say that KMeans did not got better results than Two-steps strategy. On the other hand, it is relevant that DBscan obtained good results, better than KMeans and worse than Two-steps but the quantity of Two-step combinations is a very influential factor since with 32 different different ways of binarization it is far from being an easy way to implement; this requires time and knowledge of an expert user. Which means we reduce to one single experiment (31 executions) with DBscan over the combinatorial strategies of two-step getting good results. Statistically DBscan got better performance.

As future work, we plan to reverse the strategies of KMeans and DBscan by solving the grouping problem as a matrix with KMeans and as a unique solution with DBscan. In addition, we will implement a self-adaptive approach to get better results. Finally, we will apply the online control parameters in the Two-steps strategy to reach BKS in instances that did not reach optimal values.