T-Way Combinatorial Testing Strategy Using a Refined Evolutionary Heuristic

Abstract

In complex testing scenarios of large-scale information systems, communication networks, and the Internet of Things, exhaustive testing is always prohibitively expensive and time-consuming. T-way combinatorial testing has emerged as a cost-effective solution. To address the problem of generating test suites for t-way combinatorial testing, a Logical Combination Index Table (LCIT) is proposed. Utilizing the LCIT, the t-way combinatorial coverage model (t-wCCM) is constructed to guide the test case generation process. Multi-start Construction Procedure (MsCP) algorithm is employed to generate an initial solution set, and then local optimization is performed using a low-complexity Balanced Local Search (BLS) algorithm. Further, Evolutionary Path Relinking combined with the BLS (EvPR + BLS) algorithm is proposed to accelerate the convergence process. Experiments show that the proposed Refined Evolutionary Heuristic (REH) algorithm performs best on 50% of the classic test instances, and performs superior to the average on 66% of the test instances, with a relative improvement in the maximum computation time of approximately 33.33%.

1. Introduction

With the rapid development of large-scale information systems, communication networks, and the Internet of Things, the application demand for combinatorial testing scenarios is also growing rapidly. A report by the National Institute of Standards and Technology (NIST) indicates that inadequate testing accounts for 50% to 80% of the development budgets [1]. Given that testing-related costs constitute more than half of the total budgets, improving the testing efficiency can significantly reduce development budgets. Exhaustive testing requires traversing all possible combinations of inputs and execution conditions, consuming substantial human resources, time, and financial costs, making it impractical in real-world work. Furthermore, time and budget constraints often lead to insufficient testing during development, significantly increasing the risk of defect leakage [2,3], which can subsequently lead to failures and security vulnerabilities.

1.1. Combinatorial Testing

Due to the high cost and impracticality of exhaustive testing, combinatorial testing methods have gradually developed into a practical and general approach. Its core is the automated generation of test cases, applicable to various combinatorial testing scenarios. Research [4] reveals the approximate distribution of fault-triggering conditions: most faults are triggered by a single parameter, a small number of faults are triggered by interactions between two parameters, and faults involving the combinations of three or more parameters are progressively fewer. A NASA case study [5] also confirms this distribution: 67% of faults were caused by a single parameter, 93% were caused by combinations of two parameters, 98% by combinations of three parameters, and the vast majority of faults could be triggered by combinations of 4 to 6 parameters. Other studies [6,7,8] have reached consistent conclusions, reinforcing the universality of this approximate distribution.

Relevant research concludes that most faults can be covered by testing individual parameters or small combinations of parameters. Higher strength testing (typically involving 4 or more parameters) is a necessary condition for high assurance [9,10], but leads to a significant increase in the computational cost of generating the test suite, the number of test cases, and the execution overhead. The problem of generating test suites for combinatorial testing scenarios is classified as a Knapsack-like Problem (KLP) [11], i.e., an NP-hard problem [12]. As the number of parameters n, the number of parameter values m, and the combination strength t increase, the computational complexity of finding the optimal combinatorial test suite increases dramatically, by approximately $O\left(m^{C_n^t}\right)$.

1.2. Two-Way Combinatorial Testing Strategy

Faced with complex combinatorial testing scenarios [13], the two-way combinatorial testing strategy is generally adopted, which ensures that any two values from two parameters each are covered by at least one test case. Compared to exhaustive testing, two-way combinatorial testing strategy can significantly reduce the number of test cases, thereby lowering costs. Two-way combinatorial testing is the most commonly used strategy in combinatorial testing scenarios, offering high practicality and efficiency, and has been widely applied in actual testing activities.

To date, nearly 20 combinatorial test case generation tools have emerged [14], including AETG, PICT, TConfig, AllPairs, GATG, etc. [15,16]. Existing tools primarily have four limitations:

• Some tools only support generating test suites for two-way combinatorial testing strategy, potentially missing 10% to 40% or more of potential defects, which is unacceptable for critical, safety, and high-reliability scenarios.
• There is a lack of excellent algorithms suitable for higher-strength (e.g., 4-, 5-, 6-way) combinatorial testing strategies. Only a few tools support constructing test suites beyond two-way combination strength, and the generation time for these suites is often excessively long [1,4,7].
• A small number of tools can construct combinatorial test suites within a reasonable computation time, but the results may not be optimal. More efficient algorithms are needed to improve the test suites’ quality and reduce the computation time.
• Existing tools lack interactive features and combinatorial statistical features [17]. During actual test execution, it may not be necessary to run all test cases completely, but combinatorial coverage information is crucial for fault analysis and problem localization.

1.3. Deterministic and Non-Deterministic Testing Strategy

A combinatorial testing strategy can be divided into deterministic and non-deterministic testing strategies. The deterministic testing strategy typically constructs test cases based on mathematical functions, i.e., generating test cases according to predetermined rules. Under general conditions, they can generate an optimal combinatorial test suite, but the suite is fixed and unchanged each time, making it difficult to adapt to specific testing needs or optimization objectives [8,18]. The non-deterministic testing strategy relies on random or guided selection algorithms to construct test cases and can be divided into random algorithms [19], heuristic algorithms [20], and artificial intelligence algorithms [21]. Random algorithms construct test suites by randomly sampling from the exhaustive combination space, usually requiring a random distribution as input for sampling [22]. Heuristic algorithms use random techniques to predict and prioritize the construction of valuable test cases, e.g., AETG [23] and its variant mAETG [24] aim to generate test cases that maximize coverage gain. Artificial intelligence algorithms employ evolutionary mechanisms [7,17,25] to construct near-optimal test cases, such as Genetic Algorithms (GAs) and Ant Colony Algorithms (ACAs) [21]. However, when the number of parameters and values is too large, the convergence efficiency of the evolutionary process decreases, and the computation time increases. Heuristic algorithms also include simulated annealing [26], adaptive random algorithm [27], tabu search [28], harmony search [29], gravitational search algorithm [13], swarm optimization algorithm [30], metaheuristic algorithm [9], and fuzzy adaptive sine cosine algorithm [31]. These methods typically add test cases iteratively until the coverage criterion is satisfied. Although the non-deterministic testing strategy does not always produce optimal test suites, it possess randomness, meaning that the generated test suite can vary each time.

This paper focuses on the modeling and solving of combinatorial testing problems. The main contributions are as follows.

• Constructing a Logical Combination Index Table (LCIT) to support the mathematical characterization of the t-way combinatorial testing strategy [29,32,33], capable of counting t-way combinations and guiding the test case generation process.
• Establishing a t-way combinatorial coverage model (t-wCCM), defining the t-way combinations coverage function and coverage criterion, and quantitatively analyzing the approximate size of the t-way combinatorial test suite.
• Proposing a refined evolutionary heuristic (REH) algorithm that optimizes the generation of the combinatorial test suite step by step, utilizing an adaptive evolutionary mechanism to accelerate computational convergence efficiency.

This paper is organized as follows. Section 2 builds the t-wCCM mathematical model based on LCIT. Section 3 proposes the REH algorithm. Section 4 analyzes the algorithms performance using classic test instances. Section 5 concludes this paper.

2. Mathematical Modeling

2.1. t-Way Combinatorial Testing Strategy

Assume a t-way combinatorial testing scenario contains n parameters, represented as

$$\begin{array}{c}\hfill P=\{P_i,i=1,\dots ,n\},\end{array}$$

(1)

where parameter $P_i$ has $n_i$ discrete values, and its value set is

$$\begin{array}{c}\hfill P_i=\{v_{i,l},l=1,\dots ,n_i\},\end{array}$$

(2)

$v_{i,l}$ represents the l-th value of the i-th parameter. Selecting t parameters from the set P results in $C_n^t$ combinations. These combinations can be represented as

$$\begin{array}{c}\hfill T_t=\{T_t^k,k=1,\dots ,C_n^t\},\end{array}$$

(3)

where $T_t^k=\{P_j^k,j=1,\dots ,t\}$ represents the k-th combination of t parameters.

For each $T_t^k$, assume there exists a feasible test case set

$$\begin{array}{c}\hfill S^{T_t^k}=\{S_i^{T_t^k},i=1,\dots ,m_k\},\end{array}$$

(4)

Let $|S^{T_t^k}|=m_k$ be the number of test cases required to cover all values combinations of the selected t parameters. Any test case $S_i^{T_t^k}$ consists of n values form n parameters each and can be represented as

$$\begin{array}{c}\hfill S_i^{T_t^k}=\{v_{i,l}^{T_t^k},l=1,\dots ,n\},\end{array}$$

(5)

where $v_{i,l}^{T_t^k}$ represents any value belonging to parameter $P_i$. If any combination of t values form t parameters, each is covered by the test case set $S_t^T$ at least once, and then

$$\begin{array}{c}\hfill S_t^T=\{S^{T_t^k},k=1,\dots ,C_n^t\},\end{array}$$

(6)

where $S_t^T$ is called the t-way combinatorial test suite. When $t=2$, it is called the 2-way combinatorial testing strategy, and $S_2^T$ must cover any combinations of two values from two parameters each at least once. Similarly, when $t=n$, $S_n^T$ covers all n values combinations from n parameters each at least once, i.e., the exhaustive test suite.

2.2. Logical Combination Index Table

The Logical Combination Index Table (LCIT) is used to calculate t-way combinations coverage and guide the construction process of the combinatorial test suite [34].

Table 1. Four-parameter combinatorial scenario.

Parameters	Sets of Values
	I	II	III
I (Duplex Mode)	TDD
II (Carrier Bandwidth)	100 MHz	200 MHz
III (Coding Scheme)	LDPC	Polar
IV (Modulation Order)	BPSK	QPSK	16QAM

shows a combinatorial testing scenario with four parameters, and each parameter has a different number of values.

Taking a four-parameter combinatorial scenario as an example, its LCIT is shown in

Table 2. LCIT for 2-way combinations.

Row No.	Index (Value Set)
	I	II	III	IV
1	−1(TDD)	+1(100 MHz)	+1(LDPC)	+1(BPSK)
2	−1(TDD)	+2(200 MHz)	+2(Polar)	+2(QPSK)
3	−1(TDD)			+3(16QAM)
4	−2()	+1(100 MHz)	+1(LDPC)	+1(BPSK)
5	−2()	+2(200 MHz)	+2(Polar)	+2(QPSK)
6	−2()			+3(16QAM)
7	−3()	+1(100 MHz)	+1(LDPC)	+1(BPSK)
8	−3()	+2(200 MHz)	+2(Polar)	+2(QPSK)
9	−3()			+3(16QAM)
10		−1(100 MHz)	+1(LDPC)	+1(BPSK)
11		−1(100 MHz)	+2(Polar)	+2(QPSK)
12		−1(100 MHz)		+3(16QAM)
13		−2(200 MHz)	+1(LDPC)	+1(BPSK)
14		−2(200 MHz)	+2(Polar)	+2(QPSK)
15		−2(200 MHz)		+3(16QAM)
16		−3()	+1(LDPC)	+1(BPSK)
17		−3()	+2(Polar)	+2(QPSK)
18		−3()		+3(16QAM)
19			−1(LDPC)	+1(BPSK)
20			−1(LDPC)	+2(QPSK)
21			−1(LDPC)	+3(16QAM)
22			−2(Polar)	+1(BPSK)
23			−2(Polar)	+2(QPSK)
24			−2(Polar)	+3(16QAM)
25			−3()	+1(BPSK)
26			−3()	+2(QPSK)
27			−3()	+3(16QAM)

. The numbers with −/+ signs in the cells represent indices of specific values in the parentheses. Each empty cell indicates that the value of the parameter can be any one, and does not need to be specified. Assume that a negative number in a row represents value $v_p$, which forms a two-way combination with the positive number $v_q$ in the same row. The LCIT ensures that any combination of any two values from any two parameters each are covered by at least one row.

The LCIT can be extended to $t\ge 2$-way combinations. Generating a t-way LCIT first requires calculating the exhaustive combinations of any $t-1$ parameters. There are $C_n^{t-1}$ results to select $t-1$ parameters from n parameters. Furthermore, the exhaustive combinations of $t-1$ parameters is $v^{t-1}$, where v is the number of values of each parameter. As shown in

Table 3. LCIT for t-way combinations.

Row No.	Index
	$-{\mathit{T}}_{\mathit{t}-\mathbf{1}}^{\mathit{k}}=\{-{\mathit{P}}_{\mathbf{1}}^{\mathit{k}},\dots ,-{\mathit{P}}_{\mathit{t}-\mathbf{1}}^{\mathit{k}}\}$	$+\complement \left({\mathit{T}}_{\mathit{t}-\mathbf{1}}^{\mathit{k}}\right)$
$1\sim v^{t-1}$	$-T_{t-1}^1$	$+\complement \left(T_{t-1}^1\right)$
$v^{t-1}+1\sim 2v^{t-1}$	$-T_{t-1}^2$	$+\complement \left(T_{t-1}^2\right)$
⋮	⋮	⋮
$(C_n^{t-1}-1)\times v^{t-1}+1\sim C_n^{t-1}\times v^{t-1}$	$-T_{t-1}^{C_n^{t-1}}$	$+\complement \left(T_{t-1}^{C_n^{t-1}}\right)$

, the k-th combination of any $t-1$ parameters can be represented as

$$\begin{array}{c}\hfill T_{t-1}^k=\{P_i^k,k=1,\dots ,C_n^{t-1},i=1,\dots ,t-1\},\end{array}$$

(7)

where $P_i^k$ is the i-th parameter in $T_{t-1}^k$. Let $\complement \left(T_{t-1}^k\right)$ be the complement set of $T_{t-1}^k$. In the LCIT, the values of these $t-1$ parameters are labeled with a negative sign and placed in the left columns. The values of the remaining parameters are labeled with a positive sign and placed in the right columns. In any row of the LCIT, the $t-1$ negative values and any one positive value form a t-way combination of t different parameters, representing one t-way combination covered by that row.

For any t-way combination $X=\{v_i^X,i=1,\dots ,t\}$, the coverage function is defined as

$$\begin{array}{c}\hfill f\left(v_i^X\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}X\subseteq S^{T_t}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(8)

If $X\subseteq S^{T_t}$, then $f\left(v_i^X\right)=1$ indicates that value $v_i^X$, as an element of combination X, is covered by $S^{T_t}$ once.

Algorithm 1 provides the pseudocode for generating the t-way LCIT.

Algorithm 1 Pseudocode for LCIT construction algorithm.

Require: Combination strength t, parameter set $P=\{P_i,i=1,\dots ,n\}$, value sets $P_i=\{v_{i,l},l=1,\dots ,n_i\}$

Ensure: LCIT for t-way combinations

1: Generate the set $T_{t-1}=\{T_{t-1}^k,k=1,\dots ,C_n^{t-1}\}$ from all combinations of $t-1$ parameters of set P 2: for $k=1\to C_n^{t-1}$ do 3:     Enumerate the values of combination $T_{t-1}^k=\{P_i^k,i=1,\dots ,t-1\}$ 4:     Mark these values with a negative sign and place them in the left $t-1$ columns of the LCIT 5:     Place the values of the complement set $\complement \left(T_{t-1}^k\right)$ in the remaining $n-t+1$ columns 6:     In each row, the $t-1$ negative values and any one positive value form a t-way combination 7: end for 8: Output the LCIT for t-way combinations

The number of rows in the LCIT can be approximated as

$$\begin{array}{c}\hfill m=|S^{T_t}|=C_n^t\times v^t,\end{array}$$

(9)

where v is the maximum number of values in the n parameters.

When constructing the t-way combinatorial test suite based on the LCIT, each row of the LCIT corresponds to one test case. All rows of the LCIT constitute a t-way combinatorial test suite. Figure 1 shows an estimation of the number of t-way combinatorial test cases.

As shown in Figure 1, the number of t-way combinatorial test cases is most significantly affected by the combination strength t, growing approximately exponentially. The higher the combination strength, the faster the growth in the number of combinatorial test cases. Therefore, when constructing a t-way combinatorial test suite, it is necessary to improve the quality of test case generation to reduce redundant coverage, and to reduce the computational complexity to refine the algorithm convergence efficiency.

In Table 3, each row of the LCIT covers $n-t+1$ combinations. Therefore, the number of t-way combinations covered by the LCIT can be calculated as

$$\begin{array}{c}\hfill C={\displaystyle \frac{m\times (n-t+1)}{t}}={\displaystyle \frac{C_n^t\times v^t\times (n-t+1)}{t}}.\end{array}$$

(10)

2.3. Mathematical Modeling

Based on the logical combination coverage characteristics of the LCIT, the t-way combinatorial coverage model (t-wCCM) is constructed.

$$\begin{array}{cc}\hfill max& \sum _k\sum _i\sum _{j}f\left(v_{i,j}^{T_t^k}\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& f\left(v_{i,j}^{T_t^k}\right)\le 1,\forall v_{i,j}^{T_t^k}\in P_j^k\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & T_t^k=\{P_j^k,j=1,\dots ,t\},\forall P_j^k\subset P\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & S_i^{T_t^k}=\{v_{i,l}^{T_t^k},l=1,\dots ,n\},\forall v_{i,l}^{T_t^k}\in S_i^{T_t^k}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & S^{T_t^k}=\{S_i^{T_t^k},i=1,\dots ,m_k\}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & S^{T_t}=\{S^{T_t^k},k=1,\dots ,C_n^t\}.\hfill \end{array}$$

(16)

The objective function (11) maximizes the coverage of all t-way combinations constructed by the LCIT. Constraints (12) and (13) ensure that any combination of t values is covered by the set $S^{T_t}$ at least once, while minimizing the size of the t-way combination set $S^{T_t}$. Constraint (14) ensures that every combination $S_i^{T_t^k}$ consists of n values from n parameters each, and any subset of t values $\{v_{i,l}^{T_t^k},i=1,\dots ,t\}$ is covered by the combination $S_i^{T_t^k}$. Constraints (15) and (16) ensure that the set $S^{T_t}$ covers all combinations of t values from t parameters each.

3. Refined Evolutionary Heuristic

Aiming to solve the t-wCCM problem, a refined evolutionary heuristic (REH) algorithm is proposed, as shown in Figure 2. Firstly, construct the t-way LCIT and use the Multi-start Construction Procedure (MsCP) algorithm to generate an initial solution set. Secondly, use the low-complexity Balanced Local Search (BLS) algorithm to optimize the initial solution set, generating a guiding solution set. Thirdly, introduce Path Relinking (PR) combined with BLS (PR + BLS) algorithm to optimize the guiding solution set, generating an elite solution set. Fourthly, use the Evolutionary Path Relinking combined with the BLS (EvPR + BLS) algorithm to iteratively optimize the elite solution set, converging to the optimal solution set.

Once an optimal solution is generated, the corresponding t-way combinations in the LCIT are marked as covered. The weights of covered values are set to zero, while the weights of uncovered values are increased. The algorithm iteration terminates when all t-way combinations in all rows of the LCIT are covered by the optimal solution set.

3.1. Multi-Start Construction Procedure Algorithm

The Multi-start Construction Procedure (MsCP) algorithm [35,36] aims to independently and randomly construct multiple feasible solutions, and then the feasible solutions are uncorrelated. By repeating the above procedure multiple times, an initial solution set can be obtained [37].

As shown in Algorithm 2, the MsCP algorithm constructs the initial solution set $S^I$, consisting of m initial solutions.

$$\begin{array}{c}\hfill S^I=\{S_k^I,k=1,\dots ,m\},\end{array}$$

(17)

where every feasible solution $S_k^I$ consists of n values from n parameters each.

$$\begin{array}{c}\hfill S_k^I=\{v_{i,s_k^I},i=1,\dots ,n\},\end{array}$$

(18)

where the value $v_{i,s_k^I}$ is randomly selected from parameter $P_i$.

Define the t-way combinations coverage weights $g_k^I$ of initial solution $S_k^I$ as

$$\begin{array}{c}\hfill g_k^I=\sum _{1\le i\le n}f\left(v_{i,s_k^I}\right),\end{array}$$

(19)

which is the number of t-way combinations covered by the initial solution $S_k^I$ in the LCIT. Then, the t-way combinations coverage weights of the initial solution set $S^I$ is

$$\begin{array}{c}\hfill g^I=\{g_k^I,k=1,\dots ,m\}.\end{array}$$

(20)

Algorithm 2 Pseudocode for multi-start construction procedure (MsCP) algorithm.

Require: Parameter set $P=\{P_i,i=1,\dots ,n\}$, value sets $P_i=\{v_{i,l},l=1,\dots ,n_i\}$, number of initial solutions m

Ensure: Initial solutions $S^I=\{s_k^I,k=1,\dots ,m\}$, t-way combinations coverage weights $g^I=\{g_k^I,k=1,\dots ,m\}$

1: Generate a time-dependent random seed 2: for $k=1\to m$ do 3:     for $i=1\to n$ do 4:         $v_{i,s_k^I}\leftarrow$ randomly select a value from $P_i$ 5:     end for 6:     $S_k^I\leftarrow$ construct initial solution $\{v_{i,s_k^I},i=1,\dots ,n\}$ 7:     $g_k^I\leftarrow$ calculate t-way combinations coverage weights of $S_k^I$ via LCIT 8: end for 9: Output $S^I$ and $g^I$

3.2. Balanced Local Search Algorithm

To minimize the size of the t-way combinatorial test suite and balance the coverage frequency of all values from the parameter sets, a Balanced Local Search (BLS) algorithm based on LCIT is proposed. With low computational complexity, BLS optimizes the initial solutions one by one to maximize the coverage of t-way combinations in the LCIT, as can be seen in Algorithm 3.

Given the initial solution set $S^I$, the values of parameter $P_i$ are sorted in ascending order of their coverage frequency in $S^I$.

$$\begin{array}{c}\hfill P_i^{\nearrow }=\{v_{i,l}^{\nearrow },1\le l\le n_i\},\end{array}$$

(21)

Values with lower coverage frequency (e.g., $\{v_{i,l}^{\nearrow },l=1,2,3\}$) are prioritized as neighborhood search candidates. In the neighborhood search, replace the i-th value in any solution $S_k^I$ with value $v_{i,l}^{\nearrow }$ to obtain a new candidate solution $S_{i,l}^{\nearrow }$, and calculate its t-way combination coverage weight $g_{i,l}^{\nearrow }$. If $g_{i,l}^{\nearrow }>g_k^I$, then replace $S_k^I$ with $S_{i,l}^{\nearrow }$. After performing the above neighborhood search process for each solution in $S^I$ one by one, the optimized guiding solution set is obtained.

$$\begin{array}{c}\hfill S^G=\{S_k^G,k=1,\dots ,m\},\end{array}$$

(22)

with its t-way combination coverage weights

$$\begin{array}{c}\hfill g^G=\{g_k^G,k=1,\dots ,m\}.\end{array}$$

(23)

Algorithm 3 Pseudocode for Balanced Local Search (BLS) algorithm.

Require: Parameter set P, value sets $P_i$, number of initial solutions m, initial solution set $S^I$, t-way combinations coverage weights $g^I$

Ensure: Guiding solution set $S^G$, t-way combinations coverage weights $g^G$

1: for $i=1\to n$ do   2:     Sort values of $P_i$ in ascending order of coverage frequency in $S^I$$\to P_i^{\nearrow }$   3: end for   4: for $k=1\to m$ do   5:     for $i=1\to n$ do   6:         Replace the i-th value in $S_k^I$ with $\{v_{i,l}^{\nearrow },l=1,2,3\}$$\to S_{i,l}^{\nearrow }$   7:         Calculate t-way combinations coverage weight of $S_{i,l}^{\nearrow }$$\to g_{i,l}^{\nearrow }$   8:         if $g_{i,l}^{\nearrow }>g_k^I$ then   9:            $S_k^I\leftarrow S_{i,l}^{\nearrow }$, $g_k^I\leftarrow g_{i,l}^{\nearrow }$ 10:         end if 11:     end for 12: end for 13: $S^G\leftarrow S^I$, $g^G\leftarrow g^I$ 14: Output $S^G$ and $g^G$

3.3. Path Relinking Algorithm

The Path Relinking (PR) algorithm [38] is used to construct a transformation path from one solution to another. Each path generates a neighborhood solution, which may be better or worse [39]. Therefore, each neighborhood solution needs further optimization by searching its neighborhood to overcome local optima.

The PR algorithm operations include forward and backward directions. Given an initial solution $S_k^I$ and a guiding solution $S_k^G$, bidirectional paths can be constructed as

$$\begin{array}{cc}\hfill S_k^I\to S_k^G& :S_k^I=S_1^f,S_2^f,\dots ,S_n^f=S_k^G,\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill S_k^G\to S_k^I& :S_k^G=S_1^b,S_2^b,\dots ,S_n^b=S_k^I.\hfill \end{array}$$

(25)

The solution sets generated by the PR algorithm are defined as the neighborhood solution sets

$$\begin{array}{c}\hfill N{S}^f=\{N{S}_l^f,l=1,\dots ,n\},\end{array}$$

(26)

$$\begin{array}{c}\hfill N{S}^b=\{N{S}_l^b,l=1,\dots ,n\},\end{array}$$

(27)

where the forward neighborhood solution set $N{S}^f$ is generated during the traversal of the path $S_k^I\to S_k^G$, and the backward neighborhood solution set $N{S}^b$ is generated during the traversal of the path $S_k^G\to S_k^I$, as shown in Figure 3.

Combining the PR algorithm with the BLS algorithm (PR + BLS), each neighborhood solution is optimized separately, further updating the guiding solution set to the elite solution set $S^E$.

$$\begin{array}{c}\hfill S^E=\{S_k^E,k=1,\dots ,m\},\end{array}$$

(28)

with its t-way combinations coverage weights

$$\begin{array}{c}\hfill g^E=\{g_k^E,k=1,\dots ,m\}.\end{array}$$

(29)

3.4. Evolutionary Path Relinking Algorithm

To accelerate the neighborhood optimization computation speed, Evolutionary Path Relinking combined with the BLS (EvPR + BLS) algorithm is proposed by combing the PR + BLS algorithm with the genetic algorithm [40,41]. The elite solution set is used as the initial population of the evolutionary mechanism, and then multiple groups of two elite solutions are randomly selected to generate the next generation solutions until the stopping criterion is met. Further, the stopping criterion can be a constraint on the maximum number of evolutionary generations or a computation time constraint. To reduce the computation time of the evolutionary iterations, the process will halt if the current best solution shows no improvement after three generations of evolution.

Assume that, after q iterations of evolution, the current optimal solution set $S^B$ is obtained as

$$\begin{array}{c}\hfill S^B=\{S_j^B,j=1,\dots ,q\},\end{array}$$

(30)

with its t-way combinations coverage weights

$$\begin{array}{c}\hfill g^B=\{g_j^B,j=1,\dots ,q\}.\end{array}$$

(31)

In Algorithm 4, lines 1 to 3 sort the values of $P_i$ in ascending order of their coverage frequency in $S^B$. Line 4 selects the elite solution with the maximum coverage weight from $S^E$ as the current optimal solution $S_{q+1}^B$. Line 5 defines the evolution stopping condition: if the current optimal solution shows no improvement for three consecutive generations, the evolution process terminates. Lines 6 to 21 execute the PR and BLS algorithms. If the t-way combinations coverage weight of a newly generated solution is greater than $g_{q+1}^B$, it replaces $S_{q+1}^B$. Lines 24 to 25 add the $(q+1)$-th current optimal solution obtained by evolving the elite solution set to the current optimal solution set $S^B$.

Algorithm 4 Pseudocode for Evolutionary Path Relinking Algorithm (EvPR + BLS).

Require: Parameter set P, value sets $P_i$, elite solution set $S^E$, t-way combinations coverage weights $g^E$, current optimal solution set $S^B$, t-way combinations coverage weights $g^B$

Ensure: New optimal solution set $S^B$, t-way combinations coverage weights $g^B$

1: for $i=1\to n$ do   2:     Sort values of $P_i$ in ascending order of coverage frequency in $S^B$$\to P_i^{\nearrow }$   3: end for   4: Select the solution from $S^E$ with the maximum coverage weight $\to S_{q+1}^B$, $g_{q+1}^B$, $k=0$   5: while $k<3$ do   6:     for $1\to m/2$ do   7:         Randomly select two solutions from $S^E$$\to (S_i^E,S_j^E)$   8:         Forward PR: $S_i^E\to S_j^E\to N{S}^f$, backward PR: $S_j^E\to S_i^E\to N{S}^b$   9:         for $l=1\to n$ do 10:            for $i=1\to n$ do 11:                Replace the i-th value in $N{S}_l^f$ (or $N{S}_l^b$) with $\{v_{i,j}^{\nearrow },j=1,2,3\}$$\to S_{i,j}^{\nearrow }$ 12:                Calculate t-way combinations coverage weight of $S_{i,j}^{\nearrow }$$\to g_{i,j}^{\nearrow }$ 13:                if $g_{i,j}^{\nearrow }>g_{q+1}^B$ then 14:                    $S_{q+1}^B\leftarrow S_{i,j}^{\nearrow }$, $g_{q+1}^B\leftarrow g_{i,j}^{\nearrow }$ 15:                    $k=0$ 16:                else 17:                    $k=k+1$ 18:                end if 19:            end for 20:         end for 21:         Update $S_{q+1}^B$, $g_{q+1}^B$ 22:     end for 23: end while 24: Add $S_{q+1}^B$ to $\to S^B=\{S_j^B,j=1,\dots ,q+1\}$. 25: Add $g_{q+1}^B$ to $\to g^B=\{g_j^B,j=1,\dots ,q+1\}$ 26: Output $S^B$ and $g^B$

4. Performance Evaluation

Two numerical analysis methods are used to evaluate the performance of the proposed REH algorithm (see Figure 2). Firstly, compare the size of combinatorial test suites generated by known tools using multiple classic test instances. Secondly, analyze the computation time for generating combinatorial test suites and compare the algorithms’ convergence efficiency [17].

4.1. Test Suite Size Analysis

Six groups of classic test instances are used to compare the performances of different tools in constructing t-way combinatorial test suites. Each instance can be characterized as $v^n$, where n is the number of parameters and v is the number of values per parameter. The widely used two-way combinatorial testing strategy is selected. The proposed REH algorithm and ten other tools [15] are used to generate two-way combinatorial test suites separately. The sizes of the two-way combinatorial test suites are summarized in

Table 4. Sizes of the 2-way combinatorial test suites.

ID	Instance	AETG	IPO	TConfig	CTS	Jenny	ecfeed	AllPairs	PICT	IPO-s	GATG	Mean	REH
1	$3^4$	9	9	9	9	11	10	9	9	9	9	9.3	9
2	$3^{13}$	15	17	15	15	18	19	17	18	17	19	17.0	18
3	$4^{15}{3}^{17}{2}^{29}$	41	34	40	39	38	37	34	37	32	38	37.0	31
4	$4^{13}{3}^9{2}^{35}$	28	26	30	29	28	28	26	27	23	31	27.6	22
5	$2^{100}$	10	15	14	10	16	16	14	15	10	14	13.4	11
6	${10}^{20}$	180	212	231	210	193	203	197	210	220	245	210.1	239

.

Compared to the ten tools, the proposed REH algorithm generates the smallest 2-way combinatorial test suites for test instances 1, 3, and 4, performing better than other tools, i.e., it achieves the best performance on 50% of the test instances. Secondly, the test suite sizes generated by the REH algorithm for instances 1, 3, 4, and 5 are smaller than the arithmetic mean of the other ten tools, meaning its performance is superior to the average on 66.67% of the test instances.

Utilizing the t-way combinations coverage feature of the LCIT, the number of 2-way combinations for each test instance can be quantified, annotated in Figure 4. The six growth curves of 2-way combinations coverage during the test suite generation process by the proposed REH algorithm are statistically analyzed.

As shown in Figure 4, the slopes of the two-way combinations coverage growth curves for test instances 1, 2, 4, and 5 are approximately constant, indicating that the 2-way combinations’ coverage weight per test case tends to be stable. For test instances 3 and 6, as the number of test cases increases, the slope of the 2-way combinations coverage growth curve decreases significantly, indicating that, when the number of parameters and values is too large, the performance of the REH algorithm will decline, and it becomes more difficult to search for the theoretical optimal solution in the later stages of test suite generation. The reason is that the MsCP (see Algorithm 2) algorithm and the EvPR + BLS (see Algorithm 4) algorithm introduce randomness, respectively. While the proposed REH algorithm avoids generating the same fixed test suite each time, it also introduces stochasticity into the solving process, meaning that the generated test suite may not be theoretically optimal each time.

4.2. Algorithm Convergence Analysis

The time-to-target method [42,43] is used to analyze the convergence efficiency of the proposed REH algorithm. The computation time consumed by multiple independent runs is used to fit the probability distribution curves of the computation time. The algorithm stopping criterion achieves the specified coverage target for the t-way combinatorial test suite, i.e., the algorithm terminates when the test suite covers all t-way combinations defined by the LCIT.

The experiments were conducted on a basic personal computer featuring a 2.1 GHz processor and 4 GB of memory.

In the experiments, the GATG tool was selected for comparison. The REH algorithm and the GATG algorithm were independently executed 300 times each on test instance 2 ($3^{13}$), and each time required covering 702 two-way combinations.

The experimental and theoretical probability distribution curves using the time-to-target method are shown in Figure 5. The results show that the proposed REH algorithm has a significant advantage in convergence performance. The probability of the REH algorithm finding a two-way combinatorial test suite in at most 2750 ms is about 40%, and in at most 3100 ms, it is about 90%. The average computation time for the REH algorithm is 2840 ms, while for the GATG, it is 3845 ms, meaning that the REH algorithm saves about 1000 ms on average, and its maximum computation time is about 66.67% of GATGs.

5. Conclusions

Addressing combinatorial testing scenarios in large-scale information systems, communication networks, and the Internet of Things, this paper constructs a mathematically characterized t-way LCIT and establishes the t-wCCM mathematical model for the combinatorial testing problem. Furthermore, the REH algorithm is proposed to solve the t-wCCM. Comparative analysis using six groups of classic test instances shows that the proposed REH algorithm performs best on 50% of the test instances, and performs superior to the average on 66.67% of the test instances, with a relative improvement in the maximum computation time of approximately 33.33%. In conclusion, the proposed REH algorithm offers high practicality and efficiency, and can be widely applied in actual combinatorial testing activities.